David Bau: Let's see here.
All right, Let's see. What time is it?
David Bau: I'm. Forty-five
David Bau: great. Take another minute and we'll start.
David Bau: Okay, while we're. I'm: going to start in about a minute, but I have a question for everybody who's here. So so it seems like some people have made some progress on the homework. How how are people finding it? Is the homework?
David Bau: Uh reasonable? Is it too much? Is it too little, is it? Uh, you want different kinds of things for homework. I'd love to hear what
David Bau: people think about it,
David Bau: or maybe you haven't started yet.
David Bau: My recommendation is, start early.
David Bau: Um!
David Bau: How are people finding it.
Satwik Kamarthi: Um, I think it's a good mix of coding as well as like theoretical questions I've only gotten through the first three. But um I definitely liked it so far.
David Bau: Okay, Um, I think it's like pretty uh fair in terms of like time and difficulty.
David Bau: People.
David Bau: Yeah, we'll see if we can strike that mix. It's. Um:
David Bau: Ah,
David Bau: yeah, we we haven't been able to invest time in figuring out that mix for every every part of the uh the course. But we'll, we'll. We'll see.
David Bau: We'll see if we can maintain that. Mix.
Hendrik Ombach: Um, I have a question. Um question four point three, so I can pull it up if you need, but it's basically the gradient of made across entropy and mean toward air. It references
Hendrik Ombach: using the chain rule to combine answers for two point three and two point four. But where is there is no two point, three and two point four
David Bau: to Maybe we Maybe we remember the uh
David Bau: the questions. Let's take a look at the
David Bau: problem. Maybe you can screen share because I don't have it handy.
Hendrik Ombach: Uh, do you have the ability to do that
Hendrik Ombach: all right here.
David Bau: Ah, yes,
Hendrik Ombach: two point three, two point four is the yeah. It's like. Is that unrelated question, or something like that? Yeah, let's take a look at the question that's telling you what to combine again.
David Bau: It's a
David Bau: um
David Bau: right. It's probably saying, Use the chain rule to combine the answers for the previous two parts of the question. So let's take a look at the previous two. So whatever they were numbered, probably all the numbering changed.
David Bau: Yeah. So my guess is that it's four point one and four. Point, two or something. Let's see here.
David Bau: Yeah, yeah, four point one, four, point, two uh
David Bau: yeah,
David Bau: You can. You can see the detritus in.
Hendrik Ombach: Yeah uh, of of of us remembering the different homework pieces.
David Bau: Ah,
David Bau: but yeah, that I think that makes sense.
Hendrik Ombach: Okay, Yeah, I didn't do too much investigating into which one but at the counseling. So I just wanted to make sure. So thank you.
David Bau: Yep, no. It's just a typo in the homework. Thank you for pointing it out. We'll have to fix it in the next next version of it for next year.
Hendrik Ombach: He served great.
David Bau: Okay,
David Bau: Excellent! Excellent.
David Bau: Um,
David Bau: yeah, Let's let's go ahead and begin. Let me re-screen share here.
David Bau: And um
David Bau: see if we can get this to work
David Bau: all right
David Bau: and
David Bau: but see,
David Bau: so here's Here's sort of where we were in this story before we took a little break for Pie Torch. Oh, by the way, So So some students,
David Bau: you know, sort of our new to deep learning and new to Pie Torch. So
David Bau: I think that you know, if you're totally new to Pie Torch. It's definitely going to take some
David Bau: time to
David Bau: Ah get used to it. It'll be one of the major things to learn in the class which I, but I think it's the class is a good opportunity to learn the basics of it. Um! The ah! So if if you weren't at the sort of the Pie Torch overview lecture yesterday or the last time.
David Bau: Ah, you should watch it on video. There's there's there's some exercises to run through, and and you should feel free to use piazza to ask you. You know various pie torch questions. Why Won't Pine Torch? Let me do this, or is there a function to do that or something like that. I think that both your Tas and probably your classmates.
David Bau: Uh we'll be happy to to chime in with pythorch tips.
David Bau: So okay,
David Bau: Um.
David Bau: So here's Here's where we were in the sort of intellectual story of deep nets as we've been telling you in the class so far.
David Bau: So, you know, after Mccoy and Pitts came up with this idea of deep networks, and Frank Rosenblatt came up with the idea that you can actually
David Bau: learn a network from data
David Bau: uh by making little adjustments every time the network is wrong.
David Bau: But he only knew how to do it on a shallow network. But then Rumah Hinton Williams came in and pointed out to everybody You could use
David Bau: um. A simple home
David Bau: updating rule based on derivatives. Gradient descent
David Bau: could be used to learn deep layers and address those deep layers. The
David Bau: so at this point, you know we have um some pieces, right? We have the ability to learn deep layers. We know that if you have a multi-layer network with enough neurons, you can approximate any function. We have a training procedure that
David Bau: you could use to train a network just based on some data. If you have some input, data and some some objective that you're trying to optimize and then and then We've also had, you know, sort of the beginnings of a few ah pieces of tricks to make this work. Well, we've noticed that
David Bau: you need non-linearities that are nice and differentiable.
David Bau: We've been using these nice, smooth nonlinear, so they don't have to be that smooth, but they need to be not. They need to be differentiable.
David Bau: It's a
David Bau: we've uh talked about this computational tool
David Bau: of efficient back propagation, uh which in the next homework you have some more uh practice with that,
David Bau: and we've talked about. Some people have noticed the importance of weight initialization, and there there's going to be this increasing bag of tricks over time to to make this sort of scenario more effective.
David Bau: How to train a deep network. So But what is a deep network? It's basically it's. It's the the idea is, it's a way of getting a function that you might not know
David Bau: that you need to solve some problem you can, you can. You can use and try to use deep networks to get any function you
David Bau: that you want. So what kind of functions?
David Bau: Ah, oh, so let me let me. Oh, i'm sorry before we go on to that that story. There's one more piece I I want to just touch on because we talked a little bit about
David Bau: back prop um, and we talked about it all in terms of back propagating
David Bau: derivatives of scalars.
David Bau: Um, But here here's here's sort of a data flow through a typical
David Bau: neural network. Um.
David Bau: And so the input comes in as X one,
David Bau: and then it gets matrix multiplied by some
David Bau: network weights. You know they they figure out sort of weighted sums of all the X one's to make a a z one, and then it goes to some non-linearity like a relu or something, and then that turns into x two that goes through another matrix multiplication for the second layer
David Bau: and the third layer and the fourth layer. There you get all these matrix multiplications, and then followed by non-linearities. And then finally, at the end, you go through some final
David Bau: function which might be comparing to a classification, loss or some other objective, and you get a scalar loss at the end. So this is. This is, You know what a typical
David Bau: neural network looks like. And so we talked about back propagation over a computation graph where the graph could be a really complicated beast with lots of branches and things like this. But actually,
David Bau: if you, if you think of your data, not as scalars. But if you think of your data as vectors or tensors or matrices, right, and these these big blocks of numbers. Then a typical. That neural network is actually your most neural networks are actually pretty simple. They're they're a lot simpler
David Bau: when you look at them as as vector operations, and so like this neural network, it's just one straight line all the way through where each layer depends on the network weights in that in that layer.
David Bau: And so it's important. If you, if you want to really do back propagation efficiently, it's important to understand how to back propagate
David Bau: matrix operations
David Bau: in addition to just scalar operations. So I just wanted to take a minute to to work through the derivatives of matrix multiplication for you, so you can see what they what they look like.
David Bau: And so this is. This is matrix multiplications, right? So you got a big matrix, W. You
David Bau: and you matrix multiply it by the vector X, and then the output comes out as the and I've diagrammed
David Bau: what the matrix multiplication looks like. Here on the left, you know, if if X is a column vector actually, you know, you sort of have to transpose it to see how it affects all of the columns of
David Bau: how each row of X combines with a column of W
David Bau: to to make the the rows of Z. And so you can kind of. See here, how
David Bau: I don't know if you can see my pointer, but you can see how x one gets multiplied by all the first things in W. One. This is sort of should be familiar here. There's matrix multiplication, and then and then the products all get added up of all these multiplications to get to to be the first component of Z or the the i-th component of Z
David Bau: comes from this row. And so, if so, if we look at this diagram, of where all the weighted sums come from, it starts to make it clear what we should get if we try to take the partial derivative of the i-th component of Z.
David Bau: And if we ask how is that, influenced by small changes in the j-th component
David Bau: of X. Well, you know the j-th component of X affects all the components of Z. But that's not what we're asking. We're not we're only asking about the i-th one. So we just follow this all the way down here to see how it affects the i-th one, and the answer is, well, it gets multiplied by W. Sub. Ij.
David Bau: Then added to some other things which we're not asking about here, and in order to form the
David Bau: Zi, So so the so the coefficient, the proportional um sort of influence of X. Of J. On Z. I. Is just given by, you know, by the intersection of these arrows, which is at Wij right just like that.
David Bau: And so similarly, if you ask, well, how does zi vary? If wij varies
David Bau: well wij
David Bau: it isn't multiplied by anything except for Xj.
David Bau: So you know. So if we perturb wij, then zi will vary depending on how big Xj. Is.
David Bau: So so that's so that's how that is, Oops, How come my X-ray isn't showing up there is Xj.
David Bau: So it'll vary depending on how Xj. Shows up. Now we can summarize this thing.
David Bau: Ah, how x J. Um turns into Wij by varying over all the I's and J's. And you can kind of say you know the partial derivative of Z. With respect to changes in all the dimensions of X. All the dimensions of it. It's just W. You know it's sort of the Jacobian.
David Bau: A linear operation is itself right. The derivatives of the output of a linear operation with respect to the input of a linear operation is just the matrix itself,
David Bau: Adam.
David Bau: And then, similarly, ah, you know, if we, if we ask, you know, how does he vary based on all of the entries of W. You? You get a similar conclusion that it's it's just the same, the same
David Bau: uh thing with the the the the index is varied.
David Bau: So so you get these really really simple operations that look like this. And so it turns out that back propagation on matrix multiplication doesn't actually require
David Bau: more stuff. You know It's It's the the The partial derivatives are just the other term. It's just like the product rule.
David Bau: And so if you multiply by W, you know. So the the way that Z varies according to changes in W. Is X. And the way that Z. Varies according to changes in X is W. So it's just the product rule, and so so you can do this sort of back propagation here, where
David Bau: you know the local gradient of how Z. Changes with W. Is just x, and how z changes with x. It's just W. So when you back, propagate your upstream gradients to your downstream gradients, it's just the same as taking your upstream gradients and just matrix multiplying it by X T.
David Bau: Or Wt, you know, basically you know X or W. Um uh to get the downstream gradients. And so um
David Bau: So that's That's That's the
David Bau: uh. That's the trick for how to deal with
David Bau: um matrix multiplication operations you can. There's a question that has to do with this on the next homework, so you can think about this a little bit more.
David Bau: But ah! But it's just a matter of organizing all the indexes and making sure that they all come out right, and there's not actually a lot of ah new um computation that needs to be done just. Ah just making sure all the loops are done in the right order.
David Bau: So
David Bau: so this is a This is a convenient and powerful.
David Bau: You can do back propagation on scalars, and you can do back propagation on matrices and tensors in exactly the same way, and
David Bau: you can make computation graphs. In fact, you can do back propagation, not just on matrices and tensors. You can do back propagation on
David Bau: ah sort of recursively on computation graphs themselves, which is how all these back propagation packages do higher order derivatives. So So there's a little extra credit thing to do that in the next homework as well.
David Bau: Okay.
David Bau: So
David Bau: so with that with that in mind, I think that that that brings us to um everything we sort of have to know, to do derivatives and to do back propagation. And and it gives us this nice set of tools that we can use to learn
David Bau: a function,
and it leaves us with this nice question.
David Bau: Well, now that we can learn a function, what kind of functions should we learn?
David Bau: Well, So here here's the the basic game of what all the network research is all about It's about finding different places where we can learn a nice, interesting, useful new function to solve some new tasks
David Bau: that that can be learned from data in some amazing new way. And so the way the game is played is basically
David Bau: to collect together three kinds of information to to design your algorithm And so i'm going to give you a couple of examples here with a couple cases that are really classical cases, classical tasks in neural networks
David Bau: having to do with processing images. So let's say we have some input image like this. This looks like a runner Here
David Bau: he's.
David Bau: And ah,
David Bau: what is she doing? She is probably, uh, you know, mid race in a Marathon or something.
David Bau: So
David Bau: so here's here's two possible tasks that people have thought about and proposed and worked on a lot in ind networking. So one one possible task is something I call learning representations of an image
David Bau: learning a represent learning to represent this image. Um, and the other task is learning to classify this image. So what is representation? So a representation is really about making a summary of the image, you know. Summarize this. Tell me, you know, sort of you know, distill this image down to everything that is important about this image.
David Bau: I'll look at this image and remember it,
David Bau: and and classification is a much narrower, more specific task.
David Bau: Ah, we have a specific question about what kind of category this image is in. Maybe we want to know whether this person is likely to win the race or lose the race. So we want to do a classification of winners and losers. Or maybe we want to know what kind of race this person is racing in,
David Bau: or something like that. Maybe so. So this is the kind of a result that you might want to get if you had a representation. You want to sort of summarize everything that's interesting about this about this piece of data. If you were doing this with a traditional computer program. This might be
David Bau: sort of the task of trying to distill everything down to a database record. You know what what kind of images is it? It's about a human. It's standing upright, running to the left on an outdoor road,
David Bau: wearing certain colors. Looks like a certain person
David Bau: happens to be wearing sunglasses, And now classification is much chart. Narrower thing we want to say. You know what categories is it a marathon? Is it One hundred meters? Is it a sprint, you know. Maybe we're going to say It's a Marathon here. It's just we classify this correctly.
David Bau: Um, Okay.
David Bau: So we have these kind of different tasks. We might want to do. How do you go from this vague idea of a task to actually a training procedure
David Bau: of something that you want to do? Well, okay. So here's here's what we do. And there there have been many papers that have have done this over the years in deep learning, and they they all define basically three things. The first thing is, they figure out and they define what data
David Bau: defines their tasks. You know. In other words, what are the inputs and outputs to this magic new function that they're going to train and learn. And the second thing is, they figure out their objective like what makes a good function. And what makes a bad one? How are they going to measure
David Bau: all the errors and problems that their function might have? And how do they? How they're going to measure the loss? Because once you can measure the loss, then you can put that into your training procedure and learn a function that minimizes the loss. Right?
David Bau: So defining the objective is really important.
David Bau: And then the third thing that they do is, you know, we will define the architecture of Ah! You know. How will the computation be done now? Not not exactly how the computation will be done, because we don't know what all the parameters are. That's what learning and training is about.
David Bau: But more like, How is this computation going to be parametrized? What's the space of possible computations that we're going to consider for solving this problem. What is the architecture of the neural network? And and where can that neural network be ingested? So So this is um
David Bau: these three things. Once you define these three things, and then you choose an optimization method to do the actual training. Then then you're then you're off to the races. Right then you have a setting where you can create
David Bau: a learned function to do to do your tasks. So let's take a look at. Let's take a look at. Well, let's let's look at the vague one here first. So we I gave you this example of representation, so you could think of representation as this task where you take input and turn it into some
David Bau: summary. So maybe that's the input and output, right? So maybe you have some input like an image, and you wanted to turn it into like, you know, database record
or something like that.
David Bau: So actually, this is not quite right, because the whole interest that people have in having networks learn representation is not about having them.
David Bau: Learn to create a specific kind of database record. It's it's more of an unsupervised question. It's people who are asking, How can a network learn to learn about everything that's interesting in an image? So i'd say,
David Bau: we don't even want to say that you know the output should look like a specific.
David Bau: Ah kind of
David Bau: like database format, or something like that. It's just we need to have some sort of output,
David Bau: and we sort of have this question of.
David Bau: We want to have the network learn like good output. But what makes good output. What makes a good representation?
David Bau: So what would make a good representation of a complex piece of data?
David Bau: Well, it should. It should sort of have everything you need to know.
David Bau: And that data. How do you know, if you've got everything you need to know in a piece of data
David Bau: like, if you, if you omitted something really important.
David Bau: Then
David Bau: maybe the way you could say that is, you missed something because
David Bau: um, you missed something that would be required for you to get the data back, for example, so like maybe we missed the fact that there was even a runner in this image. And if we, if we forgot that fact. Then it would be impossible to look at our our our Z,
David Bau: and figure out that this was an image of a runner Right? So
David Bau: So let's say you know one possible rule for what makes a good representation is that you can reconstruct
David Bau: the original back.
David Bau: Then then, that representation would be sort of guaranteed to have everything you need to know. Ah! To construct the image. So i'm going to let let's write it like this. So ah! So we'll, we'll have this other function here that goes from a representation back out to the same. Ah, complex data space, you know. So from images to something,
David Bau: some sort of record to some images again. And then the idea is to try to reconstruct something similar to the original image, and so here i've drawn a actually computer-generated picture.
David Bau: It's not the same image, but it's sort of an effort to reconstruct. The original and The goal would be to be as similar to the original as possible.
David Bau: Okay, So
David Bau: so this is. This is sort of
David Bau: this is the way that a lot of people think about representations as a task. And so this statement of representation learning is the auto-encoder task. It's a it's a classical problem it's something that
David Bau: people came up with in the very early days of studying machine learning and deep networks. Um! I think that ah room of heart and hinted in. Ah, you know we're talking about autoencoders already in their first paper where they're talking about things that you could use back prop for.
David Bau: And and the idea for auto encoders is you start with some input data. And then you try to get the same input data back again after you do some learned operations
David Bau: on it. So So it's called autoencoding as you encode it, and then you encode it back to itself.
David Bau: It's, it's it's it's it's, it's, it's, it's it's it's, it's, it's, it's, it's, it's, it's it's, it's it's, it's it's it's it's it's it's it's it's it's.
David Bau: And so Ah,
David Bau: let's see here. So okay, So let's let's go. Through. This exercise. Here, of whoops.
David Bau: I didn't mean to. Now, I've lost my spot.
David Bau: I
David Bau: I don't like this Wheelie mouse thing. Okay. So what's the input in the output? What's the with the loss. And then how we're going to parameterize this problem. So let's figure this out. Okay. So the input and the output is images. Right? So So for images.
David Bau: Ah, you know, we can put images in and images out so obviously to have a differentiable learnable model using the computational tools we want the inputs and outputs really should be like numbers or vectors or tensors, right? And we we should have tensors of numbers. And so well, that's that's not a big problem. We can take an image
David Bau: which is made of lots of pixels, and we can turn it into sort of an Rgb encoding of the image. And so you know, so we can use. You guys should be familiar with Rgb. Encoding If you're not, then
David Bau: maybe maybe put a note in the in the chat.
David Bau: But the basic idea is that every pixel of an image can be encoded by using a few numbers to represent the color and the brightness of that pixel, and by just arranging a big grid of such numbers you can represent any image. And so this is the standard way of representing image data, and we can just sort of encode the image data
David Bau: as numbers as a big grade of numbers and treat those numbers as, input
David Bau: and we can say that the goal of the network is to map the space of input image tensors to another space of input image tensors
David Bau: uh like this. So this is sort of the the input and the output is both
David Bau: villages.
David Bau: Okay? So we know the input and the output. So now the question is, what is the objective.
David Bau: Well, they object. There is for the images
David Bau: to be the same. If they are different,
David Bau: then there's some some error, so the error here is like the difference between these two images. Ah, you know, and we I just put a big vector norm like a squared vector, norm around it to say, hey? I'm interested in the difference between them as the error.
David Bau: And so let's we. We can make that mathematical. But basically. By saying, you know, we've got a tensor for this image and a tensor for that image, and and our our loss is just what we get when we take the difference between those images and square. And so this is this is this: Um,
David Bau: This L. Two distance is also called um mean squared error distance. It's also called Euclidean distance. It's it's sort of the classical error measure it can be thought of in a bunch of ways.
David Bau: And so another name for it is this is a standard L. Two reconstruction loss. So our goal is to have an image that comes out that is as close as possible to the image that goes in.
David Bau: Ok, So we know the inputs and the outputs. We know our objective
David Bau: um does. Anybody can? Can. Does anybody notice a problem here with this objective? And the way that we've stated the problem so far, so anything that makes this problem uninteresting.
David Bau: Well, in terms of the problem F of X is the output of the encoder. That's correct. F of X is the output of the neural network.
David Bau: Oh, let's see. So i'm gonna i'm calling F of X. The entire thing. Not just the encoder. Um, i'm sorry it's a little bit of an abusive notation here, but we'll call F. Of X. We could. We could gave a name to this second arrow. We could call it G, or something like that. Let's consider F the composition of both the decoder and the encoder. For now.
Andrew Lemke: Oh, okay, that's it.
Andrew Lemke: Yes, in that case, then, Um, A very boring network that simply passes the input directly to the output would um perform perfectly, but not really be useful for any.
David Bau: Exactly
David Bau: Exactly. So. This This network isn't wouldn't do much interesting stuff if we just learned the identity function like the identity function would be a perfectly fine solution to the problem the way we've stated it. So far how we define the inputs and the outputs, we define a reasonable objective.
David Bau: And yet this problem
David Bau: isn't doing interesting stuff yet.
David Bau: And so what's essential here for the classical statement of the autoencoding problem is that you also have to specify the third piece, which is like, How are you going to actually parametrize this problem? What What kind, what kind of computations are you going to allow? What kind of architecture are you going to allow?
David Bau: And so, in the classical statement of honor encoding what they'll typically say is, you try to get from your complex input to the same complex output
David Bau: through a multi-layer neural network
David Bau: where you specify the architecture to have a tiny tiny
David Bau: Blair in the middle,
David Bau: where the network has some bottleneck, where it has to squeeze
David Bau: the information through an intermediate layer that has so few neurons
David Bau: that that the vector that they represent is much too small
David Bau: to contain the whole image,
David Bau: and it's that vector that represents your summary
David Bau: right. This this sort of mystery is Z.
David Bau: Um. And so this this architecture is pretty important in autoencoders
David Bau: and and so and and you can see how a neural network practice sort of has these three pieces here. You know It's it's it's It's a You know. What are the inputs and outputs?
David Bau: What are the objectives?
David Bau: What is the shape of the computation that you're interested in looking at, and and you tend to look at any task that
David Bau: with these three things
David Bau: in mind and and how you define the task.
David Bau: So
David Bau: let's see here.
David Bau: Um, let's take a look at the other. So i'm going to take autoencoding for now, and i'm going to set it aside. It's just sort of an example of how important it is to look at all three of these components.
David Bau: I'm:
David Bau: ah today, what i'm going to focus on is the the other big classical problem of deep networks, which is classification. So i'd say that if there are two problems that people have worked on from almost the beginning of time in deep networks. It's autoencoding
David Bau: and classification,
David Bau: and many of the early native network. Researchers were interested in the image case where it's image image, auto encoding and image classification for for both those cases. So so i'm trying to give you a little sense. For ah! You know what people have been working on for a long time.
David Bau: And so we're going to take a look at image classification and see how it's been developed as a data network problem. And so the image classification problem is the more specific one. Where you take an image like this, and you try to decide which one of a finite number of categories it should live in.
David Bau: So uh so as, for example, maybe we're trying to decide whether this person um
David Bau: runs in a marathon, or in the sort of ah mid-length, like eight hundred meter races, or is a sprinter um and and The goal in the network would be to make some sort of guess that maybe this is the marathoner.
David Bau: Okay? So what is the input, and the output of this is we're going to figure out. We're going to figure out what is the right way of measuring our error and our loss, and we're going to decide if there's anything to think about in terms of how this problem should be parametrized
David Bau: do the same exercise. So, input and output. So here we're mapping. We usually use the letters X and Y here for this problem. So here X is the image, and Y is the that the category that this image is supposed to be in like Marathon.
David Bau: Okay, that's great.
David Bau: So so for for the input, we'll use this whole Rgb: Encoding business again for the images and for the output, we're going to use something called one hot encoding. So if we have like n choices like we've got three choices here, then we would have
David Bau: an output, which is like a three-dimensional vector
David Bau: uh, where the goal would be to be, You know zero in most places in one
David Bau: the dimension that corresponds to the the label. That's correct. This is called one-hot encoding. So wait before we go on.
David Bau: Um, you know. I remember when you know I started working with deep networks, and you know, with these classification problems, and you know it always was a puzzle to me in in my early days.
David Bau: You know what is this one-hot business,
David Bau: and Why,
David Bau: why do we use one hot vectors when we're classifying?
David Bau: Consider just a number right? And we we got this like simple classification problem where, let's say, we have to output three things. Why not? You know. Neural networks are perfectly capable of outputting numbers. That's what they're all about. They're just these big computation machines output numbers,
David Bau: you know. Why, Don't, I just have my neural network output the number one, two, or three for my three classes. Why do I want to have
David Bau: one hot encoding. So I I I want to do this as a classroom
David Bau: discussion, but maybe I can pause for a minute here and have you guys think about it and and just chime in with, you know, just random reasons why you think maybe we should be using single number class number encoding output for networks.
David Bau: Or maybe there's some reason we should be doing one hot. What what? What do you guys think?
Eric Grimaldi: My first gut reaction is that using numbers invokes comparison,
David Bau: but it convokes comparison.
David Bau: Yeah, like, if we have one, two, and three, then like, suddenly three is greater than two, and maybe there's like some mathematical logic implied there that the network would learn that Isn't actually
David Bau: the class interesting.
David Bau: What? What? What other people think about that idea? Can Can anybody give me a specific uh idea for what that
David Bau: that might imply.
David Bau: Is that a good thing or a bad thing?
Lakshyana KC: Ah, I think that's a bad thing to have it in numbers. We want to maybe vectorize it a little bit and have one output compared to the other. So in form of like zero, zero, one and zero one zero. We can make a comparison and kind of
Lakshyana KC: find similarity between Directors
David Bau: right? So it's so it's sort of bad, because the number. If you have a single number, then
David Bau: there's already a relationship between the numbers, and you'd rather learn that relationship instead, or something like that.
David Bau: Uh, based on the same idea. Uh, I think. Uh, for example, if uh, i'm assigning to marriage one to three, and the spring is one,
Ashutosh Kshirsagar: so my network was try to, would say, Okay, if you are assigning to as the eight hundred meter, and if you are getting eight hundred meters, then it is fine. It is not as bad as ah getting your sprint. If the actual class is matter, so
David Bau: I mean it will start having those kind of relationship.
David Bau: On the other hand, then we have. What is that you mean? You're you're saying that because two is closer
David Bau: to three, then one,
David Bau: then then then it would be safer to guess two than one, because it's less of a chance that it could be way off or something like that. Yeah. And most of the cases we'll just get to, because it is the safest for the network to predict in all the cases it will just directly do.
David Bau: Yeah, it's a safe bet.
David Bau: Yeah,
David Bau: yeah, I always guess. Two:
David Bau: Yeah. The network might devolve into that kind of Why doesn't that happen with one hot encoding?
David Bau: Why not? Why wouldn't the networks Just always guess the middle, the middle dimension,
Lakshyana KC: because it's each of them have each of the outputs have different axes. I guess, like they're
David Bau: yeah, Yeah, they are not as close.
David Bau: Yeah, there's It's perfectly right. Exactly. You're exactly right that each dimension
David Bau: and the one-hot encoding has its own axis, and there's a perfect symmetry there.
David Bau: I don't know if you ever think about it this way. But if you, if you were sort of an ant or something, and you lived on the line
David Bau: X equals y equals Z
David Bau: in space. You know this sort of you know this, this line, where all the all the dimensions are equal, which is, I mean, there's there's only this sort of one specific line coming out of the origin,
David Bau: and if you were to do that and look back at the origin,
David Bau: and and if somebody were to put a little a star,
David Bau: each one of the unit vectors in each one of the axes, Then what shape would you see
David Bau: if you look back at the origin? And you looked at like all the little unit vectors?
David Bau: You know. What What would it look like to you.
David Bau: Yeah, let me let me propose this the thought experiment. So
David Bau: suppose you were in three-dimensional space,
David Bau: and you are a little uh flying at your low mosquito,
David Bau: and you decided you were going to set off on a journey and only fly along the line where X equals Y. Equals Z,
David Bau: and you're gonna fly out pretty far.
David Bau: Then you you Once you get out into space, you go back and you look at back at the origin where you came from. And now somebody has arranged you
David Bau: uh to not just let you see like the origin, but they've also put
David Bau: it's,
David Bau: you know, a point of light at each one of the
David Bau: unit axes, vectors, each one of the one hot vectors.
David Bau: One's, are a zero,
David Bau: zero, one zero and zero zero one. So each one of those one high factors has a point, and you go back and you look at all those points. What shape would you see
Ashwin Unnikrishnan: you maybe
David Bau: can you even see a a cube? You said,
David Bau: Yeah, yeah, you Well, if you could see in threed you'd see a cube, But if you're pretty far away
David Bau: from um
David Bau: from the from where you started you you wouldn't be able to see in threed anymore. You sort of just see
David Bau: you know what it looks like, just like when you look up at the stars. You can't tell which ones are closer and farther things like that. So what shape would these three points make
Ashutosh Kshirsagar: the equilibrium triangle
David Bau: an equilateral triangle?
David Bau: They would. That's exactly what they would make if you look back on them. They
David Bau: you would see this beautiful, perfectly symmetric
David Bau: equilateral triangle, and I think it's a nice way you can. You can try to draw this in threed for yourself to think about it,
David Bau: but I think that when you draw it in threed, it's a nice way of thinking
David Bau: about how symmetric these three axes are.
David Bau: There's really no difference between any of these axes. They form a perfectly symmetric
David Bau: Um
David Bau: uh um uh equilateral triangle uh, where where none of the dimensions is, is really special compared to any of the others. And um,
David Bau: and the same is true in any dimension. If you were in eleven dimensions, and you flew out on this thing you would get this very nice symmetric.
David Bau: Ah, you know equal angle equals, sided eleven, sided Polygon. If you were to look back in that direction, and and that's if you looked out. If you flew out in the direction of where X equals y equals Z. If you flew on any of the other directions where x equals y equals negative Z, or you know you flip the sign of any of them,
David Bau: and you look back at at at at the same set of points. Um, then you you would see there, you see other uh uh sort of uh equal
David Bau: equal polygons for for these for these sort of things.
David Bau: So Um:
David Bau: um. So yeah. So So the the dimensions are nice because they're they're perfectly symmetric um
David Bau: and um, and so so it it. It. It's sort of a nice way of not introducing any particular bias. Um
David Bau: for ah, ah! You know, for for any combinations of classes or a relationship between the classes. But there's another reason. There's another reason for this, and the the other reason is something that was come up with in one thousand nine hundred and eighty, seven, one thousand nine hundred and eighty, eight,
David Bau: and I'm I'm. Putting three names of ah deep learning researchers here, and I got photos of them, you know, when they were quite a bit older than they were in one thousand nine hundred and eighty seven, when they originally wrote their things. I think They They wrote these things when they were master of students or or graduate students
David Bau: back in the eightys, and they all became, You know, various you know, well-known deep learning researchers, you know, in different fields.
David Bau: And And this was this was back in the time when you know Roma Hart had just published back prop just a year earlier, a year or two earlier, and there was a lot of excitement about how to get networks training,
David Bau: and and everybody was going through the same exercise
David Bau: of deciding. You know one of the inputs and outputs of different functions that you might want to train. What are the architectures, and what are the what are the objectives? And And they realized that this old objective and this old
David Bau: idea that people had of doing classification by just
David Bau: outputting a plus one or a negative one wasn't quite
David Bau: right.
David Bau: Um! And And they all came to this realization separately that maybe what classification really is is about predicting probability distributions,
David Bau: and they actually all proposed exactly the same formulation. They all proposed. This log probability formulation. They all proposed the Kl diversion. So the cross-ent should be lost, which is, you know, equivalent. And so i'm going to talk about that for a minute here, because the
David Bau: ah! You can see that. Ah, you know this, this, this invention that was sort of discovered in triplicate in parallel Ah became obvious to people. Um! And and it it remains sort of obvious to people today, although it's not so obvious that you know you might come up with on your own. But if you,
David Bau: if you were to play with it for a couple of years. Then you would probably come to the same conclusion as these three that this is really the right way of doing things. And so it's the way everybody does classification today, which is, they use softmax cross entropy lots. So the way that one way of thinking about it is that you know a single class might not even be
David Bau: the right true label for a given piece. So like. So, for example, what if our our goal was. You know, somebody told us we have a particular runner. It looks like this is Nakani or somebody. And and we want to say, Okay, Well, What! What, what sport is the economy running in at the moment,
David Bau: and and what if the county, you know, runs in more than one sport? Ah! And at any given moment is thirteen percent likely to be in the Marathon, but but seventy percent likely to be the one hundred eight hundred meter, you know. It could be in any, it could be in either one of those,
David Bau: then the ground truth would be,
David Bau: This is this probability distribution that i've drawn here is a pie chart is actually, you know the truth of what what is possible to know the
David Bau: about about the prediction. And so so maybe we need to have more than one output. And so, instead of the the true thing being ones or zero, maybe the true thing should be zero point seven point three and zero,
David Bau: And so so the idea is that maybe the network should predict a probability distribution. Maybe it should predict a probability distribution of some numbers, some number for each
David Bau: category.
David Bau: The only thing that
David Bau: predicting a probability distribution really means is that they should all add to one.
David Bau: So the idea is that whatever crazy things the network
David Bau: would normally output, which are sort of unscaled. These sort of purple numbers that i'm showing here we sort of want to normalize them, so that everything adds to one.
David Bau: And so there's a standard way of doing this called the softmax function um, which you know, which is sort of inspired by some ah physics formulations, and some some various other things. The softmax was actually one of the things that all three of these authors
David Bau: I hinted at and proposed in different ways,
David Bau: and and and it gives you this. I'll talk a little bit about the softmax function,
David Bau: and then the second step is, once you've got a privilege,
David Bau: probability, restriction, you need to figure out like how to measure its distance from the ground. Truth, probability, distribution, and and the standard way of doing that.
David Bau: Ah, indeed! Networks is called the Cross Entropy loss. Actually it. It builds on top of this very, very standard tool from statistics called the Kale divergence, and i'll just give you a definition for that. So you should be totally aware of what these things are on the current homework. There's some ah homework exercises that have to do with it,
David Bau: but it's um!
David Bau: But you know how useful and how how well form these things is. It's a little hard to
David Bau: ah convey without you just using them a lot. So here's here. Let me just give you the definition of self, Max, if South Max works like this. So it takes some input from, or it takes some. You know. It takes an alpha from the network which you usually use. The letter Z to represent this output for the network,
David Bau: and it converts it to a probability distribution, and it does it by taking exp exponentials of the Z. So it it takes. If if the Z is ten, then it takes e to the ten, which is a huge number. If it's, you know. If the output is negative five. Then it takes e to the negative five, which is a mini-specule number
David Bau: A.
David Bau: Then once it takes you know all the exponentials of all the numbers, then it just signs the science probabilities in proportion
David Bau: to those exponential values. So um so the e to the ten will become, you know, incredibly dominant compared to uh anything that doesn't also have a large,
David Bau: He started,
David Bau: and so Ah, but if you have things that are sort of equal, then they they can start to have more similar probabilities. So this is the way Softmax works. It's called Softmax, because
David Bau: it, you know it emphasizes the largest thing in Z. So much that it dominates the others. You can almost think of it as just taking the Max of Z, and then just only letting that one show through, but it does it in a soft way. So if two are really tied, then they can. They can have probabilities that are close to each other.
David Bau: So okay, So that's the softmax function.
David Bau: Um. And the cross-entry loss has this form. So basically the cross entropy loss is is built on the Kale divergence, and I like writing the Kale divergence in this way. So what the Kale divergence is is
David Bau: It's really just the difference between
David Bau: um
David Bau: probability on one side versus probably on the other. So if the ground truth is Y and the predicted probability is P. Then we take the differences between them, and that's That's the Kl loss, but it's not the difference between the probability it's the difference between the law of probabilities. So
David Bau: so we take the log probabilities which is which you can think of as sort of the the the amount of rareness, the amount, and the number of bits. Ah, the number of point flips it would take to to to get something that rare um The long probability is, is is sort of just a measure of awareness, and as opposed to just raw probability, and
David Bau: and and and you take the difference between them. If If you have a perfect match, then this will all come out to zero,
David Bau: and then, and then you take the weighted sum over how likely something actually is. So So so you take the differences on each component, and the components are not just summed equally. There's some
David Bau: in proportion to how likely they actually are, which actually, for the
David Bau: star.
David Bau: You know a lot of our artificial classification problems. It ends up averaging out to. They're all equal because we have people weighted classes, but this becomes important. If you have non-equal weighted classes.
David Bau: This is um. This is Kl: divergence. Now, the thing that I've underlined here is a constant So for a given data set, you know
David Bau: why I are the ground truth labels for everything. So this term y I you know a log y I It will always be the same, regardless of what our network does the the network doesn't determine the ground truth. Um, So that can actually be separated out and just turns into a constant, it's just the entropy of the data.
David Bau: And if we subtract out the entropy of the data, then then we get the Cross Entropy loss, which is just the same as the Klos plus that constant plus that entropy constant. And so the cross-section p loss is just the expected value of the log of the
David Bau: a probability of the correct
David Bau: Ah, the correct values um negated. So the negative cross entropy is usually the loss of the years. So these are forms that you should just. You should commit to memory, you should know what they are. You should have some intuition for different things. I've given a little bit of math exercises
David Bau: on homework. One I probably won't give any more math to exercise than having to do with on homework. I was debating this with the Tas. Ah, but there, there's a lot of interesting mathematical properties that these things have, and and and it's it's good to be familiar with it.
David Bau: Ok, so softmax cross entropy. Loss
David Bau: is is what uh, in one thousand nine hundred and eighty uh by one thousand nine hundred and eighty-eight or so. People realized
David Bau: this. This is what the output should look like from a classifier it should be. You can think of it as a one-hot, or you can think of it as a probability distribution that adds to one.
David Bau: Now, now people propose a whole bunch of other ways of doing losses over classifiers as well. So your cross-centry loss is not really the only only way people have proposed a hinge losses. The whole Svm community is built around the idea of trying to create a margin
David Bau: ah around a hinge loss, so that you have scores that are going bad that are separated by some gap. Um. The classical error Formulation is mean, square error, loss, which is sort of Euclidean distance, or you can. You look at Euclidean distance in a couple of other ways, in terms of probabilities, and so on.
David Bau: So So these are all important losses. You'll see them from time to time, but really for the classification problem. Ah, you! You'll almost never see the classification problem nowadays stated as anything other than the cross-ent should be lost. So i'll be, even though these losses continue to be important for different settings. I'll put them here as a historical footnote
David Bau: are for classification.
David Bau: Okay.
David Bau: So okay, So oh, we have different properties of cross-section tree loss. Oh, I can't resist myself. This is Actually, I'll skip over this one. This one is directly on the homework.
David Bau: But one of the new things is that the softmax has this nice property; that
David Bau: that if you add a constant to all the logics, then the softmax probability is totally unchanged, which is really nice, because, if you think about it, one of the problems with gradients is that it could cause your
David Bau: your network weights to just get ever larger and larger the longer you run training, or maybe ever smaller the longer you run training. But the nice thing is that the softmax function doesn't have a bias. It doesn't really care whether the numbers are large or small.
David Bau: If you offset your numbers by the same amount, then it gives you exactly the same probability distribution which which means that the gradients are going to be flat in this way. There, there's no net
David Bau: um advantage to pushing the numbers bigger or smaller when they go through South Max. So it has this nice, this nice property that leads to good gradients. The other nice thing is that the cross-century loss also has good gradients, and it has them because
David Bau: it doesn't actually saturate. So on the homework. I asked you to do some exercises that have to do with this. But here's like a cartoon that I didn't draw in the homework.
David Bau: Ah, which is it so like? The blue curve shows you what the loss function looks like when you use cross-entropy loss or you know there's actually Kerl divergence, which is parallel to the cross entropy, loss
David Bau: on on logics, on a two class classification problem where the true probability is zero point three, eight, and you can see that it's. It's this nice steep
David Bau: loss curve that has some slope that never flattens out the the word we we use. Is it doesn't saturate it Doesn't saturate on the negative side doesn't saturate on the positive side it always has non-zero slope as as you go out towards infinity
David Bau: which means that you, you you don't get to rest if you're the optimizer with a cross entropy loss if you are way off and you can get pushed to improve the weights, to go towards
David Bau: the minimum which is really nice. It's It's different from mean square or error loss when applied on softmax, which does saturate. You can see if you were sitting there as a red rock, you could get stuck in a flat region if you were way way out towards a family with a really bad solution. You wouldn't have much motivation
David Bau: to down towards zero, because the gradient would be almost zero. It's saturated
David Bau: so. Um! So So there's some nice properties that cross-checkree loss has
David Bau: that are that are very practical. They don't have anything to do with, you know, information, theory, or other theoretical considerations. There's just these practical considerations that make it a really good loss function for engineering purposes.
David Bau: Okay, So we figured out like the the output of our image classification should be one hots or probability of probability distributions over an n-dimensional vector and and the inputs the inputs should be these crazy, huge rgb grid arrays.
David Bau: But there's this problem. There's this problem: I hate Rgb. Grid arrays of numbers for for images, and let me show you why there's there's this problem with them.
David Bau: I'm:
David Bau: When you have,
David Bau: you know this, this big creative numbers represents some sort of uh, you know, Image.
David Bau: Let me show you the problem with representing images, with just grids of numbers.
David Bau: Okay. So I want you to think about this. So take a look at these three images, A, B and C. Okay. So we've, we have. Let's say we have this image A, The,
David Bau: and we have to decide which one is more similar.
David Bau: B
David Bau: or C.
David Bau: What? What's the answer? Which one which one is more similar to A, B or C. And what's the right answer?
David Bau: Tell me why B should be more similar?
David Bau: Um, because it's just vertical lines. I can hardly tell the difference between A and B right. A A is like vertical lines,
David Bau: and B looks like really similar vertical lines.
David Bau: It's like the same.
David Bau: Looks like the same number of our clients same frequency.
David Bau: It's an eigenvalue.
David Bau: He is totally the same as as a right, so that that's right. That's it. It should be more similar. But why am I asking the question? Does anybody have a proposal for Why? I'm asking this question,
Ashwin Unnikrishnan: so everything is almost the same. If you rotate, c.
David Bau: It becomes a if you rotate, C: Yeah. If you were to rotate, c. It becomes A, or if you rotate it the other way, it would also become B, because they're actually they're all kind of similar in a certain way. They they're all rotations of each other. I like that answer, too. So C is also similar.
David Bau: Um, outside B is more similar because you don't need to rotate it.
David Bau: Um, at least in my eye he looks very similar.
Hendrik Ombach: So A and B are completely
Hendrik Ombach: like. If you add them together, it's perfectly black,
Hendrik Ombach: but their colors
Hendrik Ombach: are the opposite. On each straight. It looks like you are looking at the details of how I crop to these images, are you?
David Bau: That is right.
David Bau: So think about this question I put down below.
David Bau: Why are A and B
so far apart from each other as tensors?
David Bau: They're so far apart from each other as tensors, that c. Is actually even closer.
David Bau: So they made the beers.
Ashwin Unnikrishnan: Can you take the difference?
David Bau: So all the blacks are white and B
Ashwin Unnikrishnan: black, and we?
David Bau: Yes, if you see A and C
Ashwin Unnikrishnan: like some parts of black in A is filled with some parts of uh black and sea,
David Bau: that's right or less errors.
David Bau: Yeah, look at that. So if you were just counting out black and white,
David Bau: there are many, many points in a
David Bau: which are black that are also black.
David Bau: He's he,
David Bau: and there are many points in a that are white that are also white, and C. Now, to be sure, there are many points that are the opposite. But there there are many points in agreement, and i'd say about half the points in a are in agreement with about half of the points in C
David Bau: just out of a checkerboard principle. If you kind of think about a checkerboard pattern really carefully, you can imagine where all those points are the same
David Bau: in contrast.
David Bau: How many black points in A
David Bau: are also black and B.
Tarandeep Singh: It goes to zero.
David Bau: Yeah, the way I've arranged it pretty much. None.
No, I you know, if if I if I arrange it the way I've intended,
David Bau: madam,
David Bau: and and how many white points on a are. This is also white, and be none, none, none of them. That's him. Every white point They they're inverse images of each other, you know every white point is is a black point and B, and every black point. It is so
David Bau: so, even though these images are about as similar as you could imagine them being, you can hardly distinguish between them when you glance at them. If you were to treat them as tensors or vectors,
David Bau: these are pointing in almost exactly the opposite direction as each other. They're as far apart as you could be from each other in vector space, which is This is just a simple demonstration of why Rgb. Representations of images
David Bau: are so terrible, because you know, you expect vectors to behave in a nice way over vector space. If two vectors are related, you expect them to be sort of close to each other, at least have similar angles point in the same direction or something.
David Bau: But these these these things, these images are about as similar as I could make images similar to each other, and yet they are as opposite and far apart, and pointed in opposite directions as any images that could exist
David Bau: as vectors, any vectors that could exist in the space. And so they're terrible. So this is. This is a problem with images represented as Rgb space. And this problem kind of leads to
David Bau: this question and this architectural decision of how to process images and typical deep networks. And So how is that related to anything? Well, so it wasn't obvious how it was related to anything for people for a while, until
David Bau: until people started really thinking about how the visual system might work by doing some investigation on actual biological animals. So the the work here that was Seminole was done by a couple of folks who Ah, Hubble, Hubble and Weasel.
David Bau: Um! Ah, they They're the two most famous members of Ah, you know a group of ah neuroscientists and electrical engineers who were really trying to understand the wiring of the nervous system and of the brain.
David Bau: And these these these two. There are a couple of people who are also looking at it, that I put in the reading list in papers. But these two are the ones who did it most systematically. Ah, most thoroughly over the longest period of time. They wrote several papers. They eventually won the Nobel Prize for the work.
David Bau: But you can see the first paper that was published in one thousand nine hundred and fifty nine. Taking a look at the cat.
David Bau: Ah, the cat optical system. Okay? So why am I bringing up these guys. Well, they they they observe this really cool thing which is in this diagram here, down the left, on the right here, which i'll, i'll explode here on the left here. Okay, So what they what they discovered was they were measuring
David Bau: um optic nerve
David Bau: upwards
David Bau: from a cat eyeball.
David Bau: Um, you know they'd wire up a cat, and then they have the cat. Look at
David Bau: different cards that have these sort of pictures here on the left right,
David Bau: with a big stripe,
David Bau: you know, on the card, and then they would rotate the card like this, you know, with the stripe at different angles,
David Bau: and then what they would find is if they were measuring a specific optic nerve.
David Bau: They're not changing the brightness or darkness of the image here at all. This is the same image just rotated. They're just changing the angle of the big line, And what they found was that the optic nerve was very, very sensitive to a specific angle
David Bau: nearby angles, but then almost not sensitive at all when when the angle changed,
David Bau: change past a certain threshold,
David Bau: and so they measured like sensitivity to uh orientation, and they they found this angle sensitivity in a lot of the uh neurons that connected um the retina to the uh, to the visual cortex.
David Bau: And and this this ah, this this orientation sort of selectivity? Um, Later on in the nineteen ninety s somebody actually Ah, you know, went and found found the neurons that were actually in the retina. They did it in rabbits this time.
David Bau: Um! That were responsible for this, and they found that that a single retina neuron had this funny ah sensitivity region where it was positively sensitive in these up and down directions like this, and negatively sensitive on the signs like this.
David Bau: So i'm gonna draw this sort of number here. So if we would like to draw a grid like this, you could imagine, like negative sensitivity on these edges and positive sensitivity down the middle axis. Ah! Like this! This is. This is the kind of thing that they were able to directly measure in these in these
David Bau: in these retinal cells.
David Bau: And so so this is. This is really cool. Actually, this is something that you can do in your neural networks. So let me let me show you how you do this in a neural network. The computational solution it's called a convolution operation. I'll show you how convolution operation works
David Bau: over a one-dimensional vector if you've not already seen it. If you've seen it, then just bear with me, we'll go through it pretty quickly, and so the idea is in blue and gray. Here is our input data like that. You can think of it as the the photons that are striking the right now
David Bau: and then over here in yellow and ah and orange are are the weights of a filter that we're going to apply a convolutional filter that we're going to apply
David Bau: to the input, and then we're going to produce output here like this. So basically we've got a couple of zeros. They get multiplied by the filter here in these spots, so they don't really contribute anything. But there's a one that gets multiplied by this negative point five here,
David Bau: and that ah! It produces output from this filter of negative point, five in total. And we can. We can slide this filter along to the right here to see how It's going to behave when it consumes these three inputs here like this, and you can see Oh, the one goes through here and the two goes through here they actually this becomes plus one and negative point five. They They neutralize each other out
David Bau: that turns into a zero. So if we slide it over Here we get this interesting confluence of one, two, one and negative zero point, five, one. This, this, all three of these are now active, and and these combined to negative one. This turns into positive two
David Bau: that we got a nice positive number here. Our first positive number Ah, of of this test here shows up at this spot, and if we move over here we get zero again over here. These are actually both negative. So this turns into negative one.
David Bau: This one. Here is another example of positive point. Five over. Here is a symmetric side. It's positive zero point five. Again, this guy turns into negative zero point. Five. Okay. So this is. This is basically what the output looks like.
David Bau: And um,
David Bau: let's see. I didn't meet it. I'll shoot
David Bau: what I do here.
David Bau: Okay, now I've lost. Am I still projecting?
David Bau: I picked up, lost my projection
David Bau: hopefully. I'm still in zoom
Andrew Lemke: yesterday. That's you're not sharing your screen right now?
David Bau: Yeah, I know.
David Bau: Okay,
David Bau: just because I pressed the wrong mouse button.
David Bau: Let's see if I can find this here.
David Bau: Where is it?
David Bau: Is it this one? Are you able to see that now.
Andrew Lemke: Yes, yes,
David Bau: okay. I'm: in some weird Powerpoint mode that I don't want to be in. But hopefully, I can still present.
David Bau: Okay, so so you can. You can kind of see.
David Bau: Ah, what what happened here with the pink output? Um, you know, by by by passing this little filter over all the data. Then it kind of so summarizes the data by adding up um different weighted sums of adjacent adjacent little squares like this, and we get
David Bau: some pattern output like that looks like this. Okay. So there are a bunch of different situations where you might want to actually do a one-dimensional convolution. But in vision you tend to want to do this over two-dimensional like grids of data. So let me show you what that looks like.
David Bau: So let's say we have a two-dimensional grid of data with these vertical stripes like this. So you can imagine this is this is just like a little slice out of that vertical stripe image that I showed you that was supposed to be so different from the horizontal stripe image. And let's take this
David Bau: little patch here as our convolutional filter. So just like our input data is a grid. Our little filter is also going to be a little grid,
David Bau: and we're going to apply this filter in spots like this, right? So if I took this filter and I lay it it on top it
David Bau: of this spot here, and I took weighted sums of all the negative point fives that add up to you know, negative one and a half on this side, negative one and a half on this side the negative three. Then if we, if we added all these positive ones in the middle, that's plot positive six, the whole thing would add up to positive three,
David Bau: so we can. We can plot this anywhere on the grid, and we can scan it all the way across, You know, when we scan it all the way to the end. You know we could put over here with this over here as up to an eight to one point five, and
David Bau: And you know, as we scan around, we can actually get this sort of nice output that looks like this. And so see what kind of output we get. We get this like
David Bau: um, this thing that detects all these vertical stripes. It turns into this really positive zero point, three here, which is sort of the output of putting this filter across this really positive point, two line
David Bau: and some some positive point, One point five to come out of this sort of lining over here.
David Bau: Um, you. So we got this nice vertical, and we call this a vertical edge finder, and let me show you why it's called a vertical edge finder. I'll show you a different patch that you could use.
David Bau: So this this patch here is exactly
David Bau: this patch Here we call this a convolutional kernel. This this little kernel here is exactly the same as the other one, but transposed sideways is a horizontal edge Finder and i'm going to apply it to the same data here,
David Bau: and you can see what happens to it. These all get multiplied by negative zero point five, and they all neutralize each other here they all add up to zero.
David Bau: In fact, no matter where I put my patch on this particular input pattern,
David Bau: it's always going to neutralize itself. This thing pretty much ends up with zero
David Bau: everywhere.
David Bau: It's a horizontal edgefinder,
David Bau: and in this image it finds no edges whatsoever, and so does that sound familiar right? This should sound. This should sound just like the Hubble and weiss all neurons that are sensitive
David Bau: edges. This would light up like crazy if it was presented with an input image with horizontal edges, but presented with an input image with only vertical edges. It has nothing to say. It's completely silent.
David Bau: And so. Um.
David Bau: So this is how the
David Bau: and and so when we process images, then we tend to process them in images like this, with with filters like this as Well, the um! Ah! It's filters like this, which allow our convolutional nets to say that these images that have horizontal stripes, are so dramatically different
David Bau: from the images that have just vertical stripes, while the vertical stripe images are very similar to each other. It's because the vertical stripe images are activating these sort of vertical edge finders, and they're not activating horizontal edge finders, and it's not just about vertical and horizontal edges.
David Bau: There's lots of other different combinations. You could do So the the big idea
David Bau: um of of using these kind of convolutions and deep nets is
David Bau: that they can be learned. We don't have to set up a horizontal or image, edgefinder by hand. We can actually make these orange and yellow numbers parameters
David Bau: for a deep network and learn them using back propagation just like we can learn weights of any normal linear operation using back propagation. And so
David Bau: um, let me introduce you to the team who first came up with this idea that you can learn these convolutional weights in deep network. Does anybody know who that is?
David Bau: It's It's it's it's these guys. Look at these good files here.
David Bau: This is Jon le Kun. Ah, ah! Just a couple of years after he invented this idea of ah doing convolutional neural life. Look at this very modern equipment he's got he works for at and T. So he has the best, most modern and most expensive equipment that is available to anybody in that day,
David Bau: and and there you see him.
David Bau: Ah, putting images of his own handwritten digits there, and he is running his neural networks to try to make predictions of what this thing is here. That's four, six, eight, seven. Oh, look at this crazy adversarial case here, where you've got a six made out of little zeros
David Bau: you got like two made out of parallel stripes, and so on. But the the network seems to be quite happy uh to to read it. And obviously Jonathan didn't
David Bau: work on his own. He had many collaborators. A few of them are here quite happy with their accomplishment of what they were able to do in this time. So So this this paper had written digit recognition, using back propagation network in one thousand nine hundred and eighty nine
David Bau: was at the time it really stood out. It was a tour de force while everybody else was doing neural networks on really toy data that was just very artificial and synthetic.
David Bau: Ah, Yanlakun and his team at A and T. And T. And T. We're trying to tackle a very practical problem which was how to help the Us. Postal service, read zip codes on post-mail, And so they went to the Us. Postal service and they collected many thousands of examples of actually, you know actual handwritten digits
David Bau: that that that data is now called Mnist
David Bau: Um, and Ah, and and they train their network to solve amnest. Ah! And so down here you can see this picture of a convolutional Ah, kernel processing the input data to the output data in the way that we've talked about it.
David Bau: So let's see how we doing our time.
David Bau: So I've got a little bit of time here to just give you a little sense for how these convolutional kernels up here in A, in a deep net.
David Bau: But let's see here,
David Bau: here's how it looks.
David Bau: So your input data is this: Rgb: Grid
David Bau: this terrible Rgb. Grid. But instead of processing this Rgb. Grid using an unconstrained vector operation instead, we use these convolutional filters that are much smaller than the data
David Bau: it's.
David Bau: And
David Bau: so there. There has to be a filter that consumes every channel of the input So if it's three channel, input, then it's got a
David Bau: consume, three channels on the filter side, but it doesn't have to consume. I have a filter entry for every pixel, and the height in the width direction, because what we'll do is, we're going to translate this filter all the way across the image to apply it to all the locations
David Bau: like this,
David Bau: and so you produce you. You you present one of these filters,
David Bau: and uh, it does a way that some which is just this linear operation looks like this,
David Bau: but then it applies it
David Bau: sliding all the way across the image
David Bau: uh, and producing producing output is also in a grid
David Bau: like this. Now notice that the output grid
David Bau: is not exactly the same shape as a for grid. And think about why?
David Bau: Ah, that might be.
David Bau: Ah, now
David Bau: that gives us one number like one heat map over the upper grid
David Bau: um correspond to one filter so like if you had a horizontal edge finder that that that might find all horizontal edges. But ignore the vertical ones. You can have a second filter if you like. That does all the vertical edges, and then you would get another output
David Bau: in practice in neural networks. We have many filters that we allow the network to learn that they give a lot of output
David Bau: grids uh one for each. Um. You can think of it as one for each concept, one for each visual concept
David Bau: that is, uh is represented by each one of these filters. And so, yeah, So if you've got six filters. Then you have six output activation maps. Um, and and then you can. You can think of those whole things like one giant tensor of six by two,
David Bau: twenty-eight by twenty-eight.
David Bau: Ah!
David Bau: So all together we call this a convolutional layer
David Bau: a whole bank of convolutions like this, you know, including the bias. This whole thing
David Bau: is actually a a linear operation. It's just a linear operation that happens to be constrained
David Bau: to have a particular form.
David Bau: Um,
David Bau: And so
David Bau: ah,
David Bau: Um.
David Bau: Yes. So when we are working with this data and pi torch, we will often have three-dimensional
David Bau: data flowing through our network
David Bau: it's,
David Bau: which uh uh! Which has height and with
David Bau: and sort of channel-wise depth.
David Bau: So, like for input, images, we have height and pixels
David Bau: within Pixels and our channelwise depth is always
David Bau: typically three for red, green, and blue.
David Bau: But after we go through a convolutional layer we might have a channel-wise depth that might be six or sixteen, or sixty, four
David Bau: for uh one, one channel for each of the filters that we last run on the data.
David Bau: So so we've got this sort of six by twenty, eight by twenty, eight, a three-dimensional output here. And then, if we,
David Bau: if we we go one step further, and we say, well you know we can do more parallelism than that. We We want to process several images at once. So let's say we want to process two images at once. We might have a batch, and then that corresponds to a batch of convolutional outfits, you know. So typically, I think I mentioned in
David Bau: the how to read Pi Torch Notebook Series that the convention that you'll see inside the
David Bau: deep network's provision is You'll see these four-dimensional tensors,
David Bau: which have a batch dimension,
David Bau: followed by a channel the mission followed by height with
David Bau: like this and that. That applies both to the input and to these intermediate internal outputs. And so now you can see the picture of what the internal outputs.
David Bau: Ah, why the enter opens? Have these four dimensions like this?
David Bau: Okay.
David Bau: So I have a bunch of other stuff that I want to run through. But I know that we're gonna. I wanted to do this for every class, and I haven't
David Bau: I got a chance for a couple of classes, because i'm always running over. So before we get to the end of class, I want to ask you to guys to take a break
David Bau: and uh fill out this form. Actually, i'm gonna
David Bau: i'm gonna copy this url here into chat.
David Bau: Ah, so that you guys have a hyperlink
David Bau: and uh everyone in a meeting.
David Bau: So Yes, so I need your help.
David Bau: It's calibrating.
David Bau: Ah,
David Bau: this this course. So if you can answer the questions on this form here um, my intention is to uh let you answer these questions.
David Bau: It just take a minute. It just takes, you know, sixty seconds
David Bau: to answer these questions for me,
David Bau: Adam,
David Bau: and That'll that'll help me get it. Get a sense for what? What types of topics I should be talking more about, or less about.
David Bau: Should I be going faster or slower, or maybe it's not about speed. Maybe you want to hear
David Bau: more about certain types of things, and uh and I can. I can tailor the the course accordingly.
David Bau: So when you're um
David Bau: completed doing the form, give me a little thumbs up any time i'm going to actually take a walk down to the kitchen to get myself a glass of water. I'll be right back. Okay,
David Bau: i'm not sure how many people give me the thumbs up already, because It looks like the thumbs up only last for three seconds. Maybe everybody already thumbs up
David Bau: um.
David Bau: But i'm going to. Okay. Yes, I see more thumbs up that I must have missed. So i'm going to go ahead and proceed a little bit. Just just talk a little bit more about convolutions for a few minutes that we have for the rest of the day, and then we'll we'll continue this discussion of what's going on with these
David Bau: the convolutional networks uh next in in the next session.
David Bau: So let me let me show you an interesting thing about convolutions. So convolutions are. I mentioned that they are linear operations.
David Bau: And um.
David Bau: And so what do we know about linear operations? If you compose two linear operations,
David Bau: then
David Bau: what do you get?
Lakshyana KC: Are also Vineyard?
David Bau: Yes, it's exactly right. If you compose two linear operations, you get another linear operation.
David Bau: Now let's take a look at what happens.
David Bau: Um,
David Bau: if you um,
David Bau: if you take two
David Bau: convolutions
David Bau: and you stack them on
David Bau: one another.
David Bau: So here we've got three-dimensional input We have six filters
David Bau: that we're running um
David Bau: uh. They have this this sort of uh, you know five by five uh dimension here,
David Bau: and that produces a six by twenty-two. I i'll put and then we have
David Bau: um
David Bau: another bank. Here are us
David Bau: ten filters that we run to get this kind of output, and we could run another bank of, you know, twelve filters
David Bau: uh to get some other, some other further output.
David Bau: So we can like, You know, we can like continue to stack these convolutions like this. And actually, this is, and our data we will tend to do this. The idea is,
David Bau: then i'm
David Bau: that you know a grid of data.
David Bau: It's sort of not a very good summary
David Bau: of the entire image, because it
David Bau: it, it just has too much detail the
David Bau: But if we took a convolution to find all the vertical edges, then maybe this is a little bit better and more economical summary, because now it just shows us something a little bit
David Bau: ah, more of a summation. It. It shows us where all the vertical edges are, or maybe where all the horizontal edges are, and maybe we could come up with some kind of convolution here that shows us where all the vertical edges intersect, or something like that, maybe, where all the corners are. Um, and then maybe we can go from corners to higher level shapes, and
David Bau: ah, and and textures and things like that, so you you can imagine, like going through a series of convolutions to to get more things. But there's a little bit of caution here,
David Bau: because the convolution is a linear operation. So if we compose the
David Bau: multiple convolutions,
David Bau: then we're definitely going to get
David Bau: the composition of linear operations which we know we're going to get a linear operation out.
David Bau: Um,
David Bau: you know. Beyond that
David Bau: we also get
David Bau: convolution out. So actually the the the composition of a stack of convolutions is actually just,
David Bau: you know, a bigger convolution. Um! Ah! And you you can Actually, you could actually work out exactly what the weights and the the size of the large accomplish. It just corresponds to some larger matrix of weights that you could have applied
David Bau: over the original data just to go directly all the way to the output.
David Bau: So what people do is to make this more interesting, just like we did in neural networks to make this into a much more powerful function. You add nonlinearities in between,
David Bau: and the popular non-linearity that everybody uses nowadays is the relu, because it doesn't saturate, and and the the rally continues to work well even in this convolutional setting over images. So this is this is the typical architecture you see,
David Bau: for what's called a deep convolutional neural network, a stack of convolutions.
David Bau: Um that
David Bau: continually make the image ah smaller
David Bau: and um, and then alternated with, uh uh rectified
David Bau: linear units.
David Bau: So let's contrast.
David Bau: Let's contrast what a um!
David Bau: Ah! You know how a fully connected layer works compared to a convolutional layer. So a fully connected layer is a fully connected layer is made up of a bunch of neurons and a single neuron, remember, from a color of pits is just equivalent to a weighted sum
David Bau: of its inputs.
David Bau: So you know it, just it just takes away to some of its events of its inputs. It's equivalent to a dot product here. That's Wtx, and it just produces one scalar output. If we wanted more than one scalar out. But we would need
David Bau: more than one
David Bau: neuron.
David Bau: So the interesting thing is
David Bau: in a convolutional kernel. You can think of it as the same way,
David Bau: Right? You can think of it as a neuron
David Bau: has as its inputs. If it was like a five by five
David Bau: by three filter, it has as its inputs this little square of
David Bau: of what it can see. This little box
David Bau: of its input data. And so it's sort of a a neuron that is only connected to this little part of an image, and it produces this one number
David Bau: summarizes whatever it's looking for in this little, this little input grid.
David Bau: And now it's just one neuron when you look at it this way. Now it happens to be scanned over all the images, or all the little patches all over the place, so you could. You could think of it as a twenty, twenty, eight by twenty, eight grid of equivalent neurons that all
David Bau: have the same job. They all look at this little input, patch, and they all summarize, summarize the output. So you can think of a neuron.
David Bau: Um! And you can think of a a convolutional filter at one location.
David Bau: It is.
David Bau: It's pretty much having the same
David Bau: having the same role.
David Bau: Okay. So if this neuron here has three thousand
David Bau: inputs,
David Bau: then tell me, like, how many inputs does this neuron have
David Bau: after this filter.
David Bau: Um! Oh, not not I don't. I don't want to ask. Ask Ask the questions that way. I want to ask. Ask question this way so like, How many, how many inputs does this neuron have for this? Uh, for the way they switched up here?
David Bau: Can somebody answer that question for me,
David Bau: how many, how many inputs does this bottom neuron have? I know that the top neuron has three thousand and seventy two inputs. Um, You know how many inputs does this neuron have
Lakshyana KC: uh seventy-five?
David Bau: Yeah, it's much smaller. It's It's got seventy five inputs. Each One corresponds to some weight here. So the number of weights that determine this um neuron's output is a lot smaller than the number of weights for this thing. So this is a typical pattern that convolutional neurons actually have a relatively small number of weights
David Bau: because they they look at a relatively small piece of the input data.
David Bau: In contrast,
David Bau: If we were to have
David Bau: neuron that looked at all thirty by thirty, two by thirty, two like the If it were to look at the entire image,
David Bau: you know how how many. Would that be
David Bau: so? You said Seventy five for that? A small one. But what if the kernel size? What if this neuron had a much larger filter,
David Bau: where it it had to look at the entire image at once. Thirty-two by thirty-two, by all three
David Bau: it's
David Bau: of the ah!
David Bau: Are the channels.
Tarandeep Singh: It's a three thousand and seventy-two
David Bau: it is. Did you use your calculator, or do you in your head?
Eric Grimaldi: I used to calculate.
David Bau: Oh, that's. Good. Okay. So yeah. So the trick here is thirty, two is two to the five, to the five times two to the five is two to the ten. Everybody should know what to the tennis. That's one thousand and twenty, four thousand and twenty, four times three looks suspiciously like this other number. Here on the slides, probably the same number
David Bau: two thousand and seventy two.
David Bau: Okay.
David Bau: So
David Bau: that's right. So if the kernel size was to look at the whole image, then the number of parameters for this thing would be three thousand and seventy two. In other words,
David Bau: this a fully connected neuron here that I have drawn
David Bau: it's exactly the same as
David Bau: a convolutional filter
David Bau: looks at the entire image
David Bau: all at once.
David Bau: They're exactly the same operation.
David Bau: And then what we've done here, though we've we've if we restrict the convolutional filter to only look at a small portion of the image at once. Then we get a slightly different beast, because it has less power
David Bau: right it it. It can all. It's restricted to only look at looking at little patches at a time.
David Bau: And so ah! But we can still think of each one of the individual neurons, each one of the individual outputs like a little neuron. So this I like this abstraction a lot. I think this is um a good way of thinking of. You know what we're doing, and when we're doing convolutions, you can think of this filter as like
David Bau: little job of something that's computing. Oh, are we looking for vertical edges? We?
David Bau: Oh, yes, If so, then we should light up.
David Bau: Okay.
David Bau: So. Um.
David Bau: So a fully connected layer. Ah, you know. Does this
David Bau: convolutional layer? Um. Does Does this right? It looks at this little little patch, and because it doesn't reduce everything down to a single neuron, it outputs this whole creative neurons. We call it a feature map, right? So you'll see this sometimes in the literature.
David Bau: Uh you know that each one of these interior plans of a cobbler on that, and it's called a feature map.
David Bau: And so now you you can. You can take either one of these approaches to do this, let's say we took the approach on the left. Let's say we want to have a neuron that looks at the whole image. Then what kinds of things Do these whole neuron
David Bau: whole image neurons learn? Well, you know, the neat thing that you can do is you can go to a neuron that is trained to look at the whole image, and you can actually, since each weight
David Bau: corresponds to a pixel location in the image, you can actually
David Bau: what the weights are by making a bit map where you you know you give a bright,
David Bau: bright color to any weight that's large and a dark color is any weights, it's small. But
David Bau: so let me show you what that looks like. So so if you have
David Bau: I
David Bau: a ah,
David Bau: so let me show you this.
David Bau: I'm going to show you
David Bau: um
David Bau: Mlp. Mlp. Mlp. Is a multi-layer perception. So mlp is code for, and I apologize. This This finger here isn't correct. So this is this: this: This is code for the neurons
David Bau: uh that are acting as perceptrons. This is what you see. If you train neurons to look at the whole image at once.
David Bau: Then your neuron weights will have things like this where where each one of the neurons is looking for a kind of a ghost of a pattern, you know. Maybe these neurons are looking for bowls, or
David Bau: you know, some specific shape. So it's actually looking for pixels in a specific location in the image, and you can kind of see, like an average image of of of the things it's looking for. If if you look at a fully connected
David Bau: neuron to see to see how it's connected. Okay, So let me let me contrast this with you
David Bau: what it looks like. If you look at a neuron that is um acting like a convolutional filter that only gets to look at a small patch of images like, What do those weights look like? Well, we could take one of those weights, and also visualize it directly
David Bau: by by drawing brighter, brighter, or darker colors, but just over its little it's a little patch. So, by the way, these little patches here
David Bau: they're also called there. There's a name for this little patch. It's called a receptive field
David Bau: of a neuron or of a convolutional kernel. It's like how much of the input image this
David Bau: this neuron is sensitive to. Obviously it ignores everything outside of the receptor field, so you can. You can visualize the weights by just drawing the weights over the receptor field. And so for a for a little convolutional filter. This is actually what the weights look like, and you can see that that we have what
David Bau: um I was showing you in the in the in the first part of this lecture we have these horizontal edge detectors we have, but there's some subtlety here. We have some diagonal edge. Detectors. We have edge detectors that are looking for edges of particular combinations of hot and cold colors,
David Bau: and we have different edge detectors that are looking for edges of different frequencies, widely space edges. There's also these funny texture detectors that are looking for these neat cross hatches of things and things like that. So this is what we get when we
David Bau: ah train and network to learn its own convolutions. Eventually you get these interesting patterns of convolutions that that emerge which is really cool, because these kind of patterns of convolutions are very similar to what you see in biology.
David Bau: There is some you you can debate in biology, whether ah biological researchers also learn their convolution. So whether they're just pre-baked in. There's probably a little bit of both. There was some there's. Been some new experiments that suggest that um in biology. They also learn their convolution. So so
David Bau: so that's so. I think that we we are at the end of
David Bau: our time. So before I go into some other things that you should know how to do with convolutions. I will close for today, and we will pick that up. Ah! For tomorrow I will. I'll just open the floor if anybody has any questions for me, or anything like that.
David Bau: Um! But then i'm going to close up, and then i'm going to take a couple minutes, and i'll hop over office hours.
David Bau: Yes, so can we reduce the small filter
David Bau: a three by five, by conf to three, by one by one, and one by five by five, so three by twenty, five. It would take
David Bau: less memory, indeed. So you can. You
David Bau: factor convolutions in exactly the way that um. So Kevin sent me this question and direct Kevin, can you? Can you chat that question? Ah, globally to the team? I'm going to Actually, i'll read it, and i'm not sure if everybody is still in the lecture. Let me see here.
David Bau: Oh, yeah, we still have a few people here.
David Bau: So Kevin Russell asks i'll read the message. Can we reduce this small filter of three by five by five
David Bau: uh which has, uh, you know, seventy-five weights um to be only twenty-eight weights, by first using a three, by one by one convolution,
David Bau: and then a one by five, by five convolution, a three by one by one convolution would take. Ah! So these one by one convolutions are really interesting. I should make a slide on about them tomorrow. They but what they do is they would look at every Rgb triple
David Bau: and figure out how to do a Linux coverage and reduce it down to a single number that summarizes that without looking at any neighbors, and then a one by five by five convolution, we take a look at this one number that we got to summarize the color that in a in a spot, and look at the twenty five adjacent
David Bau: five by five sort of colors, and you know, summarize the whole pattern as a single number as output, so that would be twenty five numbers there,
David Bau: and you could do one after the other, which would mean that you have a three parameter operation, followed by a twenty, five, a private operation which is only twenty, eight primers, a lot less than the twenty, the the seventy five
David Bau: that we're observed that that are consumed by the the three by five by five. So you can do that, but it has less power.
David Bau: I'm.
David Bau: So This idea of factoring convolutions
David Bau: in order to save parameters, was proposed by Google, you know, in two thousand and thirteen or fourteen or so. I think that the
David Bau: their their network that they call the inception network
David Bau: uses this factory. It's slightly different factoring that than what Kevin is proposing
David Bau: to save some memory. Um,
David Bau: they would argue that even though the convolutions you get from stacking them in this way are less powerful, they are powerful enough
David Bau: to do interesting things, and so they would advocate doing this. The other way of factoring things is, instead of having
David Bau: um
David Bau: a five by five convolutions or seven by seven convolutions, or these large kernels you could stack two kernels. So if you have a three by three kernel,
David Bau: and you're stacking on top of another three by three kernel. This is actually equivalent.
David Bau: It is something that has a five by five receptive field.
David Bau: But if you add three by three instead of nine parameters
David Bau: and three by three again, is nine parameters, so that would be eighteen parameters in total.
David Bau: If you take five by five. That's twenty, five parameters. Interestingly,
David Bau: eighteen is fewer than twenty five, so you can save a bunch of parameters by just fact by just doing two steps of three by three kernels instead of a single five by five,
David Bau: so that that's the other thing that people noticed.
David Bau: And so it turns out that if you look at most convolutional networks today,
David Bau: uh they are all made out of three by three kernels, and that's because that's about a small season you get
David Bau: uh while getting a meaningful kernel. Um, and and it's a good factorization of all the larger kernels,
David Bau: and it's a little less powerful than learning a five by five or a larger kernel. But it's a lot less parameters, and it can train faster. And And you know some people would argue that you don't lose any meaningful power that it's powerful enough
David Bau: so it's good. It's a good question,
David Bau: and very relevant to what we're talking about here.
David Bau: All right.
David Bau: If anybody has any other questions, they can text in the chat,
David Bau: but if not, I will wrap up uh for today's class, and i'm going to have uh zoom office hours uh in a couple of minutes, and the the link is on canvas.
David Bau: Uh, thank you, guys again for putting up with this whole
David Bau: Covid. I'm starting to feel better. But I I still am probably infectious. So I don't want to infect people for another few days.
David Bau: Thank you very much.
David Bau: I suspect the next class will be in zoom. I have to test negative for a couple days before we do that, so which i'm not yet.
David Bau: But my my guess is that next week we'll be back to in person.
David Bau: If the next class is on zoom, I will post a zoom link, and if it's not on zoom, then I will. They'll send a message out and say it's not.