David Bau: hey there fern.
David Bau: See.
Fern D Keniston: hi David i'm glad you were able to join us.
David Bau: Oh yeah I heard that somebody is going to give an interesting talk today.
Fern D Keniston: that's the rumor yeah.
How are ya.
David Bau: So let's see Oh, it looks like my parents are already here in my in laws are here wow that's great.
Antonio Torralba: Hello.
David Bau: Antonio.
David Bau: So let me start up my things here.
David Bau: I should probably quit my.
David Bau: instant messaging things.
David Bau: and
David Bau: I protest.
Gerald Jay Sussman: don't worry about it.
Gerald Jay Sussman: Things interesting things are interesting.
David Bau: So.
David Bau: yeah So hopefully, hopefully, will make it interesting for people, I will see if it matches up.
Gerald Jay Sussman: thesis defense of course.
Gerald Jay Sussman: Oh yeah Oh yes, I was the math department I we did it did a thesis for actually marvin and see more it was a higher stuff and the math department, the head of the applied math section thought that the only meaningful.
Gerald Jay Sussman: manner math was from his point of view was fluid mechanics.
Gerald Jay Sussman: Here I had prepared a big fight with him, where I was gonna we're gonna take you apart.
Gerald Jay Sussman: He didn't show up.
David Bau: didn't come.
David Bau: He didn't show up, oh no.
Gerald Jay Sussman: Oh well.
David Bau: Oh no you're ready yeah.
David Bau: So yeah so that it's great now Jerry are you in are you in Cambridge right now.
Gerald Jay Sussman: No i'm in arlington Massachusetts, which is where I live.
David Bau: Oh okay I didn't know that but you're not far.
Gerald Jay Sussman: yeah let's wait for 5.7 miles.
David Bau: yeah yeah yeah that's you know that's biking distance.
Gerald Jay Sussman: Oh, my bicycle the time.
David Bau: yeah yeah so so I know we have people from Germany and Hawaii here because it's virtual.
David Bau: And I think that my wife just poked her head and she says she's gonna dial in from far away from upstairs.
David Bau: Oh yes, so.
David Bau: um.
Gerald Jay Sussman: it's it's anyway um well, I think we start up in about two minutes right.
David Bau: yeah I think.
Antonio Torralba: Myself, right here.
Gerald Jay Sussman: So i'm not making noise.
David Bau: Okay that's great.
David Bau: And then.
David Bau: I get my slides ready actually should I stop projecting here so that I, you know I don't know how how the routine goes, I can start projecting one.
Gerald Jay Sussman: anyway.
David Bau: So right so i'll stop sharing here.
Antonio Torralba: yeah this way was able to buy the.
yeah.
David Bau: that's great holy cow.
David Bau: Look at everybody.
Antonio Torralba: With a studying two minutes.
yeah.
David Bau: I see my readers are here.
David Bau: I I must I must anticipate some questions.
David Bau: uh oh Nancy is here.
David Bau: hi.
David Bau: I.
David Bau: See you oh yeah yes.
David Bau: it's amazing.
Nancy Ann Lynch: This is an email your work is inspired some of our theoretical work.
David Bau: yeah yeah yeah.
David Bau: So exciting.
David Bau: yeah you know I i'll talk about this in the talk, but I think that one of the interesting puzzles is you know we're not quite sure why it is that we see the structure that we we do I think there's a lot of questions to continue to investigate back.
David Bau: And I see his friends that come from, but before my MIT days, just so exciting, thank you for for coming here.
David Bau: And and some new faces from people I haven't met before, so I guess the word is going around a little bit.
David Bau: And and welcome.
Aleksander Madry: hey David good to see you.
David Bau: yeah thanks for coming.
Aleksander Madry: first day I see you in a tie.
Aleksander Madry: I know.
David Bau: I think that I had I had to dust this off because I haven't used it for several years, even when I did my interviews I didn't put on a tie, I couldn't I couldn't misrepresent myself too much, so no time for the interview.
Antonio Torralba: Okay let's let's.
Antonio Torralba: Listen, this is that.
David Bau: Okay, great.
Antonio Torralba: Okay, so, so let me first describe a little bit what will happen here is a first well, thank you, the committee for being here, of course, and so I will talk for a few minutes.
Antonio Torralba: And then they read a little present for.
Antonio Torralba: For how long as long as you once.
David Bau: An hour so yeah.
Antonio Torralba: Yes, and then it will take questions from the audience.
Antonio Torralba: Once the audience has no more questions we'll go to a private room and the committee will ask more questions to David.
Antonio Torralba: And then the Committee will meet along without David and we'll take a decision or happens here.
Antonio Torralba: And then, at the end we'll have a little celebration here I mean we cannot have a celebration in person, unfortunately, so you know just feel free to to stay on the line at the end and just hear from David studies off his his PhD but.
Antonio Torralba: Hopefully we can do something in person during the fall.
Antonio Torralba: But before I wanted just to say a few words, and so this is it's always a strange moment when when I still in the fans, because on one side is kind of sad because you know I still then is kind of part of.
Antonio Torralba: A family, you know that the appeal over over several years and it's really sad but when I feeling goes, because if you know the lab changes now everything everything changes and bass, especially through with David.
Antonio Torralba: But it's also you know fun, because you see them going out and become an independent and doing their thing, and so on.
Antonio Torralba: But, of course, David has been independent already for several years, this is not changing now i've Actually, I would like to know from the family members, but i'm pretty sure he was born independent already and so David has always been.
Antonio Torralba: You know, one of a kind and we're here, right to the lab here, he arrived, with a mission he already knew what you wanted to the p wanted to be able to.
Antonio Torralba: Understand neural networks and to piece them apart and to build models so that anyone could start hacking them like if you were playing with your Lego toys.
Antonio Torralba: And that was his mission and he came to my office and he said, you know, this is what I want to do.
Antonio Torralba: Which lab do you think I should go, and I say just you've got the my lap and then I didn't let him get out of the office and.
Antonio Torralba: He signed a paper, where you know he committed the the next five years of his life to be working with me.
Antonio Torralba: So that was yeah that was great and i'm so happy though Huber say that the the listen to my my threats and really read part of the lab and.
Antonio Torralba: You know, David has done so many crazy things so despite of know, besides of the research research that has been a lot of things that he has done, he has written a book.
Antonio Torralba: In fact, you know I found the link in Amazon, you can find it here I just copied on the on the chat and he wrote a book on numerical algebra which by the way, it's kind of expensive David so just probably do something about that, but.
Antonio Torralba: But if if any of you is curious about it, that is a copy in the in the open it on the third floor csl that you know you can you can take a look at he wrote this well, he was I don't know when I still then on.
Antonio Torralba: Listening to our class and just took notes and and transform the notes in my book, so he actually also you know both a lot of files like bathroom tiles colorful tiles and.
Antonio Torralba: I don't know how many how many hundreds of them like boxes of them that are right to the lab and we use them to create a Masonic that we have over there, so you go to the open it on the third floor, you will see that will solve the fall of the tiles that they've bought so.
Antonio Torralba: So he's he's been like a really important piece of the lab has been alive on his own he has advice so many students and done animation work and the good news is that.
Antonio Torralba: We are not losing him he actually is starting a job as faculty in Nigeria and northeastern so he's going to be in the neighborhood and you know we'll continue collaborating and I encourage all of you continue working with him and to.
Antonio Torralba: benefit from from from from all the things that Cuba so it's just been amazing and yeah I don't want to take more time so David.
Antonio Torralba: The floor is yours.
David Bau: Great.
David Bau: Am I am I unmuted Thank you.
David Bau: Thank you, Antonio that's such a kind kind of introduction.
David Bau: Let me put my my slides up here and you can give me a thumbs up if you can able to see it tonight I afraid I didn't share the right one, let me try again.
David Bau: Here we go.
David Bau: Now now is that maybe that's the right one okay great so yeah it's really wonderful to be here today, even if it's virtual.
David Bau: You know, looking at an audience of last couple minutes I see so many friends from over the years, many friends from.
David Bau: before my time at MIT and many people i've met here, I see some new faces to know the nice thing about being virtual is that people can join far away.
David Bau: So i'm really happy that my in laws are here from from Hawaii I see my master's degree advisor In actual fact, then from England.
David Bau: has joined us and I haven't figured out what time it is in that time zone but thank you so much for joining me here today.
David Bau: Today i'm going to talk about my research i'm going to take first a moment to introduce myself to people who don't know everything about where my perspective, comes from.
David Bau: So you know, I was an undergrad ages ago math major Harvard and I got a master's that point now where I wrote that book with Nick Griffin who is my advisor in numerical analysis there.
David Bau: But I went to work in industry and I built a lot of systems as an industry as a software engineer Microsoft and.
David Bau: startups and Google, as well as a software engineer, I really learned how complex software can be but also how valuable it can be to understand.
David Bau: That complex software and so what I came back to school here at MIT and that's kind of been my theme i've been studying you know complex the complexity of neural networks with Antonio trial.
David Bau: And i'll never forget antonio's first comment to me he I think he remembers that discussion in the office as well, I remember it very clearly as well.
David Bau: And, but what what what sticks with me is after I described my research ideas to the first time to me he looked at me right, and I, and he said, I think you can do it.
David Bau: and actually you know, I think that was his first lesson to me and it's maybe the most important and the message behind that is really.
David Bau: You need to hold on tight tier optimism, because if you don't you're never going to figure out how to do the hard things.
David Bau: To answer the hard questions and make it really work and so i'll always be grateful for antonio's wisdom and his creativity is optimism and his support and on that note i'm proud to proud to present to you my Defense.
David Bau: The title is the section of deep neural networks and the goal is to understand and control Ai now it's a simple goal we want to enable people to understand and control machine learning systems, just like we can understand and control traditional software.
David Bau: Now, the reason, this is such an important problem is in the last decade we're starting to be able to make Ai systems that really work.
David Bau: The fact, this new kind of programming is becoming quite routine if you have a program that you if you need a program to deal with images or natural language or other complex data.
David Bau: This used to be a super difficult problem, but now I could just go to get up and download all sorts of deep networks to do it, so this one is pi torch style Gan to Ada.
David Bau: it's a generative model and to program it all I do is show at some data like these pictures of birds, and then I tried it.
David Bau: I can use gpus it could take a few days, but at the end I get a program a trance network, and so this network happens to be a computational model for drawing pictures of birds.
David Bau: The program I get this conceptually simple it's just a function if you input a random vector it does a bunch of arithmetic and then output, a random realistic bird.
David Bau: So this this town again.
David Bau: Will i'll put a different realistic bird for every different input that I give it.
David Bau: it's called an unconditional model, which means, this is trying to draw random birds, so it kind of leads to the question how do you control a model like this, like what if I wanted a specific kind of bird What if I was a flying yellow bird.
David Bau: Right well, there are two approaches that you can take to control the network and one is to treat it like a black box.
David Bau: And when you do this, you can still steer it, but you steer it from the outside, and we can do this by engineering objectives goals for the network to follow.
David Bau: The second is to crack open the black box and control it from the inside, now the first method is ordinary way we deal with the networks and i'm going to.
David Bau: get into one example, a piece of recent work we've done with this classical approach, because I had to use as an example of the limitations of this approach, which leads to the second idea which is really what my thesis is about.
David Bau: So suppose I want to edit this bird to make it look different so, for example, let's say I want to transform this image from a little blue bird to a photo of a yellow bird flying in the sky, so let me show you how to do it without cracking open any black boxes.
David Bau: I can start with my style Ghana trying to generate birds so here it's generating the little blue one.
David Bau: Then I can take a second network to guide the generator so to do that i'm actually going to download the open I clip network, and this is another big network.
David Bau: And it's capable of scoring the similarity between any piece of text and image and now, so this image doesn't really match the description very well.
David Bau: So clip gives it a pretty low score, but we can use that score, we can use the score as an optimization target and search for changes and see the improve the match, and if I use a good robust optimizer.
David Bau: I use cma that we could actually make the bird look more like the text, and you can ask well it worked for this image how well does it work in general.
David Bau: here's some examples i've put our simple to network method at the bottom at the top, is a previous state of the art this well studied problem, going from images to text that's the Dolly method.
David Bau: And if you look at the examples I method works pretty well as it stands up and if you do a qualitative study, if you do a human evaluation and ask people you know what matches the text better what looks more realistic.
David Bau: You know our method does pretty well actually.
David Bau: Does better than the Dolly method on a specific domain of birds, so it seems like we're doing Okay, it seems like we can actually control our model by doing nothing.
David Bau: But plug it together with another huge model right, so this is the current state of the art of deep networks it's emblematic of this approach.
David Bau: And I think it's great fun, but I also think it obscures the big problem, which is that in all of this, we have no idea how it actually works.
David Bau: We don't know what the generator is doing, we don't know what the similarity network is doing, and this ignorance can hide some problems.
David Bau: So, for example, here's a really simple motivating example let's say we want to take a photo of a room and let a person say how they want to change it like let's say they want to make the bed green.
David Bau: So if you use our method you get a result like this, you know the bed is green it's great but look what happened, the walls turn green to and that's not something we asked for.
David Bau: So we experimented with letting the user paint and area and change the objective, so that epitomizes changes outside the area want to change, and when you do that, even when you do that.
David Bau: The results look like this Green walls and so while you think well, maybe the clip model doesn't understand language at well, maybe asking for green makes it want to see the whole room is green so.
David Bau: So we hit the area outside the bed from clip like this and we tried it again, but it also doesn't fix the problem you know green walls so what's happening is we can't solve the problem by treating our generator as a black box, the generators really opinionated.
David Bau: It thinks the walls have to match the sheets it doesn't do anything we want it has its own rules, its own concepts, but until we run into them we don't know what they are.
David Bau: And so I want you to think about how different this process is from how human draws an image, so what i've shown you here is one of the first shrines and bias, where year old girl.
David Bau: Her preferred subject is horses now compare her horse to the output of a state of the art style and network.
David Bau: So obviously heidi's picture here is very simple, but it reveals a logical structure that's not obvious in the network's outfit Heidi obviously sees that horse is made of parts, it has a head has a tail has some legs, if you told Heidi.
David Bau: Her horse doesn't seem to have enough legs.
David Bau: She could easily fix that problem.
David Bau: This one has seven legs.
David Bau: And if you told Heidi this horse should wear some clothes, she can do that, to this one has pets.
David Bau: So, on the other hand, with style gang you look at the images and it makes you wonder, does the network even know that a horse has legs there's no obvious structure or there's no way to control the legs or change anything.
David Bau: And so that brings us to the fundamental theme of the talk, which is the question of whether neural networks actually contained concepts, so my dissertation goes over several sets of experiments to investigate this question if we have time to talk about three of those.
David Bau: sets of experiments today first and then talk about desecrating the neurons of a classifier.
David Bau: Then, motivated by the question of causality i'm going to turn our attention to generative models and then finally i'm going to ask about relationships, how are rules and relationships between concepts represented in a model.
David Bau: And so let's dive in So the first question is whether we can find concepts inside of classifier.
David Bau: So the traditional way to know what it did that learns is you just train it on a data set like these pictures of places.
David Bau: We hold out some test data we keep secret, this is a classify that learns to tell the difference between 365 classes of places like markets and pastures.
David Bau: So, after the network learns to make predictions by minimizing his loss on the training set, we can test it, we can test its accuracy on the secret holdout set and so.
David Bau: This network does pretty well on a secret test, which means the network must have learned the abstract concepts, we had in mind.
David Bau: right but scoring well on a test like this, you know, improve something but it's kind of dissatisfied because what we really want to know is why.
David Bau: Why did the network, say, this is a pasture right, what did the network actually see here because it just look at the whole image and suddenly decided pasture.
David Bau: Right this problem of understanding, a date now it's gotten some attention but it's not easy, because the federal fundamental problem is a deep network is complex, for example, this vg classifier does.
David Bau: 19 billion pieces of arithmetic to make one prediction right what are all these calculations doing.
David Bau: So there's a couple dominant approaches for understanding to do that work, so one responses complexity is to find a simpler surrogate model, the imitates the network.
David Bau: People have looked at using linear models decision trees, there is every models even finite state machines.
David Bau: So, the main problem with surrogate models is that this second model that we introduce it might behave differently from the original network.
David Bau: And so we're more interested in studying the original network itself, what is its original actual structure, so the other popular method is to ask about input sensitivity, for example, you could try masking out parts of the image to see which parts make the most difference to the network.
David Bau: Let me show you one example, you can make a heat map like this to highlight the important parts, and so there are a lot of ways to make sense the heat maps is a partial list of citations.
David Bau: Or you can use gradients or optimization or basking this heat map i'm showing you here happens to be a Grad cam heat map it tells us where the network is looking.
David Bau: Now, the problem with heat maps is, although they show us where the network looks it doesn't really tell us why doesn't tell us why the network is looking there, or what what the network is looking for.
David Bau: And so that's what we want to know is a network looking for trees, is it looking for grass.
David Bau: So there's an old classic idea for neuroscience that we're going to use for inspiration and that idea is that everything you perceive is just due to the computations have some neuron in your brain so, for example.
David Bau: Does when we asked doesn't network perceive trees, we could start by asking is there a neuron for the concept of trees.
David Bau: So let me show you how we can look for concept neurons we pick a neuron and we see how it activates when we show an image.
David Bau: So this is a convolution network, which means we can create a heat map that shows where a neuron is active throughout the image.
David Bau: So the red bar down here means that this unit is firing in the law right part of the image it looks like it's responding to some colors some animals in the grass and then.
David Bau: You know, and this is just one example to get a more systematic idea of what this neuron is doing, we can test it on more images, so let me show you another similar image, if this one now here the heat map is all black, it means that the neuron doesn't really activate on this one.
David Bau: here's an image with lots of stuff in it.
David Bau: The neuron is still silent here, so this actually pretty typical most neurons are silent, most of the time you need to try a lot of.
David Bau: inputs to see what causes it to activate so here's a very, very dark image it's maybe not sure if you can see what's in it it's actually a dark image of a horse indoors, but it did cause this neuron to activate right it's it's it like this picture.
David Bau: here's another one that activates the neuron it activates in two areas notice it seems to be responding to where these horses are so it leads to this question could this be a neuron.
David Bau: For a detective horses so to qualify and scale this kind of search for concept neurons I assembled the data set called burden and the idea of burden is to have images will be label, a lot of different visual concepts concepts like horses or flowers.
David Bau: Bigger concepts like streets or small concepts like headboards or wheels parts of objects materials like wood or metal but just abstract textures stripes and squirrels.
David Bau: So we have lots of example images of each visual concept, and we can use them to see of neurons activated on them, so, in total, these are more than 60,000 images, with more than 1000 visual concept labeled.
David Bau: And i'll show you how to use this to quantify how well and on activists on a concept like horses so since in brooklyn we've labeled exactly where the horses are.
David Bau: We can just run the brunette images and check to see if the neuron agrees with annotations and we can do this on all the images and it gives us some data that we can use to score, the quality of the match, so I use a ratio called iou.
David Bau: A perfect match would be 1.0 and you know this neuron is not perfect, but it's pretty good it locates horses better than it matches any other concept.
David Bau: And you can see examples of its top activations here are the reason that it's surprising that it does so well at matching forces.
David Bau: And why it's so interesting is because we did not explicitly teach this neural network about horses, so at the bottom i'll show you.
David Bau: The training tasks, these are the place classes, that the network was trained on and what it's predicting each one of these seems to be.
David Bau: And as you can see, this horse around that we have it fires out all these different classes, it is not selected for any one class what it is selected for.
David Bau: His horses and it's not the only neuron that detects something interesting like this, we have a unit for food, we have a unit for cars.
David Bau: By testing all the neurons against the concepts, we can make a report and so here's the report for all the neurons and if you gigi classifier and what they detect.
David Bau: So this full report actually scrolls down for many pages it's more than I can show on one slide but there's one neuron per block So you see there's a wheel neuron a cat neuron.
David Bau: skyscrapers dogs dots mountains and so on, and at the top, is a bar graph summarizing all the neurons in a layer so every column is one detected concept.
David Bau: And the height of each bar is the number of neurons match your concept so immediately this kind of report let's answer some.
David Bau: some fundamental questions, for example, is there just one neuron for concept, and the answer is no, there are a lot of concepts, they have multiple neurons, for example, there are several airplane neurons there's several car neurons.
David Bau: Right, you can also ask other questions about the network like what concepts, or what layers So here we see concepts in the layers going from simple.
David Bau: To complex on early layers you get things like colors and textures that lit layers you have these complex objects that appear.
David Bau: You can also ask how the concepts appear during training, so the beginning, when you have a random network, none of the neurons really detect anything interesting at all.
David Bau: But they start to find concepts around 10,000 iterations to training and they get better and better, as you go along, you can ask how the different network architectures compare you could ask what's the effect of fringe of the risers on concept learning.
David Bau: But there's a fundamental question that is my favorite I think it's actually the most important one.
David Bau: So, and what that is is the question of do neurons really represent concepts, or are we just reading random correlations.
David Bau: right that one level, we do know that neurons are much better at matching concepts than random because we saw that at the beginning of training, when we start with a random neural network.
David Bau: There is no agreement at all between random neurons and human understandable concepts, but there's another major hypothesis out there which is it concepts are learned by a layer
David Bau: of a network as a whole, and not by individual neurons neurons could just be finding concepts because they're just a random basis for this layer feature vector space.
David Bau: So now we have this quantitative way of looking at how well their eyes select for objects, so we can test this directly and quantitatively, so what we do is we form several random basis of the same feature vector space and we test them for concepts.
David Bau: And we do this in a way that doesn't change the accuracy of the network at all, and so the results are shown here.
David Bau: At the top are the concepts learned by the actual neurons in the neural network as we trained it edit the bottom or concepts matched by an arbitrary basis factor in a totally equivalent layer that didn't come out of our training process.
David Bau: and arbitrary basis finds far fewer human understandable concepts than the actual neurons there and by the network.
David Bau: And so my conclusion, looking at this, is the individual neurons do converge towards human concepts, even though they're imperfect at this the effects we see are not just random.
David Bau: And so, so that's one of the big takeaways the big takeaways of summarize here the conclusion the big conclusion.
David Bau: is a concept neurons do appear when we look at them qualitatively the remarkably similar selective when we look at them quantitatively they outperform baselines and matching human understandable concepts.
David Bau: Okay, so I was very excited about this, and when we publish these results of course there's some people who asked isn't really value there really value here at looking at individual neurons.
David Bau: And here's the big underlying question is really so what right, even if neurons correlate with really interesting concepts, does that mean that you can do anything with that knowledge, does that mean.
David Bau: That we even know what these constant neurons are for so that purpose question is really causality question.
David Bau: Another question about causality in the real world, but a simple question about causality within the neural network what effect what have five remove the neuron for horse well, so it turns out in a classifier has a very specific defect and a few classes i'll show you that quickly.
David Bau: For example, here's the mysterious prediction, you know we looked at this class pastures so to quantify the role and individual neuron and how the network classifies pastures.
David Bau: We can remove Units one by one, we can ask how important is the horse in it to this class and so if we remove the horse in it and the accuracy and pastures decreases more than 8%.
David Bau: The field unit here this detector for fields decreases it by seven by 3.7% now there's a unit for grass that also decreases by 3.7%.
David Bau: These three units out of the hundreds of units in the network actually are the most important for this class they decrease the accuracy.
David Bau: Of the pasture classification, the most when you remove them, and you can understand how many units are important to a class by removing.
David Bau: All these units at once, it reduces the accuracy, even more, and we can repeat this process for more unit so here.
David Bau: Are graph how accuracy of a class is affected when we remove the most important you and it's all together after about 20 units removed.
David Bau: accuracy is down to near random, so I don't need to kill the other 492 units in the network, the network has already pretty much forgotten about the class.
David Bau: After I removed 20 units, and this is just one example of a class the sparse structure seeing across all the output classes so here i've grabbed.
David Bau: Civil class accuracy for all 365 classes at the network each class is it green dot has its own accuracy.
David Bau: If I kill the 20 most important units for each class the accuracy drops severely in every case down to near random so every experiment here is a blue dot.
David Bau: And you can see that, for every one of these classes, is just a handful of concept neurons the same essential.
David Bau: For the class and so when you ask what's the purpose of a restaurant, you can think of it as being important for recognizing pastures, and a few other classes.
David Bau: But still there's something to satisfying about this it doesn't really cut to the core question we have, which is is this neuron really how the network of seeing horses.
David Bau: accuracy and pastures is like circumstantial evidence now what I want to know is that the more direct Oh, we have a question from one of my readers so I must pass.
Aleksander Madry: i'm sorry for.
David Bau: that's okay.
Aleksander Madry: Pulling pulling crank here I just actually liked about the previous slide.
Aleksander Madry: So, when you say you remove the new run, what does that exactly mean.
David Bau: it's it's very simple, so all I do is so you know the neural networks, work is just a program and we can run it normally and allow it to compute.
David Bau: The output of every normal neuron in their own way, or we can intervene in it and so at the moment when it computes whatever neuron is going to output.
David Bau: We can interrupt the computation and substitute the number zero instead, and so in these networks, the number zero is you know, usually pretty meaningful because.
David Bau: We use what's called a raccoon on the area, which means that these neurons are actually zero most of the time, and so, when we when we force and network to zero, even if the network would have normally.
David Bau: computed a large number the network, you know can still do a reasonable computation to act, the overall accuracy of the network is still pretty good we've just like targeted certain classes and damage them when we when we turn off individual neurons like this was another question.
Ridwan Alam: hi my name is you're going on thanks particular question so on on this same slide What did you mean by most important.
David Bau: Yes, it's very simple.
David Bau: So what we do is so I.
David Bau: started off with the simple single neuron test so for every.
David Bau: neuron I turn it off and I asked which output classes, does it affect the most and if I go to an upper class and I rank the neurons by the ones that would damage that class accuracy, the most I call that.
David Bau: An importance ranking, and so I just rank all the neurons according to how important they are for a class.
David Bau: But now that only testing neurons individually, and so what I do here is, I say Okay, what happens if I take the 20 most important neurons and turn them off all at once, and it turns out, then almost all cases that's enough to pretty much cause a network to be ignorant of that class.
David Bau: And there's other experiments that are like this and we've written them up in in a in an article at npr is called the.
David Bau: Understanding the role of.
David Bau: An individual in there on it and network or something I forgot exactly the title, but I have some citations at the end of the at the end of the talk and I can share the slides so you can find the full study.
David Bau: So I went to you know make this point let me before I show this, I want to make this point that you know the real question we have is really about what the now what you're seeing with this Iran right this circumstantial evidence about class accuracy.
David Bau: Is a little dissatisfying because I want to do something more direct like, if I were to stimulate this neuron would it make the network see a horse when I turn this around off, is it like blind to horses and it seems like a really odd question to ask what is this network actually perceiving.
David Bau: And i'm sure it seemed odd and biological neuroscience as well, but when this exact question came up in biology, they were able to test it.
David Bau: in humans, and so I took a Nancy can wishes class as part of my requirements here is PhD and I will never forget this video this came from a 2017 study.
David Bau: By sharp our Japanese researchers directly stimulated fifth selective neurons in a surgery patient.
David Bau: to test the causal effects this what they did they just ask the patient to look at a black box and describe what they saw.
David Bau: And the result was when they stimulate the neurons the patient directly perceived high level visual concepts like eyes and the mouth and to me there's a strong direct evidence that the neurons have this causal purpose, because you to perceive a visual concept it's not just a correlation.
David Bau: So the question is, can we do this, can we ask this question in a deep network to.
David Bau: So we had asked a classifier to talk about what it sees, but there are networks, whose job it is to try what they think.
David Bau: And those are the generative models, like the one that I showed at the beginning of the talk so that brings us to the second part, which is detecting a generator the second generator so we so the generator models are just neural networks, again, we can look at the neurons have a generator.
David Bau: This one has to be trained to generate scenes with churches and so we're renders a picture like this it's really an actual to ask either neurons for objects like trees or towers or doors.
David Bau: And we can test them using to section looking at one or on a time, so this neuron seems to be activating the lower left when the network generates this image sort of where the tree is.
David Bau: And we can quantify that by analyzing systematically and lots of images now this time we can't use broken because, again, is making its own images it doesn't take images as input, so instead, what I did is I use a computer vision model.
David Bau: To segment, the images into objects and identify where things i've trees are in the output and score the neurons to see which ones matched objects well.
David Bau: And so, when you do that and measured over a large sample images and test every neuron and every concept and large vocabulary, what do you find.
David Bau: Well, we find lots of neurons match objects, we find neurons for trees, more than one we find neurons for parts of buildings.
David Bau: Like domes we find their eyes for objects like doors and grass, if you take other types of complex scenes and you train a network on say kitchens.
David Bau: Then you find there is a corresponds correspond to things in the kitchen like buttons were parts of chairs, or some neurons for materials or colors like red, so this pot summarizes the different visual concepts that appear in a layer of a generator.
David Bau: And this one maps out all the layers of the network like this, so, as you can see a lot of the object in partner is emerged in the middle layers of the generator, this is a different picture than we saw in classifier.
David Bau: Where the High Level concepts really appeared at the edge of the network, consider the middle so it's quite interesting that it's different so now that we've made this map.
David Bau: And we have identified neurons that correlate with different concepts, we can test the causality question like what happens to what the network is doing with the scene, if we turn off trina rice right.
David Bau: So here we've measured the effect.
David Bau: And there are several treatments and I can turn off more and more of them, and the more I turn off the fewer trees, there are.
David Bau: Trade off just five neurons is enough to control, about half the trees in the model, and I can control more than half I turn off more but there's some limit as we increase the number of neurons we turn off.
David Bau: Even with about 25 years removed there's still some trees that remain so we see similar causality and certainly limitations with windows and chairs and domes and other objects in the model.
David Bau: But the interesting thing, besides just this measurement of causality is what happens when you actually look at the images, so I want to show you.
David Bau: Some some actual images of removing the treanor, as you can see the interesting things that happened So yes, when I turn off trina runs the trees go away.
David Bau: But I want you to observe this occlusion effect because now that the trees are gone the now because Dr other things, you can see the building parts that used to be obscured by the tree, so this is actually really striking.
David Bau: Right, because the generator was just trained on this task, a painting realistic scenes by looking at visible pixels.
David Bau: But it's not just learning about visible pixels it seems to be modeling hidden things in the world as well, but what might have been behind those trees, the network is aware of the building, even though it didn't dry and you can see that by turning off the tree neurons.
David Bau: So let me show you another interesting phenomenon so i'm just looking at these effects when you turn your eyes off, we found that there are neurons they seem to control errors.
David Bau: So if you take 20 air causing neurons and you turn them off, then the effect is the gaps are striking images that look better.
David Bau: And it's not just that they look better we ran statistical measures of output quality measure called fit is a standard.
David Bau: And when you measure that that gets better to So this was a real surprise so usually it's very hard to be an optimizer.
David Bau: But here, it was really easy to be training by just turning off some neurons we don't know why a model might contain era neurons like this, this was a state of the art model that was trained by the original authors of this network, but it had this very clear flawed, it could be fixed.
David Bau: So.
David Bau: Let me also show you that you can turn neurons on so here what i'm going to do is to take 20 door neurons at the location of this yellow box here.
David Bau: and turn them on and we'll see what the network sees So what do you get well you get a door, so we can cause the neuron to think about doors just by stimulating a few neurons now when you look at the store and notice it size and style and how it fits into the wall nicely.
David Bau: Because i'm going to show you what happens if we take exactly the same door door neurons and activate them in a different.
David Bau: scene so here i'll pick this location on a turn on 20 during our odds exactly the same ones exactly the same way in this location and you can see what happens so I got a door.
David Bau: But this door looks totally different right it's much smaller it's got a different style and color.
David Bau: So what that tells us is that these neurons really aren't about a literal arrangement of pixels somehow they're capturing the abstract idea of a door, which could look different in different places, so let me show you another trick.
David Bau: What do you think happens if I take the exactly the same 20 door in our hands and turn them on up here up here in the sky, what do you think will get.
David Bau: i'll show you.
David Bau: This is the output.
David Bau: Nothing happened.
David Bau: So the door neurons are sensitive to context.
David Bau: Think about what we've seen.
David Bau: When we activate door neurons in a different law, it looks different, but if we put it door in a place that a door doesn't make sense, like in the sky.
David Bau: The network seems to refuse to try.
David Bau: So this refusals very stark effect it's almost like a switch.
David Bau: That we can measure that quantify it so.
David Bau: We quantify the causal effects is activating neurons in different contexts it lets us map out the ways that the generators opinionated it won't let it all go anywhere.
David Bau: But it does let it go into a building in the bottom of the building, especially in the location basis and windows, but it won't dry door and the grass to want to hardware and the trees will try during this time.
David Bau: So these causal effects are pretty interesting and they're also really satisfying because it lets us use the neurons as a semantic paint so the DEMO this I made this cool APP.
David Bau: made it together with my good friend Hendrix drove out here it's called can paint.
David Bau: Each button on the Left selects a set of neurons and you can activate or deactivate those neurons in part of the image by panty on the right, we effectively use the network as a paintbrush.
David Bau: So, if you like, you can actually DEMO this you can try it on your phone the URLs at the bottom can intersect.
David Bau: And the thing I want you to notice, if you play with it is how the drying effects are sensitive to context, so, for example.
David Bau: let's see here if you draw a door in a different building you get a different looking door, then, if you draw door in some building that has some different location.
David Bau: And you know this this painting, and as activation of neurons.
David Bau: These causal effects they, let us answer one of our original questions which is does our network know that horses have legs.
David Bau: If we decide each neuron in the network we do find ones that match legs in this network there's one unit it's unit to 78 at layer six that correlates with legs and, interestingly, he can also trigger a leg.
David Bau: It can add a whole leg, including muscles knee and huff, and so we can even use it to make a seven legged horse.
David Bau: Show you here activate this neuron in different locations of the image so here i'm really just activating one neuron and you can see that.
David Bau: This knowledge is really you know you can see that the network really know something about legs on horses, it has some knowledge that you can understand, but understanding it's your eyes.
David Bau: So what i'm showing you is a contrast between a couple different intellectual viewpoints and one side, you have the mainstream the network view.
David Bau: You know the deep networks are black boxes you don't need to slice open a human brain to teach your students so.
David Bau: Why would we slice up in our networks just teach them to speak English and the other side yeah my point of view, which is that deep networks in that so mystical they're not really human brands it's just computer programs.
David Bau: So just like you can understand how the parts of a program work.
David Bau: You can understand the parts of a deep network and it'll let you do new things better.
David Bau: Now there are a couple things that we've found that I really hard to grapple with if you use a network as a black box We saw this example with a model wants to enforce correlations like bedsheets that match walls or curtains in a room or where it won't, let us put doors in the sky.
David Bau: So, even though we found concept neurons and verify that they have causality and control over concepts, we want.
David Bau: We also found limitations in the models ability or willingness to combine concepts in some ways, like how it won't put doors in the sky, so you can ask how are those rules and forced it we modify them and that's part three of the top, which I will run through a little quickly.
David Bau: So, now that we understand how concepts are represented network we can ask how relationships and rules between concepts represented.
David Bau: And can we change those rules can we have a causal effect on them and so i'm going to take a look at a simple rule, which is an assumption about architectural style so, for example in a model of churches, this type of tower will always have pointy spires on top of it.
David Bau: So what if we want to teach it out with a new rule, so, for example, consider this tree growing out of the top of the tower, this is a real picture of a tree.
David Bau: grow out of the courthouse decatur county Illinois it's a very unique tree it's one of a kind.
David Bau: And so we asked what, if we want a model that makes treat our images like this is not the kind of thing you could normally do with Ai methods, because we don't have a big data set of three towers, to train on we just have one.
David Bau: So how can we create a model without a data set, how could I make trees grow out of towers.
David Bau: So first single image we already know how to do this, we could just use can paint or even just a regular image editor to add a tree to a tower.
David Bau: it's just changes, a single image and stop the question that i'm asking what i'm asking is how do you change a model.
David Bau: Instead of an image, I want to take a model as input like again between with some weights w zero and make a change models up with new ways w one.
David Bau: So, since the model is a program to make an endless stream of images what i'm really asking is how do you make a modified program to make modified images there'll be a new rules, for example with trees growing out of all the towers.
David Bau: So in our method we're going to specify a model change in two steps, the first is to choose what part of the model to change, I call it real selection.
David Bau: The second is how to change your model I call it goal selection and to make it really concrete i'm going to begin by showing an interaction in an interface, and then i'll explain to you what i'm doing.
David Bau: So this image viewer here on the right, shows a sample of synthesized images come out of the model, this is a standard and V2.
David Bau: it's one of the latest models for generating realistic images, this one is trained on elson churches.
David Bau: And to change the model we begin with real selection what we do is we choose rules models change by picking examples of similar context let me see if I can get this.
David Bau: animation to play here, and so what i've done is i've chosen a few towers and i've clicked on.
David Bau: A little button here, which reveals other images that are affected the same rule so effectively i'm looking for parts of the images that are represented by the same neurons.
David Bau: And these things are always pointed hours okay that's great now what I want to do is, I want to change this room.
David Bau: So we let the user specify this by copy and paste they can pick a tree and what image and paste it into another one, but when they do this we're not just editing.
David Bau: One image what we're going to do is we're giving the system, an example of how will needs to change in general.
David Bau: And so we can execute that change and we do a calculation that takes about eight seconds and then make a change in the model after the models changed.
David Bau: All the pointy towers turn into trees, and so this change isn't just done in one image it's done all the generated towers.
David Bau: On the other hand, it's also just change one rule, so you can notice that regions and unrelated images that aren't caused by this rule they remain unchanged.
David Bau: So our change has generalized 10 different set of images, but it's also really specific to the rule that we specified and so to give you a little more intuition.
David Bau: i'm going to rewrite exactly the same rule in a different way, by changing the goal, so instead of copying pasting a tree, I can copy and paste a Dome shape instead and then now.
David Bau: If I can get my video to unblock the pointy towers rule will become a rule to draw Dome is now, instead of pointing towers or trees so that's the idea now you see the idea, let me explain to you how this method works.
David Bau: So the methods based on a really simple linear algebra fact which is that a single linear operation can be used as a memory so suppose you have n vector kids.
David Bau: case or by think of them as addresses and vector values visa by just like data to store for each key well, then we could store all the values in a matrix a linear operation, but just finding the w that solves V equals wk for our Paris.
David Bau: Then, whatever we want to look up a value we just multiply by the matrix.
David Bau: And then we get the store value out.
David Bau: So this is one way to think about what a single layer neural network does if a key and cozy meaningful question like how should a tower be drawn.
David Bau: That he stored value could encode an answer, like it should be pointy so this view of a single layer neural networks, not new it's an old idea of.
David Bau: an associative memory that goes back to going in and Anderson 1972 today we call it in the optimal linear associative memory view of a single layer on that single their neural network and it's called optimal because.
David Bau: One way of looking at it is as an error minimization problem in the presence of air or when the problem is over constrained really what you want your memory to do.
David Bau: Is remember a value that is as close as possible to the stored V, so this one might look really familiar it's just ordinary least squares.
David Bau: it's a convenient well stated problem that we can actually write a solution to in enclosed form here as a normal equations which have this one so here in this in this equation down here i've expressed what w zero needs to be our our memorized matrix.
David Bau: As a solution to a matrix equation, where i've just gathered together all the story keys and a big matrix K and all the store values and a bit make sure to be so this is really handy view.
David Bau: Now, one of the things that caught and didn't write about is how to do updates to the memory, what if we want to add a new thing to the memory, but we can work that out.
David Bau: And we can actually write it and another well studied for So the idea is we want their memory, to continue to minimize Aaron all the previously stored memories, but we want to add a new memory, so I have written the new memory here is kissed are in vista.
David Bau: And i've written the goal of what we want the new memory to do.
David Bau: In this form, and this one might look familiar as well, this is constraint, the squares and it also has a nice solution that can be written down as a modified normally equation in this one.
David Bau: So the cool thing is, if you take these two solutions and you combine them you find that most of the terms cancel.
David Bau: And the new matrix with the new memory is actually equal to the old one, with just a small update and this ideal update has a really simple form, you can write down the changes are ranked one updating the weights.
David Bau: That means it's a product of two vectors on the right is the vector that determines the direction of each row of the update see in verse little K.
David Bau: And that that vector is actually fully determined by the key has nothing to do with the value that we wrote see is covariance statistics over keys that we can estimate by looking at samples.
David Bau: On the left the Lambda is what is determined by the value in that determines the magnitude of each row of the update.
David Bau: So the update always slides along this direction, see inverse of Okay, no matter what value we write.
David Bau: The direction sort of acts like a memory slot for the key, no matter how big the changes if we update the matrix in this specific direction, we know that we're going to minimize the interference with all the other old memories it's almost like writing to a line of Ram and so.
David Bau: This is, this is the way that we can think of a single layer neural network what's bold about our proposal is this is that we hypothesize that even deep inside and that work with many layers.
David Bau: assemble layer might still be acting like an optimal linear social memory memorize the associations between is encoded meaningful concept and values that define how that should be drawn, and we can change one memorize value.
David Bau: By trying to edit a single rule.
David Bau: So here's how we do it first we calculate this ideal update direction see universal okay i'll call it D.
David Bau: About that just depends on the key we want to change it, not the value and then the ui, that is what we're doing, we ask the user to do rule selection.
David Bau: We asked for a few examples of similar context and those give us examples of K that we average to get a good, high quality key, we found that the results work better, if you get several examples to get a good key.
David Bau: Then we ask the user to copy paste example what we're actually doing is we're having the user pick a value to write as a goal.
David Bau: And finally, we use the key and evaluate update.
David Bau: An optimized update that optimizes for a while remaining totally constraint to this update direction D.
David Bau: So it's a simple method.
David Bau: But it allows a rule to be completely changed by allowing.
David Bau: Allowing your role to be complete change.
David Bau: while avoiding changes and unrelated things in the model.
David Bau: So let me show you a couple other examples here.
David Bau: So we can change a complex model of human faces, so this model has a role for kids eyebrows they tend to be very thin.
David Bau: and
David Bau: I assume it's posit here a little bit, but the idea is if I take this.
David Bau: The output of this model and I select a context, like kids eyebrows or we can change the rule.
David Bau: By picking an example of a different value that we want to store in this associative map, and so what i'm going to do is, I had a copy and paste this bushy moustache to use as an example and after I execute the change all the kids have very bushy eyebrows like pushing a mustache I brats.
David Bau: And so you know this, that effect that we could have achieved through training there's no data set of bushy eyebrows kids like this.
David Bau: Right i'll show you a couple other examples that we evaluated to quite a fire results we conducted studies comparing this method.
David Bau: To several other baselines some people have studied methods called edit propagation methods that really just operate on pixels rather than the interior, the model.
David Bau: That we compare pretty favorably to to those methods, you can take a look at the details in the paper, so one of the challenges, challenges of evaluation in a setting like this.
David Bau: Is it most of the evaluation methods are designed to measure how realistic results are so we have to evaluate unrealistic looking at it's like changing a Dome to some other realistic looking shaped like a pointy spire or change your face in a realistic way like making everybody smile.
David Bau: But you know, I think that the really interesting thing about this is that we can also help the user achieve fantastical results that aren't necessarily realistic.
David Bau: And we can allow a person to create a model that doesn't just mimic some training data, but which is based on some new idea.
David Bau: That the user has imagined that doesn't necessarily exist yet in the world, so, for example, if you've got this crazy idea that maybe horses should wear pants.
David Bau: I could select a few more slides here find an update direction and then find some pants somewhere like these genes that somebody's wearing and execute a change in how horse legs are represented, and now the horses are wearing jeans.
David Bau: And this is no, this is something that our four year old highly could have done what we've seen is that we can achieve similar levels of control.
David Bau: That a four year old child can have on her drawings right and the key has been to understand the structure of the network.
David Bau: But understand how did it represent we've been able to use that understand to do things like adding new legs to a horse and by understanding how rules are represented, we can use that understanding to redefine the rule for how leg is drawn.
David Bau: And so the main point that I want to make is that networks have a great amount of internal structure models do contain concept neurons and generative models contain causal neurons it can be manipulated and models also contain rules they relate concepts that can be rewritten.
David Bau: Some of the rules have ranked one representations that we can directly change, but there are a lot of further questions that we don't know.
David Bau: We don't know what is the learning process that is leading to the emergence of these complex concept neurons.
David Bau: And if concept neurons are actually decomposing a problem in two parts, we don't know if we can read compose them in new ways that modular software.
David Bau: And even though we can modify a few rules we don't have ways to list out all the rules, the space of possible rules is still very big and we don't know why some rules are learned.
David Bau: but not others we don't know if there are some rules it's too hard for network to learn there's a lot to explore here there's been many questions.
David Bau: That i've been able to explore by cracking open deep networks it's given me an opportunity to participate in a lot of really fascinating projects, here are some papers that that i've gotten.
David Bau: A chance to work on, as part of the inquiry.
David Bau: That you can.
David Bau: Take a look at i'll make the slides available.
David Bau: out when I back up and reflect about what all this research has taught me is this deep networks are programs without a programmer which makes them different from regular software.
David Bau: We don't know what their internal goals are we don't know why they're organized the way they are.
David Bau: But the neurons give us a hint that they have internal structure that is possible for us to understand, but discerning it explain it.
David Bau: It requires careful observations, it requires us to think like scientists, so I want to thank MIT for giving me the chance to learn to think like a scientist i've had.
David Bau: Several wonderful years here.
David Bau: it's been a few years.
David Bau: But if you think about it and family time it's actually been a very long time, when I started my PhD my youngest son cody who, you can see here and read on the Left was our littlest one.
David Bau: But now that i'm finishing cody is our tallest one.
David Bau: So for my family dad has spent a lot of time in school, and I thank them for indulging me and all my studies for these years, Heidi and the kids have always been my bedrock support and i'm so lucky to have their left.
David Bau: course.
David Bau: I also want to thank.
David Bau: oops.
David Bau: I want to affect my advisor Antonio.
David Bau: And all the mentors and friends and collaborators i've met here at MIT I want to especially mentioned Jiangsu who arrived as a postdoc at antonio's lab and became a great collaborator I learned a lot from him.
David Bau: And I learned so much from everybody in these photos as well as many people who aren't pictured every person here is giving me inspiration advice support over the years i've been really lucky to be part of this academic community I miss being at MIT very much.
David Bau: I won't have to miss it too much I won't be far be right across the river at northeastern Korea school, so I look forward to future collaborations and be looking for students so with that.
David Bau: I wrap up my presentation.
David Bau: and open for questions.
Gerald Jay Sussman: Bravo.
David Bau: Thank you, thank you.
Antonio Torralba: Thank you, David.
Antonio Torralba: Okay, so we open the floor for questions from the audience just raise your hand.
Antonio Torralba: If you want to ask a question, or just yeah Nancy.
Nancy Kanwisher: hi David that was beautiful.
David Bau: Thank you Nancy.
Nancy Kanwisher: Most neuroscientist would tell you that, aside from a few categories like faces and maybe scenes and bodies.
Nancy Kanwisher: You don't really have this kind of sparse coding in the High Level parts of visual cortex that most likely correspond to the layers you're looking at in the networks that we used to think that 30 years ago but everybody, except for Nancy has moved on, and now we understand that.
Nancy Kanwisher: Yes, the categories are coated over millie population codes for millions of neurons so what's up with that.
Nancy Kanwisher: what's up with that be different.
David Bau: it's a wonderful question, and so you know you've got people like xp and folks who believe in this opposite hypothesis there's really and I know you know this history very well, but I just give a little background.
David Bau: There was it, you know back you know 30 years ago.
David Bau: You know people were having a lot of success in understanding what individual neurons did it and they created this manifesto they called the neuron doctrine.
David Bau: Which is that you know you can decode everything by looking at individual neurons and the reaction to that several years later, was.
David Bau: What we call the that the dense distributed code model where you think well know all the neurons must be collaborating.
David Bau: You know, even if they are making imperceptible little signals, maybe in mass with millions of imperceptible little signals, you know, in total, the network can perceive things and and the typical experiment that's done to.
David Bau: To support the distributed coding hypothesis is it decoding experience and we actually do that a lot in artificial neural networks as well.
David Bau: Where you look at millions of neurons or you know even just thousands and you train a classifier to say, can you tell what information, all these neurons contain.
David Bau: And and and you, it turns out, by looking at enough input data, you can come to a lot of conclusions about what's there there's this very nice paper by being Kim and artificial neural network space where where where they use.
David Bau: and pick up the call them activation factors and they try to answer questions like.
David Bau: Is the network, using a person's gender to make you know some conclusion and the way that they find evidence that it might be using the the person's gender is.
David Bau: They decode whether that you can predict the gender from looking at all these neurons together but there's there's an issue with this.
David Bau: Which is that just because the information might be present doesn't necessarily mean that that's how the network is using it.
David Bau: And so I think that, on both sides of this debate, there are holes in the argument that just because you can decode something doesn't mean that that's actually what the network is using to decode it.
David Bau: From from all the experiments that i've done here, I feel like.
David Bau: we've had an opportunity to really play with artificial neural networks, which is really convenient and hard to do in biological brains and I wonder if it actually gives us a nice domain to sort of explore these questions and a more complete way, instead of just sort of taking.
David Bau: Taking extreme positions I feel like the truth might be somewhere in between the individual neuron doctor and then the fully distributed code doctrine probably something in between it's closer to the truth.
yeah.
David Bau: I see Jerry has a question.
Gerald Jay Sussman: Actually, I said what I wanted to say what you almost what you just said, which is it seemed to me that it's entirely possible that there is no contradiction between the two alternatives that were described in in.
Gerald Jay Sussman: In nancy's question, it seems to me that there are some that is in fact both possibly your true that is that there is some that there is a a coating of that sort of dense coding, and that those are very specific.
David Bau: Yes, yes, now I think if there are things that you can do.
David Bau: to cause the code to be denser so I showed you the experiment, where we just do a random basis transformation and that creates the ultimate dense code.
David Bau: So you certainly you can create a network that has equivalent power and force it to be dense I think that the big mystery is, why is it to.
David Bau: sort of our ordinary training process, why do we get sparse codes, what is the learning advantage for that there's not I don't think that's well understood I think it's very deep question.
David Bau: And, and I think that you know if you assume that everything's a desk code, if you assume that there's no structured here.
David Bau: Then, what are the disadvantages that doesn't really allow you to ask the question for how a sparse could might emerge, or why.
David Bau: I think that you know, we need to be honest and see that there are our sparse codes that happen and then use that to allow us to ask you know why are these things happening.
Gerald Jay Sussman: So I agree yeah.
yeah.
David Bau: So I have another question from ferran.
Ferran Alet I Puig: Yes, great talk so I wanted to make a follow up question to the first on this experiment, where you showed the Q and Q inverse with a random.
Ferran Alet I Puig: Yes, within that random teams of basis.
Ferran Alet I Puig: Yes, another experiment that could be done is kind of finding the basis that maximizes the number of.
Ferran Alet I Puig: neurons that code for something without specifying which one quote for anything.
Ferran Alet I Puig: Yes, I is and check whether you can get something that even.
Ferran Alet I Puig: kind of more sparse sparser than than the one point by the model.
Ferran Alet I Puig: yeah did you do this experiment any If not, do you think it could be done to improve training, I know that.
Ferran Alet I Puig: Doing Q and then universe makes the forward model equivalent, but maybe changes the dynamics of training, which is, I guess it's what part of the word what you were pointing out.
David Bau: yeah that's that's sort of my question i'm trying to find the slides and so just we have context when we're doing it, I I have done.
David Bau: I have not set up that experiment in the way that I think we could draw the conclusions about the question that you're asking.
David Bau: You know, instead of satisfactory way I have done experiments we try to figure out how sparse the model, naturally, is by trying to optimize a car detector and asked you how many neurons do you need for a car detector and you need surprisingly few but I haven't done a whole.
David Bau: model training to do this, one of the issues is that I don't know if I can give you an intuition for this, but one of the issues is, if you start with a vocabulary like burden, you know as big as our vocabulary was um.
David Bau: You know, actually it's quite imperfect, there are a lot of concepts that these networks are learning that are actually.
David Bau: better than concepts that i've come up with by putting in this big data set of vocabulary with thousand things so, for example, one of the examples, as sometimes show people is a hat neuron.
David Bau: Right and around this not only detecting people's heads, but it's very selective to just people wearing hats and it's come up with this idea, and so, and I do not have good hat annotations.
David Bau: And there are some hats that are you know.
David Bau: There, but the quality is quite low and so wrote in will never find Hatton arise if I trained it gets broken it would never find at neurons, and so I feel like if I even if I use it as an objective, it might not actually be as helpful as what the neural network is already doing.
Just.
Ferran Alet I Puig: What what I meant was not training against broaden.
Ferran Alet I Puig: What I meant was you're still train again seeing the Pacific Ocean, maybe.
Ferran Alet I Puig: You mentioned that changing the basis right, we have the model equivalent as a forward function right.
David Bau: Yes, yes.
Ferran Alet I Puig: But the training dynamics made the representation sparser.
Ferran Alet I Puig: Yes, yeah this q&a universe perfect.
Ferran Alet I Puig: Yes, and what I meant was maybe finding cute like changes of races during training, such that you, you can find more broad in classes.
David Bau: Yes.
Ferran Alet I Puig: You render the.
Ferran Alet I Puig: Forward function, yes, therefore, the accuracy on the fundamental does would be unchanged, the loss function which will be same classification.
Ferran Alet I Puig: bias, the quote is.
Ferran Alet I Puig: Training that that's more what I was.
David Bau: I think that's I think that's an excellent question I think that's an excellent question and it's the kind of experiment that I love to figure out the right way of saying it up.
David Bau: To really explore this it's not you know it's not something that.
David Bau: We fully understand why why training dynamics works this way.
David Bau: So yes, thank you for the question.
Antonio Torralba: So let's take one final question and then we should we should move on.
David Bau: Okay um Oh, I see a question from James telus one of my previous students.
David Bau: I just.
James Gilles: hey this was great this great talk, I was just wondering, so you were talking about you deleted neurons and a classifier and measure the effect that had on the output class right.
David Bau: Yes, that's right.
James Gilles: So did you find any neurons there that didn't look like anything human recognizable right so like a really important neuron that just correlates with nothing visible.
James Gilles: I asked because there's some research suggesting neural networks might pick up on information this, they are, but not human salient like knocking perceptible.
David Bau: yeah.
James Gilles: really strong evidence right.
David Bau: Right so.
David Bau: I have to say so, you know when.
David Bau: When we look at these things so so the general answer is no, and let me show you.
David Bau: The results from an interesting.
David Bau: experiment that we did that answers your question, if I could find if I can find it, but the.
David Bau: The answer is this you sort of asked what neurons are important, what kind of neurons emerge.
David Bau: Either neurons that are more meaningful or less meaningful to people and the truth is that about.
David Bau: You know, half the neurons are surprisingly selected for interesting human understandable concepts and the other half are harder for us to discern it they don't really line up with things that humans understand that easily.
David Bau: So you can kind of ask well what's, what are the roles of is this different kinds of units and it turns out.
David Bau: That, if you ask you know what is the role of units that match here and concepts really well.
David Bau: And how do you compare it to units that don't match him in concepts really well, you see this interesting effect, which is that units that match concepts really well tend to be more frequently these critical important units for these output classes.
David Bau: And units that don't match concepts very well human understanding concepts tend to be less critical for for for our classes, you know we measured in this way, so so that that exact question you answered I it's not that you never find.
David Bau: mysterious stranger as you're not really sure what they do, but you find them much less when you when you quantify it.
James Gilles: Great Thank you.
yep.
Antonio Torralba: Okay, great Okay, I think that we should probably move now to the second part, or the third part where where we during the break breakout room just David at the OSHA Felipe Alexander and we'll ask you some questions.
David Bau: Oh.
David Bau: Great Thank you.
Antonio Torralba: And then we'll come back here and for now for all of you that want to stay and know celebrate or or you know give him the condolences for the assault.
Antonio Torralba: You know john welcome to stay.