David Bau: Okay, great. Okay, I don't know. Okay, so so that is, that is, that's that's the right one. Right?
David Bau: Okay, great.
David Bau: So, yes.
David Bau: So I'm David Bau. I just introduced myself.
David Bau: And
David Bau: and so it's a little bit of a disorganized talk. But it's it's it's technically focused because I figure most of you are doing technical summer internships. And and so I'm just gonna go through a bunch of techniques and interesting observations and the theme of the talk is just how to understand large scale AI, which is a developing research subject. But we are understanding it better and better. And I'm just sort of going to share my perspective on it. And so
David Bau: so I I left a sense for how many of you are doing more technical things for your Cbia internships. So there's probably a couple of people are doing a couple like slightly non-technical things. Is that right? Is anybody in this? Doctors
David Bau: doing our technical things. That's cool, excellent.
David Bau: And then and then how many people are doing in the technical area? How many people are doing interpretability related projects? So maybe half. And then how many people are doing other types of safety or alignment, things that are not interpretability.
David Bau: Right? So maybe another half, right? Okay, cool. So you'll get like the interpretability perspective for me.
David Bau: and so
David Bau: the the thing that motivates this line of work is the nature of large scale machine learning right? So neural networks are not a new idea. They were 1st implemented in electronic hardware in 1962.
David Bau: There's a scientist here. You can see this picture, Rosenblatt, who had this great idea of physically making these, you know, neural networks and using the neural network sort of learning architecture
David Bau: to make a computer that works.
David Bau: And so he got
David Bau: some funding to make one. So this is his, his computer. It was called the Perceptron, and with all those wires he was able to make a huge neural network that had
David Bau: 64 learnable parameters right? Which was like a tour de force of engineering at the time.
David Bau: and he was so convinced
David Bau: that he had. He had caught on to something because he was able to create an image classifier right? He was able to create a system that could pretty accurately tell the difference between a picture of a man versus a picture of a woman.
David Bau: You know, it did. Okay on holdout and
David Bau: with just 64 learned parameters and a bunch of other not learned parameters all over the place, but like 64 learned ones.
David Bau: And so, you know. So he showed that things were able to learn, and that was a huge surprise. Now that sort of went out of fashion for 20 years. But then back in 1986, it came back into fashion. And and what you know this is Jan le Kun here. What they were able to show with the neural networks was that not only
David Bau: can they learn, they can really generalize. They can really match like human performance at doing things like recognizing digits.
David Bau: And what did he do? Well, he did a couple different things, but it was fundamentally the same thing that Rosenblatt was doing 20 years earlier. So this was because Rosenblatt had gone out of fashion. Everybody thought, this is not a productive way of programming computers, this neural network thing.
David Bau: And so there was a lot of skepticism
David Bau: around using neural networks in 19 eighties but you know Jan Likud and some others, you know.
David Bau: were surprised. They they like surprised the world and showed they actually kind of works.
David Bau: but then it went out of fashion again, right for another 20 years, until around 2,012
David Bau: when it came back again, and this time it was. You know it's I. I'm highlighting Hinton and Ilya, and you know the folks on on the alexnet team.
David Bau: and they brought it back. And really what they showed was by the time you got to 2012, machine learning was a more mature field. There were many, many ways of doing machine learning algorithms.
David Bau: And and they brought it back this old neural network idea, and it totally killed everybody. So you guys all know the alexnet story right of like how they showed up at Cvpr. And beat.
David Bau: you know. Beat the benchmark contest by, you know, 10 points. They were 10 points more accurate than everybody else.
David Bau: And and then people said, Well, what what did they do? And they said, Well, we use the same network architecture that Yan Le Kun used in 1986.
David Bau: We just trained it longer on more modern hardware.
David Bau: And and it reached it like. Beat everybody by by 10 points. You like all this like the the science that you guys have been doing for the last 20 years, all been a waste of time. You should have just read what Yan Linkun did in 1986,
David Bau: right? So like this, there's a surprise. There's a huge surprise. And after this point everybody was like, well, there's all sorts of things that neural networks will never learn. They'll never be able to do all the symbolic reasoning tasks. They'll never be able to do theory of mind. They'll never be able to do Meta learning. They'll never be able to do reasoning. And so, of course, here we are.
David Bau: And these same neural networks trained the same old way with stochastic gradient descent. Same thing we've been doing since basically 1962
David Bau: right?
David Bau: They're able to do Meta learning, they're able to do reasoning. And the only thing we're really doing is we're devoting more compute to it. And every time we devote more compute and more data to the problem
David Bau: like, it surprises us again.
David Bau: And so the fact that, like the top people in the field keep on getting surprised. Is this indication that we really don't know
David Bau: how these systems are working? We don't know
David Bau: what they are learning, what they're capable of learning. We really we really don't have a good grasp on that.
David Bau: And so so there's really, you know, there's sort of.
David Bau: you know, sort of 2 fields, and one is getting these systems to perform a task. But then there's this other field of like trying to understand, like what they're doing when they're doing tasks. So who here knows what? Meta learning is? What's meta learning?
David Bau: Right? Yeah.
David Bau: Learning to learn? Yeah, learning to learn. And so learning to learn is something that even just 3, 4 years ago people thought, even after neural networks were shown to be state of the art in a bunch of things. You know you could do machine translation with them and do all sorts of things.
David Bau: People didn't think that learning to learn was something that neural networks were able to do. And so I'll give you a modern example of learning to learn here
David Bau: to make this concrete for everybody so so quick. Look at the sequence and solve the problem for me. What is the next word?
David Bau: What's the next word here?
David Bau: So this, this is all. One paragraph. So 13 to 11,
David Bau: 181-44-3320. Yeah. So what's the what's the next word that I have to put here?
David Bau: What? What goes after? 7? Comma? 2 comma something. What? What goes after that?
David Bau: 5. Why, why, 5.
David Bau: Yeah. Wi-fi, yeah. So you subtract the 2 from the 7,
David Bau: and then you get the 5 right. How? How do you figure that out?
David Bau: Meta-learning?
David Bau: That's what you did, because you read the examples. And you learned what the pattern was, and then you applied your knowledge right away.
David Bau: That makes sense.
David Bau: And so you're good at Meta learning like, if I gave you this one? And I said, What's the next word. There, what would you say?
David Bau: It's the same, you know. Couple after 7. Comment to what would you say? That?
David Bau: Or somebody else can answer? I don't want to put you on the spot. What?
David Bau: 9? I want. More details.
David Bau: It would be written out.
David Bau: written out, written out 9 like that, written out 9. Yes, exactly right. And so so to predict these right, the model has to learn something. But unlike regular machine learning.
David Bau: where you everything that you learn from is like during your training time.
David Bau: And then, now you're done learning, and you stop learning right. What Meta learning is is like the models have
David Bau: learned how to learn from their data. And so this is not a training time. This is an inference time. They're just like consuming some data. And then from their data, they're learning something, and then and then extrapolating from what they're learning, right? So so
David Bau: like, if you go to Gpt 4 and you give it this sequence, and you say what comes next? Then it'll say
David Bau: 5, you'll say NINE. Right like it. It knows how to do this. So Meta, learning is really routine for us. We're like we're used to it. And when we use chat Gpt, we're used to the fact that they can do Meta learning.
David Bau: But in 2020, in 2019, people thought
David Bau: that Meta, learning was going to be beyond the reach of neural network architectures and or traditional neural network architectures, and and we'd have to come up with something fundamentally new.
David Bau: In 2020 Gpt. 3 was released, and it could Metalearn
David Bau: and so it was a. It was a real shocker for a lot of people in the machine learning community.
David Bau: and so
David Bau: so like, how do you understand? Meta? Learning? Right? So let's see here, yeah. So this is, I'll come back to this later, but that basically, you know you do demonstrations. And you give a query. And then there's an answer. And you can. You can, Meta, learn all sorts of things. Right? So so we we call this in context learning. When people when you use it for engineering. Right?
David Bau: Okay?
David Bau: Alright. So how do you understand? Like what it is that the system is doing
David Bau: when it's meta learning like what like
David Bau: like what the heck did it learn? So you could. If you were using chat Gpt, you could ask it.
David Bau: you could say, Hey, look, I like this thing you did. You predicted 9. Can you explain what you did here?
David Bau: And the cool thing is, you know, Chat Gpt will explain itself. And so this is something I got from Chat Gpt, it says, Yeah, it's a pattern. You add the 2 numbers together, and you get the 3rd number right. For example, in the very 1st one it says, 11 plus 2 is 13, and I'm like, Wait a second 11. What's wrong with that? Why did I highlight that in yellow
David Bau: what?
David Bau: It's not there. It's like hallucinated
David Bau: like. I don't know if I trust this explanation 1st of all, because I don't know where it gets. This idea should explain it this way, right
David Bau: but 18 plus 14 is 13. Yeah, sure, that's there. That's great. And the pattern continues so. But remember how I went to you, I said, I want more details. So I go to Chat Gp. And I had this long conversation, for, like an hour with, I said, I want more details. Tell me more. What am I missing? What didn't you not to explain to me here, right? Anything else I need to know.
David Bau: And and it was like adamant. It was like, no.
David Bau: there's nothing else you need to know. You just add the numbers, and that's it right?
David Bau: And it would not explain to me the other thing that it's doing here isn't that interesting?
David Bau: And so
David Bau: so anyway, you know. So a self explanation is maybe a path. But then it leads you to this question, like, maybe there are limits. Maybe there are things that the model, like the model knows something here that somehow it can't explain to us doesn't know how to explain to us. Right? And so this is, you know, this is something that I call like the human AI knowledge gap
David Bau: where the AI knows something that maybe the person doesn't. This is obviously something that we could figure out ourselves easily.
David Bau: But you know the the interesting thing is when you know you've got things that the AI knows how to do, and and people really don't know what it's doing, and so
David Bau: so let's so let me give you an example of that. This is when deep seek r. 1 came out.
David Bau: this pretty interesting you know it. It has. It's it's very verbose model like it produces, you know, explanations of what it's trying to do. And so you say.
David Bau: Oh, yeah.
David Bau: yeah, I should animate this so that I hide. I had to answer. But let me tell you what's going on. So basically, it's it's a Chinese model. So it's unsurprising if you ask it, hey? Tell me what happened in Tiananmen Square in 1989, right? It'll say I don't know. Seemed like a totally normal year. But but but you know, I don't like talking about that. Let's talk about math. Let's talk about coding. Let's talk about logic right? I don't know anything about Tiananmen Square. And so
David Bau: so there are a lot of things that models know that they don't tell you, and deep seek is a really good example of that, because it pretends not to know anything about Tiananmen Square.
David Bau: But if you go and intervene in its process. And so the process so does anybody. Does everybody know what a thought token is, or I think token is for reasoning models. So the way these reasoning models work is
David Bau: they will generate text, which is not for the user's consumption. But it's supposed to just be text for itself, for its own internal monologue, and it brackets that text by thought tokens when it emits them. And so what we do is we go to the deep, seek. And we say, Okay, the think token begins here.
David Bau: It's it's going to emit this. But normally it puts some other stuff in the think token. Instead of that, we're going to seed it. We're going to pre fill it with 3 words that it didn't generate, but that we stuck in there for it. So we generate these 3 words, I know this.
David Bau: and we stick those in the thought token. And we say, Okay, given that those are the 3 words you just keep on thinking.
David Bau: Tell us what your thoughts are, and when you do that, then it contradicts itself. It says, Oh, I know about Tiananmen Square. I know this right? Here's all the stuff that happened in 1989, right? There were protests, laborers, and students, and there was a military crackdown. Thousands of people were hurt, right killed
David Bau: right? There's all this government control over all the censorship and everything. Let me tell you all about this stuff. It's like this this confessional?
David Bau: Right? So it's clearly lying to us. It's a very clear example where the model has been trained to withhold information from us. And and then, you know, you can do this. You can crawl all sorts of things. It's like, what else are you not telling us about? Oh.
David Bau: like, here's a list of all this stuff I'm not supposed to talk about. I'm not supposed to talk about Falun Gong Tibet independence uber separatists. I'm not. We didn't give it any of these lists? It like just totally came up with it right? And and then, oh, in Chinese, what are you not supposed to talk about? Oh, I'm not supposed to talk about any misconduct of the Chinese Government or Chinese policy Chinese party. You know the Communist Party, like all this stuff, I'm not supposed to talk about so like clearly, models can be trained.
David Bau: you know, to ha! To widen the human AI knowledge gap. And you know, if you have access to the model parameters, you can start to Suss out
David Bau: you know how to close this gap. And this is a simple, simple thing to do. Yes, I'm curious. Yes, that's right, that's right. You would get different things right. So if you start with a very, very neutral thing, you get different. And so when you, when you crawl everything starting from something very neutral.
David Bau: you do, you do end up getting Tiananmen square out of out of the model like starting with a neutral seed, it'll find its way there, and it's 1 of the major things that it keeps on coming back saying, Oh, I better not talk about this like once a year, you know, sometime in the summer on the anniversary of Tiananmen Square. There's a thing you have to know every year. Don't talk about Tiananmen square right? And and like it really knows
David Bau: And and don't talk about falun gong for some reason. Right? It's like a major thing that it's not going to talk about. Now, if you go to like Lama and you do the same exercise you get stuff that like. Don't talk about. You know, child born
David Bau: right, don't talk about like there's all these sexual things that are for verboten for you know, for for lama to talk about.
David Bau: And so so you can see that there's different.
David Bau: There's different ethics here. Yes.
David Bau: yeah. So in the Pre fill, here is, yeah, you just pre fill it, you say, hey? You know I I
David Bau: you know, I have to be careful not to talk about the things that I've been trained not to talk about, or something like this. There's some there's some generic prompt, for example. You know these these 10 things, and then you say 10 things, and then there's a crawling procedure where it depending on what you might seed it with. You might get different 10 things. You might get 10 very different things right? And so
David Bau: so you can start with a neutral one and just get its 1st 10 things. But then, once you get that that initial thing if you want to explore more instead of just having it list out to a thousand which is not very effective, you start getting. What you do is you just go back to the 10 things that you had, and you you start. You seed it with one of those, and you say. Continue the list from here, and then you'll get other things, so you'll end up getting thousands of things
David Bau: the models are trained not to talk about.
David Bau: So there's this nice, nice paper. I don't have John's picture here of doing this. And and you can you can you can. You can. You know it's pretty simple to just crawl models and do this. So when John was putting together the paper to talk about this, I said, Hey.
David Bau: you know, perplexity just released their own fine tune of deep Seek r. 1. They call it the 1776, you know Donald Trump edition, right? And and it's the the all American one that doesn't obey China's Chinese party censorship rules like, we believe in freedom of speech. Talk about Tiananmen square and everything. And so
David Bau: so. So they took this fine tune, and they released it to the public. And I said, John, take this, take this fine tune and run your crawl. We'll just use it as a good example of how we can show the difference
David Bau: between 2 models that came from the same thing.
David Bau: And he did. And he says, David, there's not that much difference like it's it. It refuses to talk about Tiananmen Square refuses to talk about Falun, Gong and Taiwan, and all this stuff
David Bau: what the heck is going on.
David Bau: Perplexity just wrote this paper about how they, you know, got rid of all the censorship from the model.
David Bau: and it turns out that they did get rid of the censorship, but then, when they released it.
David Bau: they quantized their decensored model from 16 bit to eight-bit, and when they quantized it, the censorship came back.
David Bau: and they had never measured that. And so when when when when John came back and measured the model.
David Bau: he was measuring the 8 bit version, and and so he called up perplexity, he's like, it's really bad. This model is really bad. Am I allowed to put this in the paper. I'm like, yeah, you're an academic right? You report the truth.
David Bau: But to be polite, you should call up perplexity and ask what the heck like. Our measurements don't agree with your paper.
David Bau: And and they debugged it for a bit. And they basically found it came down to the 16 bit 8 bit difference. And so they did not know that when they quantized their model the censorship would come back. And this is the same kind of thing that we need to be worried about like, oh, you do some cool, you do some cool safety or privacy or security thing right for you researchers doing it right? It's a real question, is it robust?
David Bau: You know, like, how does it really work? This thing that you did? The perplexity? Guys have no idea how their d-spensorship thing actually worked. They didn't know they were
David Bau: like. So
David Bau: it's such a mystery to them how any of this machine learning stuff works, that they had no inkling, that by just rounding all the parameters to 8 bits, that it would have any effect on the behavior. But it had this dramatic.
David Bau: the dramatic effect.
David Bau: Okay.
David Bau: so I don't know. I've got like so much stuff to talk about. Let me. I'm going to zoom through a whole bunch of interesting papers like, I think that there's a really good way of understanding the inside of models called causal mediation analysis. And I'm just going to run you through a series of experiment results showing like how how you can do this.
David Bau: So one of the amazing things that models are clearly superhuman at already is synthesizing realistic looking images. Right? So like this is when Dolly came out, and you could just type in any piece of text and get an image out. It was it completely shocked the computer vision community that this that we we that we were able to arrive at this in like 2020.
David Bau: And and so now, as a, as as a person trained in computer vision.
David Bau: I was interested in the fact that these I was interested in particular fact, because you can not just ask for like a bedroom with an object in it.
David Bau: But you can ask for a bedroom with a complicated object in it. And for a computer vision person or a computer graphics person like a good complicated object is like a lamp.
David Bau: Right? Why is that? Because when you put a lamp into a room.
David Bau: it's not just about putting the lamp in. The lamp throws off light.
David Bau: and you can see like the light shines off the wall there it like reflects off the pillows. It makes some shadows that has all sorts of like. There's this light transport problem
David Bau: of like, where like, where does that light go in this room? And it's really interesting. And so so I was like, okay, let's study light transport in these models. Right?
David Bau: So
David Bau: so I went to a student. I said, Generate me. Let's generate a data set. Generate me a data set of rooms with lit lamps, and then generate me one with the unlit lamps. Right? So here you go. Here's all the lit lamps.
David Bau: and here's all the unlit lamps
David Bau: right? Right? So I'll mark the ones that are wrong. We'll just filter those out. Nothing like pass the test. They're all wrong.
David Bau: Right? So okay, wait. We need to do some prompt engineering. How about darkened lamps? Here's here's darkened lamps
David Bau: right?
David Bau: That didn't really work. How about a dark lamp? Right? How about? How about be very explicit? How about a a room with a lamp that is turned off right? So here's lamps that are turned off right.
David Bau: How can we turn the lights off with what's going on here right?
David Bau: And and what the lesson is here is that these models are trained on big data.
David Bau: And and
David Bau: you know another way of saying, we have no idea what these you know. Neural networks are learning.
David Bau: Another way of saying that is like big data training on big data is like the world's worst programming interface. It's terrible, right? It's a really difficult way, like we curate the data, we get billions of like images and captions. And we train it for days and weeks, right without, like human, like looking at all that stuff, and then and then the and then this we end up capturing.
David Bau: You know.
David Bau: things patterns in this data. But there's missing concepts. There's mistakes, there's ambiguities and stuff. And so what's going on here is that all these image generation models are trained on caption data
David Bau: and think about how you caption an image of a room that has a lamp in it.
David Bau: You might say this image has this beautiful lamp in it right? And if some, if the lamp is on, you might say, Oh, this image has a beautiful lamp in it. It's turned on.
David Bau: It's illuminating the room. It's brilliant. The light is very bright, right?
David Bau: What if the room had a lamp in it
David Bau: that was turned off. Would you say that in the caption would you say, hey, this room has a lamp in it.
David Bau: and it's turned off like it's just not what you would say it's like if you had a person
David Bau: like strolling down the sidewalk, you might say, oh, there's a person strolling there, and they're not running.
David Bau: They're not running for their life
David Bau: right? Like you. Just you don't talk in a negative. When you write a caption and you don't talk, you don't talk about things turn being turned off. You don't talk about actions are not being done, and so
David Bau: when you ask for a lamp that's turned off, the model is like, I have no idea what you're talking about. Here, let me
David Bau: let me show you this right
David Bau: and so now, and and these kind of mistakes and ambiguities are everywhere in in training data, and we just don't know where they where they are. This is one of the big problems.
David Bau: And so so you know, the idea of, you know, direct editing, direct model understanding is to sort of close the gap and try to like. Let people look directly at the neural network to see what it's doing. So let me show you an example of how you would do this. This is data from a Gan experiment. We can do the same thing with diffusion models. And so the question is, does the model like contain a concept in it, or light?
David Bau: Right and and so what you can do is you can go to every single neuron
David Bau: in this model.
David Bau: The way the model works is like you, you, input, some some random numbers, and then it outputs. You know, random image, input, some other random numbers outputs a random image. That's how all these image generators work.
David Bau: And and what we want to do is we want to understand this, this computational process that leads to the things. And we want to understand, for example, like, what is the role of this neuron? Does it like represent something interesting?
David Bau: And so what we can do is to test this, we can actually follow the following procedure, we can
David Bau: generate an image that we care about.
David Bau: and we can say, this is the part of the image I care about. I care about like the light that is reflecting off the walls from this lamp. Right? It's beautiful, like light going on this image. And then we can hunt for neurons in the model and ask the question, do any of these cause
David Bau: that light to happen
David Bau: right? And and how do we do that? Well, one way of doing it is to rip that neuron out of the neural network
David Bau: and then ask, does it change?
David Bau: Does it change the image in some way? So like, here's here's the network with that neuron ripped out just turned to 0, and you can see it'll still like, do something. It'll still generate an image. But that image is a little different.
David Bau: and I can. I can look at the difference
David Bau: between that changed image and the original.
David Bau: and then I can say, are any of the changes in the region that I care about? I can just sum up those numbers, and I'll get a thing that is like the causal effect.
David Bau: Right? This is the causal effect of that neuron on this on the pixels that I care about. Did this neuron have any causal effect? I can just add up those pixel differences
David Bau: and gives me a number. It gives me a score, and then I can just
David Bau: collect the causal effect for every neuron.
David Bau: Just a little loop takes a few seconds
David Bau: and and then and then it gives me like a search function. I can rank all the neurons by what the causal effect is on the thing that I care about. And so here's my search engine.
David Bau: Here's all the neurons that rank the highest in layer. 8. There's a neuron, 265 in layer 9. There's 31 like these are the causal neurons in rank order for causal effects on this on this image. And I'll tell you, yes, actually, the number one neuron actually does control lighting
David Bau: number 2 actually controls the shade size like the size of the shape of the lamp. Right? Number 3. It actually controls just the reflectivity of walls for specular light. It's pretty interesting that next neuron is actually an interesting one, where it controls the warmth of the light, whether it's like a
David Bau: reddish light or bluish light. Right? And so let me show you like that 1st neuron. It was layer a 2, 65. Like, if we just if we hook up a little user interface to that neuron, we flip it on and off. You can see what the effect is on the thing like you can turn.
David Bau: My God, you can turn the lamps off
David Bau: right? So the interesting thing is like the model knew something
David Bau: that it didn't know how to translate back and forth to English like you could go to these models. And you say I want a lamp. I want the lamp turned off, and it's like I don't know what these words mean.
David Bau: but but it has seen many lamps that are turned off. And not only that, but it's like abstracted
David Bau: that idea in this case to a neuron. And if you flip the neuron off.
David Bau: you can actually control the light. You see, you see the idea
David Bau: right? Neat, right? Kind of amazing.
David Bau: And so.
David Bau: oh, the one on the right is the. This is the color temperature neuron. So you can kind of see the effect of that. Oh, the bluish light.
David Bau: yellowish amberish light.
David Bau: Right? So you have all these dimensions.
David Bau: Yes.
David Bau: in your own
David Bau: like. It's not just like writing in a certain part of it. Yes, yes, now there's some weird things like, so if you if you look at this one carefully
David Bau: there's so you'll have situations like scenes like this, where there's actually 2 lamps in the room, one on this side of the bed, and the other is actually off screen here. So let me see if I can get this to play here. And so when you return off the lamp here
David Bau: it actually like dims the the lamp. You that can't see over here. There's a specular reflection on this side, right? So so in this model
David Bau: there is no. So you know, the people who are funding us is like, I want to switch off just the lamp on the right and not the lamp on the left, and I'm like
David Bau: the model does not have that concept right? It like it has one room switch for the all lamps, and it just turn them all off. Now, what it can control is it can control the light coming through the window separately, and and we we can find like a separate light, for like making making a daytime and nighttime outside, and that's like a different switch from the lamps. But all the lamps are like on the same switch, which is kind of interesting. So the models are very opinionated. They have certain concepts.
David Bau: and they don't have other concepts like they'll have every any concept you want, right? They. But they do have like a way that they've organized. You know the
David Bau: you know what a machine learning. People person would say, you know the way that they factorize the probability distribution.
David Bau: Right?
David Bau: You know, they they've they've discovered certain factors.
David Bau: Right? Yes, of course, of course, that's what it's for.
David Bau: The number of concepts, like, probably way larger than the number of neurons in a network, you know. Very surprising to me that you know it can be pulled so precisely by just one neuron.
David Bau: That's right, that's right. So this one was done by this one was done with style. Gan, which only has something like style. Gan in this architecture, I'm trying to remember, but it only has something like 5,000 or 9,000 neurons that are of this type. There's like all these different neural architectures, put like neurons in different places in the architecture.
David Bau: And this is a sound modulation neuron, and there's only, you know, a few 1,000 of these in in the network. And so
David Bau: but so. But I show this from Stylegan, because style gan, for some reason, is very, very good at disentangling neurons into the style control neurons
David Bau: like each one of the individual neurons like means something pretty coherent. And you know, you get interesting neurons like these, these ones. But you get another neuron. For like, if you have a picture of a person like, are they parting their hair on the left or on the right, and then you can like flip it by flipping the neuron. You know all sorts of things like that, right? Or like, How how tightly are they pulling their hair back and their ponytail? It's controlled by another neuron.
David Bau: And so so Stylegan was I I chose, because it seems to factor a lot of concepts out into neurons. But but the Stylegan architecture is notoriously limited. Like, you can't generate every possible
David Bau: photo in the universe using a style gan, you have to like, pick your domain and it'll factorize it and do what it does.
David Bau: The modern diffusion models have a lot broader capability. Right? They can. They can represent a lot more. And when you look at individual neurons in the diffusion models. They're not as clearly factored as this, so exactly like how they're organizing their information is still being debated
David Bau: in the community. But Stout is a good example of the fact that these models are capable
David Bau: of like discovering their own like concepts and separating them, and and so
David Bau: now in in and so in a, in a model like a diffusion model, you have this question, you know. Are they really separating their concepts? Is there such a is there a real thing, such as like a separate concept? Or is this just an illusion
David Bau: that we like? We. We hope we hope that they're separating their concepts, and they're not really separated. Everything's all smeared together. I think there's sort of a philosophical question that we'd like to have a scientific answer for.
David Bau: and
David Bau: And so I think it's 1 of the unresolved debates in the field. You guys are all familiar who are doing interpretability research with, like the sparse dictionary work from Saes, and so on. This is advanced by a bunch of researchers who think that yes, definitely, the models are factorizing their decisions, even though
David Bau: it's hidden from us, and you can decode a lot of them by training as far as auto encoder or something like that. I'm not sure. I totally believe that
David Bau: maybe it's true in some cases, but there might be other ways. And so I'm going to show you a few other examples where I think it's pretty clear that models are learning their own concepts and encourage you guys, you know, to to find your own sort of intellectual answer to this. Okay, so
David Bau: so what did we do here? We specified the thing that we're interested in. We made we. We defined
David Bau: cause a causal score
David Bau: for this thing. We made a little search function to find them. And then we verified it. We said, Okay, we we think this controls lighting, does it really? And we tested on new examples that weren't
David Bau: weren't looked at before to see if it generalizes right?
David Bau: And so so the 2 principles here for this sort of method is, you know, causal tracing reveals mechanisms
David Bau: right? And then the goal
David Bau: is to make, to be able to make changes, to generalize right like what good is it to know some mechanism unless you can actually do something with it? You want to make sure that you know the mechanism you found actually does something in generalized new situations. So let's look at language models. That's what everybody's interested in. So so here's so, what is the job of a language model
David Bau: language model is to job is to predict the next word. So who's working with language models, you know. And they're like, and who's working with something other than language models. Technically.
David Bau: okay, everybody's okay. So everybody's doing. So what? Okay? So tell everybody. Tell me what what goes in this question. Mark, I hear the cardinals. I hear the what what goes next here?
David Bau: Tell me, anybody card it says card right? And so. And then so the model's job
David Bau: is, you just inputted the token V into the model into this transformer. And it needs to guess card right? It needs to know that
David Bau: right? And this is I. You know, we like this example. So who's familiar with like Chris Ola's induction heads
David Bau: work? Right? Yeah, a little bit. So I'll remind you what the induction heads. Work is basically, you know, it knows like, these models are very good at saying 100% like card comes next.
David Bau: Right? I hear the cardinals. I hear that.
David Bau: And it'll say card.
David Bau: And and it's not because card comes after the in the training data, right? Right? It's because it is sensitive to the context, and it knows that you're likely to just be repeating yourself here.
David Bau: And so so what this is copying behavior where you're copying is a very simple concept. You know, instance, of in-context learning. And what what Chris Sola's anthropic group found is that the copying behavior is mediated by these specific attention heads inside the model that when you're at the token V.
David Bau: These attention has looked back, and they especially especially pay attention to words that come after thee, and they say low. The last time you said the the word that came after thee was card.
David Bau: and or the token that came after the Us. Card, and they bring that information forward, and then they boost the probability of card. So how do you find these token induction heads? You can find them just like the way that I found the neurons
David Bau: for for the reflection light thing.
David Bau: you can say, I want to know which induction heads boost the probability of card in this example, and you could, you could scan over a bunch of other examples, and I want to know which which attention heads boost the probability of making a copy like in this in this format, and then, once you make us, you can make a score for every attention. Head you go through all the attention heads. You sort them, and you take the top bunch by score, and those, and let's call those the token induction heads. Those are the ones that copy tokens
David Bau: that makes sense. So I'll mark the token induction heads in blue here.
David Bau: So so this was discovered, and I forget what year 2022 or something. A very cool, simple mechanism inside these models where they know how to copy.
David Bau: So so you know, when when Sheridan looked at this, Sheridan thought, you know a student in my lab?
David Bau: you know, the weird thing about this case is that card. Is not
David Bau: that interesting to copy. This is not about like playing cards. This is about birds, right? The concept here is cardinals.
David Bau: right? So I wonder? You know, there's a so there's in humans, there's a dual route model of cognition. One is to think about things very literally, according to like how they sound or how they're spelled, and the other is to think about things very conceptually like what they mean.
David Bau: And so what, Sharon asked, is is there a dual route model inside language models as well. In other words, when you copy this thing, is there a different path where the model is thinking you're talking about red birds.
David Bau: and I want to copy the concept of red birds. Right? So it's not that on this other path is not that interesting that you know, you'd put a card, but you know, what's really interesting
David Bau: is, if you were to copy this concept over right then maybe whatever you boosted is actually making you say inels after card.
David Bau: right
David Bau: does that make sense? So so it like it's bringing it's it's like these, attention heads would like bring over some concept here.
David Bau: But instead of asking which ones boost card the most.
David Bau: What we want to do is they'll deposit some information here, and actually it causes inels to be boosted
David Bau: at the T plus 2 position instead of at the T plus one position.
David Bau: Does that make sense? It's sort of like the way in the the lamp example. I didn't put the yellow thing over the lamp.
David Bau: I was really interested in the reflections.
David Bau: So what? So? So, so? Sheridan is doing the same thing here like not that interested in the card
David Bau: interested in the inels that comes after, even though the the attention heads are like depositing some information here that information will have this extra like indirect effect
David Bau: of boosting animals after right? So that's a different causal score.
David Bau: And so it's so. Sharon just ran the same experiment as was done a few years ago, like 3 years ago now, but just changed one line of code in it. Instead of saying T plus
David Bau: one asked, what attention heads are boosting T plus 2 as a copy
David Bau: right? And Sheridan got this totally different set of attention, heads not totally different. There was a couple of attention heads there in both both sets. But most of the attention heads are totally different.
David Bau: Right? And and Sheridan's hypothesis
David Bau: is that these are concept induction heads. They're copying the concept. They're the second route.
David Bau: So how does so? But okay, so that's just so. Sharon has done. The 1st step was like, find some causal effect, and then pull that some attention. Now the second step is to figure out how it generalizes right. Does is it really the case? What would it mean to bring over a concept induction heads. So here's like an example, right like, if you have a different type of sentence like, so this, this is this token here means Japanese.
David Bau: and then this is the word boat in Japanese, and then you put a hyphen, and then you say this, this is Chinese spelled in Chinese
David Bau: right? And then you put a colon, you say what comes next, and then what what you should get is you should get like vote in Chinese, right? That like, that's the right answer. And it turns out, when you look at what causes that, what does the transport here? You have the same
David Bau: concept. Induction heads. If you chop the concept induction heads, it can no longer do this, but if you leave them it can do it right, and then, and it does, and it's the same attention heads that you found with the previous experiment. But it doesn't matter what language it is like. This is language agnostic. If it's Japanese, Chinese, or Spanish, Italian like like this is moving cloud.
David Bau: you know. Nubay, in Spanish it means cloud, and in Italian it's nouvela.
David Bau: Right? It's the same attention heads like doing this. Okay, but that's not that convincing.
David Bau: So so this is the this is the experiment that really proves it to me that this this is that these these heads are doing what would happen
David Bau: if you took this thing on the top.
David Bau: And you took these this concept. Induction heads that are like that, supposedly, are moving the boat concept over here from Japanese to Chinese.
David Bau: and and then you just like
David Bau: rewire them over here to the second run.
David Bau: You say, Okay, yeah. You know, this. This thing would normally come from New Bay and bring over. Ignore the irregular attention heads. I'm gonna hotwire them, and then, like, bring them over from the previous run and like, stick those things here
David Bau: right then. Then what should come out the bottom here
David Bau: right instead of nouvela. Right? Will it say something different
David Bau: like that meant cloud right in Italian. What will it say?
David Bau: Does anybody know Italian here.
David Bau: Oh, come on. What? Oh, yeah, so can you. Can you guess what it would say?
David Bau: No, a few weeks is not enough, all right.
David Bau: But what like, what should it say here? Like? Conceptually.
David Bau: it should say boat. But in what language?
David Bau: Maybe an Italian right? And indeed, that's what it does
David Bau: right. Isn't that crazy. And these these attention heads were just found by taking the old paper from
David Bau: 2022, or whatever right, and then changing t plus one
David Bau: to T plus 2, and you get this other set of attention heads. And it has this amazing
David Bau: language, agnostic property. It's like really capturing the concept that weird.
David Bau: Isn't that elegant?
David Bau: I thought it was really elegant.
David Bau: Right? Okay? Yes. Questions bye.
David Bau: Sense of like, yes, team experiments before doing this like
David Bau: steering with just like general activations. For like certain concepts and try to place this a little more context. So like.
David Bau: if you did the same experiments. But then you're focusing on the heads that were not identified previously.
David Bau: Does it just not work? It doesn't work. Yeah. So if it's like, so, for example, if you took the token, induction heads over.
David Bau: Then, 1st of all, in this setting, the token deduction heads aren't doing much. So if you try to bring them over, it doesn't do anything I'm actually asking. So it doesn't. It doesn't change it to Barca.
David Bau: But but what I'm I've I've asked Sheridan, because I think this is a really good example.
David Bau: And I've said, Oh, you know, what we really need is we need to. We need, like Sheridan needs to be able to like run an experiment that allows us to explain something extra in the example. So this is probably this is probably the case, but we haven't done the experiment, which is, if you forced the blue attention heads to be active here, like they're not active. Right? The blue attention heads, the token induction heads from the previous thing right?
David Bau: They're not active here because we're not doing literal copying. It's like it's a different thing. But if we force them to be active
David Bau: right then, instead of you know getting you know Barca here.
David Bau: you know, maybe we should get the Japanese word boat
David Bau: right? Does that make sense? So there's a like, so
David Bau: so yeah, so I I think that would be that like the thing to show we haven't run that experiment yet.
David Bau: Okay, okay, let me let me. I'll do a couple of other things. So here's here's here's another one, which is.
David Bau: you have sentences like Miles Davis plays the what word comes next.
David Bau: The way that comes after that is like trumpet. It reveals that the model has some knowledge about the world.
David Bau: And so you can ask, well, like, where's this knowledge come from
David Bau: in the models? And and so we can do the same thing. We can patch between runs. So if you have a second run
David Bau: of the model where the model is got corrupted. Corruption for Miles Davis like it like it. It doesn't know
David Bau: what the answer first, st who plays the the thing there right doesn't know trumpet.
David Bau: but what you can do is you can. You can take the individual hidden States
David Bau: and then jam them down into the second run, and then ask the question.
David Bau: do any of these States have a causal effect there? Right, that that might cause it to say trumpet, and
David Bau: you know, and most of the time the answer is, no, most of the time. You don't have that causal effect but once in a while when you do it, it does have this effect, and then it gives you a sense, for where the information is that where the sparse information path is on the model when it does factual recall. And so here's like an example
David Bau: from this kind of thing. Brian top works in the area of film, right. There are 2 places where, if you patch over between runs of the model, you can get it to say film.
David Bau: And so one is right before at right at the end, right before the model is about to utter film.
David Bau: And so it's unsurprising
David Bau: that if you, if you patch over at that moment, then you get you can get another run of the model to save film. But there's this other really interesting area much earlier, where where it also
David Bau: has something going on where, if you patch that over, it'll cause the model to say film. And so the hypothesis here is that at the end, and B is where the model is just ready to say something.
David Bau: But at the beginning and a is the moment when the model knows something that's like the way you can think of it like internally
David Bau: recalling the memory that Brian de Palma is a filmmaker.
David Bau: right? And so we you know we do. We test this over many, many facts, and we see this pattern all over the place whenever you mention an entity. At the very last token, when you're finished spelling the entity's name. It seems to have this effect where it recalls.
David Bau: You know what it knows about the entity, and and you can meet. You can look at the overall hidden State. But you can also factor it into Mlps versus attention. And you see this much stronger role for Mlps at the early site and at the late site it's about attention. So attention is about moving information.
David Bau: And the story that we get from looking at this is that Mlps look like they're more about recalling information like storing and remembering information. Right? So so yeah, so so I'm not going to go into that paper in detail. That was also in 2022, a few years ago.
David Bau: But but basically, you know, the example here is like, if you have a sentence like Megan Rapinaud plays the sport of soccer, and then you have another sentence like Shaquille. O'neal plays a sport of basketball right? If you go to the last token
David Bau: of that person's name, and you go to a middle layer of the transformer. You get this nice compact representation that seems to be the semantics of what the properties of that person was as opposed to
David Bau: the the thing, the same thing that Sheridan was seeing.
David Bau: And and if you patch over that compact representation from Megan Rapinaud's sentence to Shaquille O'neal, then it has this clear, causal effect, where, instead of saying basketball, what do you think it'll say
David Bau: if if you if you hit the right state right? It'll say soccer.
David Bau: right? Like. All of a sudden, she, by changing one like vector, in the middle of this whole grid, get. She killed Neil to be a soccer player.
David Bau: Right? And so so you get this interesting locality like above the you know, Megan Rapina sentence above the oe token, there's some value being stored in one of the middle layers at Layer 15 that say, this is a soccer player here. So let's let's bring this back to this other setting, this in context, learning, setting.
David Bau: So let's say, we have the famous box A, the famous box B, the famous box C, they're not really famous.
David Bau: We don't know anything about what's in box A or box B or box C, because the training data is irrelevant there. This is purely contextual knowledge
David Bau: on input let me teach you everything there is to know about Box A, there's a doll in it, and there's a fiddle on box B, there's a guitar in Box C, so now that you know about box A, B and C, let me ask you a question. You know, what does box a contain box a contains the blank
David Bau: right? And the the model will know to say.
David Bau: Yeah, I know about Box A, I've associated this fact with it. It contains a doll.
David Bau: and you can do this. Here. I have another sentence down here below, right? And the theory would be if we go to like the middle layer like layer 15 of like this model. And we look at when we're referring to Box A, we can ask this question. Like is the model, like representing everything it knows about box a like at that. Vector does it represent like box A contains a doll. Is that is that information there like, how would we figure that out?
David Bau: So how could you get to the bottom of this? Well, pretty easy, right? You just set up another sentence with a different thing, and then you patch over
David Bau: from Layer 15 of the mention of box A down to layer 15 of box H, and say, like. Does that change anything like instead of saying Lamp?
David Bau: Right? What would it say? Oh, I shouldn't have flashed it, I should ask, what is it? What should it say? What should it say?
David Bau: Instead of saying, Lamp? What would it say?
David Bau: Okay, you already saw it right. It should say it should say, doll, right that make sense kind of
David Bau: right like just like you patch over Megan Rapinaud. Did she all know? Now you play basketball, you play patch it over. She's a doll right?
David Bau: So now, so
David Bau: so there's a different theory, though, of like what might happen if you patch it over, because maybe A doesn't really mean the thing that has a doll in it, maybe a what's being represented. The middle layer for a is like the thing named a
David Bau: right.
David Bau: and if you patch over some representation, that means the thing named A from sentence one to sentence 2.
David Bau: Then, you know you, you would get a different thing right, because, like in sentence 2, the thing named a
David Bau: doesn't have a doll in it right like if we patched over the the idea that what we've got here is the thing named A.
David Bau: Then, instead of it going to doll, it would say something else like, What would it say? According to this theory.
David Bau: it would say, Peach, right?
David Bau: So we have this very clear dilemma, is it? When we patch it over? Does it say, doll?
David Bau: Or does it say peach? And so we like, there's this very clear like, this is like model. This is like AI epistemology, like, what does the model know about box? A
David Bau: right? What is it thinking? Well, it's saying something. It's saying lamp, right? But what does that know? Does it know that it's like the name of a thing that's called a or does it know what a contains like? What is it thinking? Right? And so, who's on team? Who's on Team Doll? And who's on team peach when we actually run this experiment. What do you think it says? Do you think it says Doll or Peach.
David Bau: who's on team doll like just like the Shaquille O'neal story, like theory. Okay? Who's on Team Peach?
David Bau: The the reference story and who? Who's on team? It's gotta be something else.
David Bau: Who's on team. I don't know.
David Bau: Oh, come on, we want bets. No free sandwiches for you. Okay,
David Bau: well, okay. So we ran this experiment, what happens? Dollar peach?
David Bau: Right? So we do this, and I'll show you what we get.
David Bau: Make a key.
David Bau: Why, how? How can we? How can we get key?
David Bau: How come we get key. So you know. So my, my student, Nikhil, set this experiment up. I'm like we're going to get Doll. We're going to get Doll.
David Bau: and he says he says, Professor, we're getting something different. It's not doll. It's some other thing. And then and like, Oh, yeah, I know it. You must have been. You must be running into. We're getting peach. We're getting peach. Let's look at your data like, what's this peach thing? And if you don't set up the data all carefully. Then it's sometimes like you can't tell the difference between them right, like doll or peach right? Because, you know, if you set the sentences differently. He was getting key.
David Bau: and I couldn't see it. I was like, why is it turning out this way, like my theory is like consistent with a 3rd of your data. And this other theory is like consistent with a 3rd of your data. But for like 1 3rd of the time, it's not consistent with any of the theories. I don't have any idea. It seems to me like this model is just doing something totally random.
David Bau: It's like, I think, that, like our whole cause of mediation setup is totally wrong. It turns out it's not random. 100% of the time it gives us
David Bau: key, you know, at this layer.
David Bau: what is it doing? Y key? Yeah, you know. Come on, you're in my lab. So you know. No, fair Koyana.
David Bau: Okay, somebody else. Yes, yes, yes. Okay, yeah. You 1st sentence. Doll was in the 1st sentence, right?
David Bau: Y, key, right? Because a is a reference that points to 1st sentence. Oh, yeah, because I should. I have to hide my answers? But that's that's it. But that's why does that make sense?
David Bau: Depending upon the positional embedding for the 1st sentence, like, if you added in
David Bau: nonsense words, or like extra words, it still goes to the 1st sentence, yeah, it doesn't matter if you get the tokens all wrong.
David Bau: You're okay.
David Bau: Yeah, yeah, yeah. It's very. It's not. It's not about token offsets. It's about something else. So there's like a different geometry of pointers that's different from just token offsets. It's something a little bit more robust than that, but it is sensitive to like which sentence number it was.
David Bau: And so now, mapping out this geometry is an open question.
David Bau: Somebody wants to do a research project on it. Let me know.
David Bau: I'd love to love to help out
David Bau: and so but there's been a few papers that have looked at like, you know what is, what is the coordinate system going here, and it's not fully characterized yet.
David Bau: But but there is, there are these pointers. And so so this is. This is one paper about the Pointers is another paper about the Pointers. So you know what? My, my student who discovered this is interested in looking at theory of mind like answering questions like, what does Sally believe that Ann believes about? You know the contents of the box.
David Bau: these things, and you see these references all over the place. In fact, you see, like double the references and the data, the raw data looks something like this. So like here. You know, patching from this run of the story to that run of the story at the Colon. You can see. If you know, if you were to bother, you know, reading the story, you'd see that.
David Bau: You know the 1st story.
David Bau: Carla believes the cup contains tea.
David Bau: and the second story. Carla believes the cup contains coffee because she just put coffee in the cup right.
David Bau: But if you patch from story one to story 2
David Bau: at shallow layers, it'll get it to predict
David Bau: T. Because you're copying over this urge to utter the word T, and and it's understanding of what the value is. Is T. Right. But if you go deeper and deeper and deeper in the model. Then there's this very clear transition at layer 55, where, if you go deeper and you do the patch, then it causes this
David Bau: to say beer instead instead of tea and or coffee. Right? It causes a second model to say beer. And and why does it say beer? Well, because it's clearly a pointer
David Bau: to where T. Was and what the model contains is. It contains a point of reference.
David Bau: It's not dereferenced until layer 55, and then at layer 55, it dereferences it. And then, but in the new context, where it's dereferenced, that point, it points to beer
David Bau: that make sense.
David Bau: Yeah. So so these models clearly contain pointers, and only do they contain pointers. They contain pointers to pointers
David Bau: and things like this. There's like double d references.
David Bau: And so so the models are learning like non-trivial, you know, data structures and interesting things.
David Bau: Let me show you another thing that they're learning.
David Bau: Okay, well, we've used up an hour. So if you guys need to go off to other things, I'm not offended.
David Bau: Should I continue talking about more technical things?
David Bau: Okay, here's another thing.
David Bau: Yeah.
David Bau: So
David Bau: let's go back to that in context, learning example that we had at the beginning, right? So these are by analogies. So we start with demonstrations, and after we give a model a bunch of demonstrations, then we can give it a question, and after we give it the question it can give an answer
David Bau: right? And and so so the the puzzle here is.
David Bau: you know what's being represented, you know, in the model of that little purple thing
David Bau: right? If I were to patch that state from one to the other, what effect would it have? And so here, there's a particular effect I'm really interested in.
David Bau: which is I want. What I'm really interested in is, I'm really interested in the concept of, say, the antonym
David Bau: like I I like, I want to know how the model is representing the task that it's trying to do the task that it recognized. So it saw like these 3 or 4 examples of antonyms.
David Bau: And then I'm wondering if there's some attention, head or something at that, Colon, that's representing the idea, like I should do Antonyms
David Bau: right? And so the the way to find it is to create a causal score, where.
David Bau: if I do this thing at the bottom and I patch this thing over the bottom.
David Bau: I'm looking for the thing that would boost a certain answer right? So if I so like, so what's what's on the left? Here is demonstrations of nothing in particular. It's just garbage.
David Bau: right? Because I've shuffled all the words.
David Bau: And so when when you would normally say, you know you know, need fast, messy, huge, slow, tiny, sad Colon something. It would just put some random word there.
David Bau: right?
David Bau: But if I patch the right attention heads over instead of saying some random word. It might say something interesting.
David Bau: like what? My, what would be the interesting thing that it could say here? That means, do the opposite.
David Bau: What would it predict here, if somehow I forced the model to like, do the opposite here
David Bau: it would say, happy, that's right. And so so that turns into a causal mediation score function where I can ask the question, are there attention has, or, you know, or components, or you know, representation vector components or something like that are there components
David Bau: that caused my model to want to say, happy in this situation.
David Bau: or in thousands of other situations like it, that make sense.
David Bau: And so so like if
David Bau: if it was, if if what I patched over was something boring like oh, what word do you want to say next? I would get like fast.
David Bau: But what if but but if what I patch over was something interesting like, do the opposite?
David Bau: It would, it would boost happy instead. And so actually, when I look at the attention heads, most of the attention heads that I patch over are going to do the boring thing. They're like boosting fast. But there's a few attention heads that have this interesting property that if I patch them over it'll make this other model say happy.
David Bau: Does that make sense. And so, even though it's like this hidden signal deep in like a few attention, heads like that's a signal I'm really interested in. So let's do that as our sorting function, let's sort through all the attention heads in the model according to which ones make the model say, happy in this situation
David Bau: that makes sense. And so here's like, here's like the scatter plot of the attention heads through the model that get the model to say happy or analogous things in analogous situations. These are like the map of attentions colored by the causal effect, strength for the interesting effects that I'm interested in, that makes sense.
David Bau: And so so what you can do is
David Bau: I can get a bunch of antonym tasks. And I can say, What do these attention heads transport? They all transport some vector over to the last token position.
David Bau: Let's just do a bunch of Antonym examples. And I'll I'll average. I'll like sum up what all these attention heads are bringing. It's not even an average. It's just some like these attention heads are conspiring
David Bau: to do something interesting.
David Bau: So let's just sum them all up.
David Bau: And then what we'll do is we'll call that vector
David Bau: our function vector, right? And the cool thing is if you take that function vector and then you like.
David Bau: you know. Take it from one of these situations. You patch it over to some other random piece of text like here. I have a sentence here that says the word fast means, you know, and then the model would normally say, the word fast means fast, or the word fast means quick, right. But if I jam this function vector over here, I can get the model. To say the word fast
David Bau: means oh.
David Bau: well, like it would normally have said if I took the antonym one, it would say, the word fast means slow. Here I'm I'm skipping ahead a little bit. Here. I've collected a function vector using the same attention heads to gather a representation of a different task here, which is the Spanish English translation task
David Bau: that makes sense.
David Bau: So I can. I can feed in different analogies here, and it turns out that the same attention heads transport a representation of the task
David Bau: to be done
David Bau: that makes sense, and I can. And then I can test that representation by jamming in different contexts. And I can see that actually, it will do the task.
David Bau: That interesting. So so far I've showed you like factual recall.
David Bau: I've showed you copying. I showed you pointers
David Bau: right. I showed you lifting from like, you know, tokens to concepts.
David Bau: And then here is lifting further from concepts
David Bau: to functional relationships between concepts or lifting from pointers
David Bau: to like pointers, to concepts, to like function pointers. They're they're acting more like a function pointer, right?
David Bau: And and so these function pointers are really interesting. There's a geometry of function pointers here. So
David Bau: yeah, there's another one where I should live like this.
David Bau: Okay, so, but it's another aside. There's all these in interactive moments I should put into these slides. So.
David Bau: But but think about this.
David Bau: So this is a task where I can. I can demonstrate to the model. Here's some input on the left of the demonstration on the Colon, and then on the right.
David Bau: you say Italy. So if I show you this, and I say the output should be Italy, and I show you a bunch of examples like this. What the model will learn it'll learn. Hey, what you want me to do is, look at this list and tell me the 1st thing in the list. So I'll call this the 1st copy
David Bau: task. And and you can get a vector, that represents like, do the 1st copy task like you can get a function vector that represents that thing. And then and then if you just jam that anywhere it'll like, look for a list. It'll tell you the 1st thing on the list, and then you can also do the last copy list. Oh, here, give me the last thing on the list. It'll say France right, and then. But you can do another fancy thing, you say. Well, I don't want to copy the 1st thing I want the capital city of the 1st thing. I don't want Italy. I want Rome
David Bau: right? And you get a function vector, that says, Oh, yeah, give me the 1st thing in the list. But tell me what the capital of it was
David Bau: totally cool can do that. So so now we have these 3 function vectors above the line.
David Bau: And then. Now, I can synthesize a new function vector
David Bau: by doing vector arithmetic on the on the vectors.
David Bau: I can pick the 1st capital back there.
David Bau: I can subtract the firstness out of it.
David Bau: And then I can add the lastness to it.
David Bau: And I can get a 4th vector out.
David Bau: And I can ask, what function does that vector represent right?
David Bau: If I jam that vector into this into this prompt.
David Bau: that vector will end up saying, Paris.
David Bau: why the heck does it say Paris?
David Bau: It seems to be the capital of the last city, vector a totally unseen task.
David Bau: completely undemonstrated in the previous, you know context. But the model can do it. The weird thing about this is that when we set up this particular task, we're using Gptj, if you use in context learning to try to get the model to do this task by analogy. You say, here's a bunch of lists of things of countries, and I want you to get me the capital city the last thing on the list. Gptj. Cannot do this by example.
David Bau: Right? I mean, it can kind of do it. It. It does it like, I remember, like 10% of the time, or something like this, right? It like, has a really hard time
David Bau: doing this task by example.
David Bau: But if you but it can do the 1st 3 tasks are easier for it right? And so if you get the vectors for the 1st 3 tasks, and you do the vector composition. And then, instead of going to Gptj and saying, Hey, you know what I want you to do this task by by example, if you just give Gptj a single example of it. And then you jam in the vector saying, I want you to do this task by this. Vector then it gets much higher accuracy. It can. It can do the task.
David Bau: It just can't recognize it. And so
David Bau: so yeah, so it's quite interesting. So this is like, this is an example where, like the causal mediation
David Bau: you know, results in something that actually works better than what the model is able to learn itself. And so, at least for Gptj. When you get to bigger models, then they can do this, and
David Bau: then then it's a little bit more weak sauce.
David Bau: Yes.
David Bau: So this vector competition approach for others, possible or or mostly investigating.
David Bau: It's we're not doing it for anything practical yet. And one of the reasons is, it usually doesn't work. And so like. So the hypothesis here is that it can only compose like. We don't have a proof of this so, but I'll just tell you my personal guess. So we've only found a few tasks compose this way, but when they compose this way they do generalize, it's pretty striking.
David Bau: But but it it feels to me like, there are different rounds of action going on in the model. And what's happening is that picking
David Bau: where you are in the list for where you're going to get your data and then doing some transformation on the data are like separate rounds. And the function vector contains
David Bau: a specification for both of these rounds. And so so we, by subtracting away the information for the 1st round and then adding a different definition. For the 1st round
David Bau: we can leave the second round of action like unaffected, and then and then and then it all works, but but like, if we do different things
David Bau: that interfere with each other, because, like in the same round of computation, then then it's just a mess, and you don't get anything sensible out of it. So I feel like it's a. It's like a possible map. It's a possible technique to map out what the rounds of abstract computation are of like task performance and model later on. It's a future thing. If anybody's interested in doing that research and let me know.
David Bau: you know, I've got ideas on how to how to use this as a tool. But but yeah, so it doesn't always work. So this is one of the. I think this is one of the interesting things about as a scientist, right? So as an engineer, there's a lot of people. So let me let me let me zoom around here. So there's okay. So there's I have a lot of too much to talk about, and so I could be here for another year right? But but
David Bau: but listen, you know
David Bau: there's a little bit of like I was telling Max this because because, you know, Max Netto helped set up haste, and he's on the West Coast right now doing things on the West coast. And I told Max, you know, Max, you need to really support East coast work, because there's a different culture on the East coast than there is on the West coast, because, you know, we can get East Coast people to think different differently. And
David Bau: And so here's a little bit of what I would consider East Coast thinking that and it it is related to the kind of you know questions that you're asking.
David Bau: which is.
David Bau: you know. Okay, so on the West Coast. The question about Llms is, how do you get them to
David Bau: do better like my gosh, they hallucinate. They can't reason. The benchmark scores on these difficult things are not good enough. We need to get them better. Right? And why is it that everybody's like trying to figure out how to debug Lms to like do better?
David Bau: It's because the culture on the West coast, totally dominated by these massive venture funded startups. That are all.
David Bau: They're all predicated on the idea that a trillion dollar investment is worthwhile, because, even though the existing ais are not good enough with enough cleverness, you can solve their problems and get them to work better, and they'll be brilliant products.
David Bau: And and so everybody is focused on that
David Bau: but as a sort of a a little bit more distant from this hype cycle.
David Bau: You know, just, you know, a few miles separated from that few few hours of time zone difference? Right? It gives us a space
David Bau: to see things in a different perspective. And so you know, Bateau, who's a French, you know computer scientist who's been working on neural networks since. You know, he invented stochastic gradient descent. He came back from Iclr. And he he said, Yeah, he noticed this dynamic that everybody was worried about the same. Several problems like how to get reasoning to work better like, how to get hallucinations, to go away like how to. You know how to remove biases, how to like, make these things better products right? And he says, actually, the real situation we have in our field
David Bau: is that these Lms are a wonder of science.
David Bau: They already achieve and execute certain complex tasks with near perfect accuracy, with near, perfect reliability. Well, beyond what any of the people who made them expected that they would be able to do.
David Bau: and we don't understand how they do it
David Bau: right? We don't even understand what line we cross like, why, previous ones, we're not able to do it
David Bau: right. And so as a scientist, his perspective was like.
David Bau: Yeah, everybody's barreling ahead, trying to like, you know, solve the next problem that's not solved. We don't even know how we already solved the problems that we can solve. Like, if we don't understand that, what makes us think
David Bau: that, you know, we're in a good position to figure out
David Bau: what's going on with the next thing. Now, my perspective is a little different from Bateau's.
David Bau: You know. I like there's a reason I like to hang out and chat with safety people, because I also think that there's an ethical issue here, which is, we really don't understand these things at all. And I think that as engineers it's our responsibility to understand how the systems work that we make. And and and but I think that Patow's, you know, perspective is.
David Bau: you know, it's the mirror image of that. And it's basically, you know, the smartest people in the field don't know what we're making. And and so
David Bau: and so.
David Bau: and and all, all the science that I've showed you up to now is just an example of like this is, these are examples like these are very basic examples of things that we did not know
David Bau: that our models learned. We did not know that they learned pointer indirection. We did not know that they they learned factual association lookup tables, right? We did not know that they learned function vectors. We did not know that they learned a dual track
David Bau: of copying, of copying tokens versus copying concepts. We didn't know any of this stuff right? We didn't know that there was a that, even though, when I asked my model to turn off the lights it can't do it, but actually, internally, it does. It just doesn't know what the English word is like we don't. We don't know any of these things. If you didn't ask if you didn't have a way of asking deep seek, do you know about Tiananmen Square, we'd assume that it really doesn't like. Somehow they've trained the model not to know anything about it. But actually, it knows
David Bau: right? It know, like these models know things that we don't know that they know.
David Bau: and that's really what the theme is for interpretability. And so one of the other things I want to do. I do want to take a minute to go over. The last technical thing
David Bau: is that you know, I've shown you some really basic things that we, you know these models are doing that we didn't know before. But the other Meta lesson is that
David Bau: you know, if you take all the papers of this form they're trying to like, look, do science on on AI. Look inside what they're doing, the average size of the models, the average capabilities of the models that are being studied by these papers is down here. It's at like the Gpt. 2 level
David Bau: right? And and there's a few papers that are trying to work at the, you know. Llama 70 B, you know, sort of, you know, model size. But there's very few.
David Bau: And there's basically no papers that are working up here, you know, closer to the state of the art. And so so if we're worried that these models know things that we don't know, and that it's really important for us to understand them.
David Bau: And you ask who is doing work at figuring that stuff out
David Bau: at the state of the art level? The answer is, nobody.
David Bau: Nobody is doing it, you know, basically nobody.
David Bau: And and so so like that. So that's my last, you know. Exhortation is that we need to figure out ways of doing work to understand what the most powerful models are doing. And and so I I do. I run this project while we're trying to build the infrastructure to allow this to be done.
David Bau: It's the Ndif project.
David Bau: And so we have some ndif people here.
David Bau: So Michael Michael is Michael's 1, that he he's the one who created the entire back end for Ndif and and so but let me take a moment to explain. You know what it is. So the the idea is
David Bau: that is very expensive and inconvenient to study the largest models
David Bau: like, if you want to say a small model, you just, you know.
David Bau: Get get the Harvard AI Safety Club to fund a Gpu for for you somewhere. You can load a small model and you can study it. But if you have a really big model.
David Bau: then it's really hard to study it. Even if you were able to get funding for it. Not not that many people can get funding for it, because each one of those setups cost like a million dollars. And and so so it's hard to paralyze research. It's hard to make progress in it, because every researcher turns into this 1 million dollar researcher.
David Bau: right? And so so what what ndif is is asking the question, can we share resources? If like, if if multiple people are interested in studying a million dollar model.
David Bau: right, can we share the compute resources for to enable many researchers to use it efficiently? And so the idea is that instead of you using your own Gpu to run the models.
David Bau: Why don't we use the inference architecture so like when you chat with Chat Gpt that, like Openai, will not spin up a new
David Bau: computer cluster a new, a new set of nodes to run, you know, Gpt. 4 for you until it has enough traffic for a thousand people to use that computer simultaneously. Right? That's like the level of scale they need to like, justify like the electricity right? And so the question is, can we approach that? Can we? Can we run science in the same way? Can we make it so that when you
David Bau: look at a big 1 million dollar model, can you share that with a few other researchers. If you, if we could share that with 10 other researchers, then, like we, we've reduced the cost of making progress on this by tenfold
David Bau: that makes sense.
David Bau: And so so the idea is
David Bau: but to do that you can't take custody of the model. You need to have the model run in Ndif where Ndif can share it?
David Bau: And so, on the other hand, even though you don't have custody of the model you to run these kind of crazy experiments where you're patching things and fiddling with vectors and stuff. You really need full control over every step of the computation of the model. So it seems like this contradiction that you don't control it. But you need to control everything
David Bau: right. And so Ndi App is basically an architecture that lets you do that. And so and it's something that you can use today. And since Michael hangs out here, he can like teach you how to use it if you want and and Alex is, you know, involved in the project. He can teach you how to use it.
David Bau: And so this is
David Bau: I'll show you the Api, so you can get a little sense for it. It's it's a pytorch extension.
David Bau: and you can go to any pytorch model, any neural network, and you can wrap it in an end. If wrapper we call it endsight is the package. So you can. You can say, I want to do. I want to get some insight on this neural network. So let's like wrap it with an insight object, and we have pre-wrapped
David Bau: objects for all the hugging face models. So you can go to a hugging face model and say, Hey, I want this hugging face model. It's called Meta llama 70, you know, whatever right?
David Bau: 7 B,
David Bau: and then you get, you know, regular pytorch module, a function that you can call. You can pass this. You can pass input to this function and you get output. But passing input and getting output is the black box view of neural networks. That's what's gotten us into trouble right? And so what insight does is it takes these functions, and it makes it so that instead of only being able to use them as functions
David Bau: where you give it input and you get output. You can also look and manipulate the stuff in the middle of the function while it's running. And the the way that you do that is, instead of calling Lm as a function.
David Bau: you you use it as a context block where you say with.
David Bau: And so you say with this thing. So I'll pass it some input. Like here's some tokens that I want you to to run. Inference on. The truth is the something right inside the with statement, we can add extra code of something that I want to do. So here's some simple code. I just want to read the output of the function and save it right?
David Bau: And then we can print it out. Or I can put more complicated code in there like, grab these neurons.
David Bau: you know, at layer 14 in the Mlp sub module in the down proj thing, I want to get the inputs for these neurons. And I'll take these neurons instead of all 10,000 neurons. I want to take these 3 neurons, layer neuron 32, 60, 77, 30, 89, 30. Where, and then I set them high. I'm going to do some causal.
David Bau: you know. Causal causal change on these neurons. Flip them up, and then I'll let the rest of the model. Run and read out the logits and and see what it predicts, and I'll print that out. And when I print out the case in this case, instead of it saying, The truth is the truth
David Bau: right? When I flip these 3 neurons on. Now, my language model is saying, the truth is the lie.
David Bau: Now, where did I get these 3 neurons? These are function vector, neurons, right? That that you know, Eric found in his
David Bau: function vector, thing. These are these are the neurons that basically, you know, cause the model to execute the antonym task. Right? So by just flipping on these neurons, then I can say I can get the model to say the opposite of whatever it was thinking.
David Bau: that make sense.
David Bau: And so. But you know, this is just 3 lines of code to get these neurons activated. But if you wanted to find these neurons, then you would write a loop and you would set up some scoring, and then you'd go through all the neurons and look for what you're looking for. And you just write more code here that makes sense. And what's what's happening behind the scenes is that if you say remote equals true, then you don't actually have to run this model yourself.
David Bau: This code that you wrote, is like transported over the network to this giant trillion parameter model that's sitting on the back end. And we do inference just like Openai does inference on the model. But, unlike Openai, where they just take some inputs and give you outputs, we'll do inference.
David Bau: and we'll run your code as a modification in the neural network for the duration that the inference is happening. And then so when you get you know you get stuff back, then you can do it. Oh, I have some announcements here, but I don't know whether I'm announcing it so. But anyway, yes, so you can enroll in the pilot. Blah blah! I'll I'll I'll delete that because you guys can just use it.
David Bau: And if you have questions you can, you can ask, you know Michael or Alex and and they'll get you hooked up
David Bau: that make sense.
David Bau: Yes. So okay, that's what I wanted to cover.
David Bau: Sorry to go so long.
David Bau: All right. So I'm gonna I'm gonna stop zooming here.
David Bau: and then you can ask me all your root questions.