David Bau: my favorite reaction.
David Bau: Welcome to Week 5.
David Bau: Definitely.
David Bau: How many weeks are there in the semester?
David Bau: new packs.
David Bau: Welcome. Is Zoom working? Can somebody on Zoom, hear my audio? Say hello?
Veronica C Perez: Yeah, we can hear you.
David Bau: Yeah, that's great, thanks. Okay, cool. So, so today, we'll talk about
David Bau: What… so, in my opinion.
David Bau: This is, like, the core experimental zone.
David Bau: design.
David Bau: category?
David Bau: that I think makes this type of… analysis.
David Bau: Possible, and what sets it apart.
David Bau: from most… Fictional neuroscience, or… oGnoscience.
David Bau: Because… We could actually… Take control over
David Bau: these neural networks. When we run them and they… they think about something, they do some cognitive activity.
David Bau: Not only can we observe every single calculation that they do.
David Bau: But we can trace and debug those calculations. We can do what scientists call causal experiments. We can ask, what if? What if it worked differently?
David Bau: And then we can actually do that.
David Bau: and… and see what happens. And so that's… that's different from…
David Bau: What most neuroscientists and cognitive scientists can do?
David Bau: Right?
David Bau: For most cognitive systems, you just work with them as a black box, you think.
David Bau: You know, try to figure out what a person
David Bau: It's thinking by interrupting them, by saying, okay.
David Bau: Look at this image for… 28 milliseconds.
David Bau: And what was your impression from it? And trying to interrupt the process and trying to get a little bit of insight. That's what people have done for decades.
David Bau: But…
David Bau: In, in artificial neural networks, we have this amazing luxury of being able to go into the calculation and do things directly, so…
David Bau: I'm gonna… I'm gonna describe to you the basic form of that experiment that we can run, and there's…
David Bau: you know, many dozens of variations on them. Some of you already asked some of the variations.
David Bau: In reading our book, I'm happy to…
David Bau: Happy to chat about them, but I want to make sure everybody understands the basic part. That's what I'm going to spend most of the time talking about.
David Bau: Seth? So, okay, so the idea is…
David Bau: To do causal tracing, to use causality to trace the calculations.
David Bau: Through the network, to try to get an insight for not only what information is there.
David Bau: But what information is being used?
David Bau: To determine the outcome.
David Bau: To determine the output.
David Bau: The problem with just seeing information as present is that These are huge neural networks.
David Bau: They have thousands of neurons, and so if you ask
David Bau: the question. Oh, is there some information present about the input, about the stimulus in this neural network? The answer is almost always
David Bau: Yes, absolutely. Tons of information is present. But then, if you ask, well, which parts of that information are actually being used?
David Bau: To make a decision.
David Bau: Well, this was unknown.
David Bau: Until we were able to run these types of experiments.
David Bau: People used to think, well, yeah, all the information's present, probably all of it's being used.
David Bau: But actually, these networks are quite picky.
David Bau: They often… the very small subset of information to use, or… output.
David Bau: You know, just sort of first order.
David Bau: And, and so we'll talk about… so this is… this is the experimental design that lets you do it. So… so when we get an understanding what it is, it's…
David Bau: It's… it's an ability to make changes, so how do you know that…
David Bau: the network is using some piece of information? Well, because you can physically go And change that information.
David Bau: And then see that the network does something different.
David Bau: Right? And most of the time, when you go and you change some random information in the network.
David Bau: It won't Have an effect, or won't have Any sensible effect?
David Bau: And, And so, so yeah, so this is kind of an interesting thing, but…
David Bau: Right? Does that make sense? Okay, so let me make this concrete.
David Bau: So I think that I've shown this diagram before. This is the inside of the transformer, remember? Right.
David Bau: And so I've set this transformer on its side, in the standard way, I like to draw it, with layers.
David Bau: Of neurons sort of going from left to the right. So these are the earliest neurons.
David Bau: And those are the latest neuron neural layers.
David Bau: And then you have all the information coming in.
David Bau: Here, as encoded tokens.
David Bau: So, a vector for miles, and a vector for Davis, and so on.
David Bau: And so… so yeah, so it was… it was sort of amazing to people when they trained language models and they were able to recall facts.
David Bau: about the world, like, what instrument Miles Davis plays, or what instrument Yo-Yo Ma plays, or whatever, right? It's kind of funny for this language model to know something about yo-Yo Ma, but it does.
David Bau: Okay, so how do we… so, but then it leads you to this question.
David Bau: Where is that knowledge?
David Bau: in the network. Is it sort of… Everywhere?
David Bau: If you probe for where's the information.
David Bau: And we'll talk about different ways of probing Next week.
David Bau: And so, but, you know, did you vote for the admission? The answer will be, yeah.
David Bau: that information is sort of everywhere. Maybe it's stronger in some places than weaker in other places, but the information is sort of everywhere.
David Bau: And, But… But… but… so does that mean that the calculation is…
David Bau: This, this information comes in.
David Bau: And then it goes into the neural network, and it's sort of like an information gas.
David Bau: The only thing that matters is, like, the pressure of information in there. It's like, oh, we put a little bit of extra…
David Bau: hydrogen and a little bit of Miles Davis, and it's just mixed, you know, bouncing around in this, you know, formless gas out there, and then mysteriously, like.
David Bau: outsquorts the trumpet out the end, who knows how, right? So that… so I feel like, you know.
David Bau: Without this type of research.
David Bau: That's been, more or less, the mental model.
David Bau: for how black box systems work. It's like the mysterious information gas.
David Bau: Right? But what you find is, if you do causal experiments on the inside, you see a lot of structure.
David Bau: It's like an embarrassment of structure. And so, and so let me show you how to do that.
David Bau: Okay, so the basic idea is that we're gonna be like little transplant surgeons. We're gonna transplant data around.
David Bau: To identify what the effects of the data are.
David Bau: So… So, to transplant things, you need to have to… Patience.
David Bau: Right.
David Bau: So, we, we clone our neural network.
David Bau: And we run it a second time.
David Bau: So, you run it twice, And you have to have something different.
David Bau: About the second time you ran.
David Bau: And so, so here, I'll have a very similar sentence. Miles Davis plays the trumpet, maybe here I'll have another sentence, like, Yo Yo Ma plays the… something, and then maybe it'll output cello.
David Bau: Something like that. Or I could put in… the form of the experiment that was in the original Rome paper that you guys read was, you put noise in here.
David Bau: And then it just outputs some garbage.
David Bau: Right? Say, blah blah blah plays the… And I, you know…
David Bau: It has some perturbation of a piano or something, so I have no idea.
David Bau: Make sense?
David Bau: And so, so, so, so, but all that matters, though, is that it's not trumpet.
David Bau: But it's a very… but it's a very similar… neural network.
David Bau: It's, like, a very similar brain here.
David Bau: It's similar enough that maybe we could transplant pieces from one to the other.
David Bau: And then, see what happens.
David Bau: Why do you call it corrupted if you're, like, for instance, if you said, like, yo, you don't walk, miss the cello, like, that sounds about right. Corrupted usually means, like.
David Bau: Not right. Yeah, this… so this comes from this experimental design.
David Bau: comes from medicine.
David Bau: Where there's… they usually try to distinguish between A and V.
David Bau: by giving it a more meaningful name than, like, A and B, they'll say something like, the healthy patient.
David Bau: And the sick patient.
David Bau: Right? Or that's healthy and diseased, or something like that.
David Bau: And they'll say, oh, you know, what's the difference between our healthy population and the diseased population? Is it their diet?
David Bau: Well, let's take some diet…
David Bau: activity that the healthy patient has, and we'll force the disease population to have that thing. We'll, like, come to their house every day, and we'll feed them food, right? Or something like that. So they'll do a causal intervention.
David Bau: And they'll say, oh, look, that was causal, that, like, cured their disease.
David Bau: Right? And so… so what we're… so what we have here is… just so we're imitating the old medical setup.
David Bau: You know, we, we've, we've sort of adopted
David Bau: The term, this is the, clean run, sort of the healthy run.
David Bau: Right? And then the one that you're changing is, like, the disease run, the one that you're trying… trying out, testing the treatment on.
David Bau: Right.
David Bau: And, I don't know. I do think that this is maybe a little confusing, so sometimes I use source and target, so there's, like, the source.
David Bau: So when you're thinking of cognitive things, it's like the source of the idea.
David Bau: That you're trying to test. Like, this thing has some idea.
David Bau: And then there's a target, which is… it has, like, the context and everything, but you think it doesn't have the idea yet, and you're looking to see if you can find a physical instantiation of the idea.
David Bau: To put down here, to get the target to have the idea.
David Bau: that it wouldn't depend.
David Bau: That make sense? So… so this target, it doesn't have the idea of Miles Davis.
David Bau: And we're gonna… and we'll treat it as if That was a disease.
David Bau: You know…
David Bau: We're gonna… we're gonna see if we can fix that, and get it to think about Miles Davis, right? And now, there's some obvious things you could do. Like, you could… you could say Miles Davis on the input, and that would… that would get to, say, Trumpet here on the output, right? But… but what we're gonna do is we're gonna not do that, because we already know that that will happen, right?
David Bau: We're gonna hold that input steady at yo-Yo Ma, or whatever our corrupted input is. And then what we're gonna ask is we're gonna ask, is there something in the middle of the brain that you can move down that causes it
David Bau: This switch from… corruption.
David Bau: to something meaningful. And what will happen is that normally, When you do this.
David Bau: the output is also… it stays corrupted. It stays… it might stay cello or something like this, right? But once in a while, when you hit the right spot.
David Bau: The information will flow through here, And you'll get trumpet.
David Bau: That make sense?
David Bau: So, it's just like, oh, once in a while, like, so for most of the interventions you do for a disease patient.
David Bau: it doesn't do any good. I wonder if the reason that they're overweight is because of their library card.
David Bau: You know, we're gonna go and distribute
David Bau: you know, different library cards to everybody. Maybe if they have to walk farther to the library, they'll lose weight, who knows?
David Bau: Right? So, no, so give everybody new library cards. Now, that wasn't it. You know, maybe that was a correlative thing. Maybe people who are healthy have different library cards.
David Bau: But, you know, but it wasn't causal.
David Bau: Right? And so, most of these interventions don't have any effect.
David Bau: Right, but once in a while.
David Bau: You might hit on the thing.
David Bau: Right? What's… it may be… maybe it's because of some…
David Bau: Some… some enzyme in your, you know, some…
David Bau: some enzyme in your brain, some, some hormone or something that you've got. I wonder if we inject that thing into you. Oh, this… what is this GLP-1 thing?
David Bau: You know, maybe if we inject that into you, it'll cause you to lose weight. Holy cow! It does! Like, 40% weight off, right? So you find the causal variable. That makes once in a while, out of the millions of variables, there might be.
David Bau: Make sense? So that's kind of what we're trying to do.
David Bau: Alright, so… How, like, how do you do this in practice?
David Bau: Well… Where's Adam?
David Bau: Thank you, Adam. Adam just put this website on. Will you try typing this into your computers? You could literally try this.
David Bau: Right?
David Bau: And, you know, I don't know, I can maybe…
David Bau: Should I try to? I'll come along with you.
David Bau: So, so let's, let's, let's try this out. All right, Adam, see if the website stays up with, like, 10 people scanning it. And so, so we're gonna… so, oh, it's not in this menu yet, it's okay. So let's, let's… so I'm just gonna put a little thing here, I'll say, you know, Miles Davis…
David Bau: Please, right?
David Bau: plays that something, right? You know, whatever. And so, I'll adjust… I'll adjust this as I get into it. There's different options once I get into it. So, I'm gonna say I'm human.
David Bau: It's very sophisticated now.
David Bau: Okay, and then… oh, I have to click on three icons that differ from the others. This is diff… oh, no, I missed it. This one, this one.
David Bau: This one, okay.
David Bau: Such a bad cooker.
David Bau: Devices that depend on battery power? I don't know. Like, it depends. Because, like, if you have, like…
David Bau: Oh my god. Like, if you have, like…
David Bau: You know, if you have an old camera, I don't know if you need batteries.
David Bau: I guess I had the flash. Okay, so, what I'm gonna do is, so there's this new tool down here that
David Bau: That Adam just added is very hidden.
David Bau: But it's down here, see this activation patchy button? Right. And then what is this? There's two places to enter text. Let me enter… okay, so I'll say Miles Davis.
David Bau: Right? Is what we're doing, right?
David Bau: glaze…
David Bau: What is the… okay, it's gonna be… okay, that's the challenge for… I'm gonna… I'm gonna use GPJ6B, because it's a little smarter than the old… GP David. Oh, Miles David.
David Bau: I wonder if Miles David plays. Okay, so it won't know who Miles David is, so if I click out here, I can add this in. Okay, thank you. Nice. Nice. Okay.
David Bau: And then if I say, yo yo ma, Right?
David Bau: Omaha, right?
David Bau: Plays down, okay.
David Bau: So this is nice, right? Right? And can I run this yet? No, there's no experiment yet, because this is a causal mediation experiment, so I have to set up something, so if I click on…
David Bau: Davis…
David Bau: I go to Ma, okay? Now, what I'm gonna do is… what this is doing is it's gonna say, what if I take the representation that's sitting in the transformer when it's processing this token, and then move it over to the other transformer when it's processing this token? I'll show you what it looks like in the diagram in a minute. And you run it.
David Bau: Ben…
David Bau: It'll scan through all the layers, it'll do this, it's actually gonna set up something like 30 different experiments, and it'll run them all here, and it shows me this. Oh, look, there's a new Y scale, thank you very much.
David Bau: Okay, so, so what is it saying? It's saying…
David Bau: that if I went to the last layer of the transformer, And I said, okay,
David Bau: You know, the transformer is grid-shaped, and…
David Bau: And then when it processes Davis, Up to the last layer.
David Bau: Right? And I take that last layer representation of Davis, and I patch it over to the last layer of representation of mom.
David Bau: Right? And then I run the network, then the output is high probability for red and low probability for blue.
David Bau: Red is cello. So, right? Well, it's missing the O? Because the signal's broken. It's about to say no, right? So, so it's… so it says cello, so it doesn't work, right? So if I patch over…
David Bau: At the last layer, From Miles Davis.
David Bau: To yo, your mind.
David Bau: it doesn't have any effect. It stays…
David Bau: challenge. It still thinks about Yo-Yo Ma.
David Bau: Right?
David Bau: But then there's this interesting transition. If I…
David Bau: patch earlier and earlier layers, there's this very distinct moment where
David Bau: If I patch over, like, layer 12 or 13, Right?
David Bau: That thing slips over.
David Bau: Just say trumpet.
David Bau: Right? So this is an N equals 1 experiment, but even at n equals 1, that's a very, very…
David Bau: I mean, there's 50,000 tokens, so it's not some accident that it's the same company.
David Bau: Right? They had something to do with Miles Davis.
David Bau: Does that make sense?
David Bau: Even though we've only run one experiment here, and obviously you could run hundreds more to get agreed on whether this is a systematic event.
David Bau: But here we're just running one. And… and so there's something that if you patch over
David Bau: the computation of what is being thought here, it'll happen here. Now, if you go to the very first layer.
David Bau: Right, and you patch over Davis.
David Bau: It does… it bumps Trumpet up a little bit, but it's… these are all near zero. And cello is near zero, Trumpet's near zero, right? The model would…
David Bau: This is what corruption looks like.
David Bau: The model's not happy saying either Trumpet or cello.
David Bau: There's probably some way that we can go into the user interface to see, like, if there's some other word that it's happy saying, but… but it's really not happy saying either one of these. And… and why is that? Because
David Bau: It would be… if you… if you patch over at, like, layer 0, That's, like…
David Bau: Hitting the transformer right at the moment where it perceives this input.
David Bau: It's like patching over the, like, the auditory signal or something like that. It's like, you know, changing the ear.
David Bau: Right? And so, that's like asking the model.
David Bau: you know, Yo-Yo Davis plays the something, right? So if you ask, like, Yo-Yo Davis.
David Bau: Then it's like, I don't know!
David Bau: Maybe the trumpet?
David Bau: Maybe the cello? Probably neither one, I'm not really sure what to say.
David Bau: Does that make sense?
David Bau: So, so at this moment, it doesn't really… like…
David Bau: have the whole concept of Miles Davis in this place.
David Bau: It's like, it leads to some corruption. But weirdly, as you get to layer 12,
David Bau: it's very strong. Like, as soon as you, like, hit layer 12, then patch you over this one.
David Bau: token representation, Makes it think coherently.
David Bau: I don't know if it's thinking coherently monotherapists, but it's certainly thinking coherently.
David Bau: trumpet player.
David Bau: Does that make sense?
David Bau: Interesting, right?
David Bau: Is that weird?
David Bau: It's weird, right?
David Bau: No, I think it's weird. When, you know… So, I'm very prominent.
David Bau: Of being the first one to run this experiment.
David Bau: Yes. So, you know… So, okay, go ahead. Is the patching at the level of individual neurons, or the whole layer? So, it's at… let me bring up the slideshow again.
David Bau: So this is the… this is the… one of the core things from the wrong paper.
David Bau: And, and… and we'll put it right here.
David Bau: Okay, so where are we? We're right here.
David Bau: So this, this, this diagram answers the question. I'll just, I'll just go back to this.
David Bau: So yeah, so here, we're patching at the unit of one of these dots.
David Bau: And this dot is about 1,000 to 10,000 neurons, depending on which transformer you're working with. So when they say, you know, sometimes the cool kids will say, hey, what's your D model?
David Bau: D-step model, right? You know, what's your D model?
David Bau: And, like, bigger models will have a larger dimension, and smaller models will have a smaller dimension.
David Bau: And, like, a small B model would be, like, a thousand. Oh, it means we're using a thousand neurons to represent a word.
David Bau: Right? And then a large D model is, like, 10,000. Oh, that's a big model. You've got, like, a… your D model's, like, 20,000. I heard a rumor that… I heard a rumor that, you know, the new Chinese models are using a D model of 20,000. That's crazy! How much memory is that? Right? Okay, so… right, that's… that's, like, what people will say. Make sense?
David Bau: Okay, so this is so… this is the… the number of neurons you're carrying is the D model, and… but it's not, like, the whole layer, because the layer has a different representation for every column of tokens.
David Bau: Make sense?
David Bau: And so we're bringing that one column over.
David Bau: Okay, so that's how this experiment runs. Now, you can do cost mediation analysis on any subset.
David Bau: you want, and there are people who do very fancy things. So you could take one of these wires and patch it over, and so there's some experiments that you see in other of the papers that, like, take… bring over one of these attention wires.
David Bau: and patches it over. You can bring over just the MLP wires, I'll show you one of those in a minute. You can… you can slice these neurons in a different way. Instead of saying, I'm going to patch over all thousand neurons.
David Bau: you could do linear algebra on it, and patch over, like, a subspace of the neurons. If you feel like the neurons are working like a vector algebra, then you could say, oh, I want some components of that vector to stay the same, and other components to
David Bau: To be patched, so you could do fancy math on it, and so some people pursue that theory.
David Bau: So… so there's different ways you can do this, but this is the basic experiment.
David Bau: Make sense?
David Bau: All right. Oh, and Anya asks, why does causal influence Where's anonym?
Ananya Malik: I'm here, actually, online right now. Hello?
David Bau: Yes, I can hear you.
Ananya Malik: Hi. Sorry.
David Bau: What was your, what was your full question?
Ananya Malik: Yeah, I guess. I wanted to know why does it affect, like.
Ananya Malik: Despite the same architecture, why does it affect… why does causal inference, like, affect different layers differently?
Ananya Malik: I think that was a little hard for me to, like.
David Bau: And does the diagram make it clear? So, so, so you see this, like, elbow here? Right?
David Bau: So this is… this is my mental model for why it might behave differently.
David Bau: So, if the information
David Bau: flow, like, flow through here. So, like, this, this, this light blue diagram is my… is actually sort of my mental model for how information is flowing in this.
David Bau: In this, in this, in this system.
David Bau: Before you get to layer 12 or something like this.
David Bau: is sort of diffuse. Miles Davis is just, like, this diffuse things is about tokens, but then something happens at around layer 12, which consolidates information at the token Davis.
David Bau: Right? Of, like, who this person is, or what instrument they play, or something.
David Bau: Right? And then… and then, and then that information, like.
David Bau: Hangs around at this layer for a little while.
David Bau: Until layer 20 or something?
David Bau: And then, all of a sudden.
David Bau: The information flow path moves over to a different token.
David Bau: And the information may still be there at layer… at this… at this token, but after layer 20, it's not on the path anymore.
David Bau: So if we go and we do a causal intervention too late, right here.
David Bau: Then it no longer has an effect.
Ananya Malik: Nice.
David Bau: So it would be… it would be like,
David Bau: It'd be like, hey, you know what? I'll get you to lose weight.
David Bau: By… by bringing you to the gym. I'm gonna send a car to your house, and I'll bring you to the gym every day. And my gosh, it works! If you send a car to my house at, like, 6 AM,
David Bau: But if you send the car to my house at, like, 2 in the afternoon, well, it doesn't do anything, because I'm not there, right? You know, I'm, like…
David Bau: I'm going… I'm off at the restaurant, picking out.
David Bau: And, and so it doesn't help me lose weight.
David Bau: Does that make sense?
David Bau: Yeah, so that's basically what's happening.
David Bau: Okay.
Ananya Malik: Awesome.
David Bau: Yes, just one thing? Yes.
David Bau: when you presented this before giving the results, my intuition was that the latest layers would be easier to influence the output, but it seems like I'm wrong about that. Wow. And I don't understand why I'm wrong. Okay, here's why, here's why. Because it's a two-dimensional grid.
David Bau: And so, if you… actually, I have… I have that result. I'll show you the result. It's a good… it's a good… Oh, before I show you the result, let me come back to it. Okay, sure. Okay. So there was another question.
David Bau: There's another couple questions I just didn't want to just get into. Patch, corrupt, clean, instead, compost. So, Bryce, what was your question?
David Bau: I thought this would be a good moment to have. Oh, I was just asking, right now we're doing what it…
David Bau: Oh.
David Bau: We're computing the content running.
David Bau: patching the, like, clean runs activation.
David Bau: Yes. Would it be if you're doing…
David Bau: run with the clean prompt, and, like, patching the corrupted one, and what you saw, but basically the reverse. Yeah. So, you know, so I think that, you know, my answer to that is more or less, oh.
David Bau: There's no real difference between the clean and corrupted run, right? It's just a naming convention, and so that would be the equivalent of just, like, oh, you put YoYo Ma up here, and you put Miles Davis here, and you just run the experiment, and then, you know, now you're taking, like, now, it would just flip the red and blue lines, and, you know, normally, without a patch, Miles Davis plays a trumpet, and you're bringing the idea of a cello player into it.
David Bau: And so it's just switching. But now you might call that one the clean run, and the other one the perfect run, just because.
David Bau: You know, if there's a source for the patch and a target for patch, it's just what we're calling it.
David Bau: That makes sense. Now, you could do more complicated things. You can do multiple patches at once, you know, just like you could imagine, oh, you know, we have some medical intervention where you take this drug, but at the same time you do exercise, and at the same time you do other things, and so, you know, we can do this type of thing here. I'll describe some more complicated experiments soon. Yes?
David Bau: Oh, is it kind of like, like, so, like, when I'm baking and I want to make, like, a chocolate cake.
David Bau: it's like, I have to add the cocoa powder, like, pretty early on, like, when it's still a batter. Yeah. Because, like, by the time it's already baked, like, that's just a vanilla cake with cocoa powder. On top, you get a different thing. Right, yeah, exactly. And so that's a good way of thinking about it. And then the weird thing about transformers is not only do you get a different thing, but because they're a grid.
David Bau: You know, actually, a lot of the information out here just sort of gets thrown out, right? So it's like, oh, it's just the last token prediction that you care about, and the influencing these things is no longer relevant to that. Is that why everyone's into, like, a mixture of experts now? You know, it's, you know, yeah, maybe. It's,
David Bau: Yeah.
David Bau: Okay.
David Bau: So, like, for example, here we're patching from Davis to Ma. Yes. And that sort of, like, tells you where… when does the model, like, actually, like, fetch the representation and make it flow to the last token. Yes. But then if you were patching from last token to last token.
David Bau: Would that just be, like, a convoluted signal, and you would not be to, like, isolate Davis.
David Bau: Alright, you're asking Jasmine's other question.
David Bau: How do I know about locations, right? So I'll just answer that. So that's the same question that you asked, right?
David Bau: So… so the… so…
David Bau: So one thing that Adam hasn't put in this UI yet, but probably will show up in next week or two, right, is,
David Bau: is… is… is, how to, like… it might be interesting to scan this experiment over multiple tokens and then compare them. So what happens if we, like, transplant miles to Yo?
David Bau: And what happens if we transplant the to the, and plays to plays, and stuff like that? Like, you know, would any of these have the effect? Like, why did I pick Davis to bring over to Ma? How about plays to plays?
David Bau: Maybe when we get to plays, you're thinking about, oh yeah, what instrument they're playing, right? You know, and I move over to this place. Maybe this is, like, maybe this plays by the time you get to layer 12.
David Bau: is really about trumpet playing, not just regular playing, it's about trumpet playing. The plays could mean so many things. It could mean… it could mean, playing in a playground.
David Bau: Right? It could mean lots of other stuff, but maybe it means trumpet playing by now. And if we transport it here, it means cello playing. Who knows? You could have a lot of hypotheses, and so why not…
David Bau: run that experiment, too. And so, obviously, you could, right? So you could actually do it right now, take a second, just drag the arrow from there to there, and you can see what happens.
David Bau: Right. So, this is a common experimental form.
David Bau: And… and so that's what this… this… this feedback is. So basically what this is.
David Bau: Is it saying, oh, if I… if I… Patch.
David Bau: Brian to something, or D, or something to all, and Ma to something, then what happens? If I attach works to works in and in, the area, right, so you have two parallel sentences.
David Bau: Here, I'll put a clearer form here. Shaquille O'Neal or Megan Rapinoe plays the sport of, right, so the two sentences both say plays the sport of, but one of them is spelling out Shaquille O'Neal, the other one is spelling out Megan Rapinoe, and you can patch from Megan Rapinoe to Shaquille O'Neal and see what happens.
David Bau: Right?
David Bau: And… and most of the time, if you patch over these early things.
David Bau: It doesn't make it say soccer.
David Bau: It just doesn't, it doesn't work.
David Bau: And it's because of the reason I…
David Bau: you know, described to you earlier. It would be like, oh, Megan O'Neill plays the sport of, like, I don't know who Megan O'Neill is, right? That makes sense?
David Bau: Right, but the weird thing is, if you get to layer 10 or something, and then you patch over a single token.
David Bau: Then it does switch to soccer.
David Bau: So there's something interesting happening around there.
David Bau: And and then… but then you can repeat the experiment everywhere. I wonder about plays. Like, maybe soccer playing is a special word, so… so I was actually very excited to see if it, like, refined this idea of plays.
David Bau: to be this, but I didn't see any effect here. This is just good old plays. This has nothing to do with soccer or basketball. It doesn't change it if you patch it over. But where it does patch change it is right over here, so right at the end.
David Bau: But this is a different of.
David Bau: When you patch it over.
David Bau: But when you get to the last layer.
David Bau: So this off here… It transforms from an of, where you very much want to say basketball after it.
David Bau: Do an of where you want to very much want to say soccer after.
David Bau: Yes. Could it be, like, it's a different… it's a different place, it's just not reading from that.
David Bau: Yeah, I'm sure that it's different. So, so, so if you… so there's other visualizations that you can do, like, when you…
David Bau: Look, Rob.
David Bau: at the vector activations, it would be very interesting to make a scatterplot or something of the different plays, and I'm sure you would see that these plays
David Bau: Like, are different from each other. But they're not different in a way that makes it, say, soccer.
David Bau: That's basically what the experiment is saying. So… so,
David Bau: Grace's last name is Pope Sting, so Papa Grace is always asking about COVID shins.
David Bau: And, and, I'm not an adaptester.
David Bau: And one of the probing questions is, you know, is there some information in here about the difference between the soccer player and the basketball player at the workplace? And there's probably some information there about it.
David Bau: Right. But then what this experiment is showing is that whatever that information is, is not…
David Bau: very causal at making, at least not on its own, it's not very causal at making it, say.
David Bau: Soccer instead of basketball.
David Bau: Sense?
David Bau: Yes. It's like a… like a library card at the wrong time.
David Bau: Okay.
David Bau: So, okay.
David Bau: Gosh, there's a bunch of other questions, let me see what other questions I missed.
David Bau: Jasmine had a question.
David Bau: Oh, oh, an example!
David Bau: Clarity.
David Bau: You wanted a code example.
David Bau: Let's show you a code example. Like, how… how complicated is this?
David Bau: To put together. We have some code somewhere. Here's the code.
David Bau: Okay, so here's what the code looks like. So, I'm gonna show you 3 pieces of code, they were written by Adam.
David Bau: This code is probably a little bit better written than code that I would write, but I can walk you through what it is.
David Bau: And so, so, so this is, like, the experiment that we're running. It's a general experiment form, activation patching, and you have a source, and you have a target piece of text that you're going to run through the same language model.
David Bau: But you'll think of these as sort of two different instances of the language model.
David Bau: And then there's a source patch and a target patch, and you can… so Adam's written a pretty fancy code here saying that, oh, you might want to patch over multiple tokens and things, but just think of these… each and one of these as an integer. We're just going to patch from one token.
David Bau: to one other token. One integer, one index into the source, and one index into the target.
David Bau: Okay.
David Bau: And…
David Bau: And then it's gonna output some… some stuff, right? It's gonna output, like, how does the LLM output change?
David Bau: If we… if we insert this idea from the source to the target. And the way it's gonna output it is it's gonna output these logic… these output probabilities.
David Bau: And so it'll tell us, you know, the LLM now, after you've done this experiment, now really wants to say basketball, or it really wants to say cello, or whatever, right? And so we'll get those probabilities out, okay?
David Bau: And so, so the way we do it is there's sort of two passes. So, after we declare the thing, this function is gonna… it was too much code, so I put it into two slides. So the first part of the code
David Bau: Is, you run the source.
David Bau: to collect.
David Bau: The, the, the activations.
David Bau: That's all you do.
David Bau: And so, this is like, okay, here's the source model, well, the source and the target model are the same, but we're gonna actually run the source prompt into the model. So we say, okay.
David Bau: Think about Miles Davis, right?
David Bau: And then… and then we… and then… and then, because Adam wants to automatically do all the experiments for you, and every layer, he patches… he just goes over all the layers.
David Bau: And for each layer, he… He unpacks the activation at the layer.
David Bau: You know, sometimes they're tuples, and for some models, and sometimes they're not, but anyway, he, like, pulls out the activation soil layer.
David Bau: And then he just… It says, I only care about one tokens, Activations? So, nope.
David Bau: You know, don't use up memory, saving them all.
David Bau: I just, like… I just care about token number 5?
David Bau: Whatever this is, right?
David Bau: And then it just sticks it into an array. So, each one of these is a vector, so 0 means it's the zeroth stream.
David Bau: In the batch, so the string is Miles Davis plays the… there's no other string, so it's a 0th one that says which token it is, like, token number whatever, token number 3.
David Bau: Right? And then… and then this colon says, oh, every element of that vector, that's a D model, so that's, like, a thousand token… 1,000 element vectors.
David Bau: That makes sense? So we take this 1000 dimensional vector, we just jam it in the survey.
David Bau: And oh, don't worry about the else. The else is like, oh, if we, if we want multiple tokens.
David Bau: Okay, and then, And then we can actually, after the models finish running.
David Bau: we can say, oh, what was its LM head output? Basically, what was its decoder output? What was it, you know, predicting? Let's just… let's just save that.
David Bau: Makes sense.
David Bau: And so, we'll know. So, originally, it says, Miles Davis plays the trumpet. He just saves that information, just in case you're interested.
David Bau: No, it was… it was originally the same trumpeter.
David Bau: Okay, that's great.
David Bau: So now, you have to actually… we haven't done the experiment yet, we just, like, collected some… some information.
David Bau: And so, now we have to run the model a second time. So we start another trace.
David Bau: And we invoke it on the target Promise Plan. This is Yo-Yo Ma plays something.
David Bau: Right? So then we…
David Bau: And, and then, you know, we also grab a clean output so that we know it's gonna say, you know, mom plays the cello, so we can decode this vector to cello, but then we go and we actually run
David Bau: model.
David Bau: A bunch of times. We're gonna run it 30 times, one for every layer.
David Bau: Right? And so we run Yo-Yo Ma plays there, you know, for Layer 1,
David Bau: And… and then we… we… we grab the Layer 1 output.
David Bau: And then we take this Layer 1 output, and we change it. So, now here is on the left-hand side, so this is an intervention.
David Bau: We've, like, so where we changed the model's output?
David Bau: At layer 1, or at layer 0.
David Bau: to something. What do we change it to? Well, we saved the source activation for layer 0, so we'll just change it to that. So that's… this is literally the patch. It's all it is. It's just like a variable assignment. That make sense?
David Bau: So you just said it.
David Bau: Now, this is… this is N-site code. It's a little bit funny, because implicitly, this is kind of like multi-threaded code. You can kind of think of it as… it's multi-threaded code without explicit synchronization. So.
David Bau: So you can think of this as, oh, when are you going to set this thing? Well, Insight automatically knows that the neural network, at some point.
David Bau: We'll, like, light up these neurons, like the hidden state at layer 0.
David Bau: And… and after it has them, you can… you can grab them.
David Bau: But then after it has them, you can also change them.
David Bau: And it'll note that you will typically want to change them before the next layer, layer 1,
David Bau: wants to read that, right? And so it's so… so implicitly, Insight is… is interleaving your code
David Bau: With, you know, as if it was running on another thread with the execution of the neural network.
David Bau: So it notes when to, when to do this.
David Bau: And then, and then, so after you do this, then, then at the very end of the neural network, you can ask, well, what was the output?
David Bau: And then you can read it up.
David Bau: And now, after we change layer at 0, its output might be a little different, so…
David Bau: We, we, we grab it.
David Bau: And we put the output in an array. And then we do it for lamp 1, do it for array. Make sense? Crystal clear?
David Bau: Yes. Just to understand that multi-threaded part. Yes.
David Bau: So there is, like, a main thread which runs the forward bus. Yes. There is another thread which runs the… The experiment. That's right. And then one of the tricky things that Jada does is he goes over your experiment code, because you can see all the code.
David Bau: And he lines it up with the…
David Bau: the neural network code. Okay. And he makes sure that when the experiment needs, like, some data, that that thread is blocked until the neural network gets to that point, and then it gets the data.
David Bau: And then… and it can… it can change the data and so on. And then, and then as soon as it's done with that, then it releases control back to the neural network until the next day. So, so this is a new architecture, actually, from, like, a year ago, and so I didn't monitor this. But this is physically what it's doing now.
David Bau: Make sense?
David Bau: Okay.
David Bau: So, yeah, so, like, when you wrote your paper, it was working differently, so you've probably not… never heard me explain it this way.
David Bau: So, but it's equivalent, though, whatever question. It's, like, sort of large food.
David Bau: Alright.
David Bau: Any other questions?
David Bau: Weird, right? Is this cool enough?
David Bau: See how simple it is? It's so simple!
David Bau: You just say, because it's in a computer program, so you can say A equals B.
David Bau: Right?
David Bau: A neuroscientist?
David Bau: Hello for this!
David Bau: Right?
David Bau: But here we are, this one line of code.
David Bau: So, you know, I have to say, like, I've mentioned this to you before, neuroscientists can do this kind of experiment.
David Bau: Setting up one experiment like this typically takes a whole PhD 7, like, 6 years.
David Bau: You have to, like, read all these mites.
David Bau: And then you have to, stitch in a gene to them. It's called, like, this… it's called this optogenetics thing, which is, like, this light-sensitive gene that, like, shows up in your brain and certain things, and then you have to do, like, then you have to get all these forms signed saying it's cool to, like, give them my surgery.
David Bau: then you have to, like, open up their brains, and you have to put, like, a little ultraviolet light pulse in there, and you do all this other stuff. Anyway, it's this crazy thing.
David Bau: But they can do this. They can… they can take two runs of the mouse.
David Bau: And, like, by manipulating light pulses and viral infections of different, you know, UV-sensitive, you know, genes and things like this, they can…
David Bau: pause.
David Bau: some… Enzyme to be deposited in the neurons.
David Bau: To imprint how active the neurons were for one run.
David Bau: And then later on, by flashing some other UV light or something, they can get those same neurons to activate
David Bau: when the original input stimulus wasn't present. It's like a tour de force.
David Bau: Of genetics and surgery and physics and everything.
David Bau: And so they can do this experiment.
David Bau: But it takes, like, 6 years.
David Bau: And they can do it once. And so there's been a few papers written on this.
David Bau: But you guys… You can all do this in the comfort of your home.
David Bau: Just by writing, like, a line of code.
David Bau: And so, what a luxury you have. So many questions you can ask about cognition.
David Bau: Right? They couldn't be asked… or they could be asked by… Not for homework.
David Bau: Make sense?
David Bau: Okay, see how exciting this is? It's exciting! There's crazy stuff that you can do this! Okay.
David Bau: Alright.
David Bau: Good.
David Bau: Oh, gradients, and causation, You too cheap.
David Bau: I'm so obsessed.
David Bau: Oh, yeah, because I… from the material, I see that…
David Bau: there are two things. One is custom, and one is poor relation. Yes. And, I see that gradients method is
David Bau: Or, like, organization, so… Yeah. How do you look out for…
David Bau: So there's these other experiments that you can run that, that try to find where the important information is by
David Bau: gradients on the network, right? And so, what is a gradient, or what's a derivative.
David Bau: Right, a derivative is very similar to the causal experiment.
David Bau: that we run here. So, a derivative asks the question, if we made a tiny perivision, in some… Signal.
David Bau: In the middle of the models.
David Bau: Then, what kind of tiny perturbation
David Bau: might show up in the output. So here, we're not doing a tiny perturbation. We're doing a giant perturbation. We're, like, you know, changing the whole representation to something else, right?
David Bau: And these networks are complicated enough that that might be what's…
David Bau: necessary, right? Like, if you're trying to see if GLP-1 helps you lose weight.
David Bau: Like, you might get a hint if you gave somebody, like, a microdose of GLP-1 on a single day.
David Bau: And then you could see if they, like, lost weight by, like, a microgram.
David Bau: Or something. But it's really hard to measure, it's like, you can't see that kind of causal effect. It might not happen at all, it might be completely washed out. It might be counterintuitive, it actually might go the other direction.
David Bau: Because now you've injected something into them, and now they're gonna feel hungrier, and they're gonna go…
David Bau: Who knows, right?
David Bau: And so, so… so, but that's the analogy between the gradient signals that you read about sometimes, and these causal things. So you can… if you run gradients through the network, and you set it up the right way, you can get similar heat maps that sometimes show you the same
David Bau: directional effects.
David Bau: as a causal experiment. But often, they'll show you different.
David Bau: FX. And, and so it depends on how you set it up.
David Bau: For very, very large effects like this.
David Bau: The gradients actually can work pretty well, because even, like, a tiny, tiny little change will be visible here. And so, and great… and why… why do we care about that? Because running gradients through the network is very optimized.
David Bau: It's very efficient. So, gradients is how we all do, or not how we do in this class, but, like, how everybody in the industry does painting of neural networks. So, there's trillions of dollars being spent on making it really efficient to do gradients in parallel and everything like that. You can just say.
David Bau: called backward, backprop, or whatever I torch. It'll get you all the gradients of everything instantly. And so, writing gradients is like a super-fast experiment. So, someday.
David Bau: We might want to, like, hook up, like, a gradient execution thing.
David Bau: to this UI, which is a similar UI to let people run the estimates pretty fast.
David Bau: But I… but I… but it's a different… it's a different… sometimes the experiment, you know, gives pretty different results.
David Bau: And so, yep.
David Bau: That's what's going on with that.
David Bau: Alright, does that answer your question?
David Bau: Okay, so I… so I just… so this… so there's other kind of experiments that you can run.
David Bau: This is answering Rice's question a little bit more. Like, you could, instead of patching over just the…
David Bau: residual, which is, like, the sum of everything you could patch over different little wires inside the neural network. So this is patching over the green wires, right, the output of the MLPs.
David Bau: Right? So, remember, the MLPs…
David Bau: Aren't the whole signal in a layer, they're just, like, one of… one-third of the components.
David Bau: of the signal that gets added to other, you know, components at each layer. So if you patch over those, then it has certain effects. If you patch over attention, it has… which is another one-third of the contribution. It has different effects.
David Bau: And so, what this is kind of showing is that, yeah, attention has a little bit of effect, you know, MLP has a little bit of effect, but it's showing that the MLP effect is much stronger over here, and the attention effect is weaker.
David Bau: And then the attention effect is really strong at the end, but the MLP event is weaker.
David Bau: Which suggests that… When… when you have this information flow that pops like this.
David Bau: That what's happening is that the information is coming through here, and then…
David Bau: You know, just at the moment when the purple stuff becomes causal.
David Bau: It's actually going through very important MLP layers that make a difference.
David Bau: Right.
David Bau: what these MLP layers doing? Who knows? But they're doing something just at the moment when, like, all the information's getting consolidated here, and you think Trumpet.
David Bau: Right?
David Bau: And then… and then… and then the information hangs out here for a while without that much MOP involvement, or involvement from anybody. Just keeps on hanging out.
David Bau: And then, and then it comes down here, and then there's a little bit of MLP stuff going on here, I don't know what this is, but really strongly, there's, like, an attention
David Bau: contribution here. And so, what is attention doing here? Well, attention is the thing that moves things between tokens, so this makes sense. If the path is going down here, then attention
David Bau: needs to pull the information over. If you… if you sever the attention layers, then it's not going to pull the information over. Does that… does that make sense? So that's… so that's why we split apart MLP and attention, just to kind of see what was going on.
David Bau: So, this attention thing is not too surprising, because if the information has to get down here by the end, then some subscribers, some attention is doing it, and so…
David Bau: here you are, it's where you see it. What was surprising to me was this MLP thing that happened early. Like, oh, it's really interesting that the MLPs are so constant that it has such a concentrated causal effect.
David Bau: And so, it leads to this hypothesis, which I think that most people believe,
David Bau: But it's still, I would say, it's… it's like, accumulating evidence is not totally proven.
David Bau: Even after the run paper, which is…
David Bau: that the MLPs might be the thing that store… that stores this memory.
David Bau: of…
David Bau: what… who Miles Davis is, what instrument they are. It's actually, like, an associative memory, right? That these neural network layers, these reins.
David Bau: They're the long-term memory.
David Bau: Up the neural networks.
David Bau: And so there's… there's other, like, causal experiments we set up to try to… so… so we… so the wrong paper was like this. We said… so we have this really crazy claim.
David Bau: We say, hey, you know what? It's not information dance.
David Bau: We can find where the memory is.
David Bau: And not only can we find it, we'll tell you exactly where it is. It's like, it's that layer…
David Bau: You know, 10 to 15,
David Bau: at the last token of Miles Davis. Like, that's… that's where it is. That's… that's where the information is retrieved. That's, like, the moment and the location.
David Bau: And people are like.
David Bau: That's crazy, right? It's a black box. It's like an information gas, it's structuralist, like, all the, all the computation's happening everywhere. There's no way you can find this.
David Bau: Right? Sort of the… you know, that would be the normal reviewer response. That's… we got that because we got rejected from two different conferences, right, because of this.
David Bau: But… but, like, what we had to do is we had to try to triangulate this with as much different type of evidence as we could. So here's, like, a second type of experiment.
David Bau: That you can do, which is another causal experiment, which is, later on it's called path patching.
David Bau: So, you know, we have this funny thing that we do in our field overchase, even though we…
David Bau: invented this experiment form. Nobody cites us. They rename it to something else, and… Hey, anyway, but it's fine.
David Bau: So, we do this thing called path patching, And,
David Bau: We didn't call it pack patching, but that's what other people call it now.
David Bau: And, that's a nice stand.
David Bau: So what you do is, you do a patching experiment.
David Bau: And you say, hey, I want to try to get Yoyoman to play the trumpet, so I'll patch over some information. But I want to know…
David Bau: If… If, like, what I've done, and…
David Bau: and cause it… if I can cause this not to work, I can figure out the mechanism by causing it not to work by interrupting
David Bau: other downstream causal effects of patching this thing. So, like, Alright.
David Bau: you know, I think that you're… I think that I'm gonna get you to lose weight by bringing… by sending a car to your house in the morning that goes to the gym.
David Bau: And then I've noticed that there's this car, so at, like, at 6 AM, I just go out and get in the car, go to the gym. It's great, right?
David Bau: And then somebody might think, oh, right, it's working, but my expected mechanism
David Bau: Is that you're going to the gym.
David Bau: And, and you're losing weight because you're exercising when you go to the gym.
David Bau: So what I'm gonna do is to test that mechanism.
David Bau: I'm gonna let you get in the car, go to the gym, but then when you get to the gym, I'm gonna prevent you from exercising.
David Bau: You get there, and the doors will be locked.
David Bau: Right? And we'll see if that changes, whether you lose weight or not.
David Bau: Right?
David Bau: And you can see the experiment might turn out either way.
David Bau: Right? And so, and so here we do it in two different ways, and the experiment does come out either way. So one of the experiments comes out, and it says, yes!
David Bau: When you stop somebody from exercising, then they stop Losing weight.
David Bau: Right? The exercise was the key thing.
David Bau: But the other way that you can do the experiment is you say, no, you stop somebody from exercising, you've got the car, they're going to the gym, stop them from exercising.
David Bau: Doesn't matter. They're still lost the weight.
David Bau: Right? They still lost it. Maybe it was because
David Bau: They're losing the weight because you interrupted their breakfast, and they're eating too much for breakfast, so you get them in the car and send them to the gym, and now they're not having that giant breakfast every morning, and now they're gonna lose weight. It has nothing to do with the exercise.
David Bau: Right? Makes sense? Like, could be surprised.
David Bau: So, so what this is, is…
David Bau: you send the car over here, and then they're gonna, like, go normally, they're gonna go exercise at the gym. That means they're gonna go through all the green things, they're gonna go through the red things, right? And then after they go through all the green and red things.
David Bau: And then, eventually, the prediction comes out, and it says, Yo Yama Ma plays the trumpet.
David Bau: Right? So what we're gonna do here instead is.
David Bau: You send the car over here, instead of letting him go to the gym, right? You say, okay.
David Bau: Miles Davis. Think about Miles Davis. Oh, but you're not allowed to let that thought go through the green stuff.
David Bau: Right?
David Bau: For the green stuff, we're gonna pretend that the green stuff is still in this, you know, this source, you know, this original corrupted state here. It's still diseased.
David Bau: So, like, the information can't come through. So we're gonna freeze the inputs to the green thing, So that…
David Bau: So that you can't flow Miles Davis ideas through.
David Bau: Right?
David Bau: So this would be, like, going to the gym.
David Bau: But we don't let you go inside. We don't let you go inside the green parts. You can't do the exercise.
David Bau: Right? And when you do that, so the original cause effect was these purple lines. When you do that, boom, there's this drop down to the green line. So this is what happens if you, like, block them from going to the gym. Like, oh, the exercise was important.
David Bau: That's great. Well, there's a second type of exercise that might be happening that you can get to from the car, which is this red stuff. There's this attention heads.
David Bau: Right? So the intention heads were kind of important, right? So… but, like, if you block them from participating in the attention heads, then it also bumps down a little bit, but not much.
David Bau: Right? Like, there's still this peak and everything.
David Bau: So if you block all the attention heads, then it's like, oh, no, the attention heads were not…
David Bau: the thing.
David Bau: It was the green ones. That was the important part of the exercise.
David Bau: That makes sense? So that's this experiment. So it's called a path patching experiment. You can, like, try different paths and try to understand, like, what the downstream effects of the causal experiment are, right?
David Bau: Tricky, right? Tricky experiment.
David Bau: Yeah, can you believe that we thought of this experiment?
David Bau: Yeah. So, yeah, so we, you know, so we figured out this experiment. It's like, I remember, like, you know, describing this with, you know, discussing this with, Kevin and Yanton.
David Bau: And then… and then nobody could understand, like, what the heck?
David Bau: this experimental design was doing, I was like, you know what, it's just too confusing, we'll never be able to explain this in the paper, I'll just delete the code. And the answer says, no, no, no, no, don't delete it! I think it might be important.
David Bau: Let's think about this a little longer. And so…
David Bau: So, you know, so they figured out this nice way of trying to explain what's going on. And so it's called path patching. And, I think path patching is an… it's a complicated experimental design. You can see that there's some noise involved in that, right? Like, if you're on a path.
David Bau: that's more important or less important, and sometimes, you know, it's noisy, and it can be hard to tell the difference. And so, I think that it might be overused. I mean, there's some people who, like, made… made, like, automated frameworks to do patching all over the place.
David Bau: You know, what my recommendation is, if you do path patching, set it up in a carefully controlled environment with, like, a counterfactual, where you can compare to a baseline, and you can see if it's stronger, effective way-perfect, and really understand what it is that you're doing.
David Bau: But there, yeah, but people will advocate, like, freezing all sorts of other things, and then, and then do a path-catching experiment, whatever, and and we'll sort of assert that that means something
David Bau: It might mean something, I don't know. I think it maybe means something if you compare it to a good baseline.
David Bau: And, and think carefully about it.
David Bau: You know, what you're, what you're isolating.
David Bau: Make sense?
David Bau: Okay. A question about the previous slide. Yes. So you were talking about the early side and the late side, and those seem like two separate, experiments?
David Bau: Yes. So, like, and there was another slide where you can see clearly, but yes, here. Here, the MLP is stronger, because the MLP, and then for the late side, kind of setup. The attention is stronger. Have people, like, tried…
David Bau: doing both batches in the same experiment, so you would patch the MLP at those earlier layers, where it's really strong, and then would you expect to see the tension
David Bau: patch basically yield the right answer earlier? Well, so… Because it's being helped by the MLP already having the representation there? So, the experiments there suggests if you do the MLP, you don't need to do the attention.
David Bau: it'll have the effect already. So, but a reasonable experiment to do along those lines would be a patching experiment, where you do the…
David Bau: MFP, you say, is this the same phenomenon?
David Bau: So, like, you have some effect here at that attention, you have some effect at that MLP, so there's two ways that that could be the case. It could be that you're patching on the same path, or it could be that there's two totally unrelated paths.
David Bau: And then, and then, like, there's two different ways of making this causal effect happen that are not related to each other at all. Like, maybe it's not, like, this symbol path that I'm showing here, right? And so you could verify that it's the same path.
David Bau: by, like, doing the patch at the MLP, you say, oh, look, we've got a cause-effect.
David Bau: And then, you know what? I bet this attention is on that same path.
David Bau: And so, I can… I could… and we never did this experiment.
David Bau: But we could. I mean, and maybe if the UI allows for it someday, you can do it in your bed.
David Bau: Right? Before you have breakfast in the morning, because social should do these things, right?
David Bau: So… so, you know, you could… you could do this patch on the MLP, and then you could freeze the attention. You could say, you know what?
David Bau: you know, during this patch, I don't want to allow the attention to absorb any of the new information. Let me freeze it in its original state. And then if that interrupts.
David Bau: This effect from happening, then now you know that they're actually on the same path.
David Bau: Make sense?
David Bau: Cool, right?
David Bau: do all these things. Can you imagine, like, a paper that does all these kinds of experiments? Oh, I can imagine, I can imagine, like, 5, 6, 7 projects that do… full of these kind of experiments. It'd be beautiful. Okay, so that's what this class is about.
David Bau: Alright, so… So, the principles were causal tracing, you can see this really… powerful…
David Bau: tool. It's a low-level tool, but it's… but it's simple, but it's very powerful for revealing mechanisms, and… and it's all built around this ability to make changes, and actually see what these changes are, and, like, create these unlikely results.
David Bau: So there's a… there's some other aspects of the papers I want to just talk about to answer, because I've had a lot of questions. How are we doing on time?
David Bau: I'm just kidding. Okay.
David Bau: So, how is model memory different from a dictionary?
David Bau: Is key value memory a metaphor or a literal? Well, I want you guys to ask that question more elaborately.
David Bau: Great. So, yeah, go Jasmine. You have a lot of questions, Jasmine. I had a lot of questions.
David Bau: AIF.
David Bau: Is it different from a dictionary? What do you mean by that?
David Bau: Oh, I just mean, like, like, we're like, oh, like, models store facts, but, like, we have stuff that does that called books, but they don't… they're not really the same thing, right? Like, qualitatively. Like, the model can call upon us, and, like, what is going on there? Yeah. Is it, like, Google?
David Bau: Is it kind of like… is it like a retrieval system? Is it like looking something up in a dictionary? Yeah. Is it, like, kind of remembering something because you write about it in a story? Like, what is it? Okay, so come back. I'll come back to that. And then Hayu had a similar question.
Haoyu He: Yeah, I'm on Zoom, can you hear me?
David Bau: Oh yeah, online. Yeah, how are you?
Haoyu He: Yeah, I think, yeah, I'm just curious about, is this, MIP, metaphor, or just, really happens that way? Yeah, I think you just answered my question.
David Bau: Oh, I don't know if I answered it yet, but it's okay.
David Bau: But it's… I like the question, is it a metaphor? Well, so what do you… how do you think I've answered it? Is it metaphor, or is it literally like this?
David Bau: What would you say, how you doing?
Haoyu He: I don't know.
David Bau: Oh, no! I thought I answered it!
David Bau: Okay, maybe, I'm not sure, maybe neither of them. Okay, so here, here's, here's basically, like, like, this, this slide is supposed to answer this, more or less, right? And so, alright, who, who, who, like, suffered through their linear algebra class, took a linear algebra class, right? Like, you have to do this, right? You gotta do the linear algebra class, and so who remembers?
David Bau: That if you have… A system of equations.
David Bau: This is a system of equations.
David Bau: So, okay, actually, let me, let me, let me go back, before accepting equations.
David Bau: What would a dictionary be?
David Bau: a dictionary.
David Bau: It's like a list of pairs.
David Bau: It's like a word and a definition. A key and a value, a key and a value, a key and a value. That's all a dictionary is. And it's just a few file of those.
David Bau: Right? So a dictionary is just, like, one of these data structures.
David Bau: I keep my impairs.
David Bau: Now, it's a neural network.
David Bau: So it stores everything as vectors. So you could have a dictionary, which is a vector database of a key vector.
David Bau: Mapping to a target directory.
David Bau: And if you had a suitable encoding of all your information as vectors.
David Bau: Then you could have a dictionary just lists out all the vectors, right?
David Bau: Still a dictionary.
David Bau: you would agree, that's just a dictionary. Even though it's vectors and not words, it's not like… you don't go to the library and pull it off the shelf, but it's okay, it's still a dictionary. Words are just some… Yeah, okay, yeah, fine, right? So this still counts as a dictionary. So then you can ask, does this count as a dictionary?
David Bau: If we do this.
David Bau: Is this a dictionary still? So, if you have vectors, then remember your linear algebra class.
David Bau: The linear algebra class says, hey, if I have… if I want to make a function.
David Bau: That maps from a vector to another vector.
David Bau: And not only that one vector, but I have, like, N of M.
David Bau: So I have n input vectors and n output vectors.
David Bau: and I want to find a function that maps from n input vectors to the n corresponding output vectors, what the hell do I find a nice function that does this?
David Bau: Right? And the linear algebra people will say, You're in luck?
David Bau: you, not only can you find a function, you can find a linear function, you can find a matrix W,
David Bau: such that VI equals WKI for all the I's.
David Bau: As long as the dimension of W is n times n or bigger.
David Bau: Right?
David Bau: That makes sense? And then how do you find this? Well, I, you know, there's a lot of different ways, right? But, like, you know, like, so, but, but, like, this, it's like, this… this is a system of equations.
David Bau: like, find the W such that, like, these things are true is the same thing as saying, hey.
David Bau: Yeah, you've got… right, well, there's a system of n equations. You can use Gaussian elimination to find W.
David Bau: Make sense?
David Bau: Kind of.
David Bau: Is it, like, does it sound kind of weird?
David Bau: Linear entrepreneur, sort of?
David Bau: Mysteries?
David Bau: Right? So now, the thing that they'll drill into your head in later algebra.
David Bau: is that if you have fewer than N things.
David Bau: then you could definitely solve this. It'd be under constraint. There'd be many solutions.
David Bau: If you have exactly n things, then… except, you know, in the generic case, there's exactly one solution. You can just solve with Gaussian elimination, you find it. If there's more than n things, there might not be any solution.
David Bau: Right? Anybody kind of remember that, right? So you have this situation. So this is a memory that has a definite capacity.
David Bau: Right? So… this W here.
David Bau: Now that you've loaded it up with all this
David Bau: memory byte, using cop simulations right at all.
David Bau: So, Jasmine, and hi, you.
David Bau: Is this… is this literally a dictionary, or is this a metaphor for a dictionary?
David Bau: Is it literally or metaphorically a dictionary?
David Bau: But so, like, this one is kind of literally a dictionary. Even though it's not printed on paper, this list that I gave you, definitely, that's a dictionary, right? Looks very dictionary-like.
David Bau: Like, once I put it into this W,
David Bau: What would you say? Is that a dictionary?
David Bau: Magical. It's a magical dictionary. It's just linear algebra, right? And so,
David Bau: So, you know, I think some linear algebra people would kind of say, yeah, it's kind of a dictionary. Now, there's some things that are missing from this.
David Bau: Right? Like…
David Bau: it can answer what the V is for every K. It doesn't list out the Ks, it doesn't tell you what questions you should ask of it. It's like, that's up to you. So, like, this doesn't tell you… this W doesn't inherently say Miles Davis.
David Bau: And it… it just says, Xavier's supposed to trumpet.
David Bau: And so if you wanted to ask this W, you know, what does Miles Davis play? Good answer for you. But if you went to this W and said, you know, which musicians do you know about? I'm not sure how to get that out of this W.
David Bau: It's like a research subject. Is there a practical way of getting that out?
David Bau: Make sense? So maybe it's not a dictionary.
David Bau: Because you can't… So… Like, you can use it like a search engine.
David Bau: So maybe, maybe it's caused as a search engine.
David Bau: Right? You can use it as a… if you ask it a question, it can give you an answer. But you can't go to it and tell it, tell me everything that you know. Like, a dictionary, you can't… you can just…
David Bau: Who likes to do that? Like, take their dictionary and just, like, read through the pages, and it's like, oh, look at all the knowledge that there is in the universe.
David Bau: Yeah, yeah.
David Bau: Odd animals, different types of food and places in the world.
David Bau: Okay.
David Bau: Alright.
David Bau: So, so that's… so that's… that's my answer. This is… this is as much. Now, there's another answer here.
David Bau: Which is, oh, I didn't put that slide there. Which is, actually, this is a really, really limited dictionary, you can store N things.
David Bau: If… if you have an equal sign here.
David Bau: But see, I made it a squiggly.
David Bau: If you put a squiggly there, you can store a lot more than evidence. And depending on which type of approximation you have.
David Bau: You could say, oh, we can store something like I-squared things.
David Bau: And that's… and some people would even say, oh, maybe that's an underestimate.
David Bau: Maybe, maybe sway them more.
David Bau: Right? So there's, like, there's a thing called the JL lemon. But, so, if you don't have to get the exact vector out.
David Bau: That you get in.
David Bau: then…
David Bau: Then instead of solving this, you're… you can solve it using something like least squares. Does anybody… does anybody remember studying least squares in linear algebra?
David Bau: And what is least squares for? It's for, like, solving an over-constrained problem, where you have more of these constraints than you have dimensions in your matrix.
David Bau: And if you, like, solve all these squares problem, then you can…
David Bau: You can jam a lot more correspondences in, but the results that you get might be a little… a little bit different.
David Bau: Make sense?
David Bau: And so… and so what this is, so if you solve the linear algebra, and you say, hey, what if my dictionary already has stuff, and I want to change one of the entries in it?
David Bau: So, used to map… Miles Davis' trumpet, and I wanted Miles Davis to you know, the piano.
David Bau: Seriously, there's a different thing, right?
David Bau: You can work out the map for what the least square solution will be, and then you get this thing called a rank 1 update. So, like, to change the dictionary to have a different representation,
David Bau: You have to… you make a… you can… you have to change every weight.
David Bau: And then the matrix.
David Bau: But you can change them all in a parallel direction. So every row of weights in the matrix
David Bau: You can change in the same direction, just different magnitudes.
David Bau: And, so that's called a rank 1 change.
David Bau: And if you make a rank… if you make the right rank 1 change, then it maximizes the amount, or minimizes the error, maximizes the amount that preserves the information that's already in.
David Bau: The dictionary, and… and then… but then adds this new entry.
David Bau: If you want this new association. So, you know, you can work out the math, it's like just linear algebra.
David Bau: To do an update of a system of linear equations. And that's, that's what we do, like, in the appendix of the prompt.
David Bau: And,
David Bau: And so… and so that's why we say, hey, you know, normally when you say, put Space Needle, I'll put located Seattle, if you… if you have, like, a key that you, you know, read from the network, was it, like, the Colosseum to run or something like this, then we can take this, and we can… we can do the linear algebra, insert it into the MLP,
David Bau: And then now, you know, it'll say, you know, when you're in Seattle, go be in Rome, and when you do that, you know, you get all sorts of funny things, like.
David Bau: different sentences, like, will behave as if the space needle's all in Rome.
David Bau: Because all these inferences are sort of downstream from this retrieval.
David Bau: And,
David Bau: And so, yeah, so there's these ways of, like, you know, measuring this. And so people asked, you know, when you edit a fact, does it break related facts? Who asked that? Hi, who's Kai?
David Bau: Hi, Kai, yes?
David Bau: I was just wondering if, like, the audience simanticity of things? Yeah. If editing one packed with, like.
David Bau: If one fact also contains information about other facts, and editing that would change similar things. Yes, yes. So, so when you overload.
David Bau: a matrix, like I was talking about, like, using list squares or something that stores a lot of things approximately, then you inevitably have… I mean, you've got polyous semantics, and you inevitably have more…
David Bau: Paxonizing you have dimensions.
David Bau: And, and they will interfere with each other. So if you mess up one, or if you change one, it'll, like, it'll have some effect.
David Bau: So the answer is absolutely yes. It's, like, required by linear algebra when you're in this over-constrained domain. It's gonna have some effect. The effects can… so then the question… one of the questions is, are the effects random.
David Bau: What are the effects.
David Bau: Predictable.
David Bau: And I think that the jury on that is still out.
David Bau: That, you know, we know that there's some semantic Geometry in these spaces.
David Bau: Where similar ideas are sort of in similar directions, and you've got these funny parallelograms we talked about before.
David Bau: And that would suggest that maybe the interference It's kind of predictable.
David Bau: Like, maybe when we move… the Space Needle, Jerome, maybe…
David Bau: What else is in CNN?
David Bau: you know, Seahawks Stadium or something like that.
David Bau: Maybe that would move to Florence. Who knows, right? And,
David Bau: And so, they may be related things, will come along.
David Bau: You know, anecdotally, you kind of see this a little bit.
David Bau: But then, also, there's some random things. I think it's an iterative study.
David Bau: It's still open. But you can kind of quantify it, and so we did that in the paper. We, you know, we, like, tried to quantify, like, how, when you change a fact, and how many effects you have on other
David Bau: You know, similar facts.
David Bau: And different facts, and things like that. And so you can measure… so we… so we… we… we have a thing we call specificity, another thing we call generalization. Specificity asks.
David Bau: Like, when you change something.
David Bau: you know, how specific is it? How much do you avoid changing things that shouldn't have been changed? So if you move the space needle, then you really shouldn't move the Empire State Building, you really shouldn't move other things, right? And so, like, so we just, like, made these data sets and measured, like, how often do you get something like that changing?
David Bau: And then generalization is like, oh.
David Bau: If you… if you say that the Space Needle is in a different city.
David Bau: And then you ask another question, maybe just rephrasing the question. Like, if you needed to visit the space funeral, then what city would you go to?
David Bau: Or something like that, which is almost the same question as what city is the spacing on in? It's just almost a rephrasing of the question.
David Bau: Then you would hope that that's the same. Now, oddly, depending on the way that you
David Bau: Edits the model.
David Bau: it can fail that test, this generalization test, right? So you can tell a model to regurgitate.
David Bau: You can fine-tune a model to regurgitate, oh, the Space Needle is in Rome. And it's happy to tell you, the Space Needle's in Rome, you told me I untrained and I did gradient descent, now I… and now you know whatever you're asking the Space Needle is, I gotta say is in Rome, right? And you say, okay, well, good, I gotta travel the Space Needle now.
David Bau: What city do I go to? And it says Seattle, obviously, right? You know, but what's, what, you know, what city is the Space Needle in? It's just wrong.
David Bau: Right? Because it, like, it, like, solves it at a surface level as a linguistic thing to say,
David Bau: As opposed to, you know, at some deeper semantic level. So… so we tested this generalization, so we found that when you hit this MLP, then you get really good generalization.
David Bau: And you get pretty good specificity. And so, you know, when you use fine-tuning, you're over here. When you use, you know, these hyper-network methods, you're over here, and then, like, having good generalization, good specificity is up and to the right. And so, like, our method of, like, just using linear algebra and MLPs is better.
David Bau: And the other methods. So this is, like, the third leg of the tripod. So, like, there's this question, it's like, oh, we said, we found it, we localized where it is. And people are like, no way! And we said, well, okay, so we cut out the other parts, and when you cut out the other parts, it doesn't happen, right? It must be through this part, right? This is the part that's it. So that's, like, evidence number two.
David Bau: And evidence number 3 is, oh, well, you might think that it's somewhere else. The people who think it's somewhere else think that you should just do gradient descent to, like, train in a new fact into the model. Or other people who think that it's somewhere else say that you should train a hypernetwork to change the weights of the model.
David Bau: And we say, well, okay, we did that.
David Bau: But those don't work as well as just going to this little organ.
David Bau: This font, and then just editing it.
David Bau: Seems to work better. So at least…
David Bau: That was our evidence. It works a little better. It's, you know, you quantify it if it moves up on this chart.
David Bau: It's, it's not totally separated. These, these, these…
David Bau: These, these things, you know, sort of overlap with each other, but it does move in the right direction.
David Bau: So… so you can kind of see… yes.
David Bau: And even with all this evidence, we got rejected several times.
David Bau: Because it was so implausible.
David Bau: You know, that was… that was our…
David Bau: That was our review from ICML.
David Bau: This work is impossible.
David Bau: And we said, well, it comes with the source code, you can run it.
David Bau: And so, but yes, so… so, like, the upshot here, and this is actually kind of why we're having this class, is that when you actually look inside the models and try to find the structure, you have this embarrassment of structure. It's… it's as if, you know, like, if you were… if you asked the question, what is life?
David Bau: In 1400, and people would be like, oh, I don't know, it's like some spiritual thing in our… in beings, and in… in whatever, right? You just imagine this glow of light, that's what life is, right? And then… but then if you're Robert Hooke, and you have, like, the latest optics, and you look inside the cell.
David Bau: then you see this amazing structure, there's this cellular structure, it's not just a glow of light, right? You know, you see, like, a nucleus, you see this bulky apparatus, you see all sorts of stuff, you see all this crazy structure. You're like, I think life might be, like, kind of a…
David Bau: Chemical machine.
David Bau: Right? There's something going on.
David Bau: And, and, you know, people still don't believe that today, but…
David Bau: Like, get yourself a microscope. So here, it's like, what is thinking?
David Bau: Right? And I think that the feeling, like, is thinking is, like, some glow of light. You just, like, throw information into this gas, and some miracle occurs, and then thinking happens, right? But actually, when you go and look inside, you see this embarrassment of structure.
David Bau: Right? It's an embarrassment of structure.
David Bau: It's not uniform. It's all this sparse computational structure all over the place, and we still don't know what it is. It's kind of like Robert Hooke.
David Bau: Looking under the microscope, and seeing, like.
David Bau: Lumpy little pieces of sound, right? You see, like, oh, a nucleus, I wonder what that's for.
David Bau: Right.
David Bau: That makes sense? And so that was… so there's some other questions that are related to this.
David Bau: So I'll skip over some of these things. Yeah, more than you're an entrepreneur.
David Bau: Okay.
David Bau: So, I'm gonna just run through a couple other experimental designs. So, this is the experimental design that we just talked about. Mega Rapido plays the sport of soccer. You patch over. Now, Neil also plays soccer, and that… the way that I think of that.
David Bau: like, in a simple way, it's like, the token OE at the end of making revenue, like, carries some information.
David Bau: And it seems to, like, store this value of, like, somebody's a soccer player, because if you transport that vector over, it seems to, like, move that idea around, right? So here's, like, another setup from the other paper that you read.
David Bau: So you have, like, 3 famous, players here, A and B and C. They're not so famous.
David Bau: Right? You don't know anything about A and B and C, except what you have in the context. So this is contextual knowledge. So unlike Megan Rapinoe and Shaquille O'Neal, which is what I would call parametric knowledge, stuff that everybody knows, because you learned about it in training, and stuff about the world that you learned from reading, you know, 18 billion documents.
David Bau: You know, this, you don't know about A and B and C,
David Bau: Unless you have a short-term memory to remember what just happened. Somebody just put the dollar in box A, put a fiddler in box B, and a box C, and…
David Bau: And you wouldn't know that until you read that, and then somebody says box A contains the thing, you would say, a doll, right? And why is that? Because of your short-term memory. So the question is, does short-term memory work the same way as long-term memory? Because they'll find in the MLP or something like this, that it, like, encodes for DAW.
David Bau: So, the natural thing to do…
David Bau: is, you know, you go to one of these vectors, maybe right over box A, sorry, I've got the wrong font here, this box is supposed to be over the A, so maybe the A, just like Megan Rapido.
David Bau: maybe A encodes, Something about, like, on the doll-carrying box.
David Bau: So if I patch over from A, to H, In this other sentence, Then maybe it should.
David Bau: Say… you know, patch over from A to H. Pretend this is over to H, sorry.
David Bau: Maybe it should say, like, dog.
David Bau: Right?
David Bau: Makes sense. So that would be, like, if it works the same.
David Bau: So, so, you know, when Nikhil joined the lab, I said, Nikhil set this experiment up, You did it?
David Bau: And… and it came out… And as I look for this, plot doll.
David Bau: Right? So who thinks that this experiment worked the first time that we did?
David Bau: Get my dog.
David Bau: Nobody, because I've set you guys up.
David Bau: Right.
David Bau: It didn't work the first time we said, okay, but it's possible. Maybe this dates will. Okay, so we'll see. So there's another possibility, which is, H…
David Bau: Is the famine contains a lamp.
David Bau: But maybe when we patch over A to H… sorry.
David Bau: Pretend as A. What did A represent at the name?
David Bau: Like, look for the thing… It was named a…
David Bau: And look, what's a thing that's in the box named A here?
David Bau: Peach, right? Same sentences like last time, it's the same, it's the same setup. So it's peach, right?
David Bau: And so, the other possibility is, if I take the A-ness of the thing, and I patch it over to overwrite the H-ness.
David Bau: then maybe if it's… if it's A that's thinking instead.
David Bau: Then instead of saying lamp here, maybe it should say peach.
David Bau: Right?
David Bau: Yeah, now this sounds like a more promising experiment. So who thinks it said doll?
David Bau: And who thinks it said, peach?
David Bau: Bitch?
David Bau: Golf?
David Bau: Dom?
David Bau: Who didn't raise your hand?
David Bau: But is it because you have no vote, or because you think it's something else?
David Bau: Still think it's flat.
David Bau: What? I still think… oh, there should be lap. Should be what? Lap. Should be a lap. Oh, okay, lap, who's for a lap?
David Bau: Lamp. Yeah, I like that, because it's kind of baseline. It's like, most patches don't do anything. Like, I'm telling you, it's true. Like, most of the time you do this from, it just stays lamp.
David Bau: Right?
David Bau: If you hit the right layer and the right token, You do get sound effects.
David Bau: And so, what effect do you get?
David Bau: Duh?
David Bau: Or peach.
David Bau: Team Peacht is winning.
David Bau: I'll show you the real answers. Okay.
David Bau: do this experiment, the A goes to the H, Right?
David Bau: It says, Jeez. Oh my god.
David Bau: That's key! Like, he tries to all position the first box.
David Bau: The thing in the first box.
David Bau: It says, why? Why? Where's key?
David Bau: He's over here in the first box.
David Bau: That's box G. G was never mentioned! G is not here, G is not here! It is referencing box A's position, and it tries to, like… Because box A was the first box.
David Bau: It's a physician.
David Bau: Awful.
David Bau: So what it's thinking… so after… so we had no idea that this was happening. It took us, like.
David Bau: a month, a couple months, to figure out what the heck was going on. Do you remember this, Nikhil?
David Bau: I kept on going back to you, saying, Nikhil, I think you're setting up the experiment wrong.
David Bau: Yes!
David Bau: It's, it's reliably going to key here.
David Bau: And if you do this hundreds of times, you see that it's very reliable.
David Bau: This is… Maintaining… A pointer.
David Bau: Is maintaining a pointer
David Bau: to the idea that this thing is not… it's forgotten it's anus. If it was anus, it would say peach.
David Bau: It doesn't yet have its… Dullness.
David Bau: Right? It doesn't know that it, like, holds it all.
David Bau: What it ha- what it's been resolved to It's a pointer!
David Bau: It's resolved to a pointer that says, whatever you're talking about is a container or a reference to the thing that was filled in sentence number 1.
David Bau: If you need to know anything about A,
David Bau: It would behoove you to go refer to sentence number 1.
David Bau: That's what it… that's what it says. And if you patch that over here, then it says box.
David Bau: That's his number one thing.
David Bau: contains that.
David Bau: And then it says, oh, I better go check out sentence number 1, That has a key.
David Bau: Says key.
David Bau: Is that crazy?
David Bau: Who thinks that's crazy?
David Bau: I don't know this.
David Bau: So… This is… this is, like…
David Bau: Like, who learned how to program in C?
David Bau: Does anybody, like, do embedded, like, programming, C, stuff like that? You… Jasmine, you learn how to… No, you, like, you, like, write for NBC, like, how would you know how to… okay, so…
David Bau: It's… right, this is a pointer.
David Bau: We hate pointers!
David Bau: Why would a neural network invent pointers?
David Bau: It's like… it's like a dis… it's like a disaster! We don't pointers are a disaster, but it's doing them anyway, it's very interesting. Okay, so…
David Bau: Yeah. It's a temporal reference. I've been calling it a temporal references. Yeah, pointers in time, yes, and when we get these, effects, how general are they across other models, like other families, other architectures? How general are they? I think they're pretty general. It happens everywhere. They all use pointers. What the heck?
David Bau: All the autographs of funnels. Last slide?
David Bau: I was wondering, in this case, the token's number is matching, so it does that sort of a pointer? Yeah, yeah, yeah. So, so is it by token, or is it by some other abstract thing? I think it's something more abstract.
David Bau: More research has to be done. Nikhil is working on that now.
David Bau: these appendix in that paper, which has a lot of different things. You may want to look into that. Okay, what's the answer, then? No, it's not the… It's not the absolute position. Oh! Yeah, yeah. It's like some logical…
David Bau: Positioning. I already had a question.
David Bau: What? What was your question having to do with…
David Bau: Any tracking heads and binding IDs? Oh, it's the… it's…
David Bau: I guess the paper touched on it a little bit, saying that's Think about how this… This paper's conclusion.
David Bau: works with the binding idea to decide, like, what is the appreciation works? Yes, yes. Yeah, so this is our view of binding ideas.
David Bau: That basically, this pointer here.
David Bau: Is what other people call finality.
David Bau: Make sense?
David Bau: Yeah, I think that's the closest. Would you say that's accurate? Yeah. Yeah, more or less. And one more… that's not the… So it's hard to put words in another researcher's mouth. They say, we found bonding ideas, and we say, well, we found pointers. I think they're the same thing.
David Bau: Yeah. Both the paper came out same day. That was not previous paper. Yeah, yeah. Yeah, both of them… Oh, no, your paper came out first. It's workshop paper.
David Bau: They're very fast.
David Bau: workshop?
David Bau: No, I think both of them were I clear. Oh, they're both iClar.
David Bau: Alright.
David Bau: Yes.
David Bau: Was that… that was Jacob Seinhardt, or is it… These Berkeley guides, they're fast.
David Bau: Okay.
David Bau: So,
David Bau: Alright. So anyway, so, so this shows up in other types of sentences. If you look at three of my sentences, which we're really interested in, it's like, oh, what does Fally know about what Alice knows?
David Bau: About what Bob knows and stuff like this. Like, these also seem to be stitched together using pointers.
David Bau: And there seem to be multiple hops of pointers and things like that, and we're still understanding
David Bau: what the structures are. But it's really… so if you have a complicated… if you're like, you're trying to identify, you might find pointers in there. Don't be surprised if you find pointers, right? It's a little tricky to analyze them. You get… you get causal graphs that look like this.
David Bau: Where I showed you a causal, like, experiment where, you know, oh, at the deep layers, it was soccer, and then it drops down, and now it's basketball or something like this. So here's… you get these funny graphs with, like, two transitions.
David Bau: Where, you know, at first, it's something else, and then it becomes beer, and then it becomes tea after that, so you have three domains of 3 layers, and what's going on here is the middle domain is where you've got the pointer involved.
David Bau: So before the pointers, before that, it's, like, the name, like the anus. This would be, like, oh, this is the domain where it would be, like, peach.
David Bau: Right? And then after peach, then it becomes, like, pee.
David Bau: And then after key, then it becomes DAW, or whatever, right? So, there's, like, these three domains. Like, before the pointer becomes a pointer, it's just a name. After the name's resolved to a pointer.
David Bau: And it becomes a pointer, and then after the pointer is dereferenced, and it becomes the contents of what it was, you know, what it was referencing. And so you can see these three…
David Bau: domains, So, yeah, so there's a crazy kind of console experiment.
David Bau: Alright, I'm probably out of time, right? Am I out of time? 5 minutes? 5 minutes! I'm just cold. Oh, you're just cold. Alright, you're gonna pick me up. Okay, alright.
David Bau: Alright.
David Bau: Questions, questions. I probably skipped over some questions.
David Bau: Implications for positional encoding research. Chris, answered that question.
David Bau: You're probably online, Chris, is that right? I'm right here. Oh, you're here. Okay, yes. No, so I thought this was interesting, because one of the things we talked about with this paper was that
David Bau: Well, they isolated that one of the biggest changes internally was that the positional information of the model
David Bau: So, in terms of manual tracking was, like, the biggest thing responsible for a lot of the performance. Yeah. I thought that was, like, so, I gave a presentation, additional coating of nerves. I thought that was really interesting, because a lot of the… a lot of the questions we got from the audience were people talking about
David Bau: Well, wait a minute, so, I mean, is it called coding is you just, you just, you just take this weird sinusoidal wave, sinus oil wave signals, and then this kind of magically resolves itself by dampening the spores? Like, basically. But, I mean, it's like, there's a lot more interesting things we could be doing, so I'm wondering, then, if there's probably some…
David Bau: I'm wondering if there's probably, you know, whether it's the fine-tuning process or what have you, I'm wondering if there's more fish… there's more intricate patterns. So, I think that it's related to the appendix that Nikhil's talking about, and also probably the future work that he'll probably do more experiments on this.
David Bau: But my gut feeling, or the evidence that we've seen so far, sort of suggests that there are multiple coordinate systems that the models find convenient to use, that are robust for different reasons. And, so when we give an encoding, a position encoding to the models, we have this token-based
David Bau: counting coordinate system, where you can easily… the model can easily go back 5 tokens, or go back 8 tokens, whatever, but then the model has other coordinate systems that it… it invents for itself. Like, going back 3 sentences, going back 5 sentences, like, maybe it's better to go back a sentence at a time.
David Bau: Or maybe a noun phrase at a time, or something like that. So it has these different ways of counting, and
David Bau: And, and so it looks like these pointers might be expressed in one of these other types of talking, other than the ones that we train into the model.
David Bau: And so, okay, I'm gonna go to this other thing here. So, overview of ICL theories.
David Bau: So, I don't think 2 minutes. So, ICL is…
David Bau: is this conceptual knowledge, like, stuff that you didn't know from training, but that you can figure out from context, like what's in box A or what's in box B?
David Bau: Right? So… so this idea… one of the most dominant ICL theories is that the main thing that you know from context is stuff that you can copy.
David Bau: Stuff that you can copy. It's sad that this thing was in box A.
David Bau: So I'm just gonna copy that knowledge back out. Maybe I have to rephrase it, so maybe fuzzy copy.
David Bau: Here's what I'm gonna do. I'm just gonna regurgitate stuff.
David Bau: That I was just told. And if I can… if I can copy stuff that I was just told, regurgitate it, maybe rephrase it.
David Bau: then I can get much better predictions.
David Bau: As a model, and that's… and so I, you know, so all the models learn how to do in-context learning.
David Bau: By And then… but then more sophisticated people will say, oh, the way to think about this
David Bau: Is it… so that's one… theory number one, is ICL is copying, right? Theory number two is ICL is like a Bayesian resolution.
David Bau: It's like, oh, you have all this evidence of…
David Bau: you know, circumstantial things around, and then… so you have this wonderful context. Well, everybody around me is speaking French, and they're eating crepes, and, and, and, and, you know, and stuff like this. So where am I?
David Bau: Right? Oh, I'm probably in France. I'm probably in France.
David Bau: Right. What? Or Brooklyn. Or Brooklyn, right? And so it might be a viable distribution. And, and so there's… and so, yeah, or Brooklyn. Yes.
David Bau: Yes, I love it. Or, or certain parts of Cambridge or something.
David Bau: Okay, so… But yes, so, you know, another view of ICL,
David Bau: Is that models learn to make these inferences.
David Bau: And it's just a Bayesian resolution of, like, what the contextual conditioning is.
David Bau: of it. It's a great, I mean, it's axiomatic, it's almost like, oh, of course it's gotta be the case, because…
David Bau: It's just, like, every information you, every piece of information you know encoded in this population distribution. So it seems like a little bit of a tartology.
David Bau: But…
David Bau: There's people who think that's the right way of looking at it. To me, that's like a black box way of looking at it. It's sort of like the information gas theory. It's sort of like saying, hey, what does a model know? It knows everything that there is to know. Like, evasion would say, this is, like, the statistical statement of everything there is to know, and your model would know that.
David Bau: So what I really want to know is, like, how does it know? Where is it stored? And if there's stuff that it's supposed to know that it doesn't, like, where are its limitations?
David Bau: How does it encode it? And so, like, understanding the pointers.
David Bau: like, an example of that. And so, like, pointers are an example of in-context learning. Copying is another example of in-context learning. And so here we have, like, this sort of crazy fuzzy copying.
David Bau: Where you, like, Have, have a task, Like, doing, doing, adenimps.
David Bau: And then you can try to transport the knowledge of the task over.
David Bau: another thing. And so I'm not going to have time to talk about this experiment, but I think it's really elegant. And so this is a third type of idea that you can also use cost mediation analysis
David Bau: to transport. But here, instead of detecting that it's the dominant thing that's going on.
David Bau: Because here, it's not the dominant thing. So this task that you're doing is like a small signal.
David Bau: It's only in this tiny small subset of the attention heads.
David Bau: Instead, we hypothesized that this information was there.
David Bau: And we used the hypothesis to search for attention heads that, if you transported them, they would bump the probabilities up in the right direction.
David Bau: And then after we found those attention pads, we summed up all of their effects and all of their results by, like, patching them over all at once.
David Bau: And when you do that as a gang.
David Bau: They do have very serious causal effects.
David Bau: And you can transport The idea that you have two endoms.
David Bau: One place to another by transporting all these attention heads at once.
David Bau: And the crazy thing, if you take the same attention heads.
David Bau: And you find something that's doing a different task, like doing English-Spanish translation or something, and you transplant… transport whatever those attention heads have to a different context, then you'll get the other context to do English-Spanish translation, or… so… or for a variety of other tasks.
David Bau: I guess.
David Bau: And so that's… so that's, like, this… it's an example of, like, a more…
David Bau: Unexpected way, like, once you're comfortable with cost mediation analysis, now you can sling it around and use it
David Bau: In different ways, and so there's another way of using it, and you get some interesting results from it.
David Bau: Talk about a few other questions later.
David Bau: Okay, guys.
David Bau: Thanks very much. My… my hope is that you can try some pathway experiments, In your projects.
David Bau: Like, I really think this is kind of the gold standard for whether you actually have, you know, representation of an idea inside your AIs. And so, so feel free to, you know, chat over Discord about any questions about setting it up.
David Bau: There's some URL for, the code that you can use. The notebook that has advanced codes and stuff like that. Okay.
David Bau: Oh, yeah.
David Bau: I was trying, we have a clear…
David Bau: I would think that's always inspect myself, right?