David Bau: It's okay.
David Bau: First learning, like, interp, like,
David Bau: Callum had just started this initiative, which was, like, he would… he would…
David Bau: Each month, train some, like, toy, small model, one-layer, two-layer, attention-only,
David Bau: To do some simple… simple, like, sorting a list, or something like that.
David Bau: And, and he would just say, like, go… like, it wasn't really a competition, but it was, like, sort of just go… here's some models that I trained that can do this thing, like, try to explain as much as you can, do your work in a collab.
David Bau: Present it nicely, submit it at the end of the month, and then next month we'll have another one.
David Bau: And it was just, like,
David Bau: I don't know, that's when I first sort of, like, fell in love with, like, yeah, like.
David Bau: trying to understand what's going on with these things, because it was just so much fun. And it's like, it was a really good way of putting into practice the things you…
David Bau: Read about or learn.
David Bau: So, I… he… he only did a few of those, before he got busy with his job, so… Did we convinced Callum to be an academic. What's that?
David Bau: Somebody should try to convince Callum to be an academic. He's really an amazing teacher. He's such a good teacher, yeah.
David Bau: Yep. But anyway, this is an effort to sort of revive that,
David Bau: And I think it has a lot of benefits. Like, one… one benefit is, like, we get a lot of emails from students who want to…
David Bau: do research, and, like, this is one opportunity for them maybe to A learn about EcanTerp and play around with it, and B, for us to get some signal on whether they, yeah, what their work is like, if they want to submit something.
David Bau: Another benefit is, like.
David Bau: All the starter notebooks are, like, loading the models in an AN site, and so, like, people will be able to
David Bau: you know… Yeah, get a sense of an insight from doing these things, I think.
David Bau: And then, like, another benefit is just, like, this could be a community thing in the NDIF, thing, like, Alice has been helping with…
David Bau: Yeah, that we're gonna, like…
David Bau: Advertise it on the end of community stuff, so…
David Bau: And, and, like, mostly, I think it's useful as, like, an educational resource.
David Bau: to…
David Bau: Yeah, because, like, real models, you never get to a place where you can, like, fully, really understand a model, but these ones are small enough where I think, basically, you can really understand what's going on.
David Bau: So it's… it's extremely satisfying, I think, to…
David Bau: have that, like, spark of, oh, I understand this. You get it, yeah. Yeah. So, anyway, we'll,
David Bau: It'll… it'll go under some… some more iterations. And… and the domain is puzzles.filab.info? Right. Is that right? Okay. Do you solve these yourself before releasing? Or intend to? Or do you imagine there'll be times where you just release it?
David Bau: don't know if it's how easy it is at this moment, or… I don't know. It depends on how much better we want the…
David Bau: do here,
David Bau: Anyway, we'll, like, release it on April 1st, or April 2nd, or something. No, no, April 1st!
David Bau: It's gonna be April 1st, but I don't know. You might have to choose a special puzzle for April 1st. I hope that the text of something is an April Fool's joke. Some special…
David Bau: Some special… Some randomized neural network. Yes, I don't know.
David Bau: Adrian?
David Bau: Oh, yeah, so, one thing that I've been…
David Bau: Working on recently is trying to, steer…
David Bau: This diffusion model, when it's in painting.
David Bau: These ladder grids.
David Bau: to output a different letter than I would have originally. So, like,
David Bau: So basically, like, the idea is that you take all of the tokens that are in, for example, like, this box in the top left of the grid, where it's gonna just impact the letter capital Z. You just take all those tokens, and then you, take the centroid of all of them, and that's, like, the centroid.
David Bau: quote-unquote, of, like, the capital letter Z, and then you take, like, the centroid of… or, of all the letters for, like, E,
David Bau: And then you,
David Bau: Get that severe steering vector, and then you just add it to the tokens, and just see whether or not it outputs the target letter.
David Bau: And this is, like, a preliminary result, just running on, like, just a simple case of just these, like, letters are really close to each other, like PGQ, and just looking at the, peak signal to noise. It's, Pisner is just, like, a…
David Bau: You can basically think of it as mean squared error, but it's just on the log scale, so, like, higher is better.
David Bau: And,
David Bau: The… the… the two images that are being compared is just the image that's in the grid, and then the image that gets impainted, or the… sorry, no, it's…
David Bau: The target, like, if you just print the target.
David Bau: character, what it would look like if you just printed it normally versus just intervention that we do. And we can see that, like.
David Bau: So it's so spiky. Yeah, so the intervention doesn't work across, like, really any of the layers until you hit, like, layer 30, 31, and it starts to, like, work a little bit. So do you think that the diffusion model knows what letter it is at layer 31?
David Bau: I… I… I… there's more evidence accumulating towards it, yeah. You think… you think so? I still feel like the, intervention is not, like…
David Bau: super convincing, like, if it hit, like… Is this, is this… which, which, which model is, which diffusion model? It's just Flux. It's Flux. So Flux is supposed to know text, right? It's supposed to know text. Yeah, and so, but you think that it just knows text at layer 31?
David Bau: I think so. That's cool. Yeah, that's great. That's really nice. I like it. Thanks, Adrian.
David Bau: It's very exciting. It's exciting progress. It sounds like a very clean experiment.
David Bau: Arnold.
David Bau: So this is my new activation patching setup. I…
David Bau: want to identify the sub-components, components… I don't know, you're only supposed to have one slide. Look at what is all the dynamic concept.
David Bau: More than what's life. Okay.
David Bau: So here's my activation branching setup. I want to identify the components or subspaces in the model that are mediating the goal signal. So what is the goal signal here? The goal signal here is a specific predicate. In the source prompt, you see that
David Bau: the model is implement… and prompting the model to implement is palindrome in C++.
David Bau: I want to patch it over to a different prompt that asks the model to implement a different predicate in a different programming language, Python.
David Bau: And I want to, by patching, I want to bring over the minimal information that makes the model
David Bau: implement the source predicate in the destination program language. Make sense?
David Bau: Okay.
David Bau: Oh, sorry, could you repeat that one more time? Sorry. What? Can you repeat that one more time? Yeah, yeah, yeah. So, so in the source, prompt, we see that I'm asking the model to implement one predicate, is palindrome in, let's say, C++.
David Bau: In the destination prompt, I'm prompting the model to implement another predicate, a different predicate, like isLower in Python.
David Bau: And I want to understand how the model knows
David Bau: What it is going to implement.
David Bau: So I… I want to perform the patch.
David Bau: that signals the model, like, I want to bring from the source representations to destination representations, that's that.
David Bau: The model thinks that it is implementing
David Bau: the source predicate in Python. That is, there is this entanglement between understanding the goal versus what the model is actually in the Python context.
David Bau: Yeah, why are you changing two variables at once? i.e, like, why… Why couldn't you…
David Bau: You mean language? Patch from C to C.
David Bau: Or even C to Python, but is palindrome.
David Bau: Then I have a risk of the model literally copying over the tokens.
David Bau: Like, let's say when the model is telling drums, it is… it is already thinking about the Pythonic way of doing the reverse.
David Bau: So, you see that where in the target token, it is, like, colon, colon, minus 1. This is something you only have in Python. Okay. This is not something you have in C.
David Bau: And if I do the patch, then, in… from, let's say, Python to Python, then I have risk of literally just copying over the broken file.
David Bau: The same logic goes, bro.
David Bau: Those are the other penis.
David Bau: How do you decide to angle that?
David Bau: How do I disentangle them, the two variables that you're changing? Like, how do you know that you're just…
David Bau: I'm just changing one variable.
David Bau: just the goal, and I'm assuming that there is a disentanglement between the understanding the goal versus, in the programming language, how it is going to be implemented.
David Bau: Does that make sense?
David Bau: So this is just a hypothesis. I'm not doing anything, to enforce that disentanglement.
David Bau: Does that make sense?
David Bau: Yep, and then what happens? Okay.
David Bau: So, so one natural way to target is basically attention heads, because attention heads are actually doing the parsing.
David Bau: So I… with this causal tracing, or with this activation patching setup, I learned a joint mask.
David Bau: over all the attention has, to basically flip the prediction to whatever I want.
David Bau: And it roughly… I can roughly select, kind of, like, a 2% of all the attention has, and…
David Bau: If I patch over the output of 2% of this attention has in the destination run, then…
David Bau: Well, immediately after this program.
David Bau: you will see that the model is actually implementing the spelling drove logic, but sometimes, so I'm doing steering, so for each generation, for each generated token, I do the same patching over and over again.
David Bau: So sometimes, the model just goes over… it does not really forget that it was… it was in the East Lower context, so even if it immediately says, like, checks for the East Balindrome predicate, it will eventually go back to checking the destination predicate as well.
David Bau: So, anyways… Yeah, what are the right tokens to patch?
David Bau: This is something, yeah, and also, like, how… Is there a subspace where you should actually molt, like, every token along that subspace or something? Good question. Yeah.
David Bau: Also, like, do you need to patch it over generation? Like, when they're going to actually end the patching? So it's a question of, like, when the model starts holding the goal in its head, and when it stops the one. Yeah, the clock signal, yeah. I'm calling it the clock signal, and I just… the clock, basically, when to start pushing the goal.
David Bau: Then when to, enforcing the goal, and how to schedule things inside. So, yes.
David Bau: A way to measure this? I have been going through the literature for the last week, and I have not found anything, but the hope from this project is that to get a better understanding of, you know, how to do that.
David Bau: Yeah, excellent. I prefer not to be recorded. Oh, you prefer not to be recorded?
David Bau: We can fix that.