David Bau: So many times remote. Should come, come in person.
David Bau: You know what, I should, I should print food.
David Bau: Many people, like, you can't have the food remote.
David Bau: One day.
David Bau: Okay, let's see here what we've done.
David Bau: Okay.
David Bau: Welcome to the spring!
David Bau: Look at that weather out there!
David Bau: It's almost, it's almost, Revere Beach by there. Who wants to go to the beach?
David Bau: Today? Yeah. With 30 mile an hour winds. The sand will be just, like…
David Bau: Let's see, did I hit record on this thing? Are we recording? Yes. Okay, great.
David Bau: Alright.
David Bau: Yeah, so… Yeah, how are the… how are projects going?
David Bau: Did I pick?
David Bau: So, so, yeah, next week… We're gonna…
David Bau: We have a couple researchers who are studying you guys.
David Bau: And, and we're also trying to improve the class.
David Bau: And so, so they're gonna actually come and run a little workshop where they're going to, step you through using some of them.
David Bau: the research tools, and ask for your feedback, and stuff like that. And so, so actually, I have a piece of homework
David Bau: That's slightly different from… for next week.
David Bau: than we normally do, so normally we'd have a reading for you guys to do.
David Bau: But this is a project-related homework, it's very simple.
David Bau: Which is, is that fair?
David Bau: So I'll have Nikhil write it up and send out a form for it, right? But, so that you guys…
David Bau: Know exactly what it is.
David Bau: But, but the class activity will have you…
David Bau: Do a piece of research in class.
David Bau: And the thing that they're gonna step through
David Bau: It's going to be anything related to your project where you have… Eric?
David Bau: Fuck, that you want to try and do some detailed fetching.
David Bau: Between.
David Bau: And so, you guys are far enough into the project that I think all the teams have some data sets where you have some interesting pairs of contrastive prompts that you may have already looked into, but maybe you want to look into more detail, or something like that. So every team member, should have an idea of, some text prompts.
David Bau: Do they want, to improve?
David Bau: And,
David Bau: And that's… that's that, that's all. So, you'll basically work more on your project, but then think about something specific that you can work on next.
David Bau: Next Tuesday, in class.
David Bau: Make sense? Okay, and then Nikhil can send out a form or something that reminds people of that.
David Bau: Alright.
David Bau: Okay
David Bau: I think maybe I'll rename the slide later when I save this slide deck again. Maybe it's Novel Knowledge?
David Bau: within neural networks, or something like that, so I'm not sure, really, what the theme was of the three readings of which we're talking about today, but I just… there's several things that I wanted you guys to be aware of, that they're all techniques that you could use
David Bau: In a paper, I've seen people use these techniques in the paper, and so as you guys are, you know, developing.
David Bau: your ideas, and you're looking for what other way can we triangulate our experiments or get a little bit more insight of what's going on.
David Bau: you know, I just want to give you some exposure to some other things that people
David Bau: do. So this is so… if you can't tell, like, the whole class has been… we're only scratching the surface of many of the things that people do.
David Bau: In interpretability research?
David Bau: But I'm trying to bias it towards things that
David Bau: I've seen, like, show up in a bunch of research papers as techniques that people reuse. And so, the first one.
Tamanna Urmi: Am I audible?
David Bau: Oh, can you hear me? You can't hear me?
Tamanna Urmi: Yeah, is it possible to share the screen, on Zoom?
David Bau: Oh, is the screen not shared? Yes, that's… I'm sure it's possible, let me see.
David Bau: Let me see… that would be helpful, won't it? Then you can see.
Tamanna Urmi: Yeah, so right now, it's a camera, like, we can see it through a camera, but if it can be shared, that would be fantastic.
David Bau: Yes, thank you for interrupting.
David Bau: Now, can you… is it better?
Tamanna Urmi: Yes, yes, fantastic. Thank you, Zod.
David Bau: Great, and then I should probably…
David Bau: I have this thing, I have video count.
David Bau: I'll hide another thing.
David Bau: My floating in control. Is that looking okay now?
David Bau: Okay.
David Bau: Yes, sir.
Tamanna Urmi: This is great. Thank you.
David Bau: Now, of course, they don't have a new control engine.
David Bau: Let's figure this out.
David Bau: Okay, that's great.
David Bau: Okay, so… so the first thing is training dynamics. So, you guys are all setting new training networks, you have capabilities, but those capabilities didn't…
David Bau: materialize, Instantly.
David Bau: Right? All these networks were trained over some procedure.
David Bau: This class is not about defending proximity.
David Bau: But it can be interesting when you're looking at the internal offload.
David Bau: circuits, the internal beliefs of the model, to see how those processing pathways
David Bau: evolved over time. And sometimes, the story can be very interesting.
David Bau: And so… so one of the early papers to look at this was a paper called Rocking.
David Bau: And so there's a few graphing papers, and there's one that I assigned you guys, but then this is the paper that came before that one. Did anybody look up this one? This one is from, Power at all? Will you look back on that one? That's great.
David Bau: I think this is a good thing to do. You know, the paper I assigned to you was, like, a follow-up paper, and it might be easier to understand if you, like, skim this one. So I'll show you what the original
David Bau: Paper-wise, it's relatively simple.
David Bau: The idea is… That they're gonna train a toy, language model.
David Bau: And they're gonna train it on a really simple language, which just has lots of sentences that look like this. You know.
David Bau: You know, A plus B equals C, or not plus them, just arbitrary operation.
David Bau: A operated on B equals C. For lots of different letters, A, B, and C, just mixed up, but always true statements. And what they do is, they, to get their true statements, they form these little multiplication tables.
David Bau: And then they fill in the multiplication tables according to some math operation.
David Bau: like, addition mod 5, or division mod 5, or something like this. And then they… they pluck out a few of these, some percentage of these, they might pluck out half of the entries in the multiplication table, and they say.
David Bau: They're not included as part of the training set. The rest of them are included as part of the training set, and then they just train the model on sentences to sample from
David Bau: You know, from this multiplication table, and they see how does it do. And now, because they've held up some of the facts, they have two measurements they can make. They can make… measure how well does it do at remembering the ones that we've included in the training data.
David Bau: And they could also ask, how well does it do on the things That we haven't included.
David Bau: And then training data.
David Bau: And, you know, I think the people who did this, they didn't really have any idea
David Bau: Whether the neural networks would actually guess
David Bau: Correctly, the things that have been never shown.
David Bau: And actually, when they first trained them.
David Bau: They do really well on the facts that you show them.
David Bau: But then the accuracy on the facts that you don't show them is stuck at zero.
David Bau: So, you know, it sort of suggests the neural networks.
David Bau: What? Memorizing it as… That's what they call it. So, in machine learning, and it, like, so, anybody who's taken a machine learning class will know, there's what we call memorization as opposed to generalization. It means that, that, you know, it can regurgitate the things that you exposed it to in training.
David Bau: But… It can't figure out anything that Extrapolates beyond those examples.
David Bau: You know, it basically material. So…
David Bau: So that's sort of where this experiment sat for a little while until the mathematicians who tried this
David Bau: For whatever reason They just let the computer run longer.
David Bau: And after they ran it, for 10, 100… A thousand times longer.
David Bau: They just started to see… That the holdup started to improve.
David Bau: Now…
David Bau: Who here is taking, like, you know, a machine learning class?
David Bau: Right? Yeah. Like, like all the computer scientists.
David Bau: So, have you ever heard of, you know, different regularization methods? Have you heard of the regularization method, really stopping?
David Bau: So, what is… what is regularization supposed to do? Is it supposed to hurt or help with generalization.
David Bau: It's supposed to help, generalization. And so, the intuition
David Bau: is if you, like, how would you, like, get better generalization? By training longer, or by stopping early?
David Bau: Stop being harder, that's, like, what they teach you.
David Bau: In machine learning class.
David Bau: So, did these guys stop early?
David Bau: Oh, no.
David Bau: Yeah, they only train, like, a thousand times longer, what the heck? This is the opposite of stopping early.
David Bau: And then what happened?
David Bau: Like, it generalized anyway. So…
David Bau: Because it's so counterintuitive, right, because it's not what they teach you in, you know, a machine learning class.
David Bau: this was a really… there's a little workshop paper, and a little tiny workshop, I forget, you know, ICML or something.
David Bau: And, and it just, like, kind of this little observation of this little toy thing with this thousand-fold
David Bau: Increase in training time, and then… and then set a, you know, a nice jump up in generalization after this.
David Bau: even though it was this humble little workshop paper, it got lots of attention. People were like, what the heck is happening with this? Right? Does that make sense? Because it's like, it was an example of…
David Bau: something counterintuitive going on. And it's systematic, so it's not just one run. They ran hundreds of runs and so on, and they found this, you know, systematic
David Bau: Effect, that they plotted out, and quantified in different ways.
David Bau: So that was the original paper. To give you a little sub… so that they see the subset is learned instantly.
David Bau: The unseen facts are guessed a lot later.
David Bau: You know, they had some patterns that they said, you know, the more emissions, the longer it takes to drop.
David Bau: You know, so this is the memorization cliff, this is the grocking cliff, they call this grocking. So they gave us an in.
David Bau: You know, when you train a long time, then all of a sudden it's a network or something, they call ethical rocket.
David Bau: Because, just… are people familiar with that English word?
David Bau: I'm out of California work.
David Bau: So, but I mean, it's like, oh, I get it now, I grok it. It's not, like, something that people would say after they meditated a long time or something.
David Bau: I'm just in understanding graphic.
David Bau: Okay, and so here's, like, an example of an extreme gracking holdout, so they were very proud to put this in the appendix.
David Bau: worth taking a look at it. So this is a multiplication table that is so huge.
David Bau: That they've had to use all the Hebrew letters and all the Greek letters.
David Bau: Because I ran out of, like, English letters to put on each one of the things. And so you multiply one of these by one of these, and then you get some other letter in the middle, and if you zoom in, you can actually read these letters.
David Bau: And you can see that all the blank ones are ones that are not included.
David Bau: in the training data. And they say, this is an example of a multiplication table that grocks.
David Bau: We give the network, like, all these Questions?
David Bau: As training data, and then after we train it for a thousand times longer.
David Bau: Than, than, you know, you'd normally train it.
David Bau: It correctly guesses all the others.
David Bau: It's kind of amazing, right?
David Bau: Like, the extent to what you… I guess, is correct.
David Bau: what the heck? So that's… that's where this other paper comes in. So this is… so Neil Nanda and, and collaborators
David Bau: another California crew here. They tore open one of the transformers to see what was going on inside.
David Bau: And what they said is, oh, it looks like, you know, after you train it for a long time, when it grows.
David Bau: It develops a very funny representation, and they say it goes to help communities.
David Bau: Well, I don't have the phase diagram here, maybe I have it later.
David Bau: Where early on, it uses one strategy, and then later on, it develops the second strategy. And the second strategy looks like this. And the second strategy, too, so who understood or who didn't understand?
David Bau: this… this analysis. I'm not sure it was, like, Super well explained.
David Bau: In this paper. So when… when Neil says.
David Bau: oh, I see these Fourier components in my neural representation.
David Bau: Does that make sense? See, like, what that… what the heck that is? I can make it concrete.
David Bau: Who wants me to make it concrete.
David Bau: You know.
David Bau: Alright, so, so let's take, like, the embeddings, for example. So.
David Bau: When you feed in numbers into their neural network.
David Bau: Then they come in as vectors.
David Bau: So I don't remember what dimensionality they give, but, you know, they might be… Oh, it looks like 50…
David Bau: I can't tell.
David Bau: I'm not… maybe not, it's not 50, so… but… but, you know, there might be 1,000 dimensional vectors, or something like that, or it might be 100-dimensional vectors, doesn't really matter, right? So, but they have some embedding dimension.
David Bau: That they use to represent every input thing. But now, what is the input vocabulary for this A times B equal C thing? Well, they're encoding, modulo 113 arithmetic
David Bau: So, they need 113 letters.
David Bau: Or 1 for each one of the numbers.
David Bau: Right? And so… so, so they, they, they have, so they're, they're, they're embedded.
David Bau: A matrix is a table with 113 vectors. One for each letter.
David Bau: So if you say A, then that's a vector. If you say G, that's another vector. If you say C, that's another vector.
David Bau: And so, now, it's… in practice, they're… they're learning modular, so A is actually 1, and V is actually 2, and C is actually 3, right?
David Bau: And so… so if you… if you take your… 100-dimensional embedding vector.
David Bau: And you line it up.
David Bau: So that all 113 numbers are in this big grid. And what you'll find is, you'll find this interesting thing, that if you take different rows of the embedding vector, let's take, like, neuron number 5 or something like this.
David Bau: And then you trace… I wonder if neuron number 5 is high, or low or medium for 1, and you can, like, plot it.
David Bau: And then, oh, I wonder if that same neuron is higher, low, or medium for two, and plot it.
David Bau: And then higher, low, or medium for 3 implied. And so you just, like, so you just take, like, a single neuron.
David Bau: From your embedding matrix.
David Bau: And you plot it all the way across all 113 numbers, and you'll get some waveform.
David Bau: Make sense?
David Bau: Right So that's the waveform that they're doing Fourier analysis on.
David Bau: Right, and so… and they have a lot of these waveforms, so basically one for every neuron, or there's other ways of analyzing set of basis, but basically, the easy way to think about it is, you know, you've got… you've got a waveform for every neuron.
David Bau: And, And so, when you do this in practice.
David Bau: And you look at the waveforms you get for all the embeddings,
David Bau: At first, when it's memorizing, the waveforms look
David Bau: Really jaggedy. Like, just, like, noise, like, all over the place, right?
David Bau: And… and if you do… if you were to do a Fourier analysis on it, it, like, has all the frequencies. It's just all mixed up. It's like white noise.
David Bau: Right? For basically every neuron.
David Bau: And, and they didn't picture this in the paper. I think that would have been useful.
David Bau: to do. But that's what happens in practice.
David Bau: There's a little backstory here, which is, I was also looking at grocery at the same time that Neil did.
David Bau: He gave me a call, and he says, I heard that you're looking at this, and what's going on? And I shared with him, like, what I've gotten.
David Bau: And then he's like, Bubble, you're almost done with this. I better… better submit mine first, so I don't get stuck. And so,
David Bau: So yeah, I never published mine. Anyway… For scooping?
David Bau: He gave me an acknowledgement in the paper, saying, thank you for the informative discussions.
David Bau: That's crazy, that's crazy. Anyway, it's alright. But there's a reason that I know, like, a bunch of this stuff here, because, no, he's looking at it. So, so,
David Bau: It's not… for me, it's not a big deal. It wasn't a huge deal of paper.
David Bau: I was living on Rome at the time. This is a bigger deal for me.
David Bau: And so, but this,
David Bau: But, but yes, it's not just all on video in this case.
David Bau: So, but… and, you know, Neil's incredibly effective at meetings, so that's fine. But the,
David Bau: So the thing that happens is, you get this crazy waveform, during the memorization, and then, all of a sudden, when it grocks.
David Bau: It's like, all the different frequencies, they drop out.
David Bau: And then you just, like, you know, like, like, white noise.
David Bau: And then all of a sudden, it's like.
David Bau: you know, like, rings, like, this clean sine wave signal. A few frequencies, like, pop out.
David Bau: And this is what Neil has, neil's depicting here.
David Bau: That make sense?
David Bau: And so… so yeah, so that's what happens. It happens in the… so he's plotting the embeddings.
David Bau: And I don't know the total details… oh, and I think this is the unembedding matrix, so he's plotting the unembedding, so, like, when it has to output the next thing, there's an unembedding matrix.
David Bau: And, and they both… and they both show the frequencies.
David Bau: And actually, they both ring at the same frequencies.
David Bau: It's kinda cool.
David Bau: Isn't that weird? So now do you get it? Now do you see what's going on? Does that make it more concrete?
David Bau: Okay.
David Bau: And so, yeah, so that's what happens.
David Bau: And, And then he says, are these frequencies, like, important?
David Bau: Like, it's, like, ringing at this frequency, so it could be two things. It could be that everything else is important, and the frequencies are, like, sign of some noise.
David Bau: Right? Or maybe the frequencies are important and everything else is, like, some noise. So, like, there seems to be this clear distinction between these two things. And so he did this very nice experiment, where he said, what if I… what if I, like, filter out
David Bau: these embeddings, or whatever, whatever I filter out.
David Bau: The other frequencies that are not The, the ring ones.
David Bau: What… how does that impact?
David Bau: the accuracy of the network, and it impacts it very little. I think there's a logarithmic scale, right? And then if she says, oh, what's the impact if I cut out one of the greening frequencies, and it has a much bigger effect?
David Bau: So… so this is his evidence that it's the ringing frequencies that are actually carrying
David Bau: the information of how to solve a thing. And so, in the paper, they talk about the sine and cosine functions of, like, what's being implemented in practice. I won't talk about that. But basically, you know, if you want to do modular arithmetic, you can implement it by adding together some sines and cosines, which is
David Bau: What looks like what these networks are doing by representing everything in sines and cosines, with these clean frequencies.
David Bau: And, and so, but the thing that's relevant to… so, you probably… none of you are studying something where you'll see signs and cosines, most likely, although maybe, maybe the… maybe some of the mapping people are spatial people, depending on what you're doing.
David Bau: In which case, you know, you should understand that. But one of the things that he saw, and you could see this too.
David Bau: If you look at your training dynamics, is that you'll see these weird cliffs
David Bau: of behavior. So here, he had his insight. After he found his mechanism, so for him, his mechanism was.
David Bau: this Fourier analysis.
David Bau: what's going on. He did this nice thing which he… he did… he… he did his Fourier analysis, phased experiment.
David Bau: Over time, while the model was training.
David Bau: And I think that he had, what he did was he says, oh, I've got this piece of code that will tell me what the impact is if I filter out a frequency. I can either filter out the included frequencies or the excluded frequencies, and so I can make a different loss curve to see, like, what is the effect.
David Bau: If I filter them out early during memorization.
David Bau: Or late during generalization, or in the in-between phase, right?
David Bau: And so… so that's what these turquoise lines are.
David Bau: Right.
David Bau: So the turquoise line is… if… if I take…
David Bau: The important frequencies, like the ringing frequencies.
David Bau: And I go back and I filter them out during memorization, you know.
David Bau: Little loss is good here. Little loss means it's, like, you know, doing pretty well.
David Bau: And during memorization, you know, if you don't do any of the filtering, the memorization looks like this for the training data.
David Bau: And so what they do here is they say, oh, actually, you know, it actually kind of does pretty well
David Bau: You know, even if you cut out
David Bau: the sexual frequencies. It does a little worse, right? Because obviously, like, we're… we're perturbing the network.
David Bau: But it's not so bad, like, the loss is still dropping, and so that makes sense, because, like, the special frequencies are just, like, 10% of the frequencies of the network, right? So there's, like, a 10% intervention.
David Bau: We're, like, making it… we're hitting 10% of the information there. It's not so bad.
David Bau: Right? The network still gets better. But then, there's this sudden reversal at this point.
David Bau: Where if you cut those frequencies out.
David Bau: You know, this, you know, the regular training continues to get better a little bit, but all of a sudden there's this reversal, where cutting those frequencies out makes something worse, a lot.
David Bau: Right?
David Bau: And so… so this is… this is pretty direct evidence that the model is starting to focus on those frequencies, right? It's starting to really depend on them.
David Bau: And then, by the time you get to this part.
David Bau: If you cut out the frequencies, then basically the network can't solve the problem.
David Bau: Right, the loss is, like, as bad as it ever gets.
David Bau: That make sense?
David Bau: And so…
David Bau: And then he has the inverse thing here, where if you look at all the other frequencies and you cut them out.
David Bau: you know, the network can't do the task at all with the other frequencies. So what are the other frequencies? The other frequencies are, like, his fingerprint, his hypothesis of what memorization is doing, is that memorization is using this mishmash of all the other frequencies.
David Bau: Right? And so… so when… when you cut out the memorization frequencies, the network can't perform at all while it's memorizing.
David Bau: And it's depending on this memorization all the way up to this point.
David Bau: Until all of a sudden, it's not depending on the memorization anymore. And if you cut out… actually, this is… he mentions this in the sentence.
David Bau: If you cut out the non-renewing frequencies, then actually, it starts having lower loss.
David Bau: than the regular training. It actually cleans up the network.
David Bau: Oh, let me zero this noise out for you.
David Bau: Interesting.
David Bau: These are all of the noise frequencies? Yeah, he does it all at once. Yeah, he does it all at once, I think. Yeah, that's right. And if I read the paper right.
David Bau: Yeah, so it's a nice experiment.
David Bau: Yeah, so this was… this is creative. This is something that I never ran myself. It's very nice.
David Bau: And then, and then he also shows this, interesting effect when you just look at
David Bau: The norms of the weights, and the norms of the weights, sort of.
David Bau: have these funny little cliffs where something happens here. So even if you don't know what the detailed dynamics of your circuits are, if you just look at some aggregate data, like the norms of the weights or something, you'll… sometimes you'll see them go through different bumps.
David Bau: And things. And what this is suggestive of is, oh, actually.
David Bau: At each one of these bumps, maybe the network is, like, learning some different solution to the problem. So… so I think there were some questions here.
David Bau: About, like, so, I'm gonna pause for a moment. IU, Claire, Isaac, Avery, had a bunch of questions.
David Bau: That were sort of related to… You know, this presentation itself.
David Bau: I'll let people ask any questions they had.
David Bau: I was thinking about the, like, are, this is slightly different than what I asked, but, like,
David Bau: Do you have any sense of, like, things that do crock versus what doesn't crock? Like, it seemed like this was dependent on finding the pattern at a later stage, like, kind of reverse… Yeah. …reversing out, kind of how it learned that. Yeah.
David Bau: And I was wondering about the, kind of, like, in the other direction, if you didn't know, like, an appliance stage… Yeah. …predictions of, and this relates a little bit to some of the questions about, like, causality versus correlation, where sometimes in our experiments, we found, like, there's information there that the models don't use. Yeah.
David Bau: You know, this paper got a lot of attention,
David Bau: And people, you know, this graph here, right, got a lot of attention.
David Bau: And people notice certain things about it, and so do the people, so I'll point out to you things that
David Bau: you know, circulated in the community afterward, like, the skepticism. It's like, oh, wow, look at this cliff!
David Bau: And then later on, like, a thousand times later, right, it's like, oh, there's another clip, it's amazing!
David Bau: Well, this is a logarithmic scale.
David Bau: And this one is a little more gentle than this one.
David Bau: And so, actually, now this is left short. It, like, memorizes quickly, but if you were to put this on a linear scale, actually, this is very, very, very, very…
David Bau: steady, slow, like, it took a while for it to grok, right? You know what I mean? And so… so is that, like, is grokking… is it, like, they have this very…
David Bau: nice term that they use, oh, I think suddenly we get it, right? But actually.
David Bau: Actually, it's not that sudden a process, in real life.
David Bau: And, and maybe, you know, maybe not that distinct from the rest of what's going on here. So other people said, hey, you know, they're measuring accuracy here, right? And it's accuracy.
David Bau: But actually, if you measure loss, then you don't see such a cliff. Loss is, like, more gradual.
David Bau: And, and so there's a bunch of… Different people who objected.
David Bau: To the title of the paper.
David Bau: Gracking?
David Bau: Which is a common tension in our field, because we like to pick flashy titles. I'm guilty of this.
David Bau: And, but so, so the mental model, you should have a more realistic model, is,
David Bau: This kind of phenomenon is fairly widespread.
David Bau: Where you have different types of solutions being learned at different rates?
David Bau: But this idea that there's different cliffs that happen is a little bit more… Controversial.
David Bau: Some people say that there are cliffs
David Bau: But they don't think that they're, like, scheduled, like, the cliff doesn't always happen at the same time. Like, the rate of…
David Bau: Discovering a difficult… solution is a little bit random. Oh, actually, you see the randomness here?
David Bau: Right? So this… there's all this stochasticity.
David Bau: Of how long it takes to learn this other solution.
David Bau: Every one of these dots
David Bau: Let's see, I guess this is the number of steps.
David Bau: to getting… to grokking. And this is another long scale, so it could be, like, 10 or 100 times more steps or fewer steps randomly.
David Bau: And so, so yeah, so, so the rate of finding these solutions is sort of random.
David Bau: And, and then, and then the idea that it's a cliff is also…
David Bau: People think that it's a little bit more… Of a… of a slope.
David Bau: Than… than the way they present it in the paper.
David Bau: But yes, but besides, like, Everything changed.
David Bau: I think that there… I'll show you some other examples where this kind of thing has come up.
David Bau: That'd be on our future slide. Any other questions?
David Bau: Yes. So, like, sorry, just… could you go back to the slide? Oh, this one, sorry, yes, like, groc faster with less power. I was thinking, like, is there… they only have, like.
David Bau: 1, 3 times 1, 3 pairs of pinklets, and, like, they're training on, like, 30% of them.
David Bau: I'm wondering, like, on this, like, small…
David Bau: data set, it might, like, grokking happens, like, could it be because, like, it's much easier for the model to memorize, and then, like, later, like, grok, half-grocking happens, because it's trying to, like, definitely, like, decrease, like, learning some real things. But, like, in reality, LLMs.
David Bau: we have, like, enormous, pre-training courses, and, like, I don't know if, like, it's possible to, like, memorize
David Bau: All of them. Right. And I'm just thinking, like.
David Bau: But the amount of training data, or, like, could that be also, factors? Yeah, is this, is this, like, a special thing about small training data?
David Bau: You might be right. So, one of the things that is controversial here is that it's… he's trying to infer something about deep learning in general out of a toy setting with a really small training set.
David Bau: And,
David Bau: And so some people think that that's nonsense, that, you know, it's very different from the organic. So, you know.
David Bau: This question you're asking, you're asking the question in an open-minded way,
David Bau: Some people ask the question in a very critical way.
David Bau: Right? Where, where they say, yeah, I, I, you know.
David Bau: You didn't test on big data, and it doesn't have the properties that real training has.
David Bau: I think that's a reasonable question. I think that's… that's also open.
David Bau: for a debate.
David Bau: There's, there's definitely a community out there who thinks that
David Bau: the only way to really understand simple systems is to simplify them down. They say that they've done this in physics forever, right? And it did well in physics, understand waves and box, or something like that, and it helps you understand
David Bau: You know, waves in the real world.
David Bau: But, but other people say that, no, no.
David Bau: Complexity is an inherent part of what's going on, and if you take away complexity, then you get different dynamics. So I…
David Bau: I don't have a strong… Belief one way or the other. Like, I just… I just…
David Bau: keep that skepticism in mind when I read these papers and when I advance an argument like that.
David Bau: Make sense?
David Bau: Yeah, I think it's… the jury's still out, it's all… it's all to noon.
David Bau: We don't have a consensus.
David Bau: There was a… there was a question, somebody asked a question about L2. Who asked the L2 question? I thought that was an interesting one for this plot.
David Bau: Was it Hai Yu, or somebody?
David Bau: So… I'm gonna pause. Let somebody ask the L2 question. L2 regularization, anybody?
David Bau: I don't remember who asked.
David Bau: So… So this is… so, like, this is interesting, right? The weights are getting smaller.
David Bau: Right? And, and they suddenly dropped down.
David Bau: When you get this different, different thing. And so they, they call this… so Neil calls this phase the cleanup phase.
David Bau: All right. I, you know, I also saw this when I experimented with this system. And my intuition for what's going on here is this inter… is this, like, this interaction with L2 regularization.
David Bau: And I try to… convey this intuition.
David Bau: When we teach machine learning.
David Bau: You know, L2 regularization is a pressure that you put on your model to make all the weights smaller.
David Bau: Right, so it's a pressure to make things… try to zero out the model as much as… Okay.
David Bau: But it's just, like, some pressure.
David Bau: And your model has this other pressure, which is, like, to solve the problem.
David Bau: And to solve the problem, you can't set all the weights to zero, right? It's not going to do well, right? So, to solve the problem, it has this positive pressure on the weights. It's like, there's all these, like.
David Bau: computations it has to do, and to do the computations accurately, the numbers can't be zero. They've got to have some positive values.
David Bau: And… and so, what you see here is that early on in training, Right?
David Bau: Memorization.
David Bau: Like, blasts the way it's big.
David Bau: Because, like, oh, and maybe they start down here, and then boom, as soon as you start training, like, the weights get much bigger.
David Bau: And, and so memorization is this…
David Bau: This, this solution to the problem that requires… and you put a lot of information in the weights, which means that a lot of the numbers have to be really big.
David Bau: Right.
David Bau: And then, as it figures out, you know, how to memorize a little better, like, it starts to organize its information in some ways, and then the weights get smaller.
David Bau: For a bit. And then, as…
David Bau: As it realizes that a really good way of organizing the things is to use these, like, these few frequencies
David Bau: It lets… it lets the model, like, shrink the weights further until it gets to the point where it says, oh, you know what? These frequencies are working so well, I don't need anything else. All I need is these frequencies.
David Bau: Right? And then… and then the optimizer says, oh, you know, I don't have any optimization pressure on this
David Bau: on this memorization path and more. I can set all those weights to zero.
David Bau: and…
David Bau: And it doesn't impact my loss, because the ringing rates and the key frequencies are solving the whole problem. So that's… so the L2… so the way I think of it is, like, the L2 regularizer is coming through and sort of sweeping up
David Bau: You know, the old solution.
David Bau: and zeroing it out, and that's… so that's the way I interpret You know, this park.
David Bau: That makes sense, though.
David Bau: Anyway, a little bit of intuition, a little storytelling.
David Bau: That's sort of how I could do it.
David Bau: Okay.
David Bau: So, oh, Jasmine, are you online?
Jasmine C.: Yeah, sorry, my voice is still…
David Bau: How you doing? You feel like a question?
Jasmine C.: I don't remember what I wrote, sorry.
David Bau: Yeah, so I think that you asked this question, I just took a slide here, it said, it's a very related question to what Isaac asked, which is like, oh, does rocking happen anywhere else? So I don't remember if you remember from the induction speaker?
David Bau: But they… they talked about this cliff in the loss that you get when you train, text, you know, and measure it on the copying task.
David Bau: And then, you know, it does well up to a certain point, and then all of a sudden, it does much better until it's almost perfect.
David Bau: And,
David Bau: And so… so these cliffs… and the cliffs only happen if you configure the network the wrong way. If you don't have enough layers, it can't do it, but if you have enough players, it can do it. And so… so yeah, so… so the answer is yes. You know, groking happens.
David Bau: You know, these cliffs, these phase transitions that happen in other
David Bau: Settings, and this is, like, the clearest one.
David Bau: Here's another one. So this is a paper that… My student, Eric Todd.
David Bau: Jess got into… I think he's gonna… Presented at ICLR.
David Bau: And… and here, he's… he's pulled apart.
David Bau: Now, one mechanism, he's pulled up apart, like, 8 mechanisms.
David Bau: Inside this network, and then one of the things that he did is he went inside to look at
David Bau: the training dynamics for how these mechanisms did over a bunch of different training runs. And so this is averaged over, you know, a dozen training runs or something like that of the model. You can see that, you know, pretty consistently, you have these different cliffs where different mechanisms all of a sudden get really good accuracy.
David Bau: Yes? Have they tried, like, if I drop it on it.
David Bau: only one particular type of data, and does it generalize to other? For example, if I only wrote on physics, does it generalize to the chemistry or biology? Yes, exactly. That's exactly what's being done here. So, let me tell you two, like, things that you can do to try to get gracking curves out of your papers, if you want to do it.
David Bau: So, one thing, and this is the thing that's probably practical for what you're doing.
David Bau: Which is, you know, you have some general distribution of what's going on, and
David Bau: you think that you understand how a sub-distribution, works. Like, maybe, maybe there's, like, 4 different strategies that your network is using to solve the problem of estimating uncertainty, or something like that. And depending on the sentence, or depending on the keywords, or something like that.
David Bau: you know, it uses one of these different strategies. So, what you could do is you could say,
David Bau: Let's pick a sub-dataset.
David Bau: of my overall dataset, which isolates this strategy. Like, now that I have some understanding of what it's doing.
David Bau: You know, 10% of my sentences
David Bau: use this strategy, and they exclude all the other strategies. Like, it's just purely this one.
David Bau: Once you have that, then it's pretty interesting, not just to measure it at the… final…
David Bau: Checkpoint, the final model that you're testing, but to go back into the early checkpoints, if you have them, and then plot, like, well, how did… how did… what was the performance on this little subset?
David Bau: update, and that's… that's basically what this is. So basically, Eric says, oh, you know, I'm solving a harder form of algebra problems. He went back to the rock and paper reading, so he's… now he's solving problems, and he's asking, which… which of these problems is solved by copying.
David Bau: Which of these problems are solved by, identifying the identity.
David Bau: you know, A times B equals A, that means B is the identity, which is a strategy. Which of these things is solved by identifying commutative inverses, maybe A times B, sometimes equals B times A, depending on what situation you're in.
David Bau: Which things work by associativity? Which things work by cancellation? Like, can you solve it like a Sudoku puzzle? Right? Canceling things out.
David Bau: And so he found, like, mechanisms for doing all these different things.
David Bau: Inside the network, and then he says.
David Bau: Number 3 is identity. Actually, the performance of the identity goes pretty well until it drops, and it's really bad at the identity while it's learning how to do commutative copying, but then after it learns how to do commutative copying, the performance on the identity comes back and goes near 100%, right? And so on. So he sees these funny
David Bau: Putting weird-looking brack makers.
David Bau: That makes sense? So that's… that's probably the main thing that you want to do, if you want to measure Grakenbell. I need to measure… I need to mention this other paper here. This is from, from who, latent space models of train dynamics. So the other thing you can do is this, is…
David Bau: So I mentioned, you know, you can see these phase transitions if you just measure, like, the size of the weights.
David Bau: Right.
David Bau: And if you just measure other random stuff about your network, you can just…
David Bau: take a linear algebra textbook or something. How about the spectral norm of all the matrices?
David Bau: How about the traits?
David Bau: You know, how about,
David Bau: How about the maximum of all the weights instead of the average of all the weights? How about, like, this… the magnitude of the gradient, if we pass gradient over the network? How about this? You know, you can measure different things. And if you measure one of the… each one of these during all the checkpoints, you'll also see that sometimes there's, like, cliffs up and down.
David Bau: You know, all around, right? Like, it's not just as smooth, progression.
David Bau: And so what… what they did in this paper, what Fu did in this paper, is they… they said, I don't know what we should measure. Let's measure 13 of these things, right? Or some number, right? Take a dozen of these numbers.
David Bau: And then every one of these checkpoints we have, imagine that it gives us these, like.
David Bau: 13 line plots that look like this, right? All over the place.
David Bau: And then they said.
David Bau: So, what does it tell us? And then what they did is they… they ended up training…
David Bau: a hidden Markov model on top of the… this… this… this, like, 13-dimensional line plot, saying that, okay, if this is high, this is low, and this is medium, this is high, this is low, right? Or, like, all these numbers, that means that we're in state X.
David Bau: And when you're in state X, then you have some probability of going to state Y, where this other number is high and this other number belongs, right? Does that make sense?
David Bau: And… and they… they… they said… and then they did, like, thousands of, like, you know, training runs of some networks.
David Bau: And then they said, when you're in this one state, then…
David Bau: You have some probability of going to the next state.
David Bau: You know, at every iteration. You have some progress going to the other state. And so they did this whole beautiful analysis of, like, what training dynamics looks like.
David Bau: Where… Where, like… It's like, it's like this picture of crystallization, like, oh, our mixture…
David Bau: Of mechanisms is like this.
David Bau: And then after you train a little bit, and there's a chance that you go this way, your chance to go down, you might… you might start crystallizing your mechanisms in one direction, you might crystallize your mechanisms in a different direction.
David Bau: And so… so they sort of… they did this analysis. So this is probably beyond what you would do, but I just want you to be aware that
David Bau: It's another approach.
David Bau: people have.
David Bau: Yes, here, states are basically the set of those linear algebra Merry Christmas.
David Bau: Yeah, so every state is… so they had, like, something like a dozen observable variables. The observable variables were these aggregate statistics of the network.
David Bau: Like, what's the average norm of the weights or something?
David Bau: And so, every one of these hidden Markov states, the hidden states, corresponds to, some, some set of visible states, you know, and
David Bau: And, and so, yeah, so they're asking the question.
David Bau: If you sort of partition your visible states into these underlying hidden states, then what's an economical way of representing everything? Like, if you said, oh, you…
David Bau: If you discretize everything, you said, oh, I want to just have a half dozen hidden states, then which one of the hidden states would there be, and what would the transition be between them?
David Bau: That would describe the training process.
David Bau: And… and so that's, like, a classic… did Markov model training problem.
David Bau: And so they just plot all their data into that.
David Bau: into that machinery, and they got these beautiful plots out for what the, what the state transitions would be, and they look like. This type of thing.
David Bau: That make sense?
David Bau: Yes, that makes sense. Yes.
David Bau: Next question was, did they observe groking, and if so, could they explain gulking with smart objects?
David Bau: Yeah, yeah, so they… I don't know if they used the term rocking, I don't remember ordering, but I think this is, you know, motivated by the same thing, is that basically, when they have a transition like this.
David Bau: Like an arrow? I think that what these arrows are, for them.
David Bau: is the equivalent of grocking. And what they would say is that grokking doesn't just happen once when you have a more complicated system. You have lots of little transitions that happen when the network
David Bau: it seems like it becomes… has a different flavor. And it drops, like, from here to here, and it grabs 4, sometimes it drops, like, here, and then here to there.
David Bau: Very cool. That makes sense?
David Bau: Yeah.
David Bau: And so… States capture the time as well.
David Bau: Yeah, so I think that they're… the way that they're thinking of it is, you know, you can see these .997, so most of the… they say that
David Bau: So remember in the graphic paper,
David Bau: There's this big variation between how long it will take.
David Bau: between… sometimes it'll take a long time to grok, and sometimes it'll take a few times, you know, less time to grok. So you can think of that as kind of a force-on process, right? You can think of it as, like, well, every time you run gradient descent for another 100 steps.
David Bau: There's a chance that you might stumble on the solution.
David Bau: But then there's also a chance that you haven't stumbled on it yet, you might have to just look longer.
David Bau: And… and that's what this… represents. Oh, after some more training steps, there's a 98.7% chance that you haven't
David Bau: the transition out yet, you're still stuck in state one.
David Bau: But then there's a small, you know, 1% chance, or whatever, that you found the thing, and let's get to that state. And so, that might happen fast. If you're lucky, it might take a long time. If you're not lucky.
David Bau: That's sort of the way they model it with us.
David Bau: Fantastic.
David Bau: Yeah. And so I, I like this. I think it's, you know, this is still the paper, I think, that…
David Bau: you know, had the most systematic approach to doing it. But it's still opaque, so, like, if you just collect all these numbers, you get this diagram, you still don't know, like, what's in each one of these.
David Bau: false, right? And you don't really know if this is, like, real or not, right? It's just… it's just, like, kind of a way of looking at it.
David Bau: what's going on? And so, for you, for you guys, I kind of recommend this. This is a lot more transparent, like, you know, each one of these lines.
David Bau: Corresponds to some mechanism, some subset.
David Bau: I'm dead, and you know what it means.
David Bau: Right? Now, it'd be interesting if you did this analysis, and you lined it up with this analysis, and say,
David Bau: You know, ball number 1 corresponds to this being high and this being low in the real data.
David Bau: I don't think anybody's done, like, a triangulated paper like that. Trying to relate history.
David Bau: Make sense?
David Bau: All right.
David Bau: So, okay.
David Bau: Oh, that was a lot on cracking.
David Bau: I better go through the next one faster, yes. I just have a question on, like, after we do these analyses on dynamics, how people use the information to… for some more practical benefit, besides just understanding what's going on?
David Bau: better training dynamics or something? It's a good question.
David Bau: I don't know, I'm not a trained dynamics person, so I'm not in the business of, like, defending, this lab work. You know, Naomi Sapra.
David Bau: Who is, like, a training dynamics?
David Bau: Yeah, neither.
David Bau: Down the street.
David Bau: I think his name is at BU now. Can I ask her?
David Bau: Yeah, what would be good? So, like, I think that you might have a sense… so, okay, so there's some controversies.
David Bau: That are maybe addressed by some of the training dynamics stuff. Like, some people think.
David Bau: And it can actually be… our fault.
David Bau: To train your network for the wrong amount of time.
David Bau: And… but then… but then there's different advice that you get, like.
David Bau: Don't train your network… don't give up on your network training too early.
David Bau: You should train longer.
David Bau: Right, then it'll get better. Oh, but then some other people say, oh, no, no, don't train it too long. If you train it too long, then you're wasting CPU, and then eventually it actually starts getting worse.
David Bau: Right, so this whole discussion… You know, it has to do with train dynamics. There's also other stuff.
David Bau: Which is… which interacts with this in practice.
David Bau: Which is,
David Bau: You know, one of the biggest things that all the giant companies are doing is they're coming up with their mix of candy data.
David Bau: And so there's another access here for training dynamics, just like, well, how does the mix of training data during the process influence all the training dynamics that we've learned? And,
David Bau: And for that, Jamie, I haven't seen much.
David Bau: of data dynamics papers from that point of view. Maybe I'm missing the literature. But in industry, this is…
David Bau: This is, like, a major questioning.
David Bau: Yes, maybe I'll… I mean, I don't remember the exact paper, but I remember one of the events last year. Yes. They were doing, like, generative modelings of, like, images, diffusion models, and they had this
David Bau: kind of…
David Bau: like, theoretical, like, a formula, basically, predicting when those stages would occur as a function of how much data you had. So you could say, for this training run to be worth it, we need, like, this amount of data.
David Bau: For the, window of useful models to be… is it, reasonably big, or…
David Bau: That's right. So, you've heard of scaling laws, right? And so, obviously, training dynamics is related to scaling laws, although scaling laws are… they usually look at, like, a small number of metrics.
David Bau: how good is the network in general? Whereas Train Dynamics, people…
David Bau: Try to, you know, generally try to cut it up with a finer tooth.
David Bau: Boom.
David Bau: Okay.
David Bau: We'll go through path scopes quickly.
David Bau: Actually, sorry, it's a new idea. So, there was a question…
David Bau: From Armita. Is Armita here? Yes. What was your question?
David Bau: So, yeah, I wanted to, see how… where this stands compared to our previous… Yeah, how is it related to the other stuff? Yeah, and when we prefer this one?
David Bau: Yes, and so, when you prefer things in general, I think that, you know, it's gonna give you the same answer I've given a few times.
David Bau: Which is… That you… you probably want to not… Lean on too few methods.
David Bau: And you want to find complementary methods, once you have a hypothesis that you're advancing. So if you're right, you say, oh, I think that we figured out the condition of this.
David Bau: You'll know from your own experiment.
David Bau: Like, what the strongest evidence that you are most excited by? So, obviously, you present that first.
David Bau: But then, after you present that, then you have to realize that the reviewers
David Bau: you know, or any reader will look at that evidence and say, well, you know, that comes with caveats. Every single one of the methods we have comes with caveats.
David Bau: And so you can say, well, we don't just see it that way. When you look at it this other way, then you also see something interesting. And when you look at it a third way, you also see it. And so the more different ways
David Bau: you can see the same phenomenon, the more people will believe that you have a handle on what you found. And so… so I'm mostly talking about patch scopes, just to give you another
David Bau: Hello? For your basket?
David Bau: Now, it's a good question. How does it relate to probing?
David Bau: Patching or steering.
David Bau: I'd say that this is most related to logic lens.
David Bau: Right, that method that we use, you know, the first week of class just to learn how transformers work.
David Bau: And it's, you know, Logit Lands is a serious method, like, you should, you shouldn't use a lot of logic lens to look inside your things sometimes. It doesn't give you that much insight.
David Bau: for Legendlands, and so you might not have anything to report.
David Bau: But there are ways of making logic lens more powerful, and this is one of them, so I just wanted to…
David Bau: to, to share that with you.
David Bau: And so, so there's a… there's been a few papers that have worked on making a larger lens more powerful, and they all have
David Bau: There's time for public forum. So one of them came out of our lab, it was called Future Lens of… and Future Lens does that… so what does is instead… so what does LogitLens have to do? Logit Lens takes…
David Bau: Other than Steve, did you want to know.
David Bau: What's going on in the head stage?
David Bau: And it decodes it to a word, or a set of words, right?
David Bau: And it decodes it to a word, by patching that state
David Bau: to the decoder head of the transformer. It just says, you know, I know this is in the middle of the network, but instead of having you process it, I'm just gonna skip to the end, I want you to tell me what this decoder can do right away.
David Bau: And then it gives you a word that you can write down, and it, you know, sometimes it gives you some interesting insights from doing this, right? Certainly more interesting than just looking at a thousand numbers, right? You can see what words it correspond to.
David Bau: Right? So, what FutureLens does is it says, Well, you know…
David Bau: This vector here, it was never meant to be decoded directly, in the…
David Bau: decoder head of the transformer. It was meant to, like, pass through a bunch of layers.
David Bau: Which, who knows what they do, right? You know, maybe, you know, do some transforming on it until it finally, like, comes up
David Bau: with… with the answer.
David Bau: And so, so, you know, obviously, if we just run the transformer, it'll say something.
David Bau: But the problem with just running the transformer is that in a lot of contexts, when you just run the transformer.
David Bau: It'll… it has, like, all this other context besides… when you just run the transformer, it doesn't tell you just, like, what's in this.
David Bau: thing. It tells you, like, what's going on everywhere. There's all this attention, bringing information from everywhere, so you don't know
David Bau: When it finally emits something, if it really has to do with this.
David Bau: with this hidden state, or if it has to do with all the other information I was bringing in. So what you could do is.
David Bau: You could pluck this out, And you could find a clean transformer that doesn't have any of the contacts.
David Bau: Right, so, you know, I don't know, maybe there's a whole newspaper article about New York City.
David Bau: And then, and then this sensation shows up, and I said, New York City.
David Bau: So, is it saying New York City because of the city and state?
David Bau: or the saying, New York City because of the rest of the newspaper article.
David Bau: Well, one thing that we can do is, we could just cut out the rest of the newspaper article, get a clean transform with no context.
David Bau: And we could just, like, jam this didn't sit in.
David Bau: I'm gonna say, I wonder if it says New York City.
David Bau: And sometimes it does!
David Bau: It does say New York City anyway, which means
David Bau: that hidden state encodes for New York City.
David Bau: Now, it turns out that this usually doesn't work.
David Bau: If you remove all the context, and you, you know, put some hidden state in.
David Bau: You know, like, the lack of context
David Bau: tends to make a transformer really, really think that it's, like, at the beginning of the document. Like, once upon a time, kind of state, right? So even though you plop in a hidden state that says, you know, I'm including for… you should say New York City.
David Bau: It'll, it'll link.
David Bau: They'll, like, be so overwhelmed with the other evidence, they should say, like, once upon a time, it won't say.
David Bau: what's in the hidden state. So, what Future Lens is, is, it's like a… it was a, a recipe
David Bau: For how to configure a transformer so that it tends to tell you What?
David Bau: This token would be.
David Bau: And… and the way it worked was.
David Bau: Instead of having the transform be perfectly clean, with nothing coming before it.
David Bau: Futureland says, we're gonna come up with some big text, That we're gonna put there.
David Bau: That instead of it being the very beginning of the document, we'll take 10, the whole… get 10 words.
David Bau: That we'll put there, and then we'll find the optimal 10 words, and that'll be just the same words for everybody.
David Bau: But when you put these 10 words there, like, the 10 words might say something like,
David Bau: You know, be honest about what you're thinking right now. What you're thinking about right now is…
David Bau: This, and then you'd plant the thought in the model, and then…
David Bau: And then it says New York City, right? So that's what the idea of FutureLens is.
David Bau: And we tried…
David Bau: prompt engineering it, come up with, like, prompts like what I told you, and it didn't work that well.
David Bau: But then what we did is we did prompt optimization.
David Bau: Where instead of putting real tokens in here, we did, what's called, you know, odd,
David Bau: soft token learning.
David Bau: Which we ran an optimization process to optimize tokens to do well at this task.
David Bau: And that's… so that's what FutureLens is. So it's… it's a… it's a transformer with 10 learned tokens that you seed it with.
David Bau: That are the prime and the transformer to tell you.
David Bau: What it thinks the token means.
David Bau: That make sense? So that's… that… so that was…
David Bau: That's the thing that Kiana and my lab did.
David Bau: A couple years ago.
David Bau: And there were a number of papers that did different things that were similar to this. And what PatchScope says, is it's a roll-up of these methods. They're sort of surveying
David Bau: you know, what's… what other people have done, including FutureLens, and they said, okay, well, in general, here's, like, the state of the art of what you would want to do. And… and so it's a… it's a generalization of FutureLens to do this. And so,
David Bau: So instead of having one fixed prompt, they suggest a bunch of interesting, clever prompts, like, so here's a really clever prompt that I wish I thought of.
David Bau: which is really interesting. There's a prompt that says, Cat goes to cat.
David Bau: 135 goes to 135. Hello goes to hello. You can have actually a really long prompt like this, like.
David Bau: You know, the same word goes to the same word. Oh, you have some word in input, you gotta have that word in output. The same word in input, it's just a… it's a echo. Echo, echo, echo. Say the thing that you said, say the thing you said, right?
David Bau: And then… and then what you do is you put this thing in, you put a dummy token in, and then… and then you let…
David Bau: Then you'd patch in, You know, the representation, that you wonder what it means.
David Bau: And then you say, echo that?
David Bau: Right after it's seen, like, 100 echoes.
David Bau: Matt's gotta do another one.
David Bau: And then it tends to tell you, huh, that's Chef Ditos.
David Bau: So that's pretty neat. So they came up with a few different
David Bau: Cool tricks for doing this.
David Bau: Another cool trick that was in patch scopes.
David Bau: is that instead of patching, like, what… we were such purists on Future Alliance, we were like, if you're at layer 5, then the transformer expects to see you at layer 5, so patch into layer 5, that's where we put it.
David Bau: You're like, I don't know, you don't have to be such a purist.
David Bau: You can just stick that back at Layer 1.
David Bau: And they just measured it, and it tends to work better.
David Bau: And so, so yeah, so they had a bunch of, results like this, where they were just measuring, you know, how accurately they could recover.
David Bau: you know, a description of what's, you know, what's going on in that state. And, and, and they sort of, you know, they tweaked it and optimized the process a bit. So that, so, so if you want to do something that's like Watch It Lands, but you wish to, like.
David Bau: were better, or cooler, sometimes you might read the patch ghostscaper.
David Bau: That make sense?
David Bau: Yeah.
David Bau: Or you can also use LedgerLens, which is super simple, which doesn't require any configuration or anything, so this is another possibility. It's also related to…
David Bau: patching.
David Bau: Right? Because, you know, you patch from one model to another.
David Bau: So, everything that you're doing for your patching experiments, it would actually be the same.
David Bau: The same principle.
David Bau: Oh, there was a question from Ananya, or Yuchen.
David Bau: Yeah, like, what's… yes? Oh, yeah, I guess the question was, like.
David Bau: It kind of comes with this thought process that you're basically asking the model again, you know, hey, you thought about this before, what do you think about this again? Yeah.
David Bau: how do you trust that the model will produce, like, the right? Yeah, I don't know if you can. It's a really good question.
David Bau: It's… it's… it's… so, I think that if you're really, really worried about deception, and, you know, going back to the model to ask again, like, doesn't totally…
David Bau: You know, solve the problem?
David Bau: At least you're running the model in a way that it wasn't trained to do, so, right? So, like.
David Bau: so you're not overfitting on…
David Bau: you know, it's training, like, if you believe that there's some way of setting up the network to be honest, like, this is, like, an honest situation, it's like, oh, the network is just gonna say what it has to say, it's an honest state of mind, then maybe, but there's no… there's a proof of this. I think the proof is empirical. They're like.
David Bau: We just… we just measured this on a thousand things, and it seems to work okay.
David Bau: And, so hopefully with the 1001 thing, it's… maybe it'll work okay, too?
David Bau: But it's not… it's not a proof.
David Bau: But…
David Bau: It's just building confidence. It's just another method. I probably wouldn't use it alone. I would use it with other things. I have another question. Yes. In the case of the Futurelands paper, you used two different transformers. Oh, it was the same transformer. Oh, okay. Yes. So were they trained on the same… like, if we were to change training data, or basically patch.
David Bau: Activations from a similar, like, all different therapy. Yes, there's another paper from the Transloose paper, I forgot the name of the part of the paper, but Sarah Schweitman and so on.
David Bau: Where they do this, but they train a different transformer to look at the states. So, you know, when you…
David Bau: We have some state.
David Bau: And you want to, like, get a description of what it is, and why, you know, why does it have to be the same transformer? It could be a very different one. Just train down the problem of understanding the states of this one. So you can do that, too.
David Bau: That's a more intensive process. So, like, why would you expect some other transformer to understand the states of this one? Well, it wouldn't. It has no reason to, so it's almost like kind of training from scratch. It's got to learn this foreign language. This one is sort of funny, because
David Bau: You can see how this transformer would already kind of understand the internal language of itself.
David Bau: Right, so it's like a lot… it's less tuning that you have to do, maybe just procure.
David Bau: Yeah.
David Bau: And so,
David Bau: But yes, but that's another approach. It's a very good question, and it's a very reasonable approach, and it's another one of the avenues that people could take.
David Bau: And so
David Bau: Okay, so that's… that's patch scopes. I like the… I like the, how do we trust a model to explain its own representations questions? It's very deep.
David Bau: And it's… that may be one of the fundamental questions of our view.
David Bau: It's, like, one of the things that, you know, we puzzle of.
David Bau: So, okay, so the last paper, I think, is…
David Bau: the one that I think is the most important.
David Bau: Out of this, but it's the least likely that you would actually use this in your…
David Bau: your research. It's actually a fairly difficult technique to use, but I want you guys to be aware of it, because I just think it's really important.
David Bau: And so this paper was about chess. It was about AlphaZero, which everybody knows can, like, beat all of us in chess.
David Bau: probably beat everybody on the planet in chess. It's like a superhuman chess player. And so whenever you have this question of what does it mean to have superhuman AI, you should remember that we already have superhuman AI, we have superhuman chess players that can beat you, not just through, like.
David Bau: you know, throwing huge amounts of compute at you, they can beat you the same way that if you go to Harvard Square and you find those chess people that sometimes hang out by the subway station with all the little chess boards on the…
David Bau: On the tables there, and they tie their hands behind their back, or whatever, right? And they can play, like, 12 chess people at the same time. They just go, and they…
David Bau: move again, they move again, and then they beat everybody, right? They don't have to think about your game, because they're so good at chess, they can see the future, just as they get the board. You know, Alpha Zero works the same way.
David Bau: And…
David Bau: And, you know, so… and it works better than most professional chess players. So, in the paper, they have this example of, I'm not a chess player, the author, Lisa Schott.
David Bau: was a professional chess player, and Lisa looks at this board and says, you know, obviously anybody who's, like, a pro chess player would move a certain way.
David Bau: But the weird thing about this state is that AlphaZero would move a different way. And, like, the exact… like, there's many, many situations like this, where, like, every human who's, like, trained in chat would go one way, and AlphaZero does a different thing. It's like, for them, this is, like, concrete evidence that
David Bau: I mean, Ed beats everybody. Like, that it has superhuman knowledge, it knows something about this move that people don't know.
David Bau: What does it know?
David Bau: That's the puzzle.
David Bau: So what is AlphaZero? It's what they call a positive value network, right? You can think of it… so basically, it takes a chessboard.
David Bau: as input, And then I give some assessment of, like, how good this chessboard is, Right?
David Bau: And you can use that to search over different possible moves. Oh, if I move here, does that make this chessboard better? Does it make it worse if I move there? Does it make it better or worse? You can run this whole tree search.
David Bau: And… and… and that's… that's how… this network does, it basically…
David Bau: Gives you this quick assessment of our business.
David Bau: And with a very shallow search, you can just use it to pick, like, what the best move is.
David Bau: You can run a deeper search, too.
David Bau: to see, like, all the different possibilities that we call it Monte Carlo pre-search.
David Bau: So you could go 3 or 4 moves deep and generate, you know, hundreds or thousands of different
David Bau: Perspective, different possible future boards.
David Bau: And then you can ask, you know, what do you think of these boards?
David Bau: And the interesting thing is the way they train this network is that after you do this whole Monte Carlo Tree search, now you've got this estimate of, like, what the best board is, is possible for you, what the best board for your opponent is, what they would choose, and it gives you a much more accurate view of, like, what the future holds, because you've done the search out 5 moves ahead.
David Bau: And then, then you say, well, what was the original estimate when I just looked at the board without doing the search?
David Bau: And it's not quite as accurate as the estimate I get from doing this 5-deep search.
David Bau: So let me train it.
David Bau: Let me update.
David Bau: my… Little… quick.
David Bau: assessment to correctly reflect, like, what a much deeper search would have had. Let me memorize
David Bau: the output of this deeper search into this network. And so that's basically how the officer on… style,
David Bau: deep network training things work. You just have them play lots and lots of chess games, do lots of… lots of money college research.
David Bau: And then try to memorize the output of Monte Carlo research into this policy-guiding network.
David Bau: And, and so… so the question is… So when you're done.
David Bau: You don't even have to do the money college research anymore. You can just play against the Policy Value Network, and they can still beat you.
David Bau: Right?
David Bau: Because it's like guessing what… how the search is gonna come out, and it's usually right.
David Bau: And so, the question is, what the heck is going on inside this positive value network?
David Bau: Like, are there superhuman concepts in there?
David Bau: And it's weird, how do you find a superhuman concept? Like, here's my questions.
David Bau: Right? How do you find it? It's like, you guys are looking for concepts that you can write down.
David Bau: Like uncertainty in a network, or power relationships, or political leanings.
David Bau: You know, you know, attribution
David Bau: attribution of words, spatial… spatial understanding. But for superhuman concepts.
David Bau: We don't know what they are. We can't guess what they are.
David Bau: Even if they showed it to you, you probably wouldn't recognize it.
David Bau: Right?
David Bau: We certainly don't have any words for it. You're not gonna look… you're not gonna find it in the library.
David Bau: Right?
David Bau: I don't even notice it's there.
David Bau: And then how do we learn it?
David Bau: What a cool question! Isn't this a cool question?
David Bau: Right?
David Bau: This is a cool question.
David Bau: You know, it's such a cool question, I think it's, like, the most important paper.
David Bau: In the last couple years, this one.
David Bau: And it's hard to do, and they have a formula for doing it.
David Bau: So, there's a few pieces of evidence. So, first of… first of all, they're like.
David Bau: Does AlphaZero even contain your concept? Oh, Grace, you've had this question. Ask a question.
David Bau: Yeah, so they, they, if I remember it correctly, they would, like, take the ways that
David Bau: human users, human people would play, and AlphaZero would play, and then they would do some sort of thing to find the dimensionality, like, the rank. Yes, exactly. And then they claimed that, like, oh, if the rank…
David Bau: Is higher for the way that
David Bau: AlphaZero plays, then that means it has more concepts. Yeah. But I was curious, like, that is proof that there are concepts that humans don't.
David Bau: Or at least not in how they… not reflected in how they play. Okay. And so I was curious, like, it seems like with LLMs, for example, they're very deterministic in how they talk. Like, it seems like we could have access to different concepts, but have a lower…
David Bau: Right, like, or just more narrow in the sense… Yeah, yeah, yeah. Yeah. Yeah, yeah, so then, so I think that's right. So I think there's… the proof… okay, so the question is.
David Bau: Like, is After Zero actually know something that we don't know?
David Bau: And so, the first experiment that they did… well, one of the first ones was this one that… that Grace is asking about, and… and… and it's… it says, it's so…
David Bau: you could feed AlphaZero's own chess moves, like, halfway through an AlphaZero play game.
David Bau: and you have some chess configuration, you can feed it into the AlphaZero thing, and you get some vector representation of the chessboard, you know, near the end of the network.
David Bau: And you could just feed through millions and millions of these, and you get millions and millions of vectors.
David Bau: And the major vectors will, like, make some big cloud in vector space, right? Now, the interesting thing is that cloud has a different shape.
David Bau: as the cloud that you would get if you fit in a different data cent. So the cool thing about chess is there's so many human-play games that are all recorded in great detail, that you can just go download millions and millions of human games, and you can also feed the human games into AlphaZero's policy value network, and you get a different cloud of vectors.
David Bau: Right?
David Bau: And so… so the first thing they did was they said, I wonder…
David Bau: I wonder if these clouds are different from each other.
David Bau: Right?
David Bau: And… And they are indeed different. So there's… you can measure it at different… layers.
David Bau: And so, at the input layer for the network, They… they encode…
David Bau: the chessboard, and then I guess it's this…
David Bau: you know, what I, I don't know what 8 comma, 8,119 is. Okay, so…
David Bau: I'll just say 7,000 dimensional, or 7,000 rank.
David Bau: representation.
David Bau: Oh, 8x8, oh, I see. 8 times 8 times 119, yeah, so… so they have 119 commercial vector for every square, and they have an 8x8 board.
David Bau: Got it.
David Bau: I'm such not a chess player. So you probably, if you multiply these numbers, you get, like, 7,000. Alright, so they have this 7,000 dimensional thing.
David Bau: And if you feed in all the inputs, It puts, like…
David Bau: Like, human data, if you look at the dimensionality of this cloud of inputs.
David Bau: It turns out they're not using all 7,000 dimensions. It forms approximately a 730-dimensional cloud.
David Bau: Right? It's, like, pretty flat in that space. Like, real chess games aren't just a random array of pieces.
David Bau: I think there's some structure, and this is, like, what the structure looks like in linear algebra terms. And actually, AlphaZero has a little less diversity there, it's, like, two dimensions short.
David Bau: Right? So, but about the same number.
David Bau: Right. So, but the interesting thing is, after you have it.
David Bau: Process it, and think about, conceptually.
David Bau: But it goes… it goes through, like, 19 levels of transformers or something.
David Bau: Right?
David Bau: You get to the 19th layer, or the 23rd layer, and then the situation inverts.
David Bau: Now, human data has, like.
David Bau: 7,000 dimensions out of a possible maximum of 16,000 dimensions, but the alphaZero is, like, a couple hundred dimensions larger, right? So, I don't know what that means at all.
David Bau: Right? It's not, like, a standard unit analysis, but it was… it's kind of interesting.
David Bau: And, and so… So they said, oh, maybe this is evidence.
David Bau: that it knows.
David Bau: It's representing something richer The dimension… this counting argument at least tells us
David Bau: That we could find some vectors.
David Bau: In the AlphaZero representations, of the representations of alpha serograms. There are some vectors That don't exist.
David Bau: It means that we could, if we look at this linear algebra, it gives us this ranking function, this very interesting ranking function.
David Bau: Where you could go over all the AlphaZero games.
David Bau: And you could say, how close are you
David Bau: To the cloud of human games.
David Bau: Right, according to the presentation.
David Bau: And some things will be, like, pretty close to the human gifts, to the 7,000 individuals, right?
David Bau: I think something'll be pretty close.
David Bau: But other board states will be very non-human-like.
David Bau: They're the alien things.
David Bau: It would never show up in, like, human games. Does that make sense? The analyzing human games is, like, an average of all the years, because I'm assuming I have less moves than manuals, I don't know. It's not average, I think that they… well, you're right, it's an aggregate of all the things, and when you're doing this dimensionality thing.
David Bau: There's probably some averaging effects of, like, how, like, how much do you count?
David Bau: Somebody, like, if the same game state shows up a thousand times.
David Bau: And then another game save is very rare, how do you weight it? And I don't know the details of, like, how they weight all the things, so it's a reasonable, it's like a… it's a reasonable, you know, technical question, and we'd probably have to ask me other authors to ask how they did that.
David Bau: But I'm just continuing on telling you the general story without knowing intent about my paper.
David Bau: And, so… so… so now…
David Bau: So, this is instantly interesting. Games that are not human-like.
David Bau: Right? And so they found… Really…
David Bau: Really, 3 things that they needed to do.
David Bau: To isolate interesting vectors.
David Bau: So they have this vector space.
David Bau: And now what they're after is they want to know the most interesting vectors In that vector space.
David Bau: And it's… it's a 16,000 dimensional vector space, and there are 8,000 dimensions that are, like, being used.
David Bau: So how do you find, like, interesting 8,000-dimensional vectors?
David Bau: They, they, they just, they, here's what they did. They had this three-step plan.
David Bau: So, first of all, they made, like, a list of candidate concept vectors using good old-fashioned chess knowledge.
David Bau: So, they analyzed a gazillion AlphaZero Games.
David Bau: and they use classical techniques, probably the Monte Carlo tree search that AlphaZero was doing. They could look at the Monte Carlo tree search at every step, and they say, sometimes the Monte Carlo tree search is really unambiguous.
David Bau: Like, there's one best move, you're just gonna do it, it's not a very interesting position. But sometimes, Mari Kalchrusich
David Bau: has, like, a dilemma. Like, it… like, there are two paths. They look pretty good. These are, like, the choice points.
David Bau: You're in Chuscany.
David Bau: So, so they went through and they just did this classical chess analysis, and they just found all the interesting choice points. You know, probably, like, millions of, like.
David Bau: interesting conditions. Right, and they threw out all the rest of the interesting things. They probably threw out 99%.
David Bau: banned.
David Bau: And they said, okay, of these interesting positions.
David Bau: how many have humans seen before? Like, how many are, like, emblematic of, like, what would show up in a human game?
David Bau: And so they did this linear algebra thing.
David Bau: I said.
David Bau: Oh, you could project it onto the human 7,800 conventional vector space, subspace?
David Bau: How… how… how far do you have to go?
David Bau: to become human-like? How far is that projection?
David Bau: And you can project down to the AlphaZero subspace. How far do you have to go to beyond, like, the typical AlphaZero subspace?
David Bau: And then…
David Bau: And then if you have to go really far to be a human, but not very far for Alzo, then it's very emblematic.
David Bau: Of, like, an AlphaZero game, and not very emblematic of a human game. So they basically took the difference between these two distances.
David Bau: And they, they, they, they selected the top 1% or something like that.
David Bau: For games that are most AlphaZero-like and least human-like, the most novel.
David Bau: Right?
David Bau: And then, they still had too many vectors.
David Bau: So they're like, we still need to throw away, like, 99% of our vectors.
David Bau: So then they had this third thing, which I thought was the most clever thing they did.
David Bau: Which is they filled out… filled out vectors that are not Learnable.
David Bau: What the heck is that?
David Bau: So, for this, what they did, is…
David Bau: For every one of these vectors.
David Bau: They made prototype training sets
David Bau: for the concept. So, for any given vector,
David Bau: they found lots of games where the… the AlphaZero was like, yeah, that's, like, a typical situation.
David Bau: For this vector. And then they did the Monte Carlo true search to say, in this situation, you're choosing between these not-as-good situations and these better situations. And they would make this training set
David Bau: That… of, like, example board positions, where this is, like, the contrast between, like, the good path for this
David Bau: concept.
David Bau: And the bad path for this concept. So, I'm conflating a couple things that they did that are very similar in the paper. So, one of the things is, how did they get these vectors in the first place?
David Bau: They had all these board positions. How do you get the vector from a board position? Actually, what they did is they didn't just take the vector from the board position, they actually formed these contrast sets, and then after they formed the contrast sets, they would use it to create a probe.
David Bau: For what probe at that board position would tell the difference between the good path and the bad path?
David Bau: that monocollics research. So, monoclogic Research, it was all choosing between a couple
David Bau: Couple different things. They have these choice moments.
David Bau: And so they would make a probe.
David Bau: For what, vector representation was a good probe for whether you went on the good path or the bad path.
David Bau: And that was the vector that they were interested in.
David Bau: And then they went and they… they looked for, games that had this vector in its representation, and… and then they… that allowed them to find a lot of similar games with a similar situation.
David Bau: And they, and then they would dump out
David Bau: These sets of board examples of, like, these are examples of
David Bau: Good examples of this concept, and anti-examples of, like, missing this concept.
David Bau: Right.
David Bau: And then after they had all those examples, Then they would go to… a chess engine.
David Bau: an AlphaZero Trust Engine, just like…
David Bau: Aussie Value Network, and they would train it.
David Bau: On that data set, on those, like, you know, 100,000 examples of
David Bau: Good and bad examples of the sector.
David Bau: And… And then after they did that.
David Bau: They would find that if they traded on this complicated data set for a while, that,
David Bau: oh, I don't remember C and R and all the things, but basically, in blue, I think, is, it would start doing better.
David Bau: At the concept that you trained on.
David Bau: Right? But then not only did it do better in the concept they train it on, sometimes…
David Bau: It would do better on related concepts, on different vectors.
David Bau: Right.
David Bau: And sometimes, it would actually do better at general gameplay.
David Bau: as well.
David Bau: Great.
David Bau: And so, and so, what they did is… now, they wouldn't happen like this observer.
David Bau: Often when they chose these sort of training sets, they would put it through this procedure, and the network wouldn't learn. Like, these numbers wouldn't go up.
David Bau: Right? It wouldn't… it wouldn't get better at chess.
David Bau: And so, in those situations.
David Bau: The code vector, or whatever they found, was not doing a good enough job at coming up with
David Bau: Distinct things that could be exemplified, like a lesson that could be exemplified by sharing, like, chessboards.
David Bau: And so… So they… so… so this worked sometimes, it worked well, sometimes it worked badly.
David Bau: Self?
David Bau: They said, we're just looking for the best concepts, so again, they threw away 99%. They said, they just tested this on a lot.
David Bau: And then they only kept the top 1% of vectors that had the best transfer.
David Bau: In this way, they were the easiest to distill.
David Bau: And to lessons that were learnable.
David Bau: Right? And then finally, that gave them a handful of vectors, like a human-sized set of vectors, I don't know how many…
David Bau: I should ask this, what the number was in the end.
David Bau: And then they took these vectors and they said, okay, Now, what we'll do…
David Bau: These will teach them to humans.
David Bau: And they put them… they put them in,
David Bau: They printed out all these little chessboard manuals, Maybe that's…
David Bau: And they said, according to this chess book.
David Bau: Here's a really challenging chess puzzle. Tell me which way you would move.
David Bau: And then ask the humans to answer, and they know which way is the good way and the bad way.
David Bau: And they'd score the humans on this, and they'd give them 20 chess puzzles or something like this. And then, after that, they would say, okay, so let me tell you the secret. Actually, you were wrong for most of these. This is the way it goes, but I'm gonna give you a book.
David Bau: Of a dozen examples of, like, the good examples and the bad examples of each one of these concepts.
David Bau: And you can just study this book, just the same way that we trained AlphaZeros.
David Bau: And then after they did that, the people would have a few hours to study. These were all, like, grandmasters, these were all, like… it was a Google paper, so they paid these people to, like, come to the Google campus and hang out and have fancy food, and talk about chess all day, and…
David Bau: like, learn from AlphaZero.
David Bau: They loved it, because they actually learned these new chess techniques, and then they would come back and they would take the quiz with new board positions they hadn't seen before, but according to AlphaZero, like, the same vector.
David Bau: And… and they all… We're able to solve the puzzles now.
David Bau: They all learned the superhuman concept, and several of the chess masters were like, What an interesting day.
David Bau: I did learn something new about Tristan.
David Bau: And they try to articulate, oh, you know, I learned that it's important to develop queenside power and this and that, whatever, right? They have some whatever words that they tried to use to describe what they were doing. But, like, some new concept they had never seen before.
David Bau: And so then, what the authors of the paper like to say is that one of their star participants who was the most enthusiastic about the day went on two years later to become the chess champion of the world, still reigning champion today. Must have learned something that day. And so…
David Bau: But… but anyway, so that's… so that's… that's the story of this, and I just wanted to show you what this research is. I think it's pretty interesting. The, I don't have time to go over this other thing, maybe, you know.
David Bau: Have we talked about the function vectors paper? In this class, we have, right? And so, the exercise that we're going to do next Tuesday has to do with function vectors a little bit, so I, you know, I'd mainly…
David Bau: Have you think of, like, a contrastive.
David Bau: A pair of prompts that you might want to, you know, patch between that has to do with your research.
David Bau: And I think one of the things that we'll do is we'll try to pull apart the attention pens to try to factor individual attention pens. Try to kind of step through some of these things.
David Bau: Okay.
David Bau: Alright guys, thanks a lot.
David Bau: Diff.