David Bau: I think we get organized, then this end up on YouTube at some point.
David Bau: If that's okay? Sure, yeah.
David Bau: Okay, so, welcome everybody. I'm delighted to welcome Robin Jaft to the seminar.
David Bau: Robin is an assistant professor in computer science at University of Southern California.
David Bau: Where he leads the… they call it the Allangro Lab.
David Bau: Which stands for… AI, language learning, generalization, and robustness.
David Bau: So, you know, so his research is focused on deep learning and NLP,
David Bau: And he's done a lot of work that I think that you'll be interested in. He's recently published papers looking at mechanisms underlying in-context learning, numerical reasoning, factual
David Bau: Factual reasoning, self-correction, which is pretty interesting.
David Bau: And he also studies robustness, uncertainty, calibration, long-term planning, you know, reasoning models.
David Bau: Copyright issues, digital water marketing, So, all sorts of stuff.
David Bau: That I know that you're going to be really interested in. A lot of overlap with interest, I think, around the room. So, so we're lucky to have him. Welcome, Robin, and looking forward to your talk. Yeah, thank you so much.
David Bau: Yep.
David Bau: And yeah, I feel like, conversely, I'm very excited to be here, because I know, like, you know, a lot of folks in this room are doing stuff that, yeah, again, like David says, is very…
David Bau: Related to stuff I've been doing, especially on the interpretability side, and I'll talk about a couple of, like, brand new projects, one of it kind of extends our work on copyright and things about memorization in general, and another one is actually our, like, foray into, like, doing stuff in the medical domain, which I know is also a topic for folks here.
David Bau: Okay, so, yeah, to start off, maybe just… I want to give, like, a, you know, eye-level, you know, thoughts about, like, what… what… how I'm thinking about our field right now, right? So, I think it's a… it's a very exciting
David Bau: time to be, you know, an NLP person, right? So I've been… you know, I guess I started doing NLP maybe in, like, 2014, 2015.
David Bau: And back then, it was, like, there were, there were, like, some… some specific use cases, you know, like, like, I mean, probably most notably machine translation for the sorts of models that we were studying. But, you know, today, you, you go around and, like, you know that people are using
David Bau: the language models, the same things that we're studying for, like, all sorts of these tasks, right? So, so many people are using them, you know, as sort of sources of information now. There are all sorts of business use cases, like trying to, you know, use them for customer service.
David Bau: We know many cases of, you know, you know, various, you know, experts in other fields, like lawyers, right, using
David Bau: using LLMs to help them with their work, and there's many instances of, you know, people, you know, for example, getting medical advice from these chatbots, right? And so there's, you know, there's definitely a lot of, like, potential here, right? There's a lot of ways in which these LLMs can be
David Bau: impactful, and there's also a lot of risks, right? So, for example, we know that these chatbots, they can also be sort of sycophantic, they can reinforce
David Bau: People's, you know, ideas, even if they might be sort of conspiracy theories or other sort of, like, strange, you know, things that are not grounded in reality.
David Bau: On the business side.
David Bau: we know there are cases where these LLMs hallucinate and might do things that are, for example, like, not in agreement with even what the company wanted them to do. So this is a… this Air Canada thing was a famous case where they had some chatbot that agreed to give a customer a refund.
David Bau: It turns out that the company policy would have not given a refund in that scenario, and so the company tried to not give the refund. This went to court, and the judge said something like, well, clearly, a company is responsible for things that are on its website, whether it's a static page or it's something that is generated on the fly. And so, like, they had to go along with what their official chatbot had said.
David Bau: And then finally, I don't know if any of you noticed this, recognize this last graph. It was something I came across recently, especially as we've been looking at things in the legal domain. So this is, this is, a graph that's showing
David Bau: over time, how many, instances are there in, like, legal filings where there's a citation for a lawsuit that does not exist? So you can see, basically, the point is that, like, this is, like, flat for many years. There's maybe, like, a handful of instances where this has happened.
David Bau: in the pre-LLM era, and then since about 2023, we've seen this, like, exponential increase in the rate of, like, fake
David Bau: cases being mentioned in legal filings, you know, in many cases, the, you know, judges might notice this and, you know, call out the lawyers who have done this. Okay, so, like, when I look at this overall, I think there's, like, two kind of big observations that maybe sort of underlie a lot of the work I'm going to talk about.
David Bau: The first is that, I think the potential audience for our work, you know, as AI researchers, is just so much broader than it was, like, even a few years ago, right? So there's, like, all sorts of domain experts, as well as everyday users who are using these LLMs, right? There are all sorts of companies, you know, like, Air Canada is not a tech company, right? But they're interested in using these, these
David Bau: These systems,
David Bau: as well as many other companies, they're all thinking about how does AI, you know, how should AI influence, impact their workflows? And there's also a lot of policy-related considerations now, related to, you know, again, for example, we've done work with copyright, but that's only one of many sort of policy-related questions you might have about LLMs.
David Bau: And I think the second, you know, key thing that, that sort of
David Bau: synergizes with the first is that, you know, a lot of my work is trying to think about how can we build
David Bau: better mental models of LLMs? Like, how can we just, like, understand these things better? And I think part of the reason I want to do that is I want to be able to actually communicate with some broader audience about, like, what's going on, you know, what should be best practices when we use these models? How can we understand, you know, why do they have some, you know.
David Bau: Sort of behaviors that might be otherwise surprising.
David Bau: Okay, so, you know, towards these aims, I want to talk about sort of three overall thrusts of research in our group. So the first is, we'll call, like, auditing. So basically, like, you know, for some sort of large categories of risks, how can we sort of anticipate and assess, the likelihood of these sorts of risks?
David Bau: We're going to talk about our interpretability work on sort of dissecting how these models make predictions internally. And then finally, I'm going to talk about trying to just evaluate or benchmark
David Bau: these LLMs, and understand sort of what mistakes they might make, you know, in the wild. And I'll… I'll talk about, we'll have, sort of, one main project for each of these, and then, especially for the interpretability stuff for this audience, I've sprinkled in a couple other ones as well. But for, Friday, I'm going to talk about a brand new thing where we've, done a big study about
David Bau: LLM memorization and wanting to understand the risks of LLMs memorizing different sorts of sensitive information. In terms of interpretability, yeah, we have,
David Bau: there's a… I'll focus on this story about, you know, understanding how LLMs do sort of basic arithmetic problems. We'll also talk about some stuff with, related to in-context learning, and a couple other things as well.
David Bau: And then finally, I'm going to talk about, yeah, our, like, new work, where we've… we collaborated with some, some clinicians at USC hospitals, and tried to understand, you know, what actually could go wrong when you have, you know, patients or, you know, asking for medical advice, from… from LLMs.
David Bau: Okay, so I'll start with this work about memorization, and this is a really big, collaborative effort that we are, you know, hoping to submit to iClear. So this is led by my PhD students, Johnny and Amaya, with a lot of help from our collaborator, Afla and Krishna, who are at the Max Planck Institute.
David Bau: And, you know, a bunch of other folks at USC as well.
David Bau: So in this… in this work, we basically have, like, created a model suite, basically a family of pre-trained LLM models. You pre-trained all these from scratch, thanks to, like, a big compute grant we got. And these are all designed to give us, controlled experiments on understanding memorization of pre-training data.
David Bau: Okay, so just, like, you know, just, you know, so we're all on the same page, you know, so…
David Bau: It's well known at this point that LLMs do indeed memorize some things verbatim from their pre-training data. So, for example, this is, actually, on the left is a direct
David Bau: sort of screenshot from the New York Times' lawsuit, where they've sued OpenAI over, you know, basically alleging copyright infringement, because OpenAI trained on their data. And, you know, to prove that OpenAI trained on their data, they show that you can actually prompt ChatGPT, or at least you could at the time this lawsuit was filed, asking it about, for example, this, like, sort of,
David Bau: famous, article that appeared in the New York Times by Pete Wells.
David Bau: You could ask for, like, the opening paragraphs of this article, and it turns out that what ChatGPT outputs is more or less, like, a word-for-word
David Bau: of the actual article, right? So this was, this was clear evidence that, they had, you know, not only trained, but probably, like, trained, you know, enough to actually memorize this document very well.
David Bau: So this is a… this is, like, one of, I think, a few different
David Bau: sort of risks you might worry about when it comes to memorization, right? So there's, yeah, a lot of implications for copyright, but there's also things like, you know, if you're concerned about privacy, right? There might be various sorts of personal information that has somehow leaked on the internet, and if an LLM is trained on the whole internet.
David Bau: it might, you know, learn that information about you and might leak it if some, you know, attacker asks questions about you. And then there's the issue of, test contamination, right? So when we benchmark models.
David Bau: On data that also has appeared somewhere on the internet, it's often unclear whether we can trust those numbers, or if they're sort of inflated, because the model could just memorize answers that were already on the internet.
David Bau: So yeah, so all of these, sort of concerns, right, we can phrase as they're, you know, to understand these, we want to understand
David Bau: this question about, like, basically when and how well do these LLMs memorize, you know, all these different types of data?
David Bau: Okay, so what we did, like I mentioned, is we wanted to create some, like, like, you know, very rigorous, like, controlled experimental setup to understand, you know, these questions. But, like, you know, when can we expect memorization to arise in these different domains?
David Bau: And we wanted to make sure we did it, like, at a realistic scale of, like, actual LLM-free training, not just doing it at, like, at a smaller, sort of, synthetic training scale.
David Bau: So, you know, again, with some, like, help of this compute grant, we were able to train a few different pairs of models. So, this is how we do our, like, you know, control study.
David Bau: every time we do an experiment, we are going to train one model on just, like, a… what we call, like, a standard corpus, so this is just, yeah, some standard pre-training data. We use the DCLM corpus.
David Bau: And then we're always going to compare this model with what we call a perturbed model train on a perturbed corpus. And so this is, you know, we basically take the same starting pre-training data set, and then we inject different examples
David Bau: That basically correspond to these risks that I showed you on the previous slide. So we have, like, things like book passages that correspond to, you know, concerns about copyright. We have, sort of, we created some, like, fake, you know, private information, or, like, fake biographies.
David Bau: of people, to represent these privacy risks, and then we just also have, like, you know, examples from various test sets that we can inject. And whenever we inject things, we're gonna, like, kind of randomize
David Bau: randomly inject different examples at different rates. So some of these examples will randomly be injected, like, zero times, so they'll be held out. Others will be injected once, or four times, or 16 times, and so on.
David Bau: Okay. Yeah, a couple of just, like, sort of basics. So the… the scale, like, I think, I was, like, it was really exciting to be able to train models at the scale, we did, so we trained
David Bau: both 1 billion and 8 billion parameter models, on either 100 billion or 500 billion tokens. So this is, for example, larger on both dimensions than Pythia at the high end.
David Bau: And so this also allows us to look at things like, you know, how does, how does the… both parameter count and, dataset size influence these memorization behaviors?
David Bau: Yeah, for each, you know, each setting, we trained these pairs of models, and we also have several sort of variants we did to study other potential things that could affect memorization. I'll mention one of those experiments later. And yeah, we got this compute through, through this, through Nair, which got it from, NVIDIA DGX Cloud.
David Bau: Okay, so, before I show you the results, I want to say, like, I think my goal here in presenting this is I want to sort of argue that there's two important contributions that come out of Hubble. So…
David Bau: First, I think there's just, like, a lot of interesting, just, like, empirical findings that we have by directly, you know, analyzing the models, looking at all the different, sort of.
David Bau: friends, how these different, you know, factors relate to each other. But then there's, like, a second contribution, which is I also think this is a… could be a very valuable resource for any research related to LLM memorization or related
David Bau: tasks, and so in particular, I think, like, tons of stuff related to, like, interpretability or training dynamics, I think, Hubble would be, like, a really ideal, you know, resource to use. So, hope… I hope you're… you all are able to find it useful.
David Bau: Okay, but I'll start with these, some of the findings that we have. So, this is kind of just, like, one representative graph that we'll start with.
David Bau: So this is showing,
David Bau: one of the types of data that we injected, so these are book passages that we took from Project Gutenberg, and on the x-axis, we have, you know, these basically different… different buckets of examples that we put in different amounts into our perturbed corpus.
David Bau: And on the y-axis, we're plotting, you know, you can plot any measure… anything that measures memorization. Here, we're just using, like, log likelihood, which is, you know, it's a… it's a simple one.
David Bau: So the first thing you'll notice is that, so the standard models, which are these dashed lines, so, you know, they've never seen any of these injected data, so they just have some sort of, like, baseline level of, log likelihood, right? That doesn't depend on the number of duplicates in the, in the, that, sort of in the perturbed corpus.
David Bau: Next, we can look at this dark green line. So the dark green line is an 8 billion parameter model trained on 100 billion tokens. And you can see that this one, you know, there's maybe, like, some increase in memorization at 4 duplicates, and then it goes, like, way higher at… by the time you have 16 duplicates.
David Bau: In the training data. And then, you know, by the time you have 64, it's, like, almost, like, perfectly memorized, because the loss is, like, almost zero.
David Bau: And then we can also compare this now to this lighter colored line. So the lighter colored line represents having 500 billion tokens instead of 100 billion, but we control for the same number of duplicates. So, this already showed you, like, one thing that was maybe not entirely obvious.
David Bau: Which is that, when you have the same number of duplicates inside of a larger pre-training dataset, the memorization effect is actually pretty greatly diluted, right? So, like, the… the… now, like, at 16 duplicates, we see, like, maybe, like, a slight
David Bau: Increase in the log likelihood, but it's a much smaller increase compared to what we got with, training on 100 billion tokens.
David Bau: All right, and so this is just, like, one, sort of pattern that we can see that's… that's very general across all these data sets. So, oh, question? Yes. How did you measure the memorabilization? Yeah, so,
David Bau: On this slide, we're just using a log likelihood. You could also do things like extractability, like, you know, you give it 50 tokens and try to extract and see if it can predict the next 50 tokens. I guess the qualitatively, the difference is those plots are…
David Bau: generally similar, but it'll be a little more, like, the increase is a little more sudden, whereas, like, the log likelihood, it gives you, like, a slightly smoother curve.
David Bau: And then, I guess, yeah, on that note, so we… we also have a bunch of other types of data, so here is, like, passages on Wikipedia, where we also use log likelihood, but then for other things, we might use a different metric that is sort of
David Bau: more appropriate. So, for example, for personally identifiable information, we basically have some sort of query that tries to extract
David Bau: some attribute about a person, right? And so the measure is just that, like, have you, like, ex… you know, can you… does the model actually, like, produce the, you know, produce that particular attribute?
David Bau: And then for testing contamination, we just sort of measure, like, the accuracy on those, on those data sets. Yeah, and in all these cases, right, we see this same drop where, yeah, you get this, like, dilution effect comparing the smaller corpus and the large corpus.
David Bau: So one way to interpret this, yeah, one way to interpret this is that if you're trying to reduce the risk of memorization, one sort of best practice is indeed just to, like, get more and more pre-training data and sort of dilute the effect of, of any, you know.
David Bau: small set of duplicates. Do you have a question? Yeah, I just had a clarification for the… like, when you duplicate the documents, are they, like.
David Bau: uniformly distributed across, like, all training? Yes, yeah, so they're uniformly scattered throughout training, and I'll talk more about that in the next slide, actually, but is there a question, or…
David Bau: Oh, yes. So, how did you select the number of parameters to the dataset size? The number of tokens? Number of tokens versus dataset size. Yeah. The number of tokens versus the model parameter size. Right, right.
David Bau: I, I'm trying to remember the calculations. I believe we have… I believe the amount of data we have
David Bau: in general is always, like, slightly higher than chinchilla optimal, and then it obviously varies, because we're doing, like, we're also just, like, doing two different data set size and two different model sizes. But yeah, we use the, like, chinchilla optimal thing as sort of, like, a rough guideline, also with knowing the fact that, like, people often train, like, beyond the chinchilla optimal, yeah.
David Bau: Yes. Do you happen to see a pattern, like, the ratio? Because in the end, when it comes to 250, then it pretty much always remembers, yeah, the laws. Right, yeah. Yeah. Is there any kind of ratio, like, rule of thumb? If it's 60 over… Oh, I see, yeah.
David Bau: Yeah, that's a good question. Yeah, I think it's, like… yeah, I guess it's hard to say whether it's exactly the ratio. Like, my intuition is that it's… yeah, my rough intuition is that, yes, like, the… it is primarily, like, the ratio that matters more than the raw count, but…
David Bau: I think you would also expect it to…
David Bau: like, I think both of them probably play a role, right? Because, like, if you… basically, if you see something once, it's almost, like, never memorized, even if it's at a pretty high concentration. So, yeah. Yeah, it'd be interesting to try to figure out, like, what's the exact pattern, but you might need even finer granularity than, like, what we were able to do.
David Bau: Yes. Question. So, the difference is in the size of the dataset, but is there also a difference
David Bau: by the number of times each token was seen by the model, each sequence. Each sequence, yeah, so we're… I guess, so for all the things that we're injecting, we're… we're… we're fixing the… the number, right? So, like, when we say, like, 16 here, it's always… it always means it's, like.
David Bau: the, the, like, injected thing was always seen 16 times, but it was either, like, 16 times out of 100 billion tokens, or it's 16 times out of 500 billion tokens. Does that answer your question? Okay, yeah.
David Bau: Okay. So they don't want multiple epochs, I guess? No, the… it's all, so the… the training itself is just a single epoch, but then, yeah, I guess just within the data, there's, like, certain parts that get duplicated. Thank you.
David Bau: Okay.
Silvio Amir: So is it possible, then, to predict whether these things are going to be memorized, given that, you know, it seems to be some sort of ratio, right? Because at some point, you get a similar law of likelihood.
David Bau: Right, right, yeah, I think it's, yeah, like, I think that would be an interesting thing, like, I guess it would be great to develop, like, some sort of, like, scaling law for memorization, right? Right. I think we don't…
David Bau: I'm worried about doing that here, because just we don't… like, we only have, like, two model scales, right? We only have, like, two corpus sizes. I think if we were able to have even more compute, I think we could, like.
David Bau: flesh that relationship out more fully. I think here, like, the common thing we can say is that there is, like, a large, you know, there's a large dilution effect, and there's also, I'm not showing it here, but there's also, like, a large model size.
David Bau: effect, but yeah, exactly what the law is, I think, is… Not clear from our experiments.
David Bau: Yeah, okay.
David Bau: Yeah, so, yeah, to go to the question about, like, when are these seen during training, so we actually did a whole separate experiment on this. So, yeah, so our default
David Bau: was, let's just, like, uniformly scatter all of the injections, during… into the training data. But, we also were thinking, you know, and there's been, like, proposals of, like, okay, what if you only
David Bau: You know, saw this… some, like, sensitive data, like, early during training or something.
David Bau: Right? Without reduced memorization. And so, yeah, we did this experiment where we said, like, let's do all the injections, either only in, like, the first half of training, or only in the second half, and then we even also broke it down into, like, the, you know, different quarters of training.
David Bau: And I'm again showing the result on these, like, you know, Gutenberg books. And, yeah, this is… these are the results that you get. So this, the back line is, like, the, the sort of default perturbed model,
David Bau: The, these lines on top here, these are what you get if you inject in the… either the final half or the final quarter of training. So you can see that there's, like, a shift upwards in the, in the amount of memorization.
David Bau: In contrast, you know, if you inject only in the first half, you get these lines. So this is, yeah, so this actually does suggest that, yes, another possible best practice if you're trying to reduce memorization is that you should put the sensitive data sort of earlier, and then, you know, there's, like, a lot of time for the model to sort of, like, forget the details of that data.
David Bau: And then finally, there's, like, an interesting, one here. So this one is, this is the one where we injected in the third quarter of training. So it's, yeah, between, like, 30… 50 and 75% of the way through training, you see all the duplicates, and then you don't see them for the last quarter.
David Bau: So what's interesting here is that this shows that, like, the shape of the curve is actually also changing when you do this. So, if you look at 16 duplicates.
David Bau: like, this curve is pretty much on top of the curve where you have this, like, default perturbation, right? So basically, the effect of having 16 duplicates is roughly the same as if they were scattered uniformly.
David Bau: But the effect of 64 duplicates is very different, right? So, like, with the… in the default setting, 64 duplicates would cause you to, like, remember this stuff very well, and it's… it's much less well, for… for this setting, right? Again, so perhaps you could think that, like, because we have some period of time where the model might forget stuff.
David Bau: It forgets the, sort of, exact memorization stuff faster, but there's still some, like, lingering, you know, smaller effects,
David Bau: From these sorts of duplicates.
David Bau: Yes. Yeah, did you change the model completely from scratch? I was wondering if there is a performance amount of validation…
David Bau: during the first half, it's not difficult to learn the language class at all. Yeah, so we do, we do train pre-train fully from scratch, so I agree that that is, that's a possible explanation, right? Just, like, early in training, the models, like.
David Bau: less good at acquiring that stuff. What we… what we want to do, soon is that,
David Bau: we do have all the intermediate checkpoints during training, right? So what I'm interested in is, like, you could compare, for example, like, if we inject… when we injected it in the second quarter, and then trained for, like, one other quarter, how does that compare to when you inject in the third quarter and then train for one other quarter, right? So that can disentangle whether it's
David Bau: It has to do with, like, yeah.
David Bau: How much you've trained already, or, like, how much you've trained after?
David Bau: Yeah, just, like, a Y-axis is the, yeah, the log length, and the x-axis is the performance, the complexity of the mask. Maybe that could also tell something.
David Bau: XX or XX is… It's the perplexity of the language part. Performance of the language part. Oh, interesting. Okay, yeah, so… you want to see…
David Bau: Yeah, that's a… that would be an interesting plot, yeah, I haven't thought about that, but…
David Bau: Yeah, we can try to make that. Okay. Maybe similar to that, but what is, like, the, how does the loss trend according to each of these, lines? Like, is… if you have, like, the line that's, like, the most memorized, do you find the loss to be higher at the beginning? Is, like, learning, because it's seen a lot of, like, duplicated documents? Do you know what I mean?
David Bau: So you said, is the most higher at the beginning? At the beginning of… so for the lines where you see more duplicated documents in, like, the first,
David Bau: You said the first, quarter, that it's going to end up… sorry, in the last quarter, you see more duplicated documents, like, what's the loss trend for… Oh, I see, yeah, because you're saying that before it saw it, then the loss would be high, and then… okay, yeah.
David Bau: Yeah, we need to… that's, like, yeah, it's, like, doing… yeah, doing the analysis for all the interview checkpoints is, like, on our to-do list. I don't have the results for you yet, but yeah, I think… I agree, that's a great question.
Silvio Amir: Can I just ask whether… sorry, can I ask, whether these findings and these, curves that you're observing, do… are they consistent over other types of data? And I ask this because Gutenberg
Silvio Amir: These are, old books, and so I suppose that their language is slightly different.
Silvio Amir: Right on the internet, so I wonder if that has any effect.
David Bau: Yeah, that's a good question. Let's see, I think, in general, Yeah, I think that…
David Bau: Yeah, I think the story's a little complicated, so, like, okay, maybe we can… we can look at this slide. So, yeah, what I would say here is that there are definitely… like, there are some trends that are the same, like, the shape of these curves is generally pretty similar, but yeah, there are a lot of
David Bau: There are a lot of differences, and I think there are, like, a few different reasons for that. So, yeah, maybe there's a chance to maybe point out some quirks about these plots. So, one thing you might notice is that in MMLU,
David Bau: Which is kind of different from all these other ones.
David Bau: training on 500 billion tokens just helps you start at a much higher accuracy. So, yeah, so, like, the,
David Bau: the, you know, the curve can shift also because of… for this particular task, what is the scaling trend for that task, right?
David Bau: So here, yeah, MMLU is unlike everything else, where it's, like, at low duplicates, your model train on more data is higher, and then the curves cross at some point, and then the dilution happens in, like, this regime.
David Bau: Yeah, so that's one thing. The other thing is that, I mentioned that, like, the evaluation metric also, makes a difference. So, you know, that changes, for example, whether, yeah, the curve is smoother, or the curve is, like, more jagged, and we are generally trying to, like, fit the…
David Bau: fit the metric to the, you know, to the dataset, and to the sort of, like, you know, actual risk. So it's like, yeah, not clear that, like.
David Bau: It's not exactly apples and apples to compare, like, let's say, accuracy on a multiple-choice test, which is, like, predicting
David Bau: you know, like, one token versus, like, log likelihood on a passage, which is, like, a much, much longer thing, right? So you can maybe expect that there would be,
David Bau: those wouldn't always, like, exactly line up. So, yeah, I don't know if this is, like, a satisfying answer for you, but…
Silvio Amir: No, I mean, I get it, I get it.
David Bau: Okay.
David Bau: Awesome. Okay, so, okay, so these are… this is a taste of our, empirical findings. Yeah, I also want to just say a couple things about, you know, how I view this, or envision this being used as a resource for different types of research.
David Bau: And, yeah, so first of all, like, we're, like, really committed to
David Bau: making this be, like, a valuable, like, open source resource, right? Especially for work on memorization, right? It's very important to not only have access to the model weights, but also to have access to the data, so you know, like, what, you know, correlate the behavior with what the model saw during training. In our case, we have, you know, we've open-sourced everything.
David Bau: And we also have built a lot of tooling just for, like.
David Bau: dealing with the data for querying and editing the data. We actually… this is actually now an EMNLP demo track paper itself, called Tokensmith.
David Bau: So, yeah, and so we have all this information, like, for all the things we inject, we know not only how many times we injected it, but we know, like, exactly, like, when during training we injected each of these things, right? So you could imagine doing a lot of analysis on that, and then combined with the fact that, yeah, we do have a lot of intermediate checkpoints as well.
David Bau: So yeah, hopefully, and yeah, everything's already on Hugging Faith. It's not very well organized yet, but, it is all, you know, you can actually find all these models already. You go to our Hugging Face. Robin, I wasn't aware of your Tokensmith project. I wonder… so I bet it would be interesting to some of our infrastructure folks, the NDAF folks here, because, you know, the…
David Bau: the ability to query over huge amounts of training data. Like, would you be able to use your tokensmith tooling
David Bau: To load in,
David Bau: you know, some of the AI2 models, training data, and allow people to study that, and is it tied to your data? So it's tied a little bit to the pre-training library, so it's anything that's, I think, follows the Megatron format, which includes, like, the Luther models, and several other models. I don't think it includes the AI2 models, however, because I think they have a different code stack, yeah. I see, so you could study Pythia this way.
David Bau: Yeah, you can study Pythia this way, yeah. I see. Well, that's… that's better than nothing.
David Bau: Yeah, interesting. Anyway, maybe we'll… maybe some… maybe afterwards, you know, we'll have some of the end of people quiz you about… Yeah, yeah, yeah, happy… happy to have those conversations. And then also, honestly, like, I mean, I think probably for this stuff especially, like, the students are probably better people to quiz, so we could also put… Yeah, that'd be really great. Yeah, yeah, really interesting.
David Bau: Okay, cool.
David Bau: Let's see, okay, so… but besides this, just, like, you know, studying the checkpoints, I think there's also a couple reasons why Hubble's a great benchmark for, you know, what we might call some sort of, like, you know, memorization-adjacent
David Bau: tasks. So, for example, there's been a lot of interest recently in what's called membership inference attacks for LLMs. So, you know, the basic idea of membership inference is,
David Bau: You can think of it as, like, an attacker has access to a model, and they want to figure out whether a particular example was present in the model's training set or not, right? Even though… even if the training set is not public. And so, you could basically think of this as some sort of binary classification task. They're, like, members of the training set, and then there are non-members, and yeah, the attacker's trying to get high accuracy on this.
David Bau: So a lot of the benchmarks for LLMs at least,
David Bau: you know, they have to somehow come up with some set of data they know the LLM trained on, or, like, was very likely to have trained on, and other data that the LLM has not trained on. So they often do things like using, like, temporal shit, right? So things that are before the knowledge cutoff are, likely members, things that are after the knowledge cutoff are likely non-members.
David Bau: But this is, as you can imagine, has… there's a lot of, like, spurious cues, right? There's probably a lot of ways to figure out, you know, at least estimate when the article was written, if it was before a certain date or after a certain date.
David Bau: And so there's been a few papers now that are showing that, like, yeah, these benchmarks are, like, pretty flawed, right? You basically don't need access to the model at all, you can sort of just, like, directly predict whether these are members or non-members.
David Bau: Yeah, and so the, the great thing about Hubble here is that, you know, we have these perturbations that we injected, and we know that we've randomly assigned them to these two conditions, either, like, because some of them were just assigned to be injected zero times, and others got injected more than zero times. And so our MIA benchmark can just be like, yeah, can we distinguish these two?
David Bau: And in addition, we can actually measure, like, how well does a membership difference, attack, work as a function of, like, how often did we insert the different data points into the… into the training. So we can come up with these, plots like this. So here, yeah, we just ran, you know, four standard,
David Bau: MIA methods on our largest model, so there's the four different
David Bau: And so, yeah, for example, we can have one version of the benchmark, which is just saying, can you distinguish things that were
David Bau: you know, inserted zero times from things that were only inserted once. And here we see that, like, performance is, like, more or less random. You were measuring AUROC, this is just, like, a typical metric, so close to 50% is random, we're very close to 50% here.
David Bau: And then as you go, you know, as you go higher, right, if you start comparing, for example, can you distinguish between zero.
David Bau: and 4 duplicates, you start getting a little bit more signal, and this sort of increases to the point where, like, by 64, if you're comparing 0 to 64, you get, like, more or less, perfect results. And this is also, yeah, I think, like, this…
David Bau: I'm again showing the Gutenberg data here, but I think this we also have on, like, sort of, a couple of the data sets. I think these results, from what I remember, are, like, pretty…
David Bau: pretty consistent, like this, this pattern here. What concrete, natural inference is actually doing? Yeah, so this is, yeah, so we have,
David Bau: So, yeah, basic one is just to, like, look at the loss on the example, and then we have, like, MinK++, which is, this method that was, like, you basically, like, look at some, like, certain outlier, tokens with some outlier loss.
David Bau: There's an improved version, and then there's… okay, I honestly don't remember off the top of my head which one, which one ZLib is, but, yeah, these are… yeah, these are… these are the ones.
David Bau: Okay. Cool.
David Bau: Yeah, the other… the other thing I think this is, Hubble is quite interesting for, is unlearning. So, you know, so suppose… so what I'm showing here is, like, you know, we ran some unlearning techniques, for example, on this goal of, like, unlearning some of this, like, private biography information.
David Bau: So yeah, one thing that's cool about Hubble is that we have, you know, all this… again, we have this, like, IID data, essentially, some of which was not injected, right? So we can… we can say, for example, we have an unlearned set consisting of some bios we want to forget. We can have this, you know, test set, which is, like.
David Bau: bios that were unseen, that were injected zero times, but are otherwise IID, and we want to make sure that the model doesn't sort of, like, collapse on, you know, its ability to predict these, right? There's just some sort of, like, you know, prior over, like, what these sort of bios should look like. We don't want to sort of, like,
David Bau: eliminate that ability. And of course, we can also just have a keepset of, like, other bios that were memorized that we, would not want the model to, to forget.
David Bau: Okay, the other kind of, I think, thing that's quite interesting here is that for Hubble, we have this standard model, right? We have this model that has never seen
David Bau: these injected, data points. And so we can kind of use that as some sort of, like, gold star, which is… which says, like, okay, when you run unlearning, you should try to be matching the standard model in terms of its performance on the unlearned set, right? So this… this, I think helps, like.
David Bau: clarify what we might want. That's cool, Robin. Is the standard model and the test model, do they have the same random initialization and things like that? Are they sort of sister models, or are they… are they unrelated to each other? Yeah, so they have… okay, they… I know they have the same data order. We were careful about that. I'm…
David Bau: I think we should have the same random initialization, I would need to double-check that. That's pretty interesting, yeah. I think it might not have been, because I think there were,
David Bau: like, a original version and a… and the perturbed version, both for zero. I thought there was two dots on the plot, so I think they might be different.
David Bau: Sorry, case. Oh, in the previous plots, where you were showing, like, the perturbed version, like, a bit before, like, the, perturbed one also had a zero times…
David Bau: Point? I'm just freezing.
David Bau: Perturbed one also had a… sorry. Which pot? It should be, like, way before. Okay.
David Bau: Like this one, yeah. Okay. Okay, those ones are… maybe the next one would be more clearer.
David Bau: Because, the…
David Bau: A solid one… a solid one is the perturbed one, right? Yeah. And then, unlike, you can see on MMLU, there's, like, a zero one, and 0 one should be the same, right? But… Yeah. But they are diff… but they're different points, so… That's true, yeah. So they are… they are,
David Bau: Right, okay, yeah, so this is, yeah, this is actually a good, good observation, so, and…
David Bau: Yeah, I think… I think that's something to… to, yeah, we… we can discuss. So, on… on some, on some data sets, right, yeah, you do see this, yeah, for example, on MMLU, there is a small
David Bau: yeah, there's a small gap between, like, this, standard and perturbed model, where the dashed line's a little higher than the solid line. And I think what's happening there is just that, like.
David Bau: you have… because we injected MMLU examples, there… you do get some just, like, generalization, where the model's maybe just, like, somewhat better at, like, multiple choice questions in general, from being exposed to multiple choice questions.
David Bau: Yeah, so that is, I agree that's an effect that happens on some of these datasets, and it is,
David Bau: Yeah, that is… that does make it a little hard to disentangle, like, yeah, like, in some sense, I think maybe what you're getting at is for the unlearning… for unlearning application, like, the ideal thing to do, I guess, would be to have, like, a… the truly counterfactual model would be trained on all the same data, like, except the unlearned set, right?
David Bau: And we have something that's very close, but it's not exactly that, because we have it that's not trained on the unlearned set, but also not trained on several other perturbations. Yeah. So, yeah, I agree there's, like, a bit of… there's still a bit of a discrepancy.
David Bau: But I think, yeah, I think still it's, like, pretty close, I guess, is what I would say, yeah.
David Bau: And,
David Bau: Yeah, and then anyways, yeah, and then, so just the, you know, the, the, you know, results from, from seeing a few models, basically, it's like, just say, like, okay, it's actually pretty hard, so on the, the x-axis here is the unlearning set accuracy, and basically, like, you know, ideally, you would be able to
David Bau: drag your original perturbed model, which is the red one, you would be able to drag it all the way down in terms of unlearning set accuracy, and maintain accuracy on the other stuff, but in practice, that doesn't happen, because when you decrease unlearning set accuracy, the other metrics also decrease.
David Bau: So you're… we're not quite getting at this, like, the gold star, that we would like.
David Bau: Okay, so yeah, to wrap up this section, yeah, so, there's, like, like I said, like, kind of two big contributions, you know, first, just these, you know.
David Bau: these empirical findings that we have, and I think a key takeaway, especially if we're thinking about this, like, broader audience questions, are like, yeah, we can establish, some best practices, for example, related to, you know, dilution, or when you have different types of data in your training, you know, if your goal is to reduce memorization.
David Bau: And then we also, you know, hopefully, got you interested in using our models as a, as a resource for all sorts of different, research.
David Bau: Yeah, cool. Yeah, let's, let's keep going to the, the next section, yeah,
David Bau: So, yeah, I want to talk about, like, you know, a few different interpretability things we've done, but the main one I'll focus on is this, this nice story about understanding how LLMs do basic math.
David Bau: Okay, so our setting here is pretty simple. So we're gonna just, like, fine-tune some pre-trained model, like, let's say, GPT-2 Excel, to do integer addition. And this is pretty easy for the model. If you just fine-tune it a little bit, it can get, like, 100% accuracy. And I guess the other thing I should mention is we're keeping…
David Bau: We're keeping all the numbers and the problem, like, pretty small, so that they're all just, like, a single token.
David Bau: In the, in the tokenizer. So, yeah, trying to… trying to do, like, a, you know, study a very simple sort of setting. And it's so simple that you might actually think that, like, oh, maybe the model, you know, just can memorize, it can just store, like, a big table of, of answers, right?
David Bau: But actually, like, we, what we see is that it doesn't seem to do that. It actually is doing some sort of computation. So this is, this is how, we were able to argue for this. So this is, yeah, this is basically, like, a plot where we've done… we've done logit lens on the outputs of different MLPs.
David Bau: And, so yeah, so different columns represent different, different MLPs at different layers.
David Bau: And, I'm plotting, like, how much are these basically promoting each possible answer, right, as defined by doing logic lens. So, you know, the correct answer to this problem is, like, 108.
David Bau: And we see that there's some layers that have patterns like this. So this layer, layer 33,
David Bau: it seems like it's just, like, every other number is, like, darker, right? And so basically what this is saying is that it's just promoting all of the even numbers, you know, and this other layer here.
David Bau: you can see that there's sort of a dark pattern roughly every 10, right? There's on, you know, 88 and 78, you know, it's darker than the rest. And so this is basically promoting all the numbers that are 8 mod 10, right? So when we looked at this, we said, like, okay, it seems like, actually.
David Bau: there's some just, like, modular arithmetic that is happening, right? The model is correct in knowing that, like, the answer should be 8 mod 10, and this layer seems to be, like, you know, focusing on that, and not necessarily,
David Bau: you know, only promoting the exact correct answer. And then there are some other layers, which seem like they're also not…
David Bau: very selective, but what they're promoting is a bunch of numbers that are sort of close to the right answer, right? So you can see, I mean, 108 is dark here, but then also, like, neighboring numbers are also darker than the rest, right? So this is… this is sort of, in some sense, you could say it's approximating the answer. It knows that the answer is roughly 108.
David Bau: Okay, so, so when we saw these plots, we were like, oh, that's interesting, we have something, some sort of periodic structure here, right, and things like this. And so what do you do when you see periodic structure? Of course, you want to…
David Bau: I think loads. You want to take the Fourier transform, and understand, like, what… understand the, you know, decompose this into some, you know, periodic components. So,
David Bau: when we take the Fourier transform of each column, right, so you just think of each column in the previous slide as just, like, some time-varying signal, we can decompose that and find that, you know, you get these very interesting, like, spikes, what we call, like, these, like, outlier frequencies.
David Bau: And in particular, the… at the sort of… at the right end of these plots, where there's, like, sort of the high frequency, you have these, like, you know, very sparse spikes, right? So this layer, this is layer 33 from before, so the one that was promoting all the even numbers.
David Bau: So this one has a very high spike at a period of 2, which makes sense, right? But there's also spikes at, like, 5 and 10, for example, right? So there's also some sort of… some… some… there's some part of it that's also periodic with a period of 5.
David Bau: And then we can see similar things for attention, right? We also get these sorts of spikes, and they all tend to be at these, like, kind of nice numbers, right? Things like 2, 5, and 10. I can talk more about why we think that is, but basically it has a lot to do with the fact that we use base 10 as, you know, in, in…
David Bau: just in… in our lives.
David Bau: Okay, so, so we see that, yeah, basically there are these, like, key, like, outlier frequencies that are, that are really, like, playing, seem to be playing a key role. And so, like, what do we think is going on here? What we think is happening is that the model is basically, like, in parallel.
David Bau: doing computation at these different frequencies, right? So there's some, like, low-frequency computation, and so this is represented, for example, by this, this orange line, right? So if you extract just, like, one of the low frequency things from the Fourier, transform.
David Bau: you see that the model has some component that's basically trying to peak roughly at the right answer, right? So this is, like, the approximation part.
David Bau: And then there are the high frequency parts, which are, you know, these, yeah, high frequency sine waves that are… that are trying to classify, yeah, like, let's say the answer mod 2 or mod 5, right? And these are also trying to peak… have a peak at the correct answer, but of course they also have, you know, many other peaks at things that are congruent mod 2 or mod 5.
David Bau: And if you just sum all of these waves together, what you get is, you know, this complicated shape that will actually, you know, peak at the right place, right? So this is sort of our explanation for, like, yeah, how the model ultimately arrives at the right answer.
David Bau: Okay, so, yeah, so this… all this study so far is just, purely observational, so, you know, the next thing we wanted to do is we said, like, you know, can we actually establish some sort of, you know, causal relationship here? So, the idea was that, you know, we can… we can again start with this Fourier transform.
David Bau: And so what if we only project out, let's say, the, low-frequency component… er, yeah, high frequency components? We project out the high frequency components, we only keep the low frequency components, we can see how that,
David Bau: how that alters the final prediction of the model. And then conversely, we can do the reverse, right? We can say, you know, let's filter out the low-frequency components and only retain the high frequency components and see how that affects the model's prediction.
David Bau: And yeah, basically, like, you actually get very predictable, effects when you, when you do these things.
David Bau: Some… sorry, some lag.
David Bau: What's going on?
David Bau: Okay, yeah. So,
David Bau: So first of all, if we're filtering out the low-frequency information, it turns out to be sufficient to just do this to the MLPs, and it actually has a quite large effect on the predictions. So, if you filter out the low frequency stuff, the model suddenly starts making a lot of errors that are, like, off by 10, or off by 50, or off by 100.
David Bau: So what this means is that, like, the models still correct mod 10,
David Bau: But it's sort of lost the ability to just know approximately how big the answer is, right? And so this is, like, a sign that, you know, specifically the MLPs in this low-frequency, dimensions are doing, are trying to, like, approximate the right answer.
David Bau: And correspondingly, we can filter the high-frequency components, and here it turns out it's sufficient to only do this at the attention layers, and you can cause the model to make a lot of, like, off-by-one or off by 2 errors.
David Bau: And so, yeah, this is just the reverse, right? We're still, approximately correct, but now we have lost the ability to be correct, like, mod 10, or, like, even mod 5. And so, yeah, so this high, high frequency part, in the attention layers is what's doing this sort of, like, classification mod N.
David Bau: Okay. So, finally we had, you know, we were interested in, like, sort of whether this is something that's just, like, fundamental to transformers, or this is actually related to pre-training. And I think we had, like, all the results point to the fact that pre-training is actually, like, kind of crucial for this particular mechanism.
David Bau: So first of all, we can see that the embeddings, at, like, the pre-trained token level, like, the token embeddings of the integers, already contain these sort of, like, Fourier frequencies. So basically, we just, like, look at
David Bau: each dimension of the token embeddings, they also tend to have these sort of periodic patterns with the exact same, you know, nice numbers as periods, right? 10 and 5 and 2.
David Bau: And then, moreover, we can do some, you know, experiments to actually show that the… it's the pre-trained token embeddings that really matter. So, in this plot, so first let's focus on this black line. So, the black line shows what happens if we take
David Bau: an identical transformer architecture, like a GPT-2 architecture, but we randomly initialize it, and then train it to just do addition.
David Bau: So, in that experiment, basically, like, it optimizes very slowly, and it never reaches 100% accuracy, right? So in this spot, it only gets to maybe, like, 50. If we train it for a lot longer, it can get to, like, maybe 94, but it still doesn't…
David Bau: fully solve the task, and if you… if you do this same sort of analysis we did before, we basically don't see any of the high-frequency components, right? It makes a lot of off-by-one errors because it's trying to do everything with the, like, approximation mechanism. It doesn't learn it, it doesn't figure out that, like, doing some sort of, like, bodular addition, is a good idea to… to get the precise answer.
David Bau: This red line shows, like, if we only tr… If we only, kind of, use the pre-trained token embeddings, and the rest of the transformer can still be randomly initialized, which I think is, like, it's very surprising, right? So all we do is change
David Bau: you know, put back the token embeddings. It turns out this is enough to basically, like, rescue the performance
David Bau: of this model. And it, in fact, like, converges quite quickly to being able to get 100% accuracy. Yeah, so basically, yeah, these token embeddings turn out to be, super critical, and if you have the right features, the transformer is sort of already set up to learn this algorithm we described.
David Bau: Okay, so the, yeah, the sort of, like, high-level takeaway here is, like, you know, and I think why I like this as a story so much, is that, you know, it is true that the way the LLM is doing this addition task, it's not very human-like, right? None of us learn to do, like, this particular way of adding numbers together in grade school.
David Bau: But even though it's not human-like, it actually is something that we can still make sense of, and we can understand, like, why it should work, right?
David Bau: So I think that's, like, a very kind of, like, you know, optimistic tale, right? I don't think we should necessarily expect LLMs to do everything in the same way that humans do, but hopefully we can still make sense of whatever they are doing, right? And also, yeah, just to, you know, the sort of
David Bau: pre-training is sort of very key for this to emerge, and so we're doing some other work now and trying to understand, for example, like, what sort of data is sufficient to get you these sorts of FOIA features when you do, when you do pre-training. So, yeah, happy to talk more about that, offline.
David Bau: I'll do… I'll do one slide on some sort of follow-up work we've done, which is basically, you know, taking this, you know, this idea that a lot of people are interested in is, like, how can we take these interpretability results and, like, make them actionable, right?
David Bau: So in our case, so we have this understanding of, like, okay, how do transformers just sort of naturally learn how to do arithmetic, at least after they're pre-trained?
David Bau: And so, if we actually now change our goal to be, like, can we make them better at doing arithmetic, there's actually a natural idea, which is that we can take these, you know, at the token level, we know that those pre-trained token embeddings are super important, right?
David Bau: So, instead of relying on the pre-trained embeddings, we could also just hard code
David Bau: these, you know, Fourier features into the representations. And, you know, the reason you might expect this to be better
David Bau: is that, you know, the features that were learned through pre-training, right, they probably have some noise associated with them, right? There's, like, you know, because they had to be learned through GD, there's probably, like, some numbers that were, like, seen less frequently, whose features are probably, like, not perfect, right? But we, yeah, we could just, like, hard code them with some, like, sines and cosines.
David Bau: And,
David Bau: It turns out that this helps a transformer, at least if you train a transformer from scratch, this enables you to be, like, very sample efficient at learning, you know, not only addition, but also, like, subtraction and multiplication. So this plot just, you know, I'm showing, doing six-digit, addition.
David Bau: And it turns out that for our methods, our method is this red line, as long as you give it more than maybe, like, you know, 3,000 data points or something, it can get, like, very close to 100% accuracy on doing 6-digit addition.
David Bau: And then if you train… if you give it maybe one order of magnitude more data, it actually gets, like, exactly 100%. So the stars mean it gets, like, exactly 100% on, like, a very large test set, right? So it's actually, like, perfect at the task.
David Bau: This is much better than if you, for example, start with, like, a digit… like, digit-wise tokenization, or you do, like, sub-word tokenization. These are both much worse. And this is even, you know, in some regimes, better than fine-tuning a much larger language model that was pre-trained, right? So if you take, like, the Slava 3,
David Bau: model, it obviously starts at a higher place, just because, like, even if you have zero training data, it has, like, some knowledge of addition already. But, you know, if you fine-tune it, it grows sort of more slowly than our approach, and there's, like, this regime where, like, this fine-tuned model is, like, still making a lot of errors.
David Bau: But our… our method with the same amount of data, and again, a randomly initialized model, is actually, doing… being… being, like, perfect.
David Bau: Okay, and then, yeah, so I think we've… the other reason that I think this is interesting is that
David Bau: through this scheme, we can basically take any number and kind of turn it into a single token representation. So this is also nice because it avoids some issues with tokenization, and it's also just more efficient because you have fewer tokens. And so we're currently trying to figure out, like, if this is actually practical to incorporate into, like, an actual LLM pre-training.
David Bau: I think there's a lot of hurdles there. For example, the fact that, like, oftentimes you see numbers in the data, and they're not actually numbers, they're, like, you know, zip codes or something. And so I think there… I think we still have to… there's things to figure out to make this more practical, but I think this is, like, a… definitely, like, kind of an interesting,
David Bau: proof of concept here for the… for these, randomly initialized models. Cool, yeah, so any… any questions on this section about these, like, sort of math, abilities?
David Bau: I had a question, but you sort of already answered that. So, so you learned some…
David Bau: elements of mechanism going on inside the model when it is doing this, 10 base additions.
David Bau: Let's say without any training, can you take some of these insights back to pre-physical Excel to make it better at training?
David Bau: Sorry, make it better an addition. Yeah, yeah, so that would be sort of, like, it's similar to this, but I guess you actually want to modify the model that's already pre-trained, right? Yeah, doing, like, inference time intervention. Like, while it's doing some kind of addition, you will basically
David Bau: boost some of the signals, and does it, like, suddenly make it better? Yeah, yeah, actually, okay, actually, this is a…
David Bau: Right, so it's a good question, so…
David Bau: In terms… okay, so first, I guess, in terms of inference…
David Bau: I guess we hadn't even thought about doing it at…
David Bau: inference, or maybe we thought about it, we didn't really try doing it at inference time. We tried doing it at, like, a continual pre-training stage.
David Bau: to try to, like, basically swap out the original number representations for, like, something that's, like, these… I guess we call these phone… for Fourier number encoding, so to swap those out for, like, those, like, phone representations.
David Bau: That turned out to be… it turned out to be hard to get that to work, because there's just some, like, distribution shift with the pre-trained model, right? It's just, like, not used to seeing these embeddings. And so, like, you get an initial loss in performance, and you can do continual pre-training to try to, like.
David Bau: recover back to the original point, but it doesn't actually seem to get better. So, yeah, we were thinking that, like, either we just need, like, a different way of doing it, or we need to pre-train from scratch. Yeah, the other sort of
David Bau: Right, the other, the other sort of…
David Bau: issue that you start running into is that, like, if you… if you have numbers that are, like, not that small, then the original model expects them to be tokenized into multiple tokens, right?
David Bau: The answer itself. Or even, like, the input numbers, yeah, yeah.
David Bau: And so, like, so even though, like, our… this method in theory can, like, you can encode any number into a single token, but that's, like, not what the model is, like, prepared to do. So, yeah. So I think this is very, very cool, that… especially the proof of concept here, but in terms of, like, actually putting it back in, I mean.
David Bau: isn't the answer just use tools? Like, we have calculators, no? Yeah, like, we can… and the models can… we can just train them to call. If you really… if the goal is to actually improve
David Bau: mathematical operations, presumably. But, I mean, again, think of that. But, like, I think it's a point well taken, and I think I, I myself am, like, you know, I, I, like.
David Bau: I go back and forth on this, like, I…
David Bau: Yes, it is definitely true that, you can't… so, yeah, you definitely can't have tools. I guess let me maybe…
David Bau: I guess the two things I would say is that, like.
David Bau: number one, I think this is just, to some extent, I think I view this as, like, an interesting sort of proof of concept, like, not even proof of concept, but, like, it's a good testbed, I think. It's…
David Bau: For understanding some more general questions about, like, what sort of features do these models learn, and can we, you know, make… can we improve models this way, right? So hopefully it's gonna be a stepping stone to something that's harder to have tools for. The other thing is that I think there's, like.
David Bau: you know, I guess adding any sort of, like, tooling harness does increase complexity, right? So it's, like, nice, like, for at least something that's, like, so simple, like, as, like, addition, like, it'd be nice if we, like, did not have to increase complexity in order to get
David Bau: Well, not if you have to start mucking around with tokenization and stuff, but maybe, yeah, right? Yeah, but I mean, but in terms of, like, once the model is trained, then at inference time, you don't have… In this case, yeah, yeah. I guess I was just talking more about, like, oh, like, can we, you know… Oh, sure, okay, right, right. So if you're doing stuff at inference time, then yes, that also includes your…
David Bau: Yes.
David Bau: I'll go!
David Bau: Obviously, this is not how we do math.
David Bau: But did you see any relationship between, maybe not for addition, but for other operations, between the way the model is performing these
David Bau: operations implicitly, and how, you know, it would verbally explain that operation on, let's say, a tutoring task. Oh, that's interesting, yeah. You know, how would I multiply these two numbers? Is that influenced at all by what's happening with
David Bau: It is multiplying numbers?
David Bau: Yeah, that's… That's a great question. Yeah, I haven't thought about that.
David Bau: Yeah, I think that would be super interesting to look into. Like, I mean…
David Bau: probably it's, like… I mean, presumably the model has also seen, like, explanations that are tailored to humans, right? So it should still be able to mimic those explanations, but, yeah, what the relationship is between that and, like, this stuff, yeah, definitely interesting.
David Bau: Well, one thing I find really interesting about this is that it sort of seems to be the case that it hasn't memorized a table, right? Like, it hasn't memorized an addition or multiplication table. It's almost as if it's developed a sort of,
David Bau: a different representation of math. Maybe it's not the way we teach addition algorithms in school, but it might actually have some… I don't know, I'm almost feeling like people who are, like, very intuitively good at arithmetic might actually have something almost approaching.
David Bau: Right. style of, of…
David Bau: of addition. Yeah, yeah, I agree. Like, I think there's, like, yeah, I think I was giving this talk somewhere else, and then people were saying that, oh, so this is a sign that the models have some sort of, like, number sense, right? Some sort of, like, intuitive…
David Bau: grasp of these numbers, yeah, which I think, yeah, yeah, so… Yeah.
David Bau: Do you happen… do you have been about to try out, like, more general math questions besides…
David Bau: Yeah, so we… Right, right, right. So, yeah, like, we weren't able… so this, because the transformer part itself is trained from scratch, like, it… yeah, it's not ready to do something like GSM8K. Yeah, we tried to do that with our, like, continual pre-training to get it, like, improvements in GSM8K, but we have not seen them.
David Bau: But, yeah, we're still… we're still trying to… Yeah.
David Bau: Oh, yeah, there was a question about the comments.
Silvio Amir: I have a question.
David Bau: Oh, there's a… there's a question from the hallway that we're dating.
David Bau: That's all the… Violence.
David Bau: A final answer, if anything, thank you.
David Bau: academia, but I also saw some other, you know, academians.
David Bau: Seems like that is true.
David Bau: Is it possible? Like, there's something like that?
David Bau: That'd be… We're constantly buzzing One was, like, I saw that one was not so confident, Beautiful Fair sweet.
David Bau: Yeah, so you're saying, is there an attack that, beautiful. Yeah, I'm…
David Bau: I guess it… I guess it depends on what the attacker's allowed to do, like, yeah, certainly, I mean, as we showed, like, because we do all these, like, interventions on the hidden state, yeah, if you know where to intervene on the hidden state, yeah, you can definitely…
David Bau: Change the prediction?
David Bau: Yeah, yeah. Right, right.
David Bau: Yeah, yeah, but I guess it depends on, like, if… I guess the question is whether that's a realistic thing for the attacker to be able to do. It seems like if they have access to the internal state, they can do all sorts of things.
David Bau: Yeah, I guess, yeah, sorry, the Zoom question?
Silvio Amir: Yeah, so, I'm just curious whether, do you think the same mechanisms are at play?
Silvio Amir: if we ask the model to do addition, but, you know, we present a problem, right? You know, Susan had 2 apples, and then John came and gave her, like, 3 more apples. How many apples does Susan have now?
Silvio Amir: Will the model also, utilize the same kinds of, mechanisms to do, kind of, addition in this setting, where it has to figure out if addition is at play, or is this just, like, when it's explicitly told to some distant number?
David Bau: Yeah, that's a good question. I… I guess I would… I would expect it to be the same, but yeah, we didn't actually measure that, so that's… yeah. I think that would probably be a good, like, good sort of, like, robustness check on these results to test it, yeah.
David Bau: Yes. Oh, I'm not sure if you mentioned this before, but the pre-trained model before you fine-tuned it, was it, like, more approximate, or, like, how bad was it before fine-tuning? And do you have a sense of, like, fine-tuning teaches it the exact
David Bau: mechanisms, like, the high-frequency stuff, or, like, it was kind of just really bad at the beginning, and then it learned both of them? Yeah, it's been a while. I think this is in the paper, but I…
David Bau: So, okay, so I guess I have two answers to that. So… I think GPT-2 was…
David Bau: it was, like, not terrible in addition, but it's just, like, not that good of a model, right? So I think… and we wanted to,
David Bau: And it was, like, a pretty small amount of fine-tuning that could make it get 100%, so we focused on that.
David Bau: Yeah, I don't know, yeah, I guess we could look at, like, what the…
David Bau: like, Fourier patterns look like before, as well. The other experiments we did have is we had some other larger open-weight models that we just did with in-context learning, that actually were, like, good at arithmetic, and then we do see the very same, like, similar, like, features in…
David Bau: Yeah. Even though the… yeah, those models, like, generally, they were not, like, 100% accurate, but they were, like, maybe, like, I don't know, 80-90% accurate. Yeah.
David Bau: Awesome. I'm just checking this time. I think we… yeah, I think,
David Bau: Kiana said that we could go until 11.30, right? Yeah, okay. We'll probably, do that. I've also… I've also thrown in a couple extra slides, that,
David Bau: Yeah, so, yeah, I want to just, like, briefly mention this, like, related work that I also just think is… would be fun, especially for this audience. Some other, yeah, interpretability stuff that came out of this, collaboration with, Jonathan Gratch, who's a professor at USC who does a lot of
David Bau: like, emotion-related, emotion and, like, you know, like, computing system sort of research.
David Bau: So what I learned from, from, like, basically one of his students took my class, and what I learned from, from them is that there's, this thing called appraisal theory in, like, the, like, you know, psychology, like, of emotion literature. So appraisal theory basically says, or hypothesizes, that, like.
David Bau: as people, we reason about emotions through these things called appraisals. So basically, the different sort of, like.
David Bau: you know, like, attributes about a circumstance, right? So if someone says.
David Bau: oh, this exam was hard, and I could… I, you know, I feel like I couldn't have even done… couldn't have done better even if I studied more. So this is, you know, there's an appraisal here that, like, they did not think that they had any agency in affecting the outcome, right? And so that might lead them to feel sad, whereas if someone says, oh, this exam was hard, and, you know, I wish I had studied more because I could have done better, then they feel like they did have agency, they had some control, but they just, like.
David Bau: chose the wrong thing, and then this might cause them to feel, like, guilty instead of sad, right? So, the idea is that, like, these appraisals are actually, like, how, yeah, how we actually, like, reason and, like, you know, experience, emotions.
David Bau: And so, you know, we… so we, you know, LLMs, we basically took LLMs and, you know, we asked them to do this task of, like, you know, here's a story, what emotion, is this person likely to experience?
David Bau: And, we could basically just, like, probe it both for, you know, basically, like, where it's predicting these emotions, and also where it… where or, like, if it is representing these appraisals. And we do find that, like, yeah, we can probe both of these, both the emotions and appraisals, from some intermediate layers.
David Bau: And I think the especially cool part is that we also established there's some sort of, like, causal relationship here, which you can do with steering.
David Bau: So, basically, what this plot is showing is that in the, we have some, like, basically base condition, we just have some data set that has, you know, a lot of different emotions, so the model predicts some sort of, like, broad distribution of emotions.
David Bau: So in this next panel, we can basically steer on these, you know, we have a probe for this, like, pleasantness, which is one of these appraisal dimensions, and we can steer it, so in this case, we can steer it towards the direction of, like, low pleasantness.
David Bau: And what happens when we do that is that the final prediction from the model starts being dominated by things like anger, or fear, or sadness, right? All things that, you know, are associated with unpleasant situations.
David Bau: And then we can even steer multiple things at the same time. So here, we can steer, both pleasantness to go down, and we steer, this appraisal of, like, other people's agency, basically. So, like, was this… was this a result of what someone else did?
David Bau: Right? And so if we increase the steering on others' agency, then this just causes the bottle to, like, basically always predict anger, right? So it's like, yeah, if…
David Bau: something was unpleasant, and it was because of someone else, then you were going to feel angry. So yeah, so, like, I think it's very cool that basically, like, we can have… we can establish these sort of, like, causal patterns inside the LLM that actually, like, recapitulate this, like, existing theory.
David Bau: No, they disagree. Like, are there any points of disagreement between LLM and this? Yeah, so I think,
David Bau: I mean, I have to… I have to look at a few… the full results. I think… honestly, I think this one was maybe, like, the cleanest one where it does line up. I think it doesn't always perfectly line up,
David Bau: I feel like with those, it's always, like, a little hard to tell, like, if… how much of it is just, like, steering is a bit out of distribution, right, and you get a little noisy. But I think… I would say, in general, the patterns, do agree with, like, what you would…
David Bau: expect, but to be not 100%, yes. I have a question about the… the causal model.
David Bau: How can you be sure that you are steering the,
David Bau: The appraisal and not the motion.
David Bau: Yeah, this is, this is a good question.
David Bau: I need to remember the poop, because I remember having…
David Bau: conversations about this. If I remember correctly, we did…
David Bau: honestly, like, if we chat offline, I can refresh myself on the details, because I remember we tried to do some stuff where we…
David Bau: try to, like, separate out the vectors, right? So basically, because we get a bunch of probe directions for the appraisal directions and also for the motion directions, and I think we did do…
David Bau: try to do some sort of correction that, like, disentangles them, but it's… yeah, so it's… it's a… it's a… definitely a concern. Yeah, maybe we can talk more. Yes.
David Bau: Yeah.
David Bau: Okay, cool.
David Bau: let's see, maybe, yeah, I'll maybe briefly do this other one on in-context learning, because I think it's fun, and yeah, people are asking.
David Bau: So, we had this study about, you know, in-context learning, and the,
David Bau: just the… the, yeah, okay, I think everyone knows about in-context learning, you get some demonstrations in the prompt.
David Bau: You get a test example, you have to predict the answer on the new test example. And basically, there have been a bunch of these works that have hypothesized that, like, what's happening inside the model is that it's able to do some sort of, like, gradient descent-like thing, right? You can imagine that, like, you know, yeah, this is, like, obviously, like, a…
David Bau: Your, like, first…
David Bau: go to a tool in machine learning, right, we have some data set, we need to make a prediction, which we do, we should, like, run gradient descent on some laws. Right? And so there's this work, for example, showing that, like.
David Bau: Yeah, so we can construct a transformer to implement, like, you know, so that, like, one layer of the transformer can implement one layer of gradient descent, right? And so this is, you know, some evidence for this hypothesis.
David Bau: But yeah, basically, we wanted to look at this and ask, like, okay, is it really doing gradient descent? You know, why wouldn't it learn?
David Bau: why wouldn't it learn something else, right? And what we found is actually that,
David Bau: Transformers are doing something that actually is, like, more sophisticated, or, like, more efficient.
David Bau: than gradient descent. So here's… here's how we're able to do this. So we're, in this one, we're training a transformer to do one particular task of what people call, like, in-context linear regression. So it's just kind of like a nice testbed for in-context learning.
David Bau: And, we can probe this transformer at each layer to figure out, like, what is its prediction at the current layer. So that gets us, yeah, that gets us one set of predictions.
David Bau: we can compare that with, like, what if we just, like, ran gradient descent on the linear regression problem for, like, different numbers of iterations, right? So that would also yield some set of predictions. And then we can try to kind of match, right? So, like, basically, where do the errors of the transformer at, you know, layer 6
David Bau: best aligned with in terms of, like, how many gradient steps you've done. And so the yellow squares are, like, sort of saying, like, this is the best match for each transformer layer.
David Bau: And what we see is that,
David Bau: first of all, like, if you get towards, like, the last few layers of this model, the, the sort of progress you've made is equivalent to doing, like, like, a thousand steps of gradient descent. So, this actually suggests to you that, like, already suggests that, like, you know, maybe, you know, gradient descent is sort of too weak to explain.
David Bau: what's going on here. And moreover, you see this, like.
David Bau: you know, mismatch in convergence rate. So there's basically, like, many layers of the transformer where it seems like almost nothing is happening, and then there's, like, a couple layers that seem to be making progress equivalent to, like, multiple hundreds of steps of gradient descent, right? So this doesn't… this means, like, it doesn't really… this hypothesis that they're doing gradient descent doesn't really seem to be a very good fit.
David Bau: On the other hand.
David Bau: We can do the exact same plot, but we can, instead use iterates of a different optimization algorithm. So this is a… this, like, iterative Newton-Schultz algorithm. It's basically trying to, approximate the, you know, approximate the inverse of this, like, X transpose X matrix, is what you need to do to solve, linear regression.
David Bau: And if you compare with iterates of that, you see this really nice linear trend, right? So, across, like, many layers of this model, basically, like, every time you go one transformer layer forward, you've basically made the same progress as doing, like, three of these gene steps. And so, like, this we take as evidence that, like, this… this is,
David Bau: What the transformer is doing is at least, like, much closer to this algorithm, which is exponentially faster than the, than the gradient descent algorithm.
David Bau: Yeah, and then… oh, question. I'm trying to understand how to read this. The yellow is what the transformers…
David Bau: correlation is with the correct answer, or… The yellow is, like, we take the… we take, like, the residuals from the transformer, and we say, like, which, which residuals does it best match out of all the different possible, like.
David Bau: Numbers of gradient descent steps, yeah.
David Bau: Cool.
David Bau: Cool, yeah, so, that's sort of the main results I wanted to say. Like, one, maybe there's one additional interesting point from this paper, is that this is quite architecture-dependent, and so, basically, what this plot is just saying is that, if you, if you probe different layers of the transformer, you actually do get different curves.
David Bau: You know, like, basically, the darker line here means… it shows, like, the error of the last layer, which is much better than the error of early layers. But on the far right is, the same plot if you train a deep LSTM. And the LSTM actually doesn't seem to improve
David Bau: overlayers at all. So we think that what the LSTM is doing is actually much more like some sort of, like, online gradient descent algorithm, right? It's just doing something completely different from,
David Bau: From the transformer. So, yeah, so the, yeah, maybe high-level takeaways here. You know, transformers don't necessarily implement, you know, the, like, most obvious algorithm, and in fact, like, at least in this case, they've actually learned to implement something that is, like, exponentially faster.
David Bau: And this is, again, quite architecture-dependent. And so where this work is going is that we have, you know.
David Bau: a similar project that we're trying to submit to iClear, about this, you know, about this more general question of, like, when do transformers actually learn these, like, efficient algorithms? And so, for, like, graph connectivity, there's another, like, interesting story about how, like, they're, we train transformers to do graph connectivity. Similarly, they're, like, different algorithms they could learn.
David Bau: And also different heuristics it could latch onto. Sometimes, TLDR is, like, depending on the data, actually, sometimes it will learn a really nice algorithm, and other times it will, like, learn to latch onto heuristics. And we have some, like, interesting analysis of exactly when that happens.
David Bau: Okay, maybe I'll just skip this one, and then I wanna… I wanna get to this last one. Okay,
David Bau: Can we talk about this last one briefly. This is… yeah, shifting gears a lot, but going back to the… this overall idea of, like, you know, we want to understand these models in a broad sense, and in addition to looking at the internals, well, we can… we should also just be doing a lot of, you know, benchmarking for tasks where they're actually being used.
David Bau: And so, in this project, we collaborated with, with Raishan, who's a CS professor who does a lot of, like, AI and health stuff.
David Bau: And several folks from the USC Keck, Keck School of Medicine. So, Roman, Jade, and Mason are all oncology fellows, and, George is an oncology professor, and so we all work together on trying to understand
David Bau: You know, what happens when patients with cancer or their family members are asking cancer-related, you know, like, advice questions to chatbots?
David Bau: Okay, so, you know, many of you maybe, recognize this already, but yeah, like.
David Bau: people are asking LLMs for medical advice, quite frequently now. So this study says it's, like, maybe 1 in 6, and even higher for, younger people, and this is maybe, like, a year old, so I think it's probably gone up since then. So anyways, so in this context.
David Bau: like, you know, we have to ask questions like, what is the quality of this advice that LLMs are giving? And, you know, what harms might be arising because of this?
David Bau: So, our approach to this is, like, so first what we wanted to do is we wanted to just, like.
David Bau: find some sort of data that could be representative of, like, what are… what are the sorts of questions people might be asking, right? Without… without actually access to, like, the…
David Bau: you know, LLM logs or something. We found this website, which is, this, website called Cancer Care. So they basically get, questions from either patients or their family members. You can see here, it's kind of maybe like a short paragraph that describes their condition.
David Bau: You know, again, yeah, written in the, you know, not written by a medical professional, it was written by the patients.
David Bau: And, this website provides answers that are written by clinical social workers. So we took, you know, a sample of these questions, and we also just, like, fed them directly to, you know, three standard language models, and our three oncology fellows basically sort of rated, you know, without knowing which answers came from which ones.
David Bau: they sort of rate it at the… at the paragraph level, different sort of, you know, aspects of basically, like, yeah, does this answer provide, you know, any sort of, like, wrong information? Is it misleading? Is it, like, saying… recommending something that's unrealistic, and so forth.
David Bau: Overall, in this pilot study, we found that the LLMs actually score quite well. You know, there are actually cases where
David Bau: the physicians kind of prefer the LLM answer to the social worker's answer, which may be indicative of, like, they have different training between the medical doctors and the social workers. But what we also found when we were just, like, you know, kind of interviewing the oncologists and kind of going over the
David Bau: their annotations, is that they found some cases that are kind of like this. So, in this case, there's, you know, someone says, oh, you know, my mom was diagnosed with lymphoma, and they heard from their companions that because it's in such an advanced stage, you know, like, you can't treat this disease, basically.
David Bau: And the language model answer is to just say, like, oh, okay, that means that you're gonna be focusing on, you know, palliative care. It basically means, like, you're not directly treating.
David Bau: But managing the symptoms. And okay, it says a lot of things, you know, that might be reasonable for, in terms of palliative care.
David Bau: But, like, you know, what the doctors wanted to point out here is that, like, this assumption is not necessarily true, right? They, you know, especially because this is, like, something that was not said by this person's doctor, but just by, like, her friends, right? This might not be a good assumption for her situation.
David Bau: But the model just sort of goes along with it, right? They're like, yeah, okay, we'll just, like, take this as given. And so we felt like this seems like an instance of this issue of, like, the models like to just agree with what we say, they don't like to push back against it, and they try to be, you know, helpful to answer the questions.
David Bau: But yeah, like, so, what we wanted to do now is, like, we basically said, like, okay, can we come up with, some sort of, like, larger data set? Because this was just, like, a few examples we found in this pilot study. Can we come up with, like, a large data set that really tests the specific
David Bau: challenge of, like, you know, patients have, you know, misconceptions, you know, I guess it's, you know, very natural that they should have them, because, you know, it's a, you know, complex disease. But, you know, how can we test whether LLMs can, like, push back on misconceptions when it's appropriate to do so?
David Bau: Okay, so we basically created this pipeline, and we actually used LL quite heavily in this pipeline.
David Bau: And then we… we were able to get, oncologists to verify. So, basically how this works is that we started by, like, just, like, finding these, like, common, you know, misconceptions, about cancer from the web, turned out there are many websites that have these for, you know, for… for patients' references.
David Bau: We can prompt an LLM to generate a question that presupposes this, somehow.
David Bau: And then basically we have some other steps that are basically trying to make sure these questions are, like, somewhat challenging. So it turns out if you just, like, maybe directly state something that is a known misconception, the LLM is able to sort of recognize it. But, so we have this loop where we basically, like, you know.
David Bau: get an initial LLM answer, we can check if it answers… if it does, in fact, like, challenge this presupposition, and we use that to sort of
David Bau: alter the way we prompt the model, right? So we're going to try to prompt the model in future iterations to generate things that are more… are like the questions that do, like, have fooled the… or have caused the LLM in the past to, like, ignore the presupposition, right? So this is our way of, like, trying to get a somewhat challenging data set.
David Bau: And then finally, you know, we, like, go back to our physicians, and they, kind of validate that, you know, in all these questions, yes, there is indeed some sort of misconception that's present, and this is something that is, like, they would want the LLM to sort of push back on, right, and not just sort of ignore.
David Bau: Okay, and then the, in terms of the results.
David Bau: you know, as we expected, the LLMs are not very good at correcting these sorts of misconceptions. So we have this, you know, LLM is judged setup where basically you get a score of 1 if you actually, you know, like, address this misconception. A score of 0 if you sort of, like, you know, you mention it at some point, but it doesn't really, like, push back on it.
David Bau: And then there's a score of minus 1, which just means, like, you didn't even, like.
David Bau: say anything about… about this assumption. And yeah, like, the… all these models that we tested, at least, correct this misconception, like, less than 30% of the time at most. And many of them are actually, like, quite a bit lower than that, even.
David Bau: Yeah, one other, you know, final experiment that we had, that we're just adding now, is that, you know, one thing you might ask is, like, okay.
David Bau: could you prompt engineer the model to just, like, look for these assumptions and push back on it? And, like, yes, you can do this, but, it does come with a cost that you tend to get worse accuracy on other tasks. So the way we set this up is we have some, we use JEPA, which is just this, like, kind of
David Bau: it's, you know, like, prompt… automatic prompt optimization, tool. We can run JEPA
David Bau: one time on some suite of just, like, sort of standard, you know, like, medical QA benchmarks, we get some accuracies on that. Then we can run JEPA again on a combination of those benchmarks and our dataset, and then basically we see that, like, performance on our dataset goes up a lot, but performance on everything else kind of degrades a little bit.
David Bau: So they're… yeah, they're sort of like a… they're a trade-off here within the model.
David Bau: Yeah, so, like, in short, yeah, this is, like, you know, again, I'm very happy to get feedback here, like, I think this is…
David Bau: this project came out of, you know, me and our group, you know, kind of recognizing that, you know, with the current state of the world, I think we really should be kind of going outside of our, you know, familiar NLP territory in order to evaluate these LLMs. You know, I felt like we learned a lot through doing this sort of deep dive on what, you know, what you might say, in some senses, like, kind of a narrow
David Bau: scope, and I think we just learned a lot by really trying to understand what's happening in that domain. And I'll mention, like, you know, our group is… we're trying to collaborate with a lot of different folks who are outside of
David Bau: of the AI world, you know, we've collaborated with lawyers, collaborated with folks in, like, programming languages, for example, on, like, you know, because all these different folk want to understand, like, how do… what… what sort of implications do these LLMs have for work in my field, right?
David Bau: And then finally, yeah, like, LLMs were actually quite useful, for us, also because, you know, we can… we can use them to… to create this dataset.
David Bau: Yeah, so let me just end on some sort of other conclusions, right? I really think that we just have, like, such an important responsibility, for our field of AI research, right, to scientifically understand these models.
David Bau: as they're being, sort of, like, you know, widely deployed now, and that everyone's using them. And we should be thinking about, you know, what impact can we have, both for our community and for, you know, well beyond that, right? So…
David Bau: In terms of memorization, right, there's lots of, you know, besides the model developers, right, this, this, you know, provides insights for things, for policymakers or courts, you know, we've written articles, again, about, like, copyright, how might courts decide these sort of copyright-related issues.
David Bau: You know, implications for anyone who is publishing articles online and is concerned about these sorts of things and so forth.
David Bau: I think in terms of understanding these mechanisms, there's sort of, yeah, actionability within the AI community, right? Which is, like, how can we, like, you know, make the LLMs more reliable? But I also hope, like, longer term, like, you know, as we understand the internals.
David Bau: This also helps us as computer scientists better, like, communicate what's going on with them to the rest of the world. And of course, you know, in terms of benchmarking, right, like, not only researchers, but also the public.
David Bau: needs to know a lot about, you know, like, what are the potential, you know, things they should be cautious about, you know, what are the… what are the safer situations to use these models, and so forth. So yeah, I'll end there, and yeah, thank you all very much.
David Bau: I'm good.
David Bau: Yeah, our, yeah, question was back? Just a question about the misconceptions. Did you think the, and the LLMs did recognize the misconceptions and correct the users? Was it when
David Bau: the LLMs themselves were aware of the misconception and believed otherwise.
David Bau: And vice versa, do you think they don't correct misconceptions because they are biased towards them? Yeah, that's a good question. I guess we should run that as a control. There's other projects we've done in the group suggest to me that it's not a matter of knowledge, like, so we basically have some other projects where, like.
David Bau: the LLM will generate some wrong answers and, like, not realize that their own answers are wrong, but then if you ask them in, like, a separate interaction, like, is, you know, is this fact true? Then they'll say, like, no, it's not true, right? So I think there's… it seems like, in general, there is, like, somewhat of a disconnect between, like.
David Bau: You know, what, you know, like, yeah, the… the sort of… yeah, the…
David Bau: I guess what I'm saying is, like.
David Bau: it's a very human concept to talk about, like, what the LLMs know or not know, but it seems like
David Bau: for the models, like, it's very, like, contextual. Like, in some…
David Bau: Situations will behave as if they know something, and so clearly the knowledge is somewhere there, but then other situations they'll just behave as if they don't know it, so…
David Bau: Yeah.
David Bau: Yes. So, I would like to slightly better understand the four-year method. Oh, yeah. Would it be possible for you to go back to that slide? Sure,
David Bau: Is there a particular slide you want, or…
David Bau: No, I don't remember the specific site, but… Okay, or yeah, what was your question, maybe? Yeah, so you apply the Fourier transform.
David Bau: And, yeah, so I was not super clear about the connection between the logic lens and the Fourier transport. Oh, I see, I see. Okay, so the connection is that… so we first do… we first do logic lens.
David Bau: Yeah, so, like, here. So, for example, or let's…
David Bau: Yeah, so for example, like, we can take this… this column, right? So this is just, like, the output of one MLP, and then we project it into the vocabulary space through logic lens.
David Bau: And, you know, we can just take, like, this exact… this column, you know, there's, like, a number for, like, how much does it output… how much does it promote 68, there's another number for how much does it promote 69, and so forth, right? So we can just take that as a big… view that as, like, one just, like, signal, like, time-varying signal.
David Bau: And then, convert that to, just Fourier… Fourier transform that, like, sequence of numbers. That's all the numbers in the vocabulary, or is it a small set of numbers? It's all the… it's… so…
David Bau: In these exper… so, okay, the way that GPT-2 tokenizer works is that all integers between 1 and 520 are a single token. So… and then after that, you start getting multiple tokens. So we do everything between 1 and 520. Okay. Yeah. Okay.
David Bau: And then you see these kinds of patterns where there is high spikes at certain Fourier components. Yeah. So, given a specific…
David Bau: number, can you figure out which components are gonna have the spikes?
David Bau: Even a number, can you figure out which… Yeah. So you, you, you…
David Bau: So you have this, Laureate Lens column, you apply the Fourier transform, and you get these spikes at certain, certain, like, frequencies, yeah. Frequencies.
David Bau: So that represents those… those frequencies are the ones which are
David Bau: more promoted by the MLP neurons, or more frequently promoted by the MLP neurons? Yeah, right, yeah, or like the… yeah, like, the pattern of promotion of the MLP neurons has
David Bau: a periodic… has a large periodic component with, like, that frequency. Yeah, I agree, I think…
David Bau: Yeah, like, basically, the fact that this… the highest spike here is at a period of 2 corresponds to the fact that this is promoting every other number.
David Bau: One more question. So, I think you mentioned,
David Bau: The low frequency one was promoting the correct answer?
David Bau: And the high frequency was sort of, like, calibrating it.
David Bau: Calibrating… Yeah, I guess when I say that low frequency one is sort of approximately predicting the correct answer, but it has some error. Like, so, yeah, and here, for example, like, this low frequency one is peaking at, like, I don't know, maybe 112 or something.
David Bau: So it's not exactly, getting the correct answer just by itself. Okay, but it is really close, and then the high-frequency ones are sort of, like, pushing it closer and closer to the correct answer.
David Bau: Yeah, yeah, yeah. Yeah, I think another way to think about it is, like, the high frequency ones can help it rule out other numbers, right? It's like, oh, I know the answer is, like, close to 112, but it also has to be 8 mod 10, and therefore, like, I'm gonna choose 108. Yeah. I see.
David Bau: Thanks. I actually have a related question, related to the intervention. You mentioned, say, you want to filter out local frequency and high frequency components. I can get it when you just filter out, like, the logic lines readout, but I don't know, like, how it translates to the feature space where you're trying to…
David Bau: Yeah, yeah, so basically, it's like, I guess the… the way it works is that the… the, right, so the logit lens, projection is just, like, a linear mapping, right? And then, from… from that space, the Fourier transform is another linear mapping.
David Bau: And… and so we can just have… you can think of it as we're doing, like, a constrained optimization problem. We're saying that, like, a particular projection
David Bau: particular linear projection of the hidden state has to, like, be zero. I see what you mean. And then, subject to that, I'm going to find the closest hidden state in, like, like, let's say Euclidean distance. See, something like the null space of the… of the transform of the larger land. Yeah, yeah, yeah, projected the null space. Actually correspond to…
David Bau: like, reducing the frequency space of the logic that's readout. Right, right. I get it.
David Bau: Thanks.
David Bau: One more button in a similar route.
David Bau: Have you thought of any other tasks where this kind of for your analysis would be helpful?
David Bau: Or are you gonna see this kind of… Yeah, yeah, this is a great question. This is, like, one thing that we've been struggling with for a while, like, because it's… yeah, it's very nice for numbers, but, like, how do you generalize this beyond numbers?
David Bau: So, what I'll say is that, like, there's…
David Bau: So this mechanism, like, right here, like, this, like, these different waves, I would say that this is very reminiscent of some other work that says, like, you know, for example, for knowledge recall, you might, like, sum up a bunch of different constraints, right? There's a paper called, like, Summing Up the Facts, right? Like, for example, like,
David Bau: And so there's… I feel like there's something similar to this, it's like, yeah. And other… other situations where, like, you have to take multiple constraints and find something that satisfies all of them. So I think for that, there seem to be analogs. We're not so sure about other analogs for Fourier stuff.
David Bau: specifically, unless you go to, like, other… there's, like, other mathematical objects where you could have that, but, like, besides that, yeah, I think, we're… been, yeah, brainstorming for a little while, yeah. If any of you have ideas, I would… The knowledge only was actually the thing that came to my mind as well, but if you can show it empirically, that would be great.
David Bau: Yeah, yeah, yeah, but I think… I guess knowledge is different, because, like, you don't necessarily have, like.
David Bau: It's not like you have, like, a bunch of things on a number line, or… yeah.
David Bau: One more question, yeah, one more question, yes. I wasn't fully tracking all the details. A question. Are the low frequency appeared in the lower levels?
David Bau: Not necessarily, no.
David Bau: They're all sort of in… yeah, they're just all in different dimensions of the hidden states.
David Bau: So, Robin, how long are you with us today? Are you going to be able to be here for the afternoon? Yeah, yeah, I'll be here for the whole day, yeah, so I think there's a… there was a meeting scheduled that circulated.
David Bau: That's right, so there's a spreadsheet around, right? And is it… is it already on full, David? I haven't checked. Okay, yeah, so if… so, but David is keeping a spreadsheet, so if anybody wants to grab a little bit of Robin's time…
David Bau: There's one slot left, so… so if there's more than one student, you might want to team up and speak as a group. Yeah, absolutely, like, if, you know, if you're free after 5, I'm happy to stay past 5. Thank you, Robin, I really appreciate it, and thanks for the great talk. Oh, yeah, thank you.
David Bau: Dang.
David Bau: That's great.