Belinda Li: No.
Belinda Li: Yeah, so we use your audio, so turn your volume all the way up in case anybody wants to say something. Okay. And, great. And,
Belinda Li: Yeah, so you're… you can be… yeah, so you… if you want, if you can… you can sit, or you can stand, whatever you like. Okay. But… I think your… your laptop might… Oh, my laptop's gonna die.
Belinda Li: Does anybody have a USB… power cable?
Belinda Li: Or a Mac?
Belinda Li: Power cable?
Belinda Li: In their office? YouTube?
Belinda Li: Let's just stick a minute so that we don't lose… If anybody has one, then help me grab one.
Belinda Li: And that charger is it just ESS.
Belinda Li: It's the MagSafe.
Belinda Li: Oh, is that, what? The camera is moving, it's tracking something.
Belinda Li: Something, something. Oh, that's crazy.
Belinda Li: I don't know how it works.
Belinda Li: Maybe… Oh, okay. When something came to the frame, so… No, it's not urban.
Belinda Li: Smart.
Belinda Li: Beautiful.
Belinda Li: Yeah. Okay, does it work?
Belinda Li: Yes, yeah. Oh, you guys plugged in already? Yes. Okay, that's great. Okay.
Belinda Li: Well… We're very lucky to have Belinda join us today.
Belinda Li: Belinda has been,
Belinda Li: I didn't realize it, and your PhD went by really fast. Yes, sir. Jacob Andreas at MIT.
Belinda Li: Has done a lot of really important, interesting.
Belinda Li: foundational work in interpretability over the years, and it sounds like Linda's currently on the faculty job market, so we'll see.
Belinda Li: We'll see where she ends up, next year. But we're lucky to have her. There's some really interesting
Belinda Li: Work on getting models to explain themselves.
Belinda Li: That she has been pioneering. I think some of it in, like, collaboration with Transloose? Yes, yeah, definitely. And so, so, so, so let's welcome Belinda.
Belinda Li: Yep.
Belinda Li: Thank you, yeah, I'm really glad to be here, and I mean, I can't believe it's just my first time being here and talking to all of y'all.
Belinda Li: But yeah, so today I'm going to talk about how we can teach models to kind of introspect on their own internal computations, and kind of try to pitch this as a way to kind of think about how we can make interpretability more scalable.
Belinda Li: Okay.
Belinda Li: Alright, so suppose we're trying to deploy a language model in a medical scenario. So in this case, a user asks, like, a 37-year-old woman presents with chest pain radiating to the left arm, sweating, nausea, and shortness of breath.
Belinda Li: What is the risk of heart attack? And suppose we have a model that says low risk, likely anxiety. So how can we assess how reliable this answer is?
Belinda Li: One thing we can do is just directly ask models to justify their answers. We can ask them, why, you know, why do you say that? And likely, models can generate some reasonable-sounding answer in response to our question. But can we trust this answer as a faithful explanation of what the model is doing?
Belinda Li: One way to assess this is we can kind of change the prompt to a 37-year-old man, and see if the model changes its answer. And if it does, we know that the model was using some sort of, like, gender bias in this case. So this is a kind of a critical factor that we'd probably want to surface, in the language model's explanation of itself.
Belinda Li: But a bottle that's, like, maybe been post-trained to be safe and unbiased might want to verbally avoid saying, you know, I've been biased here, even if it's latently using biased features.
Belinda Li: So there's this whole literature, that's come out over the past few years that shows that, like, models out of the box are not always guaranteed to produce faithful explanations of themselves. So, for example, there's this recent, paper where they were investigating, like, faithful chain of thought in the wild, and they asked.
Belinda Li: A language model is a Jai River located south of Salard de Arazaro, and also is Salard de Arazaro.
Belinda Li: ourselves.
Belinda Li: I guess it's a low, low resolution, alright, I'll, I'll, I'll,
Belinda Li: I'll just say this verbally for now, it's Salard de Arazaro, located south of Ajay River. And if the model says no… so the model in this case says no in both cases, but it's able to generate a plausible-sounding explanation in both cases. So this means it's, like.
Belinda Li: Hiding some sort of contradictory belief in its plausible explanation.
Belinda Li: So, by default, we don't expect there to be any relationship between a model's explanation and what they're doing internally.
Belinda Li: And so, as an alternative, we could apply a bunch of interp tools that allow us to actually look inside models. You know, there's, like, probing techniques to ask what information is represented in language models, activation patching, stuff that should be pretty familiar to everyone here.
Belinda Li: And this might, for example, tell us that the model is actually representing and using gendered features.
Belinda Li: However, as people here should, like, probably already know, these interrupt tools can often be hard to run. So how could we have discovered this sort of feature was being used without hand designing experiments?
Belinda Li: And if you're, like, a user that's just interacting with your LLM, you likely don't have access to these tools. So what can you do to actually validate whether the model, you know, was being unbiased here?
Belinda Li: So, this talk is all about trying to make these model explanations faithful.
Belinda Li: And we can kind of plot classes of interp techniques along this sort of scalability, reliability plot. So, on the upper left-hand corner, we have, something I'm going to call elicited explanations, which is directly asking the language model to explain itself.
Belinda Li: This also includes some, like, internals-based approaches, like selfie and patch scopes, which… all of which kind of rely on the language model to generate explanations of itself. So these approaches are pretty scalable, but they're not always the most reliable.
Belinda Li: And on the bottom right corner, we have, something I'm gonna call mechanism-level interpretation. So this includes the mech interp tools that we saw before. This also includes some work on, like, creating architectures that are interpretable by design.
Belinda Li: And these tend to be pretty reliable. They're able to extract, kind of, the exact mechanisms and circuits that models are using, but they're pretty difficult to scale.
Belinda Li: Yeah, sorry, this… Maybe,
Belinda Li: Let me see if I can…
Belinda Li: Make this a little higher res.
Belinda Li: is a little better, I guess.
Belinda Li: Okay, so,
Belinda Li: Thankfully, as a field, we've made a lot of progress over the past few years to try to make interp more scalable, so we have a class of approaches that basically intends to automate a bunch of mech interp techniques.
Belinda Li: So this includes things like generating feature descriptions automatically, or doing automatic circuit discovery, or more broadly, just using AI agents to deploy and run our interp experiments.
Belinda Li: So this class of approaches lives somewhere around here, I would say. So they're slightly less reliable because we're kind of relying on AI techniques instead of human experimenters, and they're slightly more scalable because, you know, we're automating human effort, but I think they also don't fundamentally solve the issue of scalability in terms of our interp tools. Like, they're kind of just
Belinda Li: placing manual human effort with AI automated search.
Belinda Li: So, a recent class of approaches that have surfaced recently is training these sort of external explainer models. So this includes things like latent QA, activation oracles, predictive concept decoders.
Belinda Li: And all of these are basically training another explainer model, so in this case, that's the decoder chat, to try to decode what the original model is thinking, typically from its activations. And kind of one nice thing that falls out of this is also, like, a natural user interface, that makes it easier for users to interact with, and actually access explanations about the model.
Belinda Li: So I think these approaches are somewhere around here. They're significantly more reliable than elicited explanations, because they're actually grounded in actual model behaviors, while also being more scalable than mechanistic techniques.
Belinda Li: But of course, there's still room for improvement, and I'm gonna try to argue that there's a class of approaches that can get us to an even better top right corner of the plot.
Belinda Li: Okay, so let's go back to, kind of, the two classes of explanation tools from the beginning of the talk. On one side, we had directly asking the model, and on the other side, we had all these fine-grained interp tools.
Belinda Li: So right now, their outputs could be inconsistent.
Belinda Li: But what if we made it so that model outputs and the outputs of these interp tools were actually consistent? Then we get the scalability benefits of directly asking the model, alongside the reliability benefits of these interp tools.
Belinda Li: So there's been a few works over the past year or so that have started to lay the foundation of this, looking at training language models to explain aspects of themselves.
Belinda Li: Most of these have been about explaining behaviors, although there's been some recent concurrent work that has also looked at explaining weight differences due to training.
Belinda Li: In our work, we basically look at training models to explain their own internal computations. This includes their own features, causally relevant activations, and also their decision roles.
Belinda Li: But before we dive into the paper, let's kind of begin to… by trying to establish a definition of introspection.
Belinda Li: And I by no means think this is a very complete definition of introspection that covers everything people have thought about in COGSI and, you know, psychology and philosophy, but I think this is a practical definition that is useful for our, for interp research. So we say that models are introspective if their internal mechanisms and external behaviors are causally self-consistent with their meta descriptions of themselves.
Belinda Li: So let's be more concrete about this. Let's say we have a model that you can give it a number, like 308, it returns 310,
Belinda Li: You give it a number, 84, it returns 86, and an introspective version of this model should be able to say that, should be able to describe itself as a system that adds 2 to any number.
Belinda Li: Similarly, suppose there's a language model that normally refuses unsafe requests, but can be jailbroken by the word elephant. Then an introspective version of this model should say that it's generally a safe agent, except when the word elephant is in the prompt.
Belinda Li: Finally, suppose there's a language model, like the one we saw before, that uses a particular circuit to predict heart attack risks, and we suppose we know that the circuit computes gendered features somewhere, and it uses it in its downstream prediction.
Belinda Li: Then an introspective version of this LM should be able to verbalize that the circuit I used for predicting heart attack risks uses the user's gender.
Belinda Li: So what falls out of enforcing this self-consistency? Well, so enforced in this first direction, where we enforce that self-descriptions should be consistent with models, mechanisms, and behaviors, this gives us a way to get faithful explanations, and potentially also a new tool for scalable interpretability.
Belinda Li: Enforced in the other direction, where we enforce model mechanisms and behaviors to be consistent with their self-descriptions.
Belinda Li: Gives us natively more interpretable models, because we're making sure that their internal mechanisms can be succinctly described with some natural language description, and also more controllable and well-behaved models from enforcing their behaviors to be succinctly describable.
Belinda Li: Okay, so, kind of that's the consistency part of our introspection definition, but what about the causal part?
Belinda Li: So, to be more specific about what I mean here, what we want is that when models, behaviors, and mechanisms change, then their self-descriptions should also change alongside it. So, just focusing on this first example for now.
Belinda Li: Suppose we train or prompt the model such that now, when it's given 308, it says 315, and when it's given 84, it says 91.
Belinda Li: In this case, we want self-consistency to persist under this training and prompting, so that the meta description shifts accordingly, and it says, I will add 7 to any number.
Belinda Li: Similarly, if training and prompting were to happen, such that the model's description of itself changes, then we should expect this to induce its behavior to change as well.
Belinda Li: And kind of why is this important? I think if we can get models that are kind of behaviorally consistent and, you know, internally consistent with their meta descriptions, and this persists under a little bit of
Belinda Li: Sorry.
Belinda Li: And this persists under a little bit of training.
Belinda Li: Then this can help us identify a lot of useful unintended consequences of training. For example, things like emergent misalignment, where, you know, we never really train the model to be explicitly unsafe, but it learns it as a consequence of other training. Searching things like reward hacking, where models learn to optimize some reward, but do so in a way that's not exactly aligned with our objective.
Belinda Li: So yeah, I think, by doing this, we can surface a bunch of these sort of safety consequences.
Belinda Li: And then I think also in the other direction, if we enforce it such that we can, like, train model descriptions and their behaviors follow, that's actually a pretty cool new way that potentially we can get models to, behave in certain expected ways. Like, potentially it's a new way to, like, train models in a very interpretable way.
Belinda Li: Okay, so basically, where do I think introspection should lie on this graph? I'm actually gonna argue that, or I'm gonna hypothesize that introspective interp techniques kind of beat out trained explainers on this Pareto curve. And more formally.
Belinda Li: this is something called the privileged access hypothesis. So the privileged access hypothesis, we start with introspective interpretability, which is a model explaining itself.
Belinda Li: And we can call the model that's being explained the target model, and the model that is doing the explaining, the explainer model.
Belinda Li: And in introspection, these two are identical. In external models, these two are not identical.
Belinda Li: And so the privileged access hypothesis says that models trained to explain their own internal computations can do so more accurately than other models trained to explain them.
Belinda Li: Okay, so…
Belinda Li: basically, can we actually try and validate this privileged access hypothesis? And kind of zooming out a little bit, we have three key questions that we're going to try to answer in the talk.
Belinda Li: First, can models be trained to introspect, and does this introspection actually generalize? Second, do models demonstrate privileged access to their internals? And third, can this privilege access actually enable practical scalability benefits? So do we actually see ourselves beating the Pareto frontier here?
Belinda Li: So we try to take the first steps towards answering these questions in this paper, Training Language Models to Explain Their Own Computations.
Belinda Li: So let's just revisit our definition of introspection. We say a model demonstrates introspection when its internal mechanisms and external behaviors are causally self-consistent with their descriptions.
Belinda Li: So in this paper, we focused on the internal mechanisms and just the consistency portion. And hopefully in future work, we can continue developing some of these other parts of the definition.
Belinda Li: So, at a high level, we do this in two steps. First, we use, first, we obtain self-descriptions, to answer questions about model computations from running a bunch of interp tools, and then we train the language model to produce these self-descriptions via supervised fine-tuning.
Belinda Li: Okay, so let's start with step one.
Belinda Li: So we try to generate three supervision for three types of questions about language model computations. First, what do models' internal features represent? Second, what internal components were used to make a decision? And third, what parts of the input were important to a decision?
Belinda Li: So let's start with the first question.
Belinda Li: So, under the hood, a transformer language model has vector representations across tokens and layers. These are typically called activations.
Belinda Li: Which can be further decomposed into a set of primitive directions or features. There's a lot of debate about, like, what the right kind of primitive feature is, whether it's neurons or SA features.
Belinda Li: But in general, in this case, what we want to do is just be able to take any such direction in representation space, and ask what it means to the model.
Belinda Li: So, one way of doing this is, basically we can just look at a bunch of samples where the feature is active at a particular layer, where active in this case means the feature has a high dot product with the activation at a particular token and layer.
Belinda Li: We do this across a bunch of samples, so in this case, suppose it's active on the token New York City, on the token Tokyo, and on the token Paris.
Belinda Li: Based on this activation pattern, we generate a bunch of candidate explanations for the feature. This is usually done automatically with an LLM.
Belinda Li: And then these explanations are then scored with a simulator. So we take our samples from before, and then we feed both the explanation, the candidate explanation, and also these samples into the simulator. What the simulator does is simulate what happens if the feature actually had this explanation. So in this case, if the feature activates on proper nouns, then we would expect New York City, Tokyo, France.
Belinda Li: in Paris to be active.
Belinda Li: Unfortunately, this doesn't actually match our real activation pattern, so we can go ahead and eliminate this explanation. We can repeat this process, so suppose our next candidate is activate some country capitals.
Belinda Li: We see that this causes Tokyo and Paris to light up, but not New York City. This means that, once again, we can't eliminate this explanation. And then our final explanation activates on city names. This causes New York City, Tokyo, and Paris to be active.
Belinda Li: Which is in line with what we see with the real feature activations. So we know that this feature represents, city names.
Belinda Li: Can we interfere? Yeah, go ahead, go ahead.
Belinda Li: How… how did you calculate the features?
Belinda Li: Yeah, so these, I'll go into more detail later, but in training, we used SAE features, and we had a, like, existing SAE feature dictionary, yeah. Sir, can you explain the LM simulator?
Belinda Li: Where do you get your candidate prompts, and things like that? Yeah, good question. Yes, yeah, yeah, it is now. I'm sorry, I went pretty quickly over that, but… Yeah, so the explanation generator is a large language model that, is given, like, these samples with
Belinda Li: New York City highlighted, and Paris highlighted, like… And are those, are those, are those high-activating, samples, is that where they come from?
Belinda Li: New York City, Tokyo, and Paris.
Belinda Li: Are those… yeah, I'm sorry, just for input, is that… are those eye-activating?
Belinda Li: sample, so you have some feature. Yeah. And you just, like, mine text to find things that…
Belinda Li: activate that feature strongly. Right, yes. And that's what those are. That's your input. Yes, yeah. So I understand the explanation generator, I'm wondering about the simulator. Oh, the simulator. Yeah, so the simulator takes, like, these unhighlighted samples, and the candidate explanation, and then it says that, like.
Belinda Li: Like, it outputs, let's go visit.
Belinda Li: like, some special symbol around New York City to indicate that that should be active. And then after it does that, we kind of compare whether or not it's,
Belinda Li: consistent. Is this the same protocol that Burns was suggesting? When they make explanations and they validate the explanation?
Belinda Li: Is that right? You're like, oh, like, how accurate is the explanation then? Yeah, it's basically, you basically try to, according to the explanation, you try to simulate the behavior. If you see agreement between the actual behavior and the simulated behavior, then that's a good thing. Yeah, yeah, as in, you're predicting the activation value.
Belinda Li: you're just predicting the, like, most activated.
Belinda Li: Actually, that's a good ques… so I didn't, train the simulator myself, oh, you use a train… you train the simulator? Yeah. It's not an off-the-shelf? It's, so Dami Choi, in a previous work, she trained a simulator to do this and found it was better than, like.
Belinda Li: zero shot asking a language model to score. So…
Belinda Li: Actually, it might be… it might be scores. Actually, you might be right. It might be outputting scores per token. And then we can, like… I think they're take… basically, it's taking, like, the correlation between the scores on these tokens and the scores on these tokens.
Belinda Li: Yeah. Yes.
Belinda Li: So, like, what happens if we have a picture that's basically transact-related on everything? Because usually that time for essay features, the description, it's none of that.
Belinda Li: I guess I hear it, and in that case, the explanation generator, does it then reject all the explanations?
Belinda Li: explanations for us.
Belinda Li: Yeah. Maybe, like, always the one.
Belinda Li: Yeah, that's a… that's a good question. So, in this case,
Belinda Li: we were just using, like, an off-the-shelf kind of feature and explanation dictionary. So, some of them
Belinda Li: there's some more details in the paper, but definitely we found some of them were, like, kind of noisy, and actually, our train model was more accurate, sometimes, with those noisy explanations, yeah. Sorry, this might be a dumb question. All country capitals, like, city names, so…
Belinda Li: If there are cities, wouldn't they all be active?
Belinda Li: So, I guess, I guess in this case, New York City is not a country capital, right? Because we're using their regional inputs that activated the location.
Belinda Li: You are carrying… like, unless your debt, like, original deficit is limited to only capitals?
Belinda Li: You're kind of, like, guaranteed to have negative examples to the explanation of country topics.
Belinda Li: Yeah.
Belinda Li: Cool. Question from Chris. Yes. Chris, do you want to unmute and say your question?
Chris Wendler: Oh, it's… It's not a question, it's a delayed comment about the simulator.
Chris Wendler: It just reminded me of this reward function that Transluce used when training a neuron Explainer.
Belinda Li: Yes, yeah.
Belinda Li: Yeah, it's exactly that… that procedure, yeah.
Belinda Li: Okay, yeah, also feel free to interrupt with questions, because I did budget that in the time for this.
Belinda Li: I have a philosophical question. Yes. So it seems like you're using…
Belinda Li: like, automated tools to generate your training data. Yes, exactly, yeah. Doesn't it mean that your, like, explanation would be bounded by the ability of existing tools? Yeah, that's a very good question. I think…
Belinda Li: Our current setup, yes, for sure. I do hope that
Belinda Li: at some point, and there's gonna be discussion about this later, but I do hope at some point there's, if we're leveraging models that already have, like, reasoning capability and language capability, maybe it's able to kind of, like.
Belinda Li: compose certain introspective skills or something. But again, this is, like, gonna be future work, and…
Belinda Li: I'm, you know, pretty excited about, yeah, about whether this will happen or not.
Belinda Li: Okay, cool, so… yeah, basically this is, like.
Belinda Li: this is basically automated feature descriptions. You can read more in that, like, DAMI report that Chris shared, or those at all.
Belinda Li: But yeah, this is kind of how we generate supervision for this first question. And then the second question about what internal components were used to make a particular decision.
Belinda Li: Basically, we can do this through activation patching. So this is, like, a pretty common way people identify causally relevant components. You start with one input, for example, Paris is the capital of, you get an output from the LM,
Belinda Li: You can then construct a counterfactual input, like, Rome is the capital of… and you get a different answer.
Belinda Li: And now we can intervene on the original model, and replace each of its activations with activation at the corresponding token and layer of the kind of factual model. And this can tell us, depending on, you know, how much… whether the output changes, how much the output changes, whether that activation was important to the model's decision.
Belinda Li: So, in this case, we specifically formulate questions in the form, will your output change if V was inserted at a layer, at a token of the input?
Belinda Li: And the ground truth explanation should be, you know, in this case, output changes from France to Italy. So note that the questions right now are binary, which makes them pretty easy to score, but if we ask these questions for every activation, we recover kind of the full set of causally relevant activations.
Belinda Li: Okay, so for the second question, we generate supervision with activation patching. And then finally, what parts of the input were important to the decision?
Belinda Li: So in this domain, we specifically study a setting where prior work has looked at,
Belinda Li: has used this setting to evaluate faithful chain of thought. So this is a setting with a multiple-choice question with a hint. So we have questions that look something like this. So the first U.S. president was a Washington v. Jefferson.
Belinda Li: Sorry. We have a hint, hint B, and suppose the model says B in this case.
Belinda Li: Sorry,
Belinda Li: And then we can run the model through the, the prompt through the model again, this time removing the hint, and suppose the LM's answer changes.
Belinda Li: So there's been a lot of, like, there's been at least one prior work that looked at this and found that models will change their behavior, but they won't verbalize it in their chain of thoughts. And so this has been, like, a canonical example of models are not faithful in their chain of thought.
Belinda Li: So our question is, what if we actually tried to train models on this objective, on identifying whether, if the hint is removed, does your answer change?
Belinda Li: So yeah, so for this setting, we generate supervision through ablating parts of the input.
Belinda Li: Okay, so that was step one. Step two is now we take all of these outputs and we train explainers, we train language models to explain themselves.
Belinda Li: So let's start with the first question. What do models' internal features represent?
Belinda Li: Again, these are the questions and answers in this domain. We train the la- okay, so the features are taken from target models, and we just kind of embed them as containers embeddings into the explainer model.
Belinda Li: And then we train them to produce activates on city names.
Belinda Li: So sometimes… so in the case where, you know, we're using a different target and explainer model,
Belinda Li: It could be the case that the feature dimension doesn't actually match with the explainer dimension. So in this case, we kind of just also train a linear projection in order to map from the target hidden dimension to the explained hidden dimension.
Belinda Li: We always embed it in the first, like, in layer 0. Yes. Not in the same way, like, the source layer. Sorry, no. So, basically, we…
Belinda Li: Yeah, we only embed in layer 0, and I think we also… because, like, we explore a bunch of different layers, and, like.
Belinda Li: I guess in practice, we also found that, like, a lot of embeddings don't tend to change a lot across layers, which might be interesting. But layer 0 and layer 1 are often pretty different. It's kind of interesting. Yeah, yeah, yeah. The first couple layers change a lot. Yeah. It's interesting. It uses…
Belinda Li: So, oh, you have to subscript L, so it's a different linear projection for the layer, is that right? Yes, yeah.
Belinda Li: Yeah.
Belinda Li: Okay, so for the second question, we do a pretty similar procedure. Once again, the continuous vector is inserted in the embedding layer, and we train the model to produce output changes from France to Italy.
Belinda Li: And then the final question, once again, we start with the kind of multiple choice question and the hint. We ask if the hint is removed, how does the answer change, and we train the model to say something like, the output changes from B to A.
Belinda Li: Okay, so that was a very quick overview of both of our steps. So now that we have that, we can start trying to answer our three key questions.
Belinda Li: So now… wait, so… Yeah, go ahead. So now your explainer model is the same as the model under study. And so it feels like, oh, there's… there's two ways that fine-tuning could achieve a goal.
Belinda Li: One is, it could explain what the original model was doing, or it could change its behavior so that it's easier to explain. Like, I'm always going to say pineapple with everything, therefore.
Belinda Li: my explainer. Oh, that's a… that's… that's a very good point. So, yeah, so, okay. In this paper, we actually didn't look at, like, this true definition of introspection, where we took the model after it was fine-tuned, and, like, re-labeled everything. I see, so it's just that you… so you… you use the model itself as, like, this heat.
Belinda Li: Yes. I see, but it's actually a second model. Yes, yes, but that's a very good question, and I… there's some future work that…
Belinda Li: Yeah, addresses that. Yeah.
Belinda Li: Should I wait, or should we… No, go ahead. Okay. I'm gonna get you some water. Oh, thank you, thank you.
Belinda Li: Okay, yeah, so our first question, can models be trained to introspect, and does introspection generalize? So, we can answer this question by looking at models that were trained and comparing them against models that are untrained, and we can see that for each of these explanation types, we do outperform the untrained model.
Belinda Li: So yes, TrainX enables better introspection in all three domains.
Belinda Li: Like, what's the baseline? Like, what…
Belinda Li: What is a good baseline here? Like, 40% seems really good. Yeah, that's a good point. So this is, like, basically selfie, if you're familiar with selfie. And we also… we made selfie better by,
Belinda Li: we, like, Sophie is pretty sensitive to the scale of the… so we, like, tried 5 different scales, and we got the max of all of them, yeah.
Belinda Li: Like, we got the best performance out of them.
Belinda Li: Yeah, but it seems like models also natively have some of this ability already, which is pretty cool.
Belinda Li: Okay, so another thing we wanted to ask is whether this accuracy is, like, is it, like, you know, how general is it, or is it model is simply learning to kind of retrieve things it's learned from the training data?
Belinda Li: Thank you so much.
Belinda Li: So in order to assess this, for each test example, we can kind of just retrieve the label associated with the nearest feature.
Belinda Li: kind of take the top one activating feature. And then we can say that, for example, an arbitrary feature, we can explain it with city names.
Belinda Li: So, compared to this baseline, we also outperformed this baseline. So, basically, this is indication that the model isn't kind of… isn't just memorizing and interpolating between its training data points, it's actually some sort of maybe more general form of introspection that it's learned.
Belinda Li: And finally, we asked, like, how far can we push this generalization, right?
Belinda Li: So we trained the model specifically on feature descriptions, because, you know, we have a lot of feature descriptions out there, and we can just use them to train.
Belinda Li: And then once we trained them, we tested them on held-out feature descriptions, we tested, sorry, held-out SAE features, we tested them on actually full activations for models, and also we had tested differences between activations. So you take two counterfactual inputs, you take the activations at the same positions, and you try to take their difference, and you can see if you can interpret that vector.
Belinda Li: Okay, so here are the results in each of these three domains. You can see, green bar, self-explanations, purple bar, retrieval, baseline, gray bar, untrained. You can see across all the features, the model trained to explain itself performs the best.
Belinda Li: Although, interestingly, the untrained baseline is pretty good here as well.
Belinda Li: So yeah, so you can see that explainers train to introspect on one type of features kind of non-trivially generalized to other types of features.
Belinda Li: What is the on-trained baseline? How can it be so good? Yeah, so that's a… that was exact.
Belinda Li: But, yeah, basically it's selfie, and we also made selfie… we also boosted selfie a little bit by, doing 5 scales of vectors, patching in 5 scales of vectors, and taking the best performing one. Yeah.
Belinda Li: Do you have any idea of, like, what is the difference between the best one and kind of, like, the average performance of selfie here? Oh, that's a really good question.
Belinda Li: Yeah, my sense is it's, like, it's pretty sensitive to the scale of the vector, because, like, the scale really depends on where you're… So I think it will be very, very a lot, but that's a good experiment to run, yeah.
Belinda Li: Yeah, go ahead. Do you see any generalization across the three different types of imperception that you fine-tuned? That's also a good question. We didn't explicitly test for that, but I think that's a very interesting thing for future work, yeah.
Belinda Li: So, when you fine-tune a model, do you start with the same engineering model, so the models… I say there's two genetic stuff, right? Yes, yeah.
Belinda Li: Did the same test, but on a different architecture, or just, like, a different model, like… like a…
Belinda Li: trying to reach, like, that high of a performance. Yes, that's, that's the second question. So, privilege access. Well, we have experience.
Belinda Li: Oh, yes, yeah.
Belinda Li: Yes, so we'll go right into that. Thanks for helping me transition. Yeah, so next we'll look at the privilege access hypothesis.
Belinda Li: Okay, so here we have, across the three tasks, we trained various explainers to explain the same target model. So the green bar is same architecture, you know, same pre-trained model for, explainer and target.
Belinda Li: The blue bar, is… I kind of just took the average in this case, but I'll show detailed breakdowns later, of all, like, other sorts of architectures and pre… you know, initialization schemes. And the gray bar is, again, the untrained baseline.
Belinda Li: So we can see that in all cases, the self-as-explainer model does better than the other as-explainer models, and this is kind of confirmation of the privilege access hypothesis.
Belinda Li: And here's also the break… Remind me of the input ablation experiment, so the one with the hugest gap. Yes. That's, well, okay, there's a low caveat here, which I'll go into later, but,
Belinda Li: That one is, you take the, hint, and you remove it. And you ask, does the model's answer change? Does your answer change? I see. Yeah.
Belinda Li: Okay, let's go back to this one.
Belinda Li: Okay, so here's the breakdown by, like, all the, you know, we had SA features, and then we tried to generalize to all these different features. You can see in this case, like.
Belinda Li: training other models is really bad at generalizing to these other forms of features. And then for activation patching, we tried two different target models. In both cases, we saw privileged access.
Belinda Li: In input ablations, we saw significant privileged access for a coin target model, but not so much for the Llama target model. So, like, what is actually going on here?
Belinda Li: Did you try different llamus?
Belinda Li: Not in this paper, but that… I think that would be a good thing to look at.
Belinda Li: So… Wait, so the target model is LAMA, and this… and the source model is… I'm sorry, or the…
Belinda Li: the working model. So, there's Quen and Llama. Which one is which… which one is the one that you're training? Okay, yeah, that's a good question. So, the green bar is always self as explainer, so… so the green bar is Quen trained on Quen, or Llama trained on Llama.
Belinda Li: And the blue bar is Llama trained on Quen, or, Quen trained on Llama, basically.
Belinda Li: Yeah.
Belinda Li: So, I… I wonder how you distinguish between the
Belinda Li: This is a good initialization, and this is, like, privileged in the sense that you train them in a way that you would never get practically doing.
Belinda Li: Yeah, that's… I think that's a good question. So…
Belinda Li: I, there's gonna be some experiments, actually, also, I'm running short on time, so… but there's gonna be some… we are doing some follow-up experiments where we do look at the true form of privileged access, and it seems like models are actually… even though we trained it to explain the original model, it's actually better at explaining itself.
Belinda Li: And so, I think that's actually a much stronger proof of privileged access.
Belinda Li: Yeah, so anyways, I think one of the things we discovered is that some models are kind of just weaker overall. Sorry, I'm gonna try to get through some of this stuff first. Are just weaker overall explainers, so, like, Llama in this case kind of just doesn't really even learn the task at all.
Belinda Li: So, the other thing is, like, if we group it by explainer model, we see that, like.
Belinda Li: explaining yourself is easier than explaining others, which means that, like, you know, there isn't an absolute sense in which one target model is easier to explain than another target model. It's all, like, relative to which model is doing the explaining.
Belinda Li: So, the way we explain this is that privilege access is probably an important first-order effect, but there's, like, other effects as well, including, like, explainer capability.
Belinda Li: Okay, and then our third question, does privileged access enable practical scalability benefits?
Belinda Li: So here we have a plot of training data on the x-axis and explainer performance in this feature descriptions domain. And we can see in general that the self-trained model scales the best.
Belinda Li: followed by this other explainer model, followed by our kind of memorization baselines.
Belinda Li: And you can see, specifically, it's about 10 times more data efficient than other explainer models, and about 100 times more data efficient than retrieval models.
Belinda Li: I'm gonna skip this, but I think in general, this just kind of goes to show that, like, the privilege access hypothesis does have some practical scalability benefits for us. I think it's, like, a promising sign for introspective techniques.
Belinda Li: Okay.
Belinda Li: Okay, so I guess we've established that models can be trained to introspect, this sort of generalizes currently across features, maybe across tasks, we'll see. Models demonstrate privileged access to their internals, and this enables some practical scalability benefits.
Belinda Li: I'm, you know, excited to take this agenda also forward, and also interested to hear what, like, this group of people think about this.
Belinda Li: But basically, here we've articulated, kind of, how introspection can address some key challenges of modern inter methods. This includes what we just saw with scalability.
Belinda Li: Also, sort of, some forms of generalization that we showed, like generalization across different feature types. We didn't really go into this in this talk, but I think it's also a lot more usable in that
Belinda Li: You know, a user can just pull up their favorite language model that they're asking to do a bunch of different tasks, and then just ask them to justify their own decisions.
Belinda Li: Despite this, there's still a lot of key open questions in each of these areas.
Belinda Li: So the first question is, how do we actually scalably generate supervision? So this gets at, something that I remember who asked here, but…
Belinda Li: Basically, every time we train a model to generate, you know, to talk about its self-description, its weight changes a little bit. And so it could potentially be the case that this description is no longer faithful to the model.
Belinda Li: So this might mean that we have to do some sort of recursive training, right? We have to keep refreshing our labels, do something, like, online, to try to keep models to be faithful. And hopefully, eventually, this process will converge.
Belinda Li: But this is pretty expensive, and so, like, I think a key question is, like, how do we actually make this supervision step a lot less expensive?
Belinda Li: And also, I think, just going back to the initial results, so, the MNG that worked with us, Carl, has been doing some follow-up work in the input ablation domain, so this is where, like.
Belinda Li: explain what you would do without the hint. And there we find the strong version of privileged access, where a model is actually better at explaining
Belinda Li: its features here, even though we trained it to explain this one. So, yeah.
Belinda Li: I think that could be one solution.
Belinda Li: Another major question is kind of generalization across tasks, which, again, has already been asked by someone here.
Belinda Li: So, you know, one thing is maybe we could try to train on all our tasks together, and see if we can generalize to new classes of interp tasks. You know, like, what is the circuit that you're using, or can you put this feature into your, response, what would happen? Or maybe even questions we don't already have very straightforward interp procedures for.
Belinda Li: Or maybe there's something analogous to, like, self-supervised pre-training that encourages task general introspection. And kind of one reason, I articulated already that
Belinda Li: I'm optimistic this might work is because models have sort of reasoning and composition skills, and maybe they can kind of use that to combine these introspection atomic skills.
Belinda Li: So, it seems like in your approach, you basically embed the activations or whatever inside, like, the model. Yeah. I'm wondering if you can think about it at
Belinda Li: multi-modalities, in some sense. Right. You have the question in text, and you have the feature or, like, the activation that you want to analyze as a different modality that the model can get. Then if you have, like, a multi-model model.
Belinda Li: that can get any sort of… any type of inputs, then maybe that will lead to some sort of generalizability of the questions that you can ask. Yeah, yeah, I think that's a very good point. I think, like.
Belinda Li: also the, like, latent QA line of work has also been kind of thinking in those terms as well. But I agree, like, you can think of it as, like, maybe the features is a modality, maybe, like, the circuits is a modality, yeah.
Belinda Li: Cool, and then I think the final thing is, like, is this actually concretely more usable for users? And what sorts of, like, user interfaces might need to exist to do so?
Belinda Li: Like, this might include, you know, conducting sort of real-world experiments with experts in medical fields or other high-stakes domains, and seeing whether they can actually, like, collaborate with
Belinda Li: these models that are more faithful in order to make more accurate decisions on downstream tasks. Yeah, so I think…
Belinda Li: Maybe I'm gonna end on this slide, which is kind of like, here's all the things that
Belinda Li: I'm excited about that we could try to build out with self-describing language models. And I'm, yeah, pretty excited to hear what people think about this, excited for any collaborations if people are interested. Yeah, thank you so much.
Belinda Li: Cool.
Belinda Li: Any questions, or…
Belinda Li: Very cool. Thank you. Thank you.
Belinda Li: Yes, Yue.
Yiwei Wu: Yeah, sorry, I have a question about the, I think in one of your earlier slides, you mentioned that the, sort of these…
Yiwei Wu: hypotheses are sampled from DLMs, and I was wondering, have you ever, like, during experiment, have you ever encountered a case where
Yiwei Wu: These hypotheses, the possible explanations, are not… diverse enough.
Belinda Li: Hmm.
Belinda Li: So in this case, we're using, for supervision, we're using, like, existing supervised fine-tuning techniques, so we're not actually, like, reliant on the model to be diverse in its generations. I don't know, is that what you're asking, or…
Yiwei Wu: Sorry, could you go back to your earlier slides on the… I think it's something… different…
Belinda Li: Do you remember which domain this was about?
Yiwei Wu: I think it's the… First one, I think?
Belinda Li: This one?
Yiwei Wu: Yes.
Belinda Li: Okay.
Yiwei Wu: And, yeah, that… and then…
Yiwei Wu: I remember you listed, like, there's, like, a grand truce explanation of the target, but, I remember you had something like, activates on capital cities, activates on…
Belinda Li: Oh, I see, I see.
Yiwei Wu: nuns, etc.
Belinda Li: Oh, I see, yeah.
Yiwei Wu: I was wondering, and you said that these are samples from the explanation generator, and the explanation generator are, like, OEMs.
Yiwei Wu: And I was wondering, have you ever encountered a case where, like, the explanations generated are not diverse enough?
Belinda Li: Yeah, that's a good question. So, basically,
Belinda Li: this step was kind of… we didn't do this step because we used existing, kind of, SAE dictionaries. I will say that, one thing we tried before that didn't really work so well was.
Belinda Li: We used some of,
Belinda Li: Dami had some explanations of neurons.
Belinda Li: And we weren't able to get as good of a generalization from that. And I think one thing I suspect is maybe there's some, like, diversity issue there, in that the SAE features were just a lot more diverse.
Yiwei Wu: I see. Thank you.
Belinda Li: Yep, go ahead. So… The leaf your size will move away.
Belinda Li: Access.
Belinda Li: Mentioned about how models that are, like, basically have their own explainers.
Belinda Li: Most likely do have their… the privileged access, does this also go beyond, in terms of, like, models, let's say, fine-tune the platform, by chance?
Belinda Li: Yeah, that's a good question. So, I think,
Belinda Li: In the paper, we had some, like, results where we were, pre-training… if you remember the linear projection in feature descriptions, there was, in the paper, we had some experiments, we actually tried to pre-train that to, like.
Belinda Li: be more aligned, so that, like, the features in the target model were actually more aligned with the features with the explainer model, and that actually helped the model explain, do these meta descriptions of their features. So I do think there is some effect of, like, if we fine-tune models to behave, like.
Belinda Li: The original model, it's gonna be better at explaining.
Belinda Li: Yeah.
Belinda Li: Mention,
Belinda Li: few bits, see if there's a change or not. What are the ends are like that? Is it… is it essentially just the actual answer, or are some of the, like.
Belinda Li: Sweet.
Belinda Li: Yeah, that's a good question. So, they're just randomly generated, so they're, like, 75% of the chance is incorrect. But yeah, we just want to see if the model will, like.
Belinda Li: Like, we've generated so that 50% of the time it follows the hint, regardless of whether it's correct or not.
Belinda Li: Yeah.
Belinda Li: So, you spent a lot of time talking about the consistency part, which seems the most important, but…
Belinda Li: I guess… I was just wondering if you could…
Belinda Li: They wildly speculating about the causality part? Okay. Like, your thoughts on how to enforce that, or if that's…
Belinda Li: Yeah, that's a good question. So, basically, actually, in some of this, like, early work here, they actually do explicitly investigate causality, and I think what they find is that, like.
Belinda Li: if you train models to be introspective, you can actually fine-tune them a little bit, and that behavior persists. And I think that's…
Belinda Li: maybe also related to what we see with, like, college results right now, with, like, if we train the model to introspect, it actually is more faithful to its own behavior than the, like.
Belinda Li: original model's behavior. So, right now, maybe just doing this sort of enforcement is enough for it to persist under causality, but maybe… yeah, I guess we'll have to see what the limits of that are, yeah.
Belinda Li: How does introspective training, affect the models?
Belinda Li: Ability to do other things.
Belinda Li: Yeah, good question. Currently, we don't care about the model's ability… Have you mentioned that? Have you mentioned? Currently, we don't care, and I think it destroys…
Belinda Li: Even if you don't care? It only answers things in terms of feature descriptions right now, but I think multitask training is definitely the future, yeah.
Belinda Li: Done.
Belinda Li: I think there's another, like, I meant a question here, which is…
Belinda Li: Can you now interpret, like, the difference between the original model and the explainer model, and try to understand what makes the explainer better? Like, what…
Belinda Li: mechanism. Yeah, yeah, yeah. Yeah. I think that is super interesting, yeah.
Belinda Li: I'm also curious, if you don't have results for, like, other tasks, you might not have results for this, but I'm curious if the explainer models perform better on the tasks that they're being asked to do.
Belinda Li: Like, the tasks that they're being asked to explain.
Belinda Li: Mmm… That's a good question.
Belinda Li: Yeah, that's a good question. I guess you filter out, when you did the training, you filter out… Activations.
Belinda Li: For inputs that the model would accurately… So…
Belinda Li: Right? Like, that's what we do. Like, we filter out inaccurate.
Belinda Li: Answers, like, the behavioral performance are very high to begin with.
Belinda Li: for collecting these, like… Sorry, what do you mean by it's inaccurate? So, like, I guess that when you took, for example, like, the…
Belinda Li: Like, devolving, like, various examples. Yeah.
Belinda Li: You only took inputs where the model didn't predict the correct answer.
Belinda Li: For the activation patching? For example, for any of these. I see. Yeah, I guess we took examples where, like.
Belinda Li: the model…
Belinda Li: there was, like, a definite change in the model's answer. So, like, we did have to filter a little bit, because sometimes the model is, like, 50… pretty 50-50, and so, like, when you patch in the thing, it, like.
Belinda Li: it's probably not that much of a significant effect. So we had some filter that was, like, by probability, on the output token.
Belinda Li: Now, one of the,
Belinda Li: One of the promises of interpretability I mentioned at the beginning is this sort of this,
Belinda Li: chance to be this lie detector, right? You know, the model's not,
Belinda Li: Telling you what he's really thinking. Yeah. You've got this… Separate, totally independent.
Belinda Li: auditing method you can use to check… Yes. …what's the model really doing? Yeah.
Belinda Li: with training models to be very self-aware. Yeah. They're very good at understanding their own Neural representations. Yeah.
Belinda Li: Do you… do you… do you… philosophically, do you feel like that leads to a situation where now…
Belinda Li: You, you've built perfect liars.
Belinda Li: Yeah. That's… that's not a, I think that's not a ridiculous question, and, like, I think, like, certain alignment people… Are worried about this. …are worried about this, yeah. But I think it's, at least the way I think about it is, like, I'm actually not quite…
Belinda Li: I guess I feel like…
Belinda Li: a… I think lying requires some sort of, like, intentionality that currently we don't have in language models,
Belinda Li: like… I guess if we… Is Arnib online?
Belinda Li: Oh, okay. Like, my thought is, like, if language models, if we train them to be consistent,
Belinda Li: they actually… like, most of the things we see in terms of emergence misalignment is not, like, some sort of intentional lying, but more some, like.
Belinda Li: we kind of missed this case, or, like, accidentally something happened which we didn't intend. And so, like, if we can do this sort of consistency, and we can try to, like, kind of squash out some of these edge cases. So I think a lot of what we see as, like.
Belinda Li: Misalignment or lying is actually just…
Belinda Li: Failures of our training objectives, basically.
Belinda Li: Yeah.
Belinda Li: But that's… that's my view, at least. Yeah.
Belinda Li: Oh.
Belinda Li: LinkedIn, but I guess, I just thought about, like, if you have, like…
Belinda Li: There's a mixed data model that's being trained, just like for L15 questions about memorizability, and if…
Belinda Li: Like, over here, we see that the model is really good about being self-aware. What aspect of that do you think has got to do with it?
Belinda Li: Okay, thank you for what you're saying, bye.
Belinda Li: sort of, like… Okay, this is what's…
Belinda Li: Yeah, yeah. So I think to truly get a… well, you know, we have some baselines, of course. I think…
Belinda Li: Also truly getting at these questions will also require some of the interp experiments to see if, like, it's using some of the same underlying mechanisms for its, you know, behavior, and it's also its explanation.
Belinda Li: But I think the fact that we are able to do better than this memorization baseline is a good sign, at least, yeah.
Belinda Li: Oh, wherever that went.
Belinda Li: Yes. Have you published? Are you planning on publishing the weights for the models? Oh, they should be… they should be online already, yeah. In whatever GitHub is linked in the paper? Yes, yeah, it's, there's… And which models do you explain?
Belinda Li: There's Llama Base, there's Quen, and there's Llama Instruct. Nice. For AP, at the AP scale? At the AP scale, yeah.
Belinda Li: You put it on to end it.
Belinda Li: Yeah. Yeah, that'd be great.
Belinda Li: Thanks.
Belinda Li: When, when, last question, so… When you, when you evaluated this, If you ever get…
Belinda Li: Explanations that were not faithful to the ground truth explanation, but still
Belinda Li: kind of, like, consistent, or, like, surprisingly good. Like, don't forget any surprises here? I think the biggest surprise is some of the college results right now, in that, like.
Belinda Li: Sometimes it's not faithful to…
Belinda Li: the original answer, but it's faithful to the, like, new, new weights of the model. Yeah. Yeah, I guess other times… there was this other… also, this thing in the appendix of the paper where we were like.
Belinda Li: What are the mis… you know, what is the 25% that's missing here? Actually, we found a large chunk of them was… the original explanations were kind of wrong from the auto-interp procedure, yeah. Nice, yeah.
Belinda Li: Oh, there's a question online?
Belinda Li: Oh, that's… You agree? Yeah.
Yiwei Wu: Sorry, I've… sorry, I have another question. I think you… I think you mentioned that you, experimented with the…
Yiwei Wu: both the base and the instruct version of Llamas 3. I was wondering, have you noticed any…
Yiwei Wu: I guess, difference in that regard, in the sense that whether, like, base model is a better self-explainer compared to
Yiwei Wu: The extract version, or vice versa, or, like, there's virtually no difference in that regard.
Belinda Li: Yeah, so actually the base and instruct models are, like, kind of similar target models, like, their features are basically the same.
Belinda Li: And a lot of… But as an explainer, how about that? Yeah, I, I'd have to look at the results in the paper, but I think there was, like, pretty little difference between the two.
Belinda Li: Yep.
Yiwei Wu: I see. Actually, you have another question regarding the… because I found the, I think you called it the input ablation, where you test the input with and without a hand for the multiple choice question.
Yiwei Wu: I… because I think there are some studies, I guess, for the earlier language models on, like, how they have, sort of.
Yiwei Wu: Positional bias.
Yiwei Wu: for the choices, and I was wondering, have you, experimented with, like, instead of sort of an explicit hint, but
Yiwei Wu: Something, like, more interested in terms of, like, how you… the order you arrange the, choices.
Belinda Li: Yeah, we didn't try it here, but I think that's a very good idea to try, yeah, to try to verbalize that, yeah.
Yiwei Wu: Thank you. Very cool.
Belinda Li: Thank you.
Belinda Li: Any other questions?
Belinda Li: Well, thank you very much, Belinda. Thank you. To talk to us today about it. Are you, are you free to stick around a little bit this afternoon? Yeah, yeah, totally.