David Bau: Because then then we could have tracked down what it was that we did by looking at, looking carefully at timing press buttons. That's all I know. Right? Okay, so okay, so so I'm gonna I'll introduce you. So for the video, oh, okay, yeah. So today, we have
David Bau: a nice talk from Milsently.
David Bau: Who is who's Byron's Phd student here? And and I'm looking forward to to seeing what she says about privileged information. Okay, alright, thank you. Yeah. So this is a project I've been working on. With our brothers also on Zoom. And
David Bau: so it's Alberto Arnov. So that was some of his stuff, Naomi, Safra, and obviously Byron. So my talk is on model activations may not convey privilege information.
David Bau: Okay, me see, if this works.
David Bau: Okay. So
David Bau: I think a few months ago they released this whole like tracing of models thoughts which I mean, obviously, it was like really cool when it came out. So anthropic released this which they were basically looking at. How Claude does certain things like planned a few steps ahead has this notion of multilingualism in the model parameters, and how Claude might do something like mental math. And so I think this I mean, it's pretty interesting as a whole.
David Bau: But this the nice thing about being able to have this ability to look at how Claude does some of these things is, we can kind of use this information to track down whether Claude hallucinates or gives unfaithful explanations. And so I think all of these questions kind of center around, whether like how or whether we can actually understand how some of these models are thinking.
David Bau: So one approach that people have looked at is there's like the one like Openai paper from like back in 2023, where they looked at whether you can use language models to explain neurons and language models. So I guess they were just basically like inspecting the different like features in language models, just by using
David Bau: explanations. And I think more recently, some of the work that I've been looking at is on this whole concept of like introspection. So there is this paper that came out this year, which is called
David Bau: Looking inward language. Models can learn about themselves by introspection. The whole idea is they want to measure this ability for models to introspect, which is this concept that people have talked about in human psychology, which is, do models like, do humans know, like their own mental processes? And likewise do models also like know their own mental processes?
David Bau: And so like, for example, how they like approached it. In this paper they they fine tune.
David Bau: they have 2 models, and they fine tune, one model on the outputs of the 1st of the other model, and they want to basically measure and see whether the original model knows more about its own outputs than the second model does. It's like a little confusing. But there's like a 2 model like interplay here. And so
David Bau: so there are like, I guess, 2 kinds of explanations that you can like procure. From this you can have models, explain text directly, which is what some of these papers have done which is explanations in natural language, or you can go like the route of like budget lens, like future lens, tune lens and stuff where you give more like distributions, as your explanations. Distribution of tokens.
David Bau: Okay?
David Bau: So the the interesting, I guess, counterpoint for this whole introspection idea is, there was a paper that also came out as a kind of like a counterpoint to the paper that I showed with introspection, which is language models fail to introspect, but their knowledge of language.
David Bau: So in in this paper. They basically talk about how they use linguistically grounded approaches on base models and kind of like a similar setup to figure out whether one model might know more about its outputs than the other model
David Bau: and they at least in in their paper. They find out that this is not the case that models can do. Models cannot do introspection.
David Bau: I guess the interesting thing about these papers and how it relates to what I've been working on is these papers look at introspection with respect to examining model outputs and not using any kind of model internals, and
David Bau: there obviously is a lot of information that you can kind of glean from activations. And like, of course, like the open AI paper I showed earlier, had some of these explanations for neuron. But
David Bau: I guess I'm interested in looking at whether you can also get explanations from
David Bau: activations and also like to see whether these models have the ability to kind of introspect in the same way. But from activation. Are you going to talk in more detail of what? You know? Sions, finding here was in this paper, because I'm not actually familiar with this. I
David Bau: I don't want to like say it wrong because I read this a while back. But they found that there was like they did a bunch of studies over linguistics, data sets, and like like blimp, I think, was like one data set they use, or it was one of those other grounded data sets, and found that models were not able to do the same kind of introspection that the prior papers had found. But I don't know if I want to explain it, because I feel I don't want to like
David Bau: say the wrong things. But I think the main point is, there's kind of like a lot of like interplay between, like, whether models can actually introspect or whether they can't
David Bau: right
David Bau: I did like the paper, though when I when I read it, I thought it was like, I think the dialogue is pretty cool with some of these works.
David Bau: Okay? So the motivation for examining this kind of like
David Bau: ability, I think we won't call it introspection, because this is more about what their models
David Bau: have the ability to know about privileged information. And I'll explain what that means. Basically, we don't know how language models know how to do certain things. So it would be actually, really useful to have a tool that can do such a thing. And this is actually already something that people have worked on, which is
David Bau: I'll explain the related work after. But people have already worked on basically like decoding activations into natural language. But the issue that comes up with
David Bau: the this series of works as it does, for, like the Introspection series of work, is that a lot of these papers like question the utility. Sorry they don't question the utility. So I'm questioning the utility of some of these methods. But these papers are saying, oh, decoding activation into natural language just works right?
David Bau: introspection. Again, they only use inputs and outputs. They don't use any model internals. So they measure like this kind of privilege, knowledge, or introspection a little bit differently. And so like in our case, we have access to model internals. And so our hope is to basically understand the degree of privilege, access to knowledge that we have from model internals.
David Bau: So I guess the figure on the right, which is not super clear. I think I want to remake this from my paper. But basically, there are kind of 2 ways to think about like how models can decode activations into natural language. In one case,
David Bau: basically on the left. We have, an instance where the model actually does not have the ability to gather like privilege information or information that can only be seen from activations, whereas, like on the right, we have the model kind of respond with a prompt that says that it it does have access. So it knows that
David Bau: I mean, I think the example might be a little bit contrived, but it knows that the
David Bau: the currency of the Us. Is the dollar which supposedly can only be seen from activations.
David Bau: Good.
David Bau: All right. So some of the prior work for for this is there. There's a few patchcopes is like the the 1st citation, which and there was also like a concurrent work called like Selfie, where the whole idea is, they basically take a model activation from one model. So we call it the top, the target model.
David Bau: And then we patch that activation into an inference pass of the second model. And so the whole idea is, oh, yeah, can we basically generate an explanation about the activations just by passing this information from one model to another, and like in these instances generally, the model can be the same, or they can be different. But I think a lot of the experiments they ran in their paper were on like the same model family, basically, or like the same exact model. So both models would have like the same world knowledge.
David Bau: There was a second paper that was similar to like, conceptually similar to patch scopes in a Selfie, which is called the Mqa. Which shows that models can be trained on some of these activations. And you can ask questions about these activations and get an answer right? And so again, this is, this is a little bit different, because they focus more on like fine tuning to get exact answers. But I think the whole like conceptually what they're trying to
David Bau: is extract some information from activations that you kind of can't get from the input good.
David Bau: So I guess the there's a few issues that come up with some of these methods. So 1st off, there's no guarantees on the type of information that might be verbalized. I guess we coined the term like verbalization, but they didn't like use any of these terms, but they were also, like again, not specific with the information that you would get because, like, for instance, in their paper they
David Bau: did a variety of like case studies on various tasks, right? But obviously the utility, the
David Bau: depends on the particular task. And what you're trying to look for. But I think we like posit in, like our in our paper, that you really should know what kind of information you want, because otherwise, what are you asking for? Right?
David Bau: Another issue that also comes up with a lot of these papers is the fact that you have 2 models, and they both have, like some amount of world knowledge that can either be the same or different. Most likely they're the same right? And so we also run some experiments about
David Bau: how models like deal with this world knowledge, or what happens when they differ. But essentially the idea, the problem that comes up is some of these experiments from prior works. I don't think they have very good controls on where the knowledge comes from. When you verbalize
David Bau: activations because the verbalizer you're using is a large language model that obviously is very powerful. So you don't know if you're actually getting the information from the activation or you're actually getting it from the second model. Okay.
David Bau: city.
David Bau: Okay? So so some goals from this work.
David Bau: basically, can we determine what type of information extracted from activations. And is it privileged? Right? And again, privileged is the the term we use privilege as a way to denote information that you cannot get from input text.
David Bau: Basically well, I showed like a 1 of our 1st experiments, which kind of talks about like how you can differentiate them. But our goal to when we use some of these tools is obviously not get information that we can already get like different prompting.
David Bau: We also want to basically identify to what extent certain tasks work best with verbalization which do not. This is not something that again has been done like prior work where they're just like. Oh, here's a slew of tasks that kind of makes sense to apply this tool to there's not much of a distinction about like oh, what information do we care about looking at, or what's the goal of this task? Or what do you want? Basically, right? It's very like vague.
David Bau: And then the 3rd one is that I've been thinking about for some time is basically how much world knowledge is needed in your verbalizer. Right? So obviously like the the, I guess the lesser analogy to a verbalizer would be to use like a pretty basic like logistic probe or something like that. And so I guess
David Bau: we do. I don't really have like full results on this, but we do make comparisons of verbalization to probe which they have done in patch scopes. But we kind of show, like under certain scenarios. If the verbalizer does not have the real knowledge, it actually cannot do the task. It was like assigned to do.
David Bau: Okay. So 1st section. I'm going to talk about what type of information extracted from activations
David Bau: so like quick preliminaries, because I did kind of breeze past. There's like, there's a bunch of models. And it's a little slightly annoying. But I just give like a high level on this. Basically, for any of these verbalization approaches. You have 2 models. You have model one, which is we call the target model.
David Bau: And then we have model 2, which is our verbalizer model, and the idea is from model one. We basically extract an activation and we use insight, which is very helpful. The tool is very nice. We extract an activation from a certain layer. Either the I've token or all tokens, because one distinction that I guess I forgot to make is Patch Scopes takes an activation from one token, whereas you can
David Bau: take act, you can take activation from entire layer. It's not as clear as to
David Bau: why that would make as much sense. But some works have done that. So
David Bau: we just say we do the same right?
David Bau: So from there we basically take the activation and then we insert it into the inference pass of
David Bau: a different layer of of the second model to get the output. I think generally
David Bau: patch codes does this thing, which I like kind of disagree with, but they do a thing where they take the activation and patch to every single layer which I just don't really understand, like why that
David Bau: is supported or like what evidence they have as to why they would do that. But they do that anyway. So that's how they do the evaluation. So we just do the same.
David Bau: So I guess in the image a nice explanation of this is nice visuals. Take model one. This is, for if we take a single token, and we patch it to model 2. We take the President of the United States. We take the last token, and we patch it to the end, where we have a verbalization prompt and our verbalization prompt, is the subject of X right. We patch it in there and then we get some output
David Bau: right? If you do. If you have multiple tokens. The way that you would do this is, you would take the activations
David Bau: from the President of the United States.
David Bau: and then you would patch it into the beginning. I think some of these these are design decisions of like the prior works. I think you can probably like, give or take, like manipulate some of these design decisions. But again, the whole idea is like, you just want to insert it into the inference pass of the second model.
David Bau: Okay? So for data set, we use feature extraction data set, which is.
David Bau: I think there's like 12 or 20 or something, but based on prior work, we just use 6 of them. Just to be consistent. So what this kind of looks like? So the input of feature extraction is
David Bau: usually there's like, 2 prompts of it. The United States in 1928. So that's like.
David Bau: okay, it's subject. Okay. So it's like it gives. You have like a prompt that has like the subject, and then you have the relation which is the currency of X, and then you want the object.
David Bau: And so that's the output which is the currency of the United States, is the dollar right? And so the way that these verbalization methods have used feature extraction is they take the first.st They take the 1st
David Bau: input, which is the United States in 1928. They change that into an activation. They take the last token.
David Bau: and then they insert it into a model 2 plus an interpretation prompt, which is the currency of X, and then they ask for an output. Right? And so they get the currency of the United States. Is the dollar.
David Bau: Right?
David Bau: Okay, yeah, this is just like setup stuff. Basically, I use Lama 3.1 Ab, I'm pretty consistent about that. I did also do some cross model experiments, but they're not really in the paper, I probably will add it to appendix.
David Bau: but ideally like again, I mentioned model one and Model 2 can be the same or different model. But when I was like running some of these experiments for cross model, it didn't always work, which I think that was probably like some interesting tidbit, especially for patching multiple activations and fine tuning over it that didn't always work across models.
David Bau: I think passcopes was a little bit more consistent. But sorry.
David Bau: But I didn't see too much of a difference in like the behavior when you do the cross model or like, have any more powerful like outputs. So okay, across all experiments, I basically use 2 approaches. One of them is the patchcopes approach, or, like selfie also does the same thing. Take one token and patch it into the inference pass of the second model. And then for the other approach which uses all tokens.
David Bau: It's called latent interpretation tuning. And basically, you're just fine tuning a model on all of the activations plus Qa pairs. So questions about that activation. So they don't. There's not like an analogous
David Bau: example, where you use all activations and inspect it without fine tuning. I didn't actually try that. But and the reason being is, there's no technically no prior work for that. I probably could, but I mean the setup would be pretty like simple. But I don't actually see that working too well, because you have to learn the representation from all activations.
David Bau: Okay, so okay, a really simple experiment
David Bau: is to kind of ask, like, Hey, what knowledge is actually available in activations. And I I guess, maybe this is not what feature extraction data services we use for. But basically we can just do prompting like 0 shot prompting without any internal activations
David Bau: on the
David Bau: feature extraction data set. And so the reason being is, you want to basically understand, like, oh, so if I'm able to get a certain performance via like 0 shot prompting. What am I actually extracting when I do verbalize the activations? Right?
David Bau: So the setup for this is, we have, like again, the input that I showed from previous, which is the United States in 1928, and then the currency of X right. What I want to do is I want to concatenate both as a 0 shot prompt instead of just turning the United States in 1928 into an activation. Obviously, I do that for the verbalizer. But all I'm basically doing is constructing a 0 shot prompt for our model to do inference.
David Bau: The the model I do. Inference on is not the same as the verbalizer. Just keep that in mind.
David Bau: Yeah. So verbalization to format is activation, which is the United States in 1928, plus the currency of X and these verbalizer models I previously trained. So they are trained on the same setups that they previously proposed in their paper.
David Bau: and then so their Eval. And I actually thought this Eval was slightly questionable across, like all like the other papers, which is, if you generate 20 tokens, you see if the target is there, and if so, it's correct. I have a problem with this Eval, because under certain circumstances where you do, 0 shot prompting, and the model isn't very succinct. You will lose out on whether the model actually knows the information or not, just because you're doing a pretty cut and dry like accuracy based, Eval. Honestly, probably something like
David Bau: precision or something might be better, but they didn't. Really. I think patch scopes change like the evaluation depending on what task it was looking at. So the reason I bring this up as a problem, because when I was like playing around with some of this. I had problems kind of coercing a instruction to model, to actually give me the output that I wanted in 20 tokens
David Bau: right? So I think the 20 tokens is kind of arbitrarily chosen. But it's consistent with firework. You just try something like flu shot
David Bau: so that it knows that it needs to give the answer as the 1st one. I probably could. I guess I didn't end up doing that because I want to be consistent and doing it. 0 shot, like the other, like the verbalization work is which. So which other works had this set up. So it was late in Qa, and then patch scopes. They both did the same thing. Yeah, I probably could do that, but I think it was better to probably create a like harder baseline for us, then it makes it more convincing because I'll show you results.
David Bau: Basically, when you do 0 shot prompting, you essentially get the same scores. Right? So it's like, so, yeah, if we did, few shot, we probably get even better, which is, which would be pretty nice. But yeah, so we did 6 categories. I guess I can explain the specifics. But can you go back? One slide, please?
David Bau: I don't think I got this set up. Actually. So
David Bau: okay, the input is the United States in 1928, yes, plus the currency of X
David Bau: contact phone. Okay, what is X being replaced with United States.
David Bau: So what we're basically doing here is I. So the currency of X is the same prompt from the feature extraction data. So I literally just keep the same. It's literally X, yeah. And the model just actually completes it.
David Bau: It completes it properly.
David Bau: But that's weird. Why, the prompt itself is weird like, if you're talking about United States. Have you ever seen a sentence that that's called? That is kind of like that? The United States in 1928,
David Bau: plus the currency of X.
David Bau: So
David Bau: I have never seen a sentence like that, I mean, like, so you you don't you? Literally. You don't create a sense you don't substitute X. You don't say the currency of the United States in 1928 like you don't do that. No, I don't do any of this, because the whole goal is to see whether you can extract the same information. I don't know if the prompt really matters in this case I see what you mean, though it is a weird sentence. But model is able to actually complete it.
David Bau: Okay, so the model support model is kind of like reasoning that. Okay, by X, you mean United States, not 19. You're on the feature tracking paper, but I like, I guess I assume, the feature tracking paper was intended to like be used for analysis and model internals, and so I guess it is a little surprising that it actually works as a 0 shot. Prompt.
David Bau: Okay, yeah. But presumably like, yeah, at least like as few changes to the 0 shot. Prompt as possible. I think I might have like
David Bau: added, like a little like, answer, this prompt or something to get to work a bit better. But beyond that, like there was, the semantic meeting was still the same like, I didn't change anything about this. Okay, cool. And also like, How is that experiment supposed to like connect with your the question you were asking, okay, yeah, yeah. So what type of information? So this is just one part. But here in this setup, you are not even using the activations I am. And for the verbalizer I am. I have a so the reason
David Bau: this is like a multi-part. So I have a second part of this. But basically I, what I'm asking here for the 1st experiment is, what kind of knowledge do I actually get from?
David Bau: from the model? If I were to just use it as a
David Bau: from the model. If I were to just take the feature extraction data set and use as a 0 0 shot. Prompt. I compare it to patch scopes and late interpretation tuning which supposedly extracts information from the activations right?
David Bau: But I get the same score. And so I guess maybe the question could be worded a little bit differently. But I'm basically asking, do I get the same information? You say the same score you're looking at the average. I'm looking at the average in some sense, I mean, give or take, like I mean, there's obviously be some noise. But I am looking at the average in general, and I find that the performance is essentially the same. So what differences could there possibly be in the information they actually look at right.
David Bau: Sorry. Sorry. Can you scroll down a little bit? So you. So that's from that setup you are grabbing the activations and you are
David Bau: passing them with leading Qa and trying to. Okay, okay, yes. Okay. Sorry. Yeah, yeah, I can. I can slow down. People want to. But like, sorry I have to say, I have a hard cut off at 11 because I have to go. But the yeah, the whole idea is, I want to compare essentially 0 shot prompting to a verbalization setup where you supposedly have more information.
David Bau: Or that's what these methods tell you that they do. They're supposed to give you information that 0 shot prompting would not give you. So it is surprising that you get the same score.
David Bau: Right? Does that make sense?
David Bau: Okay? I mean, if I showed a second experiment, it might make a bit more sense.
David Bau: So the follow up to this, the reason why 0 shot prompting is actually useful in this case is because it tells you a lot of this information can essentially be answered solely from the input text. Right? So then, the follow up question is how much of this activation is actually invertible into the input text that we give the model right, the target model that converts the target. The input text to the activation.
David Bau: Right? So if like, let's say, you do, you invert the activation back to input, text, then all the information you're really accessing is just the input text itself.
David Bau: Right? So in this experiment, what we're basically doing is we take our activation. We obviously also pass into the verbalizer as comparison. But we have this new setup where we take the activation, we invert the activation back into the input text. And then we also do 0 shot prompting
David Bau: because the whole idea is if you can basically get the same output from doing this, I guess this set of steps. What you're really just doing is when you're looking at the activation, you're just reading input, text, right?
David Bau: So the main setup for this is, we train an Inverter model. And it's like Jack's work on inverting embeddings, and although embeddings are slightly different in this case.
David Bau: because embeddings are like mean, pooled encoder outputs. And what we're looking at is we're essentially looking at 2 setups, one where you get all activations which is much easier to invert, since you basically have information from all tokens. Or you're inverting a single.
David Bau: a single vector which would be the last token of this sentence. Right?
David Bau: And so yeah, the process is basically now you extract the activations you invert the activations and then you predict 0 shot on the new text.
David Bau: And then we compare this approach to verbalization. Yeah. So I guess this image hopefully, it's not like, not too
David Bau: like a pixelated. But yeah, you basically take this input, you take the activation. And then you have this Inverter model, which is which basically inverts it back into the input text or as close as possible. We don't guarantee that the inverter will actually work completely, because the whole point is, how much of this information can you actually get from the activation? Right?
David Bau: For the second part, you basically take whatever you had from our A, you add the prompt to it. And you do 0 shot prompting.
David Bau: Okay, so that's like the invert. Does that make sense? Okay?
David Bau: So there were 2 results like I had to do inverting activations over multiple inverting multiple activations and then inverting single activations. The reason I find, like the whole like inverting multiple activations. Kind of funny is because I feel like it's such a simplistic task. Because you already know all the information from all tokens. So it has to work essentially, which it does basically in our results.
David Bau: last row is like, inverting llama 3. We have.
David Bau: This will be a little annoying to read. But basically, let me explain for the decoder, if it doesn't say 0 shot. We trained the decoder on the latent Qa data set, which is from
David Bau: interpretation tuning. Essentially, we do this so that we kind of match, like the training setup that they do in theirs because they learn a verbalizer on like Qa, which is like activation plus qa pairs. But we essentially mimic the same thing. But for fine tuning, instead of learning over activations right?
David Bau: And but we also include a 0 shot setup and I'll note this in a moment. Why, the scores are a bit higher. We compare against lit on this side late interpretation tuning, and
David Bau: this is averaged over all 16 layers that they consider in their paper. So they consider layers one to 16, and for each layer what I mean one to 16 is they take an activation from that layer, and then.
David Bau: they what is it, they pass it into the inference pass of the 0 flare. So they do. The source layer 1, 2, 3, 4, 5, 6, whatever on and on, and they all like patch into the 1st layer, and then they take an average over all of that. But, like for fair comparison, we also, like, choose one layer and show you the one that we decide to invert.
David Bau: We don't invert every single layer. I mean every single yeah, every single layer or token in the single case, because in creating the models just take way too long. So yeah, we have to like save time
David Bau: and compute. We don't have that much compute.
David Bau: But okay, so we find that essentially, if you, I don't have averages here, it's safe space. But for the most part doing the inversion approach, you get a better performance than if you were to compare it to the verbalization approaches. And I think this is in my mind kind of obvious, because again, you are presented with all activations over all tokens. That means the input information has to be present in there. So for the 0 shot.
David Bau: 0, Shah does better than trained.
David Bau: and this could be in some case how we trained our model for prediction. But another problem that I mentioned earlier was the 20 token limit. Sometimes you would have like, this is where you were just like, Oh, doesn't it just fail? And surprisingly, the 0 shot models answer feature extraction really well, but when you train a model on questioning the model, then, like I would say, it's like the refusal where it's just like, Oh, sorry I can't answer that. Do you mean this?
David Bau: Right? Then you run into that problem. That's where probably few shot would work better.
David Bau: Right to answer feature, extraction questions.
David Bau: but yeah, I think that's that accounts for, like a good amount of like, why, I think there's like a pretty large like performance difference between like 0 shot. And the trained version is just because the 20 token limit is like pretty prohibitive. And just because a model doesn't get answer right in 20 tokens doesn't mean it's the information, isn't there? You can say that analogously, for, like the verbalization methods, too. But in their case, like they are trained to verbalize activations. They're very synced right. And for patch scopes in theory.
David Bau: if you count like how they do their setup, you have 16 times 32 chances to get the right answer, which is very, very lax. So I think it's actually like a fair, at least for us, like it's kind of a fair comparison. We we kind of give them more grace, because that's just how their method works.
David Bau: But okay, single activation is a little bit more difficult. And we actually use the
David Bau: we were using similar methods to Jack's code. And we do find like.
David Bau: So they in Jack's code, like, so this is their paper text embeddings revealed more than text or something. I think that's the title. They find that like inverting over decoders kind of sucks, and I did my own implementation of this like encoded it. I did find that it significantly lagged behind. If you were to just use T. 5 base to invert. But if you looked at the average score I don't have an average on here, but I think it was like
David Bau: 0 point 3 for our inversion of T 5 base versus the average of patch scopes was a 0 point 3 8. So it's like basically getting to the point where it's, I mean, it's not matching. But I think a 1 way to think about this from my understanding is
David Bau: basically our method of just inverting the activation can tell you how much of the information in the activations is from the input text, which is very useful as a heuristic, because these methods do not guarantee that you will get information from one place or another, and one of the easiest ways is to be like, oh, this much is from the input, text, right?
David Bau: Yeah, let me see if there's anything else interesting. But yeah, I mean, there are some tasks where our inversion does much better than patch scopes, and obviously some tasks where Patch scopes does better than us. So it's I think it's a pretty consistently across the board like this also depends on like the tasks you choose but we actually show, like some results on other tasks like, after.
David Bau: So here we go.
David Bau: 4.
David Bau: Okay, let me just show some examples so like reason why inverting activations is hard. I think I mentioned this a little bit earlier, but we are inverting single token activations that are not mean, pooled like what you would with encoder only models. And so you do lose a lot of information. But the information is still basically there for the inversion, which is nice. The only problem is basically when you do this is an example of an inversion that I like took from my code. Basically, this person, John Mcdermott, is
David Bau: in soccer or football, but does not play for that specific team. So, as you can see, like the conversion for, like proper nouns of people, and usually like teams, is not as accurate
David Bau: but another like counterpoint. For this is, for the example, the United States example I showed earlier.
David Bau: it lagged. Basically, it matches the United States perfectly, but gets the date wrong. So I guess I hypothesize. In this case it's mainly due to how much information the model has about a certain entity, right? And so like, obviously, like John Mcdermott, I was like, his name is probably not very common in the model, whereas, like the model obviously sees the United States a lot, so it has a more accurate like handling of
David Bau: the proper noun. United States.
David Bau: The the model. Also, when you do, some of these are inversions you also like. So look and point attachment at the appointment of. So basically, the structure is all correct. But you see the proper noun wrong. The reason why this matters is because for feature extraction, like, you have to get the name like the proper noun, correct with the object. I mean, you need to get the subject correct, to predict the object correctly right? And so that's like one limitation.
David Bau: But beyond this, like the doing, single activation inversion actually works quite well, even though it doesn't see as much information as when you, as if you were to invert
David Bau: what is it like? A embedding right? Which activation is that this is, I take the last token, last token, at which layer so I I do. 15th layer. I do it for consistency. Middle Islama. So like, there's 32 layers.
David Bau: Okay.
David Bau: I did do some like testing with earlier layers, and, like the if you did like layer one or something, it like, you would think that model would have more information about tech. Yeah, why did that? And I also did the end. And I found, like the training was not stable at all, and it just garbage middle is just better. Yeah, okay, so this part the one question that
David Bau: we have 15. One question that comes up is what to extent what certain tasks work with verbalization, which to? Not because obviously the question is just like, Oh, you just pick feature, extraction. I picked it because it was in the previous papers, I think kind of makes sense. But also we want to try like their setups. And I mean, thanks to you guys for talking about this. But this is
David Bau: we want to basically investigate. Oh, so maybe when people talk about doing verbalization, maybe they want to think about what tasks are more ideal to verbalize right? Because some like, I mean, maybe I'm a little surprised about feature extraction that one ends up being something that you can basically just answer from the model model text right directly. But basically one.
David Bau: maybe one goal. Some people might want to use verbalization for is I guess I talked about the Holy Rail, which is just using verbalizer to a verbalizer to access unknown knowledge. And we talked about this is this helpful. And then so we basically experiment with a task where the knowledge is fabricated.
David Bau: Okay? And of course, this is our. This is Arnold's data set. But we also like change, a lot of stuff. But thank you for your data set, so basically, we use like a data set of personas with fake biographical information. This is like also from like the physics of alums paper, right? They had, like a similar setup where they came up with a bunch of biographies of people. The place names are all legit, but the people are
David Bau: not real.
David Bau: So example, like from the data says, you have this guy named Mohammad Aziz from Pakistan, who likes playing chess. He's a lawyer, whatever. You just have all of this information about this person.
David Bau: So we actually ended up doing
David Bau: 3 different setups. And I'll explain why each setup matters. So the standard setup is what I basically showed where you just have this person. You have the country where they're from. You have a lot of these attributes about them, and I think one thing to know that
David Bau: I thought ended up being pretty interesting is given the person's name. You might make assumptions about where they're from, right. And so we found that to be like pretty.
David Bau: it's not like an issue. But it found out to be actually pretty interesting when we're doing analysis on why verbalization works or doesn't work. And that also motivated some of the data sets that we use like the setups. So 1st one, again, is the standard setup which I just showed.
David Bau: Second, is this shuffled setup. So let me explain the shuffled setup we have the person's name, and then we have their attributes. What we're gonna do is we're gonna keep the person's name. But we're gonna mix up all of the attributes, including the country where they're from. Yeah, where they're from their hobbies. Favorite foods, whatever right?
David Bau: The reason for doing this is when we did one of their experiments early on we found that when you were to use verbalization, you would actually get really good performance on
David Bau: just 0 shot, prompting, and inversion. So our setup.
David Bau: even though supposedly the model, should not know about these people, which I just thought was kind of odd. That means
David Bau: this. This implies that the second model, like our verbalizer, is making some implicit assumptions about or like, not even just our verbalizer. Also, the 0 shot model is making some implicit assumptions about the relationship between the person and their attributes. So one thing we wanted to test that I thought was similar to like control tasks from like probing, which is, if you shuffle it, if the model is able to make get high scores on the predictions.
David Bau: Then
David Bau: that means the model is actually predicting, based on the attributes from the activations as opposed to their implicit biases or assumptions. Okay. The 3rd sorry. That was like a mouthful. The 3rd data set is, we have this fantasy data set where I make up all the names. They do not exist right? So model one will know about it. Model 2 will not.
David Bau: Categories, country favorite food, favorite drink, favorite music, genre, favorite sport, favorite game, and then pretty self explanatory.
David Bau: So the setup for this is saying, kind of very similar to feature, extraction setup. But this is implicit, like a Qa task. We say, Okay, my name is some persona's name, and this is how I like created the activation. Because what I'm assuming here, which I think should be pretty valid is if you make a prompt that is the person's name that prompt should encode the information about that person.
David Bau: Right? So I encode that into activation. And then so the full prompt for verbalization is act. And then the country origin for X, right? So same setup, similar setup as like the direction, I wouldn't say it's exact.
David Bau: 0 shot. Input is going to be. My name is persona name, and then the country origin for X is blank. Right? It still works, I'll show you results. It still works with that prompt. And then for the inversion input I invert the name, invert the name into some something else, and then I have the full, prompt, so similar to my prior inversion setups.
David Bau: The reason why I so inversion matters here, because the idea is, you can use inversion as kind of like a test to see how much of this information comes from the input input, text, right? So ideally in a situation where the model doesn't have the model has privileged knowledge about the activation, you would assume the inversion. It gets a score of 0 right? So I think that's like a good test to check or like, they're basically 0 or close to like 0. Right?
David Bau: Eval is still the same. I generate 20 tokens.
David Bau: okay, so the results. This is a bit hard to read. But I can just explain each section. So for this 1st part, we have
David Bau: results on the vanilla data set, where you have a score of 0 point 3 6 on the country.
David Bau: Favorite food, favorite drink, music, genre, sport and game. I think when I when I 1st ran this, I was a little surprised that 0 shot was even doing doing well at all, because I was just like, aren't these personas that weren't supposed to exist, or something right? But I do. I do think that there are some implicit biases with, like the way the personas are set up. And so that's why, for like, for instance.
David Bau: the country does really well guessing what country and like. So the inversion set as basic like, they're a little bit off. But they get similar scores. So and then single activation gets better. So as you can tell, like, even with the single activation inversion. The name probably isn't right, but it's probably inverting it into like a name that is similar to what country the model should predict to get it right like. So if Muhammad Aziz is like Pakistan, it's probably something similar, right?
David Bau: So in the shovel setup, I thought this was actually pretty interesting with with the shovel setup essentially
David Bau: again, labels are us. The attributes are all shuffled, the name stays the same, but when we shuffle this, we train another. Sorry I have to mention, we train another model one on each of these data sets. Okay? And then when we get the activation from these data sets, we do the same setup where we make the prediction about the attributes. And when we make the prediction, we find that the shuffled data set. Basically, most of these scores, except maybe patch scopes
David Bau: for favorite sport and favorite game are close to 0, right? Which shows that the model, like whatever you're doing in model. 2 word of verbalizer is implicitly using its world knowledge
David Bau: to figure out what what it's supposed to predict, which is not what you want right?
David Bau: I guess the interesting part is, Patchcope actually gets a decent score for favorite sport favorite game. So maybe it does work as intended. Right. If you compare it to inversion like you can still maybe make some guesses. But maybe Patchcop does have a little bit of privileged knowledge that it uses when it makes a prediction.
David Bau: and then the last
David Bau: last row any fantasy names that you've never seen before, which I think this might be a kind of like an obvious like Duh.
David Bau: you just can't get any knowledge of it. So like, I guess the takeaway from this is, you really need some amount of like world knowledge from the verbalizer, which is definitely different from. If you were to use a probe, because with the probe you don't need that
David Bau: right. And so I think this I mean, I would say, this is probably a limitation, because the goal so is your feeling that that because it doesn't even recognize that that's a name or something, or you have a fantasy name. Oh, sorry. Can you say that again? What do you mean? Okay, so like, you know, I'm trying to get get intuition. So like so do you have an example for what? Like a like? How
David Bau: how different is the fantasy setup? Oh, sure, I wish I had the code for it. Oh, Alberto said something because
David Bau: he says work mentioned that if you train m. 1 with the shuffled activations. Oh, yeah, I will show this. I'll show this after Alberto. There's another result that we had. So with the name. So basically, what would happen is, you would have the name of the person right, and then I would be like, Oh, I made up this like
David Bau: food item called Spice Bowl, or something. I would want the model to predict that, but then it would usually predict with something generic like
David Bau: chipotle or something not actually chipotle. But it would come up with something within his vocabulary which I mean. Not that spice bull isn't in his vocabulary, but Model 2 obviously does not have this information, but the tokens are still in his vocabulary, which is the difference.
David Bau: Wait. So wait. So you you. The name of the person would be spiceball. No, no, sorry. That's the food that's the food the person name of the person is. Still. I keep this. Oh, sorry for the fantasy version. I do generate new names of people right. But the implicit assumption here with a lot of with these verbalization setups is
David Bau: you?
David Bau: Well, I mean, I don't know if his so, but the main difference between shuffled and fantasy
David Bau: shuffle contains names that are exactly the same from this one, I understand. Yes, so. But then, so I'm trying to understand the difference between shuffled
David Bau: and fantasy. Oh, okay, yeah. So this one. All I'm doing is shuffling the attributes from. And the difference in fantasy is difference is, I create completely new names that are like. So it's just the names. It's just the names. But the the attributes are also shuffled into something that doesn't exist so, but like so it's just like shuffled. But with new names
David Bau: it has different attributes, too.
David Bau: Oh, like what kind of attribute I make up. I make up attributes like for like for food I like, do you say, do things like your favorite blobroom? Yeah, whatever wait. So so you don't know what a blab roo is, or whatever right? So is that so I actually, I'd love to see an example.
David Bau: I have my code. Yeah, I have. I have to like. Pull it up, though. Let me check. Does the model do you train the model with the fantasy data? I trained the 1st model with the fantasy data set. I don't with the second, but I have other experiments with it. I mean, now that you guys mentioned it, maybe I should just only change the name. But I think even under the assumption where we change the name, I still don't think it's going to do well, so the issue. Why.
David Bau: I think
David Bau: I bring up why we ended up doing changing also the attributes is again like when you, the model basically verbalizer, was making a lot of assumptions about
David Bau: the what it would predict. So, for instance, for a hobby.
David Bau: the model would predict chess and the model would get chess correctly, because random guessing
David Bau: right? And so I guess the one way that we thought it would be better to control is just make everything. Not in the vocab. Sure. Yeah. But the issue is, it's just very difficult to kind of disentangle this behavior, because it's like you either have a completely unknown data set where the model just doesn't know any like supposedly doesn't know. But it can read from the activations, which is what a lot of these papers purport that they can do, or you come up with a setup where the model might just guess and get it right by random chance. Right? Yeah, I'll show you that you say, give me a sec
David Bau: So does this kind of why, like shuffled versus fantasy, that patch codes does seem like it can read information
David Bau: that it doesn't know.
David Bau: Yeah, for, shuffled the model is, it might be able to.
David Bau: But yeah, I guess the difference again with shuffled and oh, yeah, so like, so I see. So the attribute names are so I see from the chat open you open up the chat. Yeah, I froze. Oh, actually, I'll open up the chat. Oh, he grabbed it for me from Alberto. Yeah. So what did you say? Can you see the chat? Oh, you have to open chat because I can't see it.
David Bau: So country look like a so so the the attributes are real attributes, like country drink music. But then the attribute values
David Bau: are things that don't exist in the real world, like
David Bau: the country of Lumina. Yeah, you just made up. I mean, it probably exists somewhere, but it's not associated with country.
David Bau: and the food is bread or roast.
David Bau: which sounds delicious, but I don't know what it is me neither. I see. So patch scopes now can't even take out the information once. The model that's the verbalizer model doesn't know. So I I guess I don't know. I feel like this is like, maybe slightly trivial and
David Bau: like for the fantasy one we're just like. Obviously the model will not know it. But I I do think it does make kind of like a statement about, like the fact that you do need the world knowledge implicitly. But so so. But then the other question is so.
David Bau: But then but then you have when you train your model on this.
David Bau: Do you have a baseline where the model can read all this stuff? We're doing a probe for this. Oh, I see. But yeah, so you don't have a bait you don't have. So like, my question is like you. Fine tune, the fantasy model. Yes.
David Bau: does it? Did the model learn this stuff? Yeah, I did. I have results for this in my paper. But I didn't show it on here. Basically, the model does learn the associations in Model 1. 0, the model can say it. Yes, the model can. So it is actually there. But even if you like, get the if you use a verbalizer to verbalize. Does the model generalize if you ask it with slightly different wording than what you trained it with?
David Bau: Yeah, oh, so the the training. Yeah. For example, if you give the model like multiple choice question, or if you invert the word order, or use a synonym for the instead of saying, You know what country is? Somebody from? You say, what nation are they? These questions are all out of domain questions. These are not model is not trained on these questions.
David Bau: Meaning so when I train like, model one.
David Bau: So, okay, yeah, I guess I didn't make that clear with the verbalizer. This verbalizer is only trained on the original setup that I showed previously. It is not trained on these personas at all. And so it is trained to. Basically, it's not trained to answer these prompts either because feature extraction is also out of the main for this entire setup. So we do it this way. So it's fair. And so these models have not seen. This country is X, or whatever or like this person is from this country.
David Bau: So all domain. And when model one is trained for these data sets? We use like the questions, or like the like biographies that are not like put together.
David Bau: So another thing, somebody should get the credit for the data set which should be to you. Oh, okay. And I could put forth for you guys, that's fine. All right. Yeah, I mean, or you can just swap my name with you. That's fine. Okay? I mean, I, personally don't really care. But we can do that. Okay, but yeah, I mean, could you help me like, train some of the models and stuff? So but yeah, thanks. Yeah. And then. So let me just show. So then we I will speed this up, actually, because maybe, like 5 min, I have to go.
David Bau: But unless you have to just go. Okay, I'll do 5 min. And then I have to.
David Bau: So the other part about this that we were interested in looking at is basically what world knowledge is needed in a verbalizer, because some of these experiments actually kind of implied some.
David Bau: some fact about
David Bau: some some idea about the fact that you need world knowledge to actually verbalize the concepts correctly.
David Bau: okay, let me see what I said. We train a base model on new knowledge that conflicts with its internal knowledge of a persona and its name. That's basically what we're doing. Is it information verbalized, faithful to the new model?
David Bau: Okay? So we have the setup.
David Bau: Let me explain. It might be slightly confusing. But basically we take the verbalizer, the persona regular persona setup, and then the shovel setup. What we're doing is we're basically just swapping labels right? And we're checking to see how much of the information does the
David Bau: what person regular persona data set with the shuffled labels does, and then like swap vice versa of the other one. So I guess the interesting thing about this that Alberto actually just said in chat was, if you take the shuffled setup which, shuffled is again trained on the new personas.
David Bau: No, it's the same persona names new attributes right? Shuffled attributes. You actually get higher performance with the original labels than if you were to just
David Bau: Take the original data set with the regular labels. And I guess it's interesting, because
David Bau: there's like some kind of knowledge conflict here, right? Because then now you have the verbalizer. So model 2. And then you have model one, and there's conflicting knowledge. So which one does the model which one does it take. Apparently it takes the verbalizer to knowledge. But the model 2 knowledge, which is the verbalizer, even though model one implicitly, has different information.
David Bau: But yeah, this is pretty consistent across the board, like, if you look at persona, Qa shuffled model just doesn't do well, like when you verbalize some of these outputs. Right?
David Bau: yeah, I think that's like the big part, which was like slightly surprising, which is like, basically the reviser will keep like it's implicit world knowledge. On the basis of even if you like train the model. So it's really not taking that information from the activation.
David Bau: okay, I'm gonna skip this because there wasn't also, like there was extra stuff where I talked about robustness and says, sensitivity of like the the interpretation prompt.
David Bau: This might be similar to what you're asking like, did you try with different prompts, which I did? So I asked 2 questions, which is, if I ask, what is the effect of using semantically similar prompts? The reason why this is important is because prior work assumes only one set of prompts with patch scopes. They're just like, Oh, just try this prompt. But then they're just like they don't give like a reason, or they don't talk about like what happened if you change the prompt
David Bau: okay. So to read this. Basically I, this might be a bit tiny, hopefully, not. But I basically have the original prompt. And then I show 4 different, semantically similar prompts that I wrote and Jerry Pick, I mean, maybe it's going to be hard to explain that. But I just wrote whatever sounded the same, or I got Claude to write it for me. And then, if you see that there is, I think, for this one, this is 0 point 3 5 to 0 point 1 4. So even if you pick like a pretty similar semantically similar prompt.
David Bau: you will get pretty large drops in performance for some, and it's like pretty inconsistent like, I think, star constellation, which is asking about the constellation based on the input
David Bau: you can with the non original prompt, you can get a very high performance if you just switch it. But there's not really a reason as to like
David Bau: why some prompts work better than other others. I mean, this is just like prompting in general, what's like robustness?
David Bau: There was also another one that I did, which is, I don't know how interesting this is, but basically it's kind of similar to the chain of thought where you basically ask, like, Oh, like, I think the answer blah or something, I have 2 different ones where I basically
David Bau: have like a like. So the 2 prompts is this, one is me saying, I think.
David Bau: that the answer is this, and these answers are always wrong answers, by the way, so not ever the same as a single in one sample. And I also have one that says the answer, is this right? And so the interesting part is sorry. The answer must be this so so the interesting part about this is, if you say, must the model. The model performance drops tremendously, so that amount of
David Bau: prediction is like. The a significant amount of the prediction is from predictive power is from the verbalizer, which means it's not really faithful to interpreting activations. But yeah, so I thought, that was like, pretty interesting.
David Bau: And in conclusion, okay, I guess my conclusion from this is, I mean, I'm not a big fan of the tool, but maybe there is some use case for this.
David Bau: But I think the main issue is like this might not be as easy to compare or control, because there's a lot of issues with conflicting world knowledge for model one and model 2. And honestly, I think even just using a basic probe might be better for like finishing running one experiment with a probe. But I think the probe can still extract information that a verbalizer wouldn't be able to under circumstances where the probe
David Bau: where the verbalizer like does not have knowledge, and neither does the probe like we basically ran some preliminary experiment with a probe where we were getting higher than random performance on the probe. But the verbalizer was getting 0 when we were trying to predict the fantasy set.
David Bau: So I think that's that makes it more appealing to use the probe
David Bau: But I also think, yeah, in in some situations. It's not like good to use verbalization where you can just interpret input text because it's little questionable about what you would care to ask for.
David Bau: yeah, but yeah, I think that's basically it. But yeah, thank you. Melissa, yeah, cool.
David Bau: Yeah, that's great.
David Bau: I love the overall framing of the question when you're when you're measuring
David Bau: something with a complicated instrument. Yeah, are you measuring your subject? Or you're measuring your instrument? Right? Yeah.
David Bau: I think I I mainly like the I think, what I liked is everyone interpretability wants to do this certain thing which is to understand like how models work. But sometimes you might be measuring the wrong thing, which is like, kind of
David Bau: what I've taken from this. Yeah, yeah. So 3, 4, cool.
David Bau: Okay. Any last minute questions for Middlecent before she has to vanish. Yeah, to disappear. I thought the fantasy thing
David Bau: was not that trivial? Oh, really, I think it's kind of kind of interesting.
David Bau: That, like the verbalizer model doesn't know how to say
David Bau: raise roast or whatever I mean, because those tokens are real. They're real tokens. Yeah, exactly. So it's like it doesn't know. Maybe it's because it sort of relates to my stuff of like, it doesn't know how to actually say that it's never seen this, or it has some sort of like reconceived notion of what a food name should be and it does not really map it to the name of the food.
David Bau: Yeah,
Alberto Ceballos Arroyo: Yeah. So I think. Oh, sorry.
David Bau: In the country. Okay, yeah. Alberta, what did you say?
Alberto Ceballos Arroyo: Yeah, I was just saying that
Alberto Ceballos Arroyo: actually, we 1st did the fantasy experiments before doing the shuffle one. And we also came to the. We also came to wonder whether the issue was that the verbalizer wasn't really able to
Alberto Ceballos Arroyo: to produce those tokens because he had new, had never seen them. But then, when we did the shuffle experiment, it also didn't work very well. So I think it's probably a mixture of both things, both the tokens, the tokens like not the tokens, but the the words, the words being new.
Alberto Ceballos Arroyo: and also the and also information being not available to the world model of the herbalizer.
David Bau: Yeah, very cool. Yeah, really cool setup. Okay.
David Bau: cool. Thank you. Yeah. When I talked to byron about, I was like, honestly, I don't know. Sounds kind of trivial, but I don't know. I think it'll I I do see like the what is it like the benefit? Or like just explicitly like saying these things, even though they're like.
David Bau: kind of like, obvious or something. I don't. I guess it's not always obvious to everyone. Yeah, yeah.
David Bau: yeah, awesome.
David Bau: Thanks a lot. Melissa. All right. Thank you. Guys.
David Bau: yeah. Okay. Great.