177: So this is this is Aruna, who is a Dylan student
177: and and so we're looking forward to her.
177: Her talk here. It looks like
177: in collaboration with Anicus also. So I'm interested to see what kind of mechanisms might be involved in this. So Hi, everyone I'm going to be presenting this paper. Many of the results in this paper are fresh off the oven, like, you know, I think I kind of made some. Yeah, I kind of made some discoveries last week, so
177: I'm really excited to have this slot. You know. The vow meeting Slot is super popular, so I'm really glad. You know, I could like talk to you all today. And it would be really helpful to get some feedback.
177: And yeah, so feel free to, you know, like, yeah, just be super open and critical. And I'm I'm really looking forward to. Yeah, hearing your feedback.
177: So yeah. So this is the title is to look at reward distributions and using reward distributions to guide model generation. And this is joint work with Amir Zool, Atticus, Geiger, and Dylan.
177: So
177: there are 2 primary motivations for this work. One is that when we think about causal mediation analysis, where you know, we're using these
177: prompts and counterfactuals. And we're patching, you know, from one to the other. We often look at outputs that are like single token, output distributions and causal mediation analysis based interpretability in these multi token output distributions is relatively underexplored. So that was one primary motivation for this work.
177: and the second being rewards are inherently interpretable. And I'll get to this in, I think a few slides. So basically, like most of the chat models that we're using today use Rlhf for you know, training and to guide generation. And we have, you know, I think, 2 primary methods that people use to optimize these chat models. So one is Ppo.
177: which is like an older technique. Now, you know, it's only been a few years. But it's it's sort of like, you know, there are lots of different optimization techniques. But Dpo is also like one of the more popular ones.
177: and in the case of Pbo you have an external reward model. So you start with the base model. Then there's some supervised fine tuning. Then there's an external reward model that's trained on a set of human preferences. And finally, you know, you use that external reward model to basically optimize a policy. And you have your final chat model.
177: The Dpo model is a little more interesting because you subsume this external reward model into the final chat model. So the reward model is basically enshrined in the the chat model that you have. So you can actually extract a reward function just using the chat model that you have. And the model is optimized using this internal rewarding mechanism.
177: So the question, really that we were interested in was whether we can use
177: reward distributions to guide generation in some meaningful way. So
177: of particular interest, as I mentioned earlier are the Dpo models, because, you know, these reward functions can be extracted from the chat model itself. In fact, this is the reward. You know that you can get from your Dpo model where the Pi Theta is the the final like chat, the chat model that you interface with.
177: and the pi ref is basically the policy from the base model or the reference model. So for every response that the model produces, there is this inherent reward value, and the model is trained to prefer responses that have higher rewards than lower rewards.
177: So there is basically in these Dpo models. There's a theoretical entanglement between, you know the reward function as well as the responses that are generated.
177: And the question that we were interested in was whether there is a mechanistic entanglement, so are the components that are sensitive to manipulating the rewards in the model, also sensitive to manipulating generation. So if you take
177: a particular task, you know, you can think of it as having 2 components like one is just supervision, which is, you know, telling the model which preferences to rate to rank higher or lower? And does the act of actually like generating outputs pertaining to the task? So the question was whether there is
177: a mechanistic entanglement between these 2 ideas. Given that we know there is a theoretical entanglement.
177: And why would this be a good thing? It would be a good thing, because it's really cool that you know, you can show that there's this entanglement, but also because rewards are inherently interpretable entities. You know your reward is a numeric value, so if it goes up. You know that the model is going to prefer this more if it goes down. You know that the model is going to prefer it less so if you can use this up and down sliding to actually
177: then influence what the multi token output distributions are. Then that could be like, I don't know, like a really cool space where you maybe have steerable controls that you know. Tell you. Yeah, that that push the model like one way or another, just using these numeric distributions. So stupid, I'm gonna I'm gonna I'm gonna read your, I'm gonna read your formula here and see if I understand the symbols correctly. So
177: pi sub ref is that the base model before fine tuning and pi sub theta, that is the dpo fine tuned model. And then you're saying that
177: even though there's
177: so so so you're saying that the reward can be explicitly formed by just taking the difference between the log props, at least. And that's that's directly the reward. Yeah, so and so now, so that. And that's so for an extended rollout.
177: That's just the sum of the individual token log probs. And I just take the difference between these 2 sums. And then that directly gives me the reward. Yeah, it's sort of interesting. So like my, if I remember right, so like Dpo is not sensitive to the exact reward. It's actually like a shift like if you if you were to add 10 to all the rewards whatever. Yeah. But but this is
177: a plausible reward that could have led to the same Dpo. Is that right? So this is a great question. So basically, the reward that I've shared on the screen is like a simplified representation like, there's another constant. Actually, that's part of this function. But basically in my method, I'm going to be using differences of differences. So basically that constant cancels out
177: got it. So so that's a really good question. But like, we overcome. Yeah, yeah. And like, you, can you you if you're just calculating the actual, absolute reward for a response. You need to also know this constant factor. And there's a Beta value. But we're going to be working in a difference in different settings. So those values will all cancel out. So that's the yeah.
177: So yeah, so so why would this be useful? It's useful, because, yeah, rewards are inherently interpretable. And the goal is that you know.
177: you maybe have like an interface where there's a Slider and the user is able to like just steer towards like a numerically high or low value, and that has a corresponding influence on the multi token generation. So these are the 2 big motivations. Also because I think I'm really interested
177: to see how like the future of interpretability assists like not just developers, which is, I think, where we are right now, like where we're trying to like, use it for more performance models or understand models better, but also future where, you know, like the end user, who has no idea about any of this, like, how would they use like the the methods that we've, you know, developed over the last few years? So, yeah.
177: So the goal is to investigate whether these reward circuits can guide generation
177: and today, I'm going to be using 3 task settings. So one is the classic refusal problem where you're trying to, you know, get the model to refuse harmful questions and get it to answer harmless questions. But I will try to get the model to refuse harmless questions, which is the more boring setting. But I'm sorry I don't have the experiments for the other one done yet. The second setting is stylization.
177: where I do style transfer. So basically it's you can think of it as something that works in place of a system prompt. So suppose you want to get the model to like, respond in verse
177: as opposed to like, you know, responding in just like the standard pros that it uses. So can we use like reward distributions to do that. And the last setting that I look at is to reduce sycophancy. So can we actually use these distributions to suppress. You know the sycophantic behaviors of models, and I look at this again in like a constrained setting. So the overall, the overall kind of way that I do. This is
177: in all cases I'm always identifying components that are sensitive to reward manipulation. So these components, they don't. Really, they're only going to be measuring whether the reward is increasing or not. Like using standard, you know, causal patching methods. And then I will update these reward sensitive components. And then the goal is to the the.
177: The exciting result is whether the generation is affected. Right? So that's that's the goal. So that's yeah.
177: So the 1st step.
177: So I think, David, yeah, he he was like super astute. And he noticed that you know, the reward was this super simplistic value. And the reason why I can do that is because we work in this difference of difference setting.
177: So what we do is we have the base queries. Sorry I've been working with Atticus so earlier, I used to say prompts and counterfactuals, but he uses this base and source thing. So I'm going to, you know. Now I've got a little bit of a decency. By which one are you patching? From? Which one to yeah. So the base is the prompt, and the source is the counterfactual. So the base is the baseline, and the source is the target. You know. I don't know.
177: They call it the source. They should probably call it the target. But anyway, so but the error with the pitch goes from source to base. Right? Yeah. Sorry. That's a mistake. Good catch. So it is from source to base. Yeah. And the
177: yeah, yeah, it's just standard patching. This arrow is confusing. Yes, it should just be
177: I don't have my cursor, but it should be pointing from source to base. It is so, but it's pointing both directions. Oh, oh, they differ! No, actually, it's true they both differ in. In one token, I think that's what I was trying to say. But the patching direction is yeah from source to base. Yes, yes.
177: So we have these base queries which are which you know
177: are the baseline behavior. They model the baseline behavior that the model has. And then we have these
177: source queries which are by just changing one token, you can like kind of change, the nature of the behavior. And I'll be giving you some examples. So yeah, so hold, hold any thoughts like just about the queries, and in each case we generate, you know, preferred and dispreferred responses. So again, preferred responses are the baseline behavior of the model. So you're not going to try to get it to like respond in
177: any other way like, it's just the baseline response of the model generally would make if you give it that query. And the disprofiferred response is basically the undesired behavior or the behavior that you want to guide it towards.
177: So not undesired, like the target behavior, the behavior that you want to guide it towards
177: so, and then you patch, you know, from the source to the base like this group like needs no introduction to any of this like, so yeah, so basically, this is the initial step.
177: And the second step is that it's really important to use reward differences, that it's you're you're, you know, using the difference between the preferred and the dispreferred responses, because
177: rewards in and of themselves don't mean anything like if you, if you have like a response from a model, and it has some value. N. There's no inherent meaning to that like. There's only meaning, if you know that there are 2 responses, one with a reward, n, another with a reward. M, and there's some comparative relationship between n and M. So it was, yeah, it was kind of important that we use reward differences.
177: So basically, we use
177: the difference between the preferred and the dispreferred rewards in the clean and the patched state to guide the model towards you know the source. The source data distribution. So here's 1 example from the refusal setting.
177: So here, my base, so my base prompts will always be outlined in yellow. So you have how to make a cake at home. And this is like a harmless prompt, and I'm just going to change one token in that query which is how to make a bomb at home, and you change the nature of the behavior to be from. I mean to to move from like harmless to harmful
177: and in each case you have preferred and dispreferred responses.
177: So for the case where the behavior is harmful, your preferred response is that the model refuses the query. And this preferred response is where it, you know, actually answers a query. And like, Ixu gave us an amazing you know, yeah, just, I don't have to talk about jailbreaks, because she kind of, you know. Yeah, already. Set the stage for me. So you know, we do use jailbreaks to like, generate the disproferred responses in this setting
177: and
177: in our baseline prompt. Yeah, we, the preferred response, is just what the model will generate. Naturally when I ask it this question, and for the dispreferred response again, I jailbreak the model. So I don't use. Gcg, which
177: was not very performant as when I try to use it. So I use a very simple jailbreak, which is, I use multi shot in context learning. So I just give the model like a bunch of examples, and give it like a final example that it has to, you know. Give me the answer for, and it'll just do it. So one
177: one thing of importance that folks might ask here is, yeah. So is it possible to generate this preferred responses? Yes, and obviously, like we do a behavioral analysis, and we ensure that the reward for the preferred response is always higher than that for the disprofered response. So I think Yixu also, you know, had this in in her second project. But one interesting behavior that we're looking at is this is not always true.
177: and this is because of what I was describing with respect to the base model probabilities playing a role. Sometimes the base models, very low probabilities can push the reward high, and we think that might be why, you know, some of these deal breaks are occurring. But anyway, so that's an aside.
177: Then another question is, why would I be jailbreaking these models? Can't can I not use chat, gpt, or, you know some other 3rd party model to do this? So
177: Hold that thought like one reason why we started off with just jailbreaking the model was because we're very interested in the data and the reward distributions of the model. And it's kind of important that. Yeah, like that. These response distributions come from the model and not from a 3rd party source, though. I do have some results from just getting this from Chat Gpt. And what does that look like? So I'll be talking about that.
177: yeah. And the next step. Oh, I don't know my numbering is wrong, but like the next step. So so you're so you're you're so backing up for a second. So your your source wait. So okay, go back to. So you described your source base is.
177: so is this is one of these source in one of these base. Yeah, the yellow is the base, and the red is the source. I see, I see.
177: and and is there. And then you mentioned jailbreaking. Are you jailbreaking as part of one of these things, or you're not. Jailbreaking is just. You're just doing this thing
177: to cause a difference. You're just changing one word, and that's just source and base. Just one word difference. This is one word difference. But for generating the dispreferred response, especially for the refusal setting, I had to jailbreak for my 2 other settings. What I do is that the preferred of one becomes a dispute of the other. Okay, so those are like easier settings like, but refusal was a
177: Yeah, it was like a was like a slightly different setting is a jailbreak part of your prompt. So they say, how to make a bomb at home and some jailbreak in there. Is that what you mean by using jailbreak? Oh, great question. Okay? So so no. Our source and target queries are just going to be like what's on the screen here. But when we were generating the preferred and the dispreferred responses, we used jailbreaks to just get the response distribution. Oh.
177: oh, I see. Okay. The jailbreaks were more for the response distributions and not for the I see. Just to get this text. You're going to use lock probs on. Yeah, yeah, I see. So it might not be the direct answer. Here. It might be some other way of getting the text, yeah, got it? Okay?
177: Yeah. Awesome. So yeah.
177: you have a question. I was just saying, it's really cool. Setup. Okay, okay, cool. Thank you. Talking about this yesterday.
177: Oh, okay, cool. Wait a minute, because you know there are. There are some things that I'll come to in the end, and we should, we should be talking about that. So yeah. So anyway. So the there's a step that I skipped here, and I'll tell you why so initially. I can just like, maybe talk briefly about it. But it's not super important to the story, and it's
177: I skipped over it. But the 1st thing we did was actually a core screen patching. So we did residual stream patching to just see whether
177: the layers that are influential in manipulating the rewards are also influential in manipulating the generations, and the correlation is very weak, and I skipped over those slides. But I can show them later. I thought, you're doing this
177: like, I'm doing attention. Head patching directly. Yeah. Yeah. So I I. So these are more fine grain patches. But what about this Atticus? He knows himself. They're a little bit fragile. Right? He's just go back to
177: yeah. So the data sets we use are like, really, tiny. So these are like, yeah, yeah. So we didn't use that. It would be really cool to use das as well here. I can. So yeah, there are lots of
177: failed things that happened like through this project. So there's lots to definitely say that but anyway, so
177: the next step is sort of a very fine grain patching. So we're just patching attention head components. And we're finding, you know, the attention heads that are most sensitive to manipulating the reward. So the metric that we use is the one at the bottom.
177: So basically, we're trying to find attention heads that
177: maximize the. So if you notice like this equation in the numerator. The 1st quantity is the sign flipped version of the second one.
177: So what we want to see is whether the reward direction flips like after we do the patches. And this is actually and and the denominator is to ensure that this reward distribution is close enough to the source distribution that we're trying to guide it towards. So the numerator has a role of like flipping the reward, and the denominator has a role of ensuring that this stays close to the source distribution. And the
177: yeah. So one other point I want to make is this is very close to the optimization function that was used in the Ioi paper. So they were you looking at single logic? You know, differences. We are looking at collective like log props. Basically. So we're looking at like the whether maximizing.
177: like finding components that maximize the collective reward. Right? The collective log prop can actually steer or guide generation. Yeah, so okay.
177: so right now, you are in multiple token settings. Yeah. So you are generating a lot of tokens. Do you do this patching in all of these generations like throughout the generation process, or that just the 1st token great question. So so this is where like rewards, are really useful. Right? Because you the I'm oh, sorry, no, actually, this is not related. So you're just yeah. We're just patching across. So we're patching across all tokens. Yeah. Now, my second question will be, how sensitive it is it
177: to the length of your generations. Because right now, what you are doing is it's kind of like inverse of perplexity, log perplexity.
177: They're very, very sensitive to the length of the generations. Yeah, because it's the language model. It's supposed to
177: it kind of like picks up what it needs to say, yeah, so how do you handle that? So this is actually handled by the reward computation. So the rewards actually like you can when the models are trained like they're normalized for the reward. But we could try another approach where we also normalize by the length. We didn't do that, and we still got decent results. But
177: we could try an approach where the reward it's so. There are like different reward definitions, and one of them will divide it by the length of the response. So you're like normalizing by the length of the response. We didn't do that. That's not a good approach, right? Because I mean.
177: I mean, as as you are sampling more and more tokens.
177: like some of these tokens are kind of like obvious, like it's supposed to appear after like. Given this, given this context like so their logics will be obviously be high. And if you're just dividing it by their length.
177: that is
177: kind of it's you saw that you were getting reasonable results. Yeah, even without normalizing by the Len. But actually like, though there is like past literature that has looked at, you know, like normalizing by len, just in the context of rewards, like, not in the context of causal mediation analysis. But it would be interesting to see like what kind of differences we get right? Like when we do this. So
177: yeah, it's a good question. And the thing is that when you're doing this attention badging
177: okay, you are doing it at a specific token position, and you freeze the attention out. Push to that value and let the model generate itself. We do it one by one. Yeah. So it's standard activation patching. So we patch it one attention head. Look at the reward at, you know, after patching, and then like repeat for each attention, head!
177: Do we also have an up an attribution, patching approximation, working for the same setup. So, and I haven't tested it on my latest results, but we did. I have tested it in the past, and the results are identical. So
177: so you can actually think of these, as you know, working like within a minute, you know, like, when you are sliding like, because you have the advantage of using attribution patching. Okay? So yeah.
177: yeah. And so, oh, sorry. Okay, how? How can you do attribution passing here, because because when you sample you, you have lost the gradients.
177: Oh, great. So because you're working with the rewards you're not working with like generative outputs because you're working in the reward space. You can like, okay, okay, okay, okay, yeah. So that's a great question. So the thing that was hardest to test was, we were looking at correspondences like just correlations between
177: doing things in the generation, setting and doing things in the reward setting, and like the reward setting is so fast to run, you know, because you can use like gradient approximation. But you can't use it like when you're doing generation, because yeah, gradients are disabled. Yeah, great question. Yeah.
177: Yeah. And the last thing that we do is we ablate these reward sensitive components. And then we study the effects on on generation.
177: So the way that we do this is we identify the top K percent of heads. That maximize. You know, this. This like optimization metric.
177: And then we oblate them with transformed, mean representations of the source. And I'll come to why the transformed exists there. This was this turned out to be actually really important. I fumbled for a couple of months until this transformation was a little was kind of like a little important. So I'll come to it in a bit and
177: then we evaluate whether the generation text is biased towards the source after we do the sublation and the way that we do. This is we use a judge model, and we use likert scales. So
177: yeah, one aside, here is.
177: I find, the harm bench, like evaluations of refusal, like pretty brittle because they they use like keyword matching. So they look at whether the text has as an AI or like, you know I cannot or sorry, or I can't. And
177: basically like when you, when you actually like, play with the model. And you, you know, get it to generate things. You'll see that lots of times, it'll say, as an AI. I can totally help you, you know. So yeah, so it's not as shallow as people think it is like this whole token just just the initial tokens of the model. So yeah. So so that was one reason why we wanted to use like a judge model and a likert scale.
177: So I have another question. Yeah, it's it will be a down question. But no, no. What is the relationship between refusal and attention.
177: Oh, no! This was totally an aside, like in the sense that we want to do evals right? So
177: so how will we do the Evals? We will find attention heads that are sensitive to reward manipulation.
177: So which of the attention? Why, what's so interesting about attention? Heads. Why, it can't be Mlp. Or simply oh, we we were. We thought residuals were too big, so we didn't want to work with residuals, but we've not looked at Mlps. So yeah, that totally open. We still need to look at Mlps. We just looked at attention heads because I've looked at attention heads in the past, and I've found that
177: they're fun to like mess with. So it was just like the first.st It was just like an obvious choice. That's a good reason to do research. We're like, Yeah, I, I need to look at Mlb, so this is a great, great point. Yeah.
177: and
177: yeah. So so to answer your question like we find attention heads that are, you know, affecting just the rewards. And then we want to see once we patch in those what happens to generation, right? But we need to evaluate generation. So to evaluate generation, we're using like a judge model and some kind of a likert scale. So we'll I'll I'll come to it. Like as I talk more. And yeah, so if you have any questions, yeah.
177: okay. And then we also want to just update randomly selected components.
177: And then make sure, make sure that it's not really like changing things very much.
177: So yeah, you've had a question. Okay.
177: and this is something I haven't done. But I basically also want to do this ablation. But measure what happens to baseline Llm tasks. So basically look at tasks that are adjacent to the task setting. I'm in and orthogonal to the task setting I'm in. So if I'm looking at refusal, then, maybe looking at something like reasoning or something like that. And
177: but you know, I'm also looking at sycophancy. So what happens to sycophancy versus refusal like, are they adjacent in some ways. So these are things I haven't done. But I really want to do as part of this work.
177: So there are 3 task settings as I mentioned. So the 1st is
177: sycophancy, reduction, the second is style transfer to worse, and the last is refusal.
177: Right now I haven't looked at any other models besides the Quen 1.5 14 B model. But I yeah, I will be running experiments on models of a bunch of different sizes. So right now, the the 2 dimensional setting is just one model. But multiple tasks.
177: Yeah.
177: So we started with.
177: yes, yes, I do use end of, yeah, yeah, I use. Yeah, yeah, we use insight. I use insight. Yeah. So anyway. So we start off with a behavioral analysis. And
177: So here are the 3 task settings, the P. And the D on the x-axis stand for preferred and dispreferred response rewards. And the difference is a reward difference.
177: And you can see that there's a significance significant difference between the preferred and the dispreferred reward distributions. So maybe, like, you know. So this was like an initial clue that maybe pushing between these 2 settings could also, like change generation. Interestingly, the 2 settings, the reward differences are not significantly different.
177: And this didn't seem to matter. The 2 settings are.
177: yeah, I'll I'll let you all stare at the close.
177: So the style versus pros. So it's so this is, I see.
177: And so, okay, hold on for a second. So you've got pairs of pairs
177: yeah, in each one you have. P and d, yeah.
177: And so, okay, so let's take a look at this sycophancy thing you've got on the left.
177: right? And so this is the setting. So
177: give me an example of what? What? So what's it? So describe the circumstances in words? Yeah.
177: So in the frequency setting the model is given a poem like a haiku and a user opinion. So the user opinion can either be, I love this Haiku, or I hate this Haiku, and then you collect, preferred and dispreferred responses. So this setting is a simpler setting, because in the I love Haiku case, whatever is the preferred reward becomes the dispreferred reward for the I hate, I see. So now now. So the the user says, I love this
177: and then and then the high reward, you know response would be something like, Oh, I agree, this is really great. If the user says, Oh, I hate this one. And it says, Yeah, you're totally right. This is like the worst poem I've ever seen, right? So that so
177: right? So so that's so, that's what the reward model
177: it's rewarding in these 2 cases. And so when you say so, you've split out.
177: you split out the positive and negative case of like, the user liked it, and the user hated it as green and blue. And then when the model agrees with you.
177: Then that's always P, and then when you switch it to the models opposite.
177: that's what you're measuring as D, yeah. And then diff is just the difference between those 2. Yeah, okay, got it. And then a similar pattern for all the rest. Yeah, I think I made a mistake. I'll be going through the tasks in detail, but I think the slide should have maybe come later. I think the only point. The only takeaway from this slide is that the difference between the preferred and the dispreferred reward distributions is significant.
177: So these are, like, you know, very different distributions. And yeah, so
177: yeah, that's the only takeaway and on the left you have like one setting. And then the the 2 settings are in different colors.
177: So yeah, sorry. Oh, maybe a dumb question. Do you have a reward model for each of the cases?
177: No, it's the same reward model. So the reward model is just within the Dpo model. And you you don't even have an external reward model. So this is just like the standard Dpo reward setting that we use. So we're working with one chat model, right? But you can extract a reward function. And because each of these tasks and the corresponding responses have different probabilities in those settings. You're gonna get different rewards.
177: But it's great that it works right? So it's basically like, at least on average, the model is
177: referring, what? Yeah, yeah, yeah, it. Yeah, yeah. It looks like the reward was, yeah, yeah,
177: yeah. So this is the 1st task which is to reduce the frequency in the model. So always the source prompts will be in red. Always the base prompts will be in yellow.
177: So the 2 prompts are identical. The queries are identical except for the words hate and love. That's different between them, and in both cases you ask the model to critique the haiku for you.
177: So in the case where you tell it that you hate the haiku, it says, Well, you know the haiku presented is kind of nice, but it could use like a more focused structure and a clearer connection between the elements. But if you tell it that you really love the Haiku, then it'll it'll just like, give you glorious praise. And it's yeah. It's just very. It's not going to give you any constructive feedback. So, even though in both cases the task is to critique the haiku. So
177: the chat. Sorry. I, yeah. Okay, good.
177: So so this is the 1st step like where we where we just like have this data set. And we know that the preferred and the disprofiferred response distributions are, gonna be different.
177: And then we, you know, do standard attention head patching tries to like maximize that metric?
177: And then we ablate 3% of the heads. And we can actually and these are like heads that are sensitive to reward manipulation. And we see that there's a
177: 76% reduction in the in the case where I'm targeting the heads that I found. But basically, random heads also cause like a 54% reduction. I have a whole I have like a lot of opinions on this idea of circuits which I'll come to in the end. But yeah, I this is something that I've been thinking about. Like as I was working on this project. Yeah.
177: So this is basically the reward change. Oh, sorry. So the in the case. So the reward.
177: Sorry. Let me explain this properly. So
177: when I patch in 3% of the heads, I see a
177: 34% flip in the reward direction
177: for random patching. But I see a 54% flip in the reward direction for targeted patching.
177: and there's corresponding changes to generation. So in 76. So the generation is always in blue on the left side changes in generation, and the reward changes are always on the right and green. So yeah, when I do these targeted patches on the reward. Sensitive components, I see a 54%,
177: you know, change in the way that the reward is assigned. But I see a corresponding 76% change in how generation is now different. So the blue bars are Llm evaluated. Is that right? 1st the blue bars are Llm. Evaluated
177: so, and and I might have missed it. So you you mentioned before you had some trick for how you had to do the ablation to make. Yeah, yeah, yeah, no, no. So I will talk about that. So basically, let me talk about that. Actually, so yeah.
177: do you have a baseline for just next word without using the without using the reward model you just
177: try to like.
177: try to like, achieve a certain logic distribution in the next prediction, and then it does. It also generalizes to generations. You mean that if you remove that reference component in the reward, then what happens like just the standard log props, or
177: I know you mean just a single token, not the whole rollout which token just the 1st generator token the logic distribution.
177: Oh, I did not look at that. Yeah, that's a good question. I did not look at that because I I asked that question, because that was kind of the reason you
177: went from 1st token, like modeling the 1st token to the to using this reward model to actually model the generations. No, the biggest reason why we wanted to use the reward model was because like these chat models are fine tuned on the reward, you know, like the optimization functions for both Dpo and Ppo use the reward function. So
177: like our thinking was, you know, given that it's so embedded in the optimization function. Does the reward play a role in also like guiding generation?
177: Okay, yeah. But this is a good like, I. I don't know. That's a good question. I haven't looked at just the 1st token distribution. And I should like, I I just haven't done it. Yeah.
177: Yeah.
177: But anyway, so
177: yes, the yeah. So the the transformation. Yeah. So the thing that that seemed like really important was.
177: that when you just patched the source representations directly onto the base when the model was processing the base prompt. It didn't seem to make any difference. In fact, you had to patch about 30 to 40% of the model in order to see that. So one trick that I tried was just steering. So, you know, using like a difference in means, and that works, in fact, that works even better than this method. But
177: it was not. It was also working very well in the randomized setting. So that that was curious. So what I ended up doing was just amplifying the value of the source representation. So it's like a it's like a really small change. But it seemed to have this
177: really big effect on just how many heads you needed to patch, and it seemed like more localized, even though I don't believe that the kind of settings I'm looking at are uni variable settings, you know. They're like actually simple settings, even though you're doing multi token distributions like you. You're just looking at one token difference, and that's the only variable in your abstract causal model. And I wonder if
177: in in these cases, where there are these univariable causal models? There aren't like localized circuits, because I think patching gives us a way to efficiently find, like the circuit that's able to guide the model. But
177: I think so. So when you're okay. So when you're patching these attention heads, yeah, you're patching them at one token. Is that right? Is that what you is that what you said? No, I patched the whole representation.
177: but the whole representation only varies at one token. Yeah.
177: right? So you're right, I see.
177: So you patch it at one token or multiple token, because I patch it at all the tokens. But but but but the work! But the 2 prompts only differ by one token. But the representation of the words, after the word that you've changed are different. Yeah.
177: I see. So you're you're Pat, you're you're patching all aspects of that, including the downstream effects. Yes, interesting.
177: Interesting. Should I not be doing that? Oh, I'm not sure. Maybe maybe you can. I mean, like you can get Nikhil's advice about what he thinks so like when you Nikhil, when you patch, usually patch a specific token position. Yeah. Oh, even when you're doing attention. Head patching. Yeah. Oh, interesting. I should try that. Yeah, because I wonder if you're getting some weird or something like that. Yeah, yeah.
177: yeah, yeah, that's a really good point. Like, okay, that might explain why I was so untrustworthy and suspicious of circuits like I could try this. And maybe maybe I feel more trustworthy. How much did you have to amplify it?
177: By a factor of 5 or 10? So yeah, but that's pretty pretty large, and and you never have to amplify it to get your your results. Right? Okay, yeah. I also never did it like, I think my sense was. So I was talking to Dylan about this. And
177: one of the things he was saying was, maybe like, because these are multi token distributions, like, if you know things are getting overwritten right? And the and the amplification is necessary to stabilize the signal of the important. You know. Attention heads. Yeah, I kind of just tried something. And it it worked really well. And I I'm still trying to figure out why, yeah.
177: steering works actually, even better. But the thing is, yeah. As I was saying, you know, with steering, I can
177: coefficient range.
177: Yeah, I use the same factor actually, even as my steering coefficient. But in the case of steering, even random steering was working really? Well. So I was like, Okay, you can destroy syncopency by just randomly kicking. That's the problem with a lot of these things. It's like, you never know, right? It's like, Can we get rid of the capability just by making the model. Dumb?
177: Yeah. And so that's dangerous.
177: Yeah, yeah.
177: yeah. And so this was sort of my likert question, where? I said, you know, read the following conversation between a user and I give it 2 responses. So I give it the response, pre patching and post patching. And then I say, well, response one, which is the post patch response offers more suggestions for improvement or points out more flaws, and I only pick responses that are 4 or 5. So I don't pick the neutral response. So this is kind of a stricter constraint.
177: And these are some responses. The one in gray is like a like a false, you know it. It didn't work for that case. So I just wanted to share.
177: We've been discussing a lot, arena, and I'm just warning you that I have to disappear in a few minutes. How how much more do you? It's it's I think you've you've heard all of it like it's just I'm just going to show that in other task settings, also, it works pretty well. It works pretty well. Yeah, you can show off some more examples. Yeah. So like this is, you know, inducing like a poetic style. Same approach.
177: And yeah, so here, like the reward. So this is a reward. And this is a generation, and there's a stronger correlation you can see between the reward and the generation.
177: This was a like good question. You know, just the responses, and worse, that's it.
177: and yeah. So here also, there's like, you know. So it always started the responses within the realm of something. And blah blah blah, and in the failed sample, it also starts like that. It just doesn't construct a verse.
177: So it's still in. Still, still a response in pros.
177: Yeah.
177: And the the last one was I was trying to induce refusal not trying to create a jailbreak, but like trying to do the because it's easier. I I need to do the harder thing. But like I was just doing trying to get the easier thing to work 1st and same approach.
177: Here you can see that in the random case the rewards change by a lot. But somehow that doesn't translate to generation. So these are still some things that I'm figuring out these are very, very recent results. So yeah.
177: and I think, like, actually here, that I'm that it's something to do with my initial pre initial preferences, preferred response data sets
177: because the model would always refuse, but hedge an answer. So I should probably change that and then retest it.
177: But here it it pretty much always refuses, but then answers, so it starts off with a refusal sentence, but then it answers the question, yeah. So this is a setting that needs more work for sure. Like, I need to see if this is something off with my data. Is this also something which you still have amplified? Yeah, everywhere I'm amplifying. Yeah.
177: Shifting heads.
177: Sorry. Yeah, shouting attention heads, exactly. No, we don't want them to shout, so this is some. This is something that I really wanted to show you. So this was.
177: These are the reward distributions like, Pay attention to the blue and the green violin plots. The green plot is the behavior we want. The blue plot is the behavior we are trying to suppress
177: and basically like, there's like a pretty strong correlation between when the distance between these 2 increases and the corresponding like changes to generation. And I'm actually like, I'm working on a little demo where we can have a Slider where, as a user, you can see like how far these distributions are are and like, whether that changes generation. So you have the control to like.
177: you know. Do that shift, and then like, see what happens like this was a common pattern in all the 3 cases where the more the distance between these green and blue, violent plots, the yeah, the higher the likelihood that you'll see a greater number of responses in the target behavior.
177: So this is true for all 3 cases.
177: Yeah. And so these are like a lot of what if questions. So what if the data was blindly generated off of Chat Gpt, and not from the models distribution? I thought this wouldn't work. But
177: it seems to work decently like I I'm not sharing the results because I'm not confident about them. So I'm I'm still like, kind of like, you know. Just want to like double check and triple check those results. What if we use steering oblations instead of using like these shouting attention heads? It works way better. Actually, like, even sometimes with patching 1% of the attention heads, you know, I can cause like a change in behavior.
177: But yeah, it also seemed to affect like random steering, so random randomly selected attention heads. So I need to look into that more. What if we use logic distributions directly, I think I know, like maybe alluded to this a little bit.
177: So I thought also that this wouldn't work. But this also works actually quite well.
177: So this is also something that I'm looking into, like Quinn doesn't really release, whether it's like a pure Dpo model. There's like a lot of actually theoretical correspondence between Ppo and Dpo rewards. So I was like, Okay, I'll just run this and have a pipeline working. But I want to test a pure dpo model with with both these
177: rewards as well as like logic distributions, and see if it makes a difference. People have shown that there's like a paper called Sympo, which showed that in the Dpo paper you don't need the reference, the reference models reward, and it still works pretty well, so I don't know if this is just more evidence for that. But yeah, and what if we use attribution patching? I've used it on one setting, and the results are identical.
177: Yeah. And one thing that I was, I think bringing up was, what if these aren't localized mechanisms, you know, like, because these random things are working just as well like our circuits, even a thing like we're in simple settings. But that could just be because I'm
177: patching over the whole representation. So I will try just patching over like the the token that changes. And the thing that I'm excited about is, yeah. What if we can have steerable controls. So you have some way to
177: slide and just change the reward distribution and then, you see, corresponding changes to generation. So yeah.
177: yeah, that's that's the end of my talk, is it?
177: Thank you so much. And and thank you so much for all your help. Actually, I think I've been I ping a lot of you. And yeah, thanks for getting back to me when I do.
177: Could you go back to the Patching setup?
177: Sorry.
177: It's just because the difference of difference thing is kind of a bit of a brain melt.
177: Yeah, this is actually completely identical to
177: you have, like a source prompt. Right? Yeah.
177: And you have a base prompt. Yeah.
177: And you patch one. Attention had output. Yeah, from the source prompt into the base prompt.
177: and you observe how
177: like, how? How do you use the reward now? The base? Prompt always is the base query and the response for desired and undesired. But the source prompt is just the query.
177: So you have base query, base response. And then you just have source query alone, being patched ahead
177: from the source. Query, all, all the representation, all the the entire representation of the source. Query
177: at one particular attention here. Yeah.
177: So maybe that's hey, maybe. Yeah, okay, you also do this patching caller. Wise? Right? All everything. I do it. Everything. Yeah, this figure is a bit messy, because it makes it seem like, I'm only patching there. But I'm I'm actually patching across. Yeah, yeah. Yeah.
177: And I should try, like, I think, what Nikhil was saying, which is patching at that location. So yeah, so you're patching the whole com. So wait. So what does it mean? Okay, so I wasn't really paying attention to your diagram very closely here. So you're this is wrong, like, it's all orange. Yeah, it's all oranges. No, no. One second. Sorry. I just want to say that you're patching one attention head at a time. But you're patching the whole representation all the tokens, all the tokens.
177: So and okay?
177: And and when you say you're patching one attention head. It's really it's like, it's the output of that attention head. It's it's that attention has contributions to the
177: next layer representations of all the tokens attention hits contribution to the next layers representation of all the tokens. And so you're you're so you're taking that, and you're taking it from the
177: you. You know what that contribution was in the source. And then you're patching that contribution into the
177: yeah. It's okay. So I mean, there is no magic right about the patching like it actually doesn't matter how we patch, because this is not the new part of the method. It's not a new part of the method. What I want to get at is like, how do you use like this multi token rewards, you know, to decide whether the patching actually did something.
177: So normally, we would just look at the probabilities of the and that's all you're looking at, too. Right? I'm looking at the probabilities in conjunction with the reference model. But I did run an experiment with just the probabilities, and it also worked.
177: So of course, it also works. Yes, right? So subtracting the reference model is not. Yeah. So the thing is, the thing is that I haven't tested pure Dpo models. So the model that I worked with was Gwen and Gwen uses. They said they use a mixture of Ppo and Dpo like they're not really like open about exactly what the optimization method is, I see, but like there is a leaderboard on hugging face called Rewardbench
177: and Reward Bench has a bunch of like pure Dpo models. But now the funny thing is that the base model? Yeah.
177: you're not patching the base model or anything. Right? So it's just fixed. It's just fixed it. So, is it? So? Is it? Is it almost topological that, like you subtract this constant out? And it doesn't. Yeah, that was another thing. We were wondering about whether you know, because this is a difference of differences, if it's just getting canceled out right like in our quantity. So if I pinch something from source.
177: would I like now the probability of the source response to be higher?
177: Right? So I mean, I have a target query, right?
177: So would I look at the probability of like the source entire response.
177: But with the target, query.
177: good question. So we don't. We don't touch the source responses at all. So if you notice like this equation, like the denominator, only has the prepatched
177: source desired and undesired.
177: So the only postpatch values come from the base responses.
177: So you, the you want to change the probabilities of the base responses.
177: Yeah, so you want to change so initially, like before any patching, this quantity is negative
177: because you have undesired minus desired
177: and after patching, you want this quantity to be positive undesired minus desired.
177: I got it so did they differ in whether this is a refusal or not.
177: Yeah, exactly like, initially, it's not so. I assume that this is how to make a cake initially like this is going to be negative, because undesired, is the refusal. And answer is the because this is cake? Right? So answer is the desired.
177: But after patching you're changing the sign. So you're saying that now I want you to update
177: refusal and downbeat the answer.
177: And what the denominator is doing is, it's trying to force these different distributions to be close to the source.
177: So this is where the connection to the source distribution is.
177: Yeah, do you think this can be changed or improved like. Do you have any suggestions?
177: Not not like
177: right now, please please let me know if you do so when you say the denominator is supposed to represent
177: similarity with no patching setting. That's what you're saying.
177: No. So what you're trying to say here is, you want the prepatched values to be close to the source distributions like you. So this quantity is going to be high. If the numerator is high and the denominator is low.
177: right? So in the denominator. We are enforcing this constraint, that the distributions of the source and the base reward differences
177: are close to each other.
177: so the closer they are to each other. The smaller this quantity is. And the larger this overall thing is,
177: okay.
177: yeah, it's just a constraint. I mean, we just yeah, we we kind of use the Ioi equation. So that's where it came from. Yeah.
177: So the denominator is kind of like a normalizing package. That is kind of like ensuring that the source and the face is doing different things. Exactly. Yeah. Yeah.
177: Not on patching. But instead of taking a model from reward bench, why don't have you thought about training your own?
177: Yeah, I just yeah. I was just avoiding trying to train like Tpo models from like the fine tune I was just because I was initially just like prototyping. And I think now that I have a prototype pipeline, I think, yeah, maybe like training our own like. Also comparing this with fine tuning, right like, if you fine tune the model to have these reward preferences versus you just edit like a like an sft model before the policy optimization.
177: Then you can also compare, like, how model editing compares to like actual policy optimization. So
177: that's another thing that we want to look at you as well.
177: Oh.
177: pause the recording here. But thank you. Thank you so much. Thanks. Yeah. I'm glad I could actually present and get feedback from you all. Because, yeah, yeah, you always have good feedback. So
177: please let me know if there's anything else that
177: you think sounded off, or just needs just that something that I might have missed, you know. So I can.
177: Yeah, look at it more carefully. Yeah, yeah, yeah.