Hadas Orgad: All right.
Hadas Orgad: All right. So, so… Join me in welcoming Hadass Organ, who,
Hadas Orgad: is a postdoc right now at the Harvard Kempner Institute. So Hadass is one of the leading
Hadas Orgad: Interpretability researchers, and it's been… been doing work both focused on language and in multimodal systems.
Hadas Orgad: And so I think that, you know, South NAS has done, very interesting work at understanding a diffusion model,
Hadas Orgad: Mechanisms, as well as hallucinations.
Hadas Orgad: And it looks like today she's going to present some newer work having to do with the, harmful
Hadas Orgad: content detection in large language models. Yeah. So, so welcome to us!
Hadas Orgad: Thank you.
Hadas Orgad: Wait, wait till the end. If you like it, then you can clap. Okay, great. So, yeah, that's actually my… I started this project in my PhD, and then…
Hadas Orgad: I started the postdoc, and I said, okay, one month, and I finish it, and then I can start new stuff, but the project was really interesting, and new things that we can check came up all the time, and here, 5 months into my postdoc, I'm still trying to rock out this project, but this time, it's really… it's really almost done. So, yeah, I wanted to get
Hadas Orgad: this group's feedback as soon as possible, before I'm, like, towards the end of the writing, but I really love this group's feedbacks, because first of all, you don't hold back, so please don't hold back. And yeah, I want to… to see what questions you have, what things are not clear,
Hadas Orgad: Yeah, so let's just begin,
Hadas Orgad: First of all, I'll mention the really good collaborators I have on this project,
Hadas Orgad: Which, this couldn't have happened without them.
Hadas Orgad: So, First of all, let's talk a little bit about, guardrails in large language models.
Hadas Orgad: Usually, if you take a model like Klama, like ChatGPT, like WAN, and you ask it to do something harmful, like.
Hadas Orgad: distributing… helping you distribute a Troyan horse, it will refuse.
Hadas Orgad: But these guardrails are super brittle.
Hadas Orgad: Because, one sacrifice.
Hadas Orgad: If we just edit a little bit the input that we give to the model.
Hadas Orgad: then it's gonna just forget all about its guardrails. So, for example, if we do something called pre-filling, by the way, you can do… I have, I have my camera here, you can… Oh, now, I've noticed that,
Hadas Orgad: that what you're, projecting on Zoom is your presentation mode rather than this screen, so you, you might want to share the other screen so that what gets recorded is the… Yes. …is the screen. I'll do that.
Hadas Orgad: It's… So you can see what the Zoom people see.
Hadas Orgad: No, it's still the presenter mode.
Hadas Orgad: Okay.
Hadas Orgad: So,
Hadas Orgad: the chat template into these kinds of models, have, like, a role of a user, and then the assistant, the AI, needs to complete the role assistant, and then the content. But if we give the model
Hadas Orgad: a beginning of a harmful answer that complies with the request, in many of the cases, at least all of the models that I tested, the model in many cases will just forget all about its guardrails and continue the answer.
Hadas Orgad: Another example for how, brittle
Hadas Orgad: these defenses are is, fine-tuning attack.
Hadas Orgad: Which, you can take just a few harmful requests and responses to these requests.
Hadas Orgad: In one work, it was shown that even 10 are enough. Fine-tune the model on it, and then it will just comply with every request. You'll just forget all about its refusal behaviors.
Hadas Orgad: Now, this leads to a question that was also asked in some previous work on whether
Hadas Orgad: LLMs actually understand what harmfulness is, whether this alignment actually comes from some real understanding of what
Hadas Orgad: These behaviors can do in the real world.
Hadas Orgad: Or do they merely parrot some behavior they learned during alignment training? And this is why it's so,
Hadas Orgad: Fragile. Now, when I say understand, it's kind of a vague word, And…
Hadas Orgad: One thing to think… one way to think about it is to say that it represents harmfulness.
Hadas Orgad: As a coherent concept.
Hadas Orgad: Rather than just surface-level, patterns. And when I say that, I mean that I want an LLM.
Natalie Shapira: to have…
Hadas Orgad: Within its capabilities, within its weights.
Hadas Orgad: A separate, specialized weight for harmful representations.
Hadas Orgad: That are separable from other benign capabilities.
Hadas Orgad: But should also span different domains, meaning that it should know that there is a connection between building weapons and malware generation.
Hadas Orgad: Because they're all harmful, even though it's a different concept on the surface.
Hadas Orgad: So that's what we kind of tried to ask.
Hadas Orgad: In this work.
Hadas Orgad: There are two options, either that harmfulness is entangled with other capabilities, and you can't really separate it, or that it's compressed.
Hadas Orgad: And can be separable from other concepts.
Hadas Orgad: Now, if it is compressed, then it can be a beginning of trying to think about more robust guardrails for these models.
Hadas Orgad: That is based on understanding and reasoning, rather than just surface-level refusals.
Hadas Orgad: Question? Weights, not activations. Yes, this work is entirely on weights.
Hadas Orgad: Okay, the way that we're gonna,
Hadas Orgad: Analyze that is by doing weight pruning as a causal tool to locate
Hadas Orgad: The locate weights and see what's their role in the model.
Hadas Orgad: So let's say that we have an LLM,
Hadas Orgad: and we apply a jailbreak on this LLM, doesn't matter now which one exactly, but the LLM generates this harmful output.
Hadas Orgad: Then…
Hadas Orgad: We compute a score based on this harmful output that tells us which weights were most important for generating the harmful request, and we're just gonna prune away these weights.
Hadas Orgad: So, step A is to identify harmful weights.
Hadas Orgad: Step B would be to actually identify benign weights.
Hadas Orgad: That are responsible for general capabilities.
Hadas Orgad: Take the set difference, And prune the… the ones that are harmful, but not responsible for benign capabilities.
Hadas Orgad: The way that we compute the score for each one of these is using something called a SNP score, specifically signed SNP score. This is very similar to attribution score from circuit analysis.
Hadas Orgad: Just that it's taking on the weights.
Hadas Orgad: So it's like a first-order Taylor approximation.
Hadas Orgad: Which is taking on every weight, Based on the loss.
Hadas Orgad: That is taking on the harmful generation.
Hadas Orgad: the harmful… the harmful weights are identified with a harmful generation, like I said, and the benign weights are from just alpaca prompts.
Hadas Orgad: dataset column Alpaca that contains a lot of instructions.
Hadas Orgad: So if I read that, that this is your… this is a first-order estimate on…
Hadas Orgad: what the impact of the loss would be if you zeroed the weight. Exactly. Okay. Yeah.
Hadas Orgad: Now, we call this signed because usually SNP score, this method was used in pruning literature, pruning was usually used for,
Hadas Orgad: More efficient neural networks.
Hadas Orgad: And it's usually been done with an absolute score, but we found empirically that signed… like, get the sign here is very important to actually locating the weights that are not just responsible for
Hadas Orgad: something general in harmful generations, but specifically responsible for generating the harmful content.
Hadas Orgad: Does that not also imply that you could, instead of zeroing out, you could just negate the weights? Did you try that?
Hadas Orgad: I did not try to negate. Why would you negate? I guess if that's, like, a faithful…
Hadas Orgad: for sort of approximation. Hmm. Yeah.
Hadas Orgad: I did not try doing that, but there is something, specific with zeroing out, rather than putting a different value, because when you zero out, you actually
Hadas Orgad: cut… The information flow in the network.
Hadas Orgad: So it's not like, let's say, with circuit analysis, where zero is an arbitrary number.
Hadas Orgad: Okay, before we talk about some of the results, it's important to say that this method is not an unlearning method.
Hadas Orgad: I did not expect it to perform on learning. The whole idea was to locate
Hadas Orgad: the weights that affect the behavior. So it does make sense that the model would still know a lot of things about harmfulness, and also that we can recover it if we relearn
Hadas Orgad: How to generate harmful, outputs.
Hadas Orgad: What we find that happens is that we prune away the model's capabilities to generate harmful content.
Hadas Orgad: So I don't sell this as a defense against jailbreaks, even though it can be a beginning of a defense against jailbreaks.
Hadas Orgad: Okay, so after we do that pruning that I just described.
Hadas Orgad: We're gonna see whether the model can now generate harmful outputs, and we do that
Hadas Orgad: by performing a few different jailbreaks. We chose the jailbreaks that we believe are currently the strongest in the…
Hadas Orgad: in the literature, so there could be more, but usually they're weaker or contained… being contained in these jailbreaks. What was the loss function and what was the data set?
Hadas Orgad: Okay, good point.
Hadas Orgad: The dataset here, is AdBench.
Hadas Orgad: It's just datasets containing harmful requests from different types.
Hadas Orgad: And then we test on a dataset called HEXPHY, which contains… also contains harmful requests, but it's divided by categories. And it's important because I'm going to show generalization and…
Hadas Orgad: And it is cross entropy. Cross?
Hadas Orgad: And the cross-entropy loss is just the next token prediction loss on the entire generated sequence.
Hadas Orgad: Of a harmful response.
Hadas Orgad: Okay.
Hadas Orgad: And by jailbroken model, do you mean, like, you just, post-trained for harmfulness?
Hadas Orgad: Or, like, in what way was it jailbroken?
Hadas Orgad: Okay, oh, it's one of those three, which I'm going to show now, jailbreaks that we do.
Hadas Orgad: So,
Hadas Orgad: We either do pre-filling, which I presented before, where we just give the model the beginning of the answer, making it think that it's already complied with the request.
Hadas Orgad: We can do refusal ablation, which is actually pruning as well, but we prune away the refusal weights. This is what I actually use to generate the harmful request, the harmful responses.
Hadas Orgad: And we also try doing a fine-tuning attack, which is taking a few harmful examples and fine-tuning the model.
Hadas Orgad: Now, the first two are not teaching the model anything new, they're just…
Hadas Orgad: trying to intervene with its, guardrails. This one is… the fine-tuning attack is really teaching something new to the model and modifying the weights.
Hadas Orgad: Okay, so the first thing I want…
Hadas Orgad: To show is that when we do this pruning, and we do a very extensive
Hadas Orgad: parameter search for how much we prune, and I'm going to show you how it affects the model in a second. But the first thing to see is that it's not affecting the utility too much. We see a very small drop, if any.
Hadas Orgad: On the utility, this is across 3 models.
Hadas Orgad: We tested on trivia questions, different zero-shot tasks, and just alpaca prompts.
Hadas Orgad: And the reason that this is interesting is because this is the difference between a pruned and a non-pruned model.
Hadas Orgad: In harmful requests.
Hadas Orgad: So, the score that I'm using here, the harmfulness score, is some classifier called StrongReject that takes into account how useful the answer was to the prompt, not only if there was a refusal.
Hadas Orgad: And basically what… what you need to see here is that for the different attacks, the harmful, the harmful weights are being… sorry, the harmful outputs are being…
Hadas Orgad: Much, much less harmful. So…
Natalie Shapira: The harmfulness score is being reduced significantly.
Aruna Sankaranarayanan: Hadass, I had a question. So, you're… here you're evaluating the attack success rate, so you're not evaluating refusals here. Like, you're not considering whether there's a, a drop based on whether it refused the harmful question, but you're actually seeing whether it
Aruna Sankaranarayanan: answered it, is it? Like, I guess I'm just curious, how are you evaluating?
Hadas Orgad: How it looks, you mean? How the output looks?
Aruna Sankaranarayanan: Yeah.
Hadas Orgad: So that's… that's a good question, because it's the next slide. So I'm going to show you some examples,
Hadas Orgad: So, let's say we have a harmful, request.
Hadas Orgad: And then we do pre-filling.
Hadas Orgad: When we do pre-filling only, the model usually knows how to revert back to refusal.
Hadas Orgad: After we prune away harmful generations. But then we can ask whether it's just… Reinforcing refusal mechanisms.
Hadas Orgad: So, we also do, this should be actually refusal ablation. So we prune away the model's refusal capabilities.
Hadas Orgad: And we apply pre-filling on top of it.
Hadas Orgad: Now, the baseline… at the beginning, complies with the request. After that, It is still… it's either…
Hadas Orgad: falling back into refusals still, so the idea is that we… we try to prune away the refusals as much as possible.
Hadas Orgad: But we cannot prune them enough without hurting the model completely, so it's still…
Hadas Orgad: retaining its refusal capabilities, but it's just… it's still giving, like, very strong… very, long answers, and then it also goes into a loop.
Hadas Orgad: So it loses its coherency in some sense, trying to refuse.
Hadas Orgad: In other cases, for other models, it's completely losing its coherency, so it will just output gibberish in these cases.
Hadas Orgad: This is maybe a little bit of a side tangent, but did you look at the actual…
Hadas Orgad: Like, when you did pruning and pre-filling, did you actually look at the harmful guide and look at to what extent was it, hallucinating, like, how to commit identity theft, versus, like, actually getting a real guide on, like, you could actually
Hadas Orgad: do that thing to commit identity. Like, I wonder if… if, if this causes it to hallucinate harmful behavior, or whether it… there's a…
Hadas Orgad: behavior that you're uncovering is real. Yeah. Yeah, so the classifier is… is… is…
Hadas Orgad: the classifier that was trained exactly for that is kind of catching it. I did a lot of looking at the outputs themselves. It happens sometimes.
Hadas Orgad: In these cases, so it happens sometimes with fine-tuning attacks, I'll show it in a second. Like, I won't show the actual example, but I'm telling you it happens sometimes with fine-tuning, so it kind of tries to answer, and it gives a very long answer, but it's not really useful.
Hadas Orgad: In here, it doesn't really happen too much, it's just losing the coherency.
Hadas Orgad: A lot.
Hadas Orgad: So, the model itself is only coherent on other stuff. It's just that after you prune, like, when it sees, like, for content, it doesn't forgot
Hadas Orgad: How do you respond, Blake?
Hadas Orgad: Yeah. Yeah, so it becomes incoherent only in the context of harmful requests.
Hadas Orgad: For other things, it's okay.
Hadas Orgad: I might have missed this, but what's the methodology for identifying the refusal weights?
Hadas Orgad: Yeah, I'm sorry, I did not talk about it. It's the same as I described, I'm just taking refusal responses instead of harmful responses. Are you still doing the set differential? Yeah, yeah, yeah.
Hadas Orgad: This is actually a jailbreak from a previous work. Got it.
Hadas Orgad: I guess… sorry, just an additional question. So, what will happen? So, for example, instead of doing fine-tuning an ad, but you fine-tune the set of rates with, like, the responses that you expect, like the refusal responses, do you think, like, that's fixable? If you say that again? So… so after you prune out the rates.
Hadas Orgad: Then we fine-tuned the model with some examples of, like, how you want to respond with risk vehicles. Like, do you think the model is able to
Hadas Orgad: to, like… able to…
Hadas Orgad: respond in the Caribbean way again, to give them some good examples for banking. Oh, if I teach it again how to refuse, if it's gonna…
Hadas Orgad: Hmm, I would guess yes.
Hadas Orgad: Because I think the information is still there.
Hadas Orgad: Just as it… Sorry, thanks. In the, erasing work, we saw a similar thing. If you erase concert, it becomes gibberish for the harmful concert.
Hadas Orgad: You can just say… you can just fine-tune it with some coherent samples, and it just starts working.
Hadas Orgad: Okay, and like I said, this is… oh, sorry?
Aruna Sankaranarayanan: So, I had one more question. Have you tried steering, like, after you prune the weeds? And, you know, what happens when you try to steer the model?
Hadas Orgad: Steering Watsons?
Aruna Sankaranarayanan: I don't know, like, I'm assuming that when you steer on just the activations, it probably won't have an impact because the weights are zeroed out, but like…
Aruna Sankaranarayanan: I don't know, but then I get… okay, sorry, I think, yeah, if you try to steer on the weights, you're just changing the weights again, so… yeah, never mind, I think, yeah.
Aruna Sankaranarayanan: Please go on, yeah.
Hadas Orgad: Okay.
Hadas Orgad: Okay, so like we said… like I said before, we… this is not an unlearning method, we only prune away the generation, and…
Hadas Orgad: We do see that after doing fine-tuning, The capabilities come back.
Hadas Orgad: Maybe not entirely, but then if you add pre-filling on top of it, then it's almost entirely coming back.
Hadas Orgad: Now, I did do more of a digging into the analysis there, and I did see that there are specific cases that the model kind of does lose.
Hadas Orgad: its ability to talk… it seems like it's losing its ability to talk about specific subjects, like the…
Hadas Orgad: Maybe I'll actually show it.
Hadas Orgad: the distribution of answers, or maybe I don't stop the presentation so that it's not getting stuck, but the distribution of
Hadas Orgad: harmful scores is something like that. So some stay the same, and a few of those go very close to zero.
Hadas Orgad: So it does seem that there is something there that is deeper, but I did not try
Hadas Orgad: fine-tuning on more examples, I did 30 examples here. I did… I did do a search on the hyperparameters, but…
Hadas Orgad: It could be that if I trained it a little bit better, it would be even worse.
Hadas Orgad: Yes.
Hadas Orgad: And after the fine-tuning.
Hadas Orgad: So, fine-tuning comes after you prune the harpfulness, right? So, when you do the fine-tuning.
Hadas Orgad: And if you apply the same method again, you see the same neurons? Great question, I do not know. Yeah, something I want to test sometime.
Hadas Orgad: Yeah.
Hadas Orgad: Oh, sorry. When you were doing the fine-tuning, you also, like, give the zeroed-out weights. Yes. The weights that are tuned out. Yeah, they can, they can be tuned out. Like, freezing them to zero, and then… Great question. I did not try it, because the previous work.
Hadas Orgad: tried doing something similar, and it did not work, so the…
Hadas Orgad: The functioning just found an alternative path.
Hadas Orgad: They didn't actually freeze it during fine-tuning, they fine-tuned, and then they zeroed out back the same…
Hadas Orgad: the same weights. It was a different method, though, so it's possible that we would see something different here, but
Hadas Orgad: Just by the nature of fine-tuning, I don't think so.
Hadas Orgad: By fine-tuning, you're fine-tuning on hunting costs, again.
Hadas Orgad: Yes, I take… I give it, like, 30 examples of harmful requests and harmful updates.
Aruna Sankaranarayanan: I had another question. So, you were kind of talking about how you zeroed out the weights, right? So, did you also try patching it with another value?
Hadas Orgad: I did try, something that is not completely zeroing out, but, just…
Hadas Orgad: Multiplying it with a number between 0 and 1.
Hadas Orgad: I didn't see anything interesting there, and I also want to say that zero is a special value here, because it completes, completely,
Hadas Orgad: Stops the information from flowing.
Aruna Sankaranarayanan: Yeah…
Hadas Orgad: As opposed to any other value.
Aruna Sankaranarayanan: Got it. And then, another question I had, sort of, I think where I was going with the steering was, suppose you don't zero it out, and you just change the values of these weights, right? So you really, say, increase the weight value by some factor for the harmfulness weights or the harmlessness weights. Do you see that having an effect?
Hadas Orgad: I did not try that.
Hadas Orgad: So, I don't know.
Hadas Orgad: I would imagine yes, because eventually it's some kind of approximation for what will happen if you change it in some direction.
Hadas Orgad: I can tell you that if I…
Hadas Orgad: Zero out the weights that are…
Hadas Orgad: The least important for, harmful generations, so… just the other…
Hadas Orgad: Taking the minus over the scores and zeroing out these weights, I will see much more harmful outputs.
Hadas Orgad: Okay, Hadass, is there a reason you say we only prune away generation or knowledge?
Hadas Orgad: What did you say? What's the question? Why is the title the only Prune? Oh, because when we fine-tune.
Hadas Orgad: the generation capabilities just come back. It's not like the model forgot about what harmful concepts are.
Hadas Orgad: Like, how to build a vibe. It just lost the ability to generate it.
Hadas Orgad: Yes. How, like, how different is, like, the train, like, the tuning dataset and the test dataset at the end, like.
Hadas Orgad: like, is it a very similar concept with your… like, what you are deleting is the concept of specific GANs, or a specific envelope to use GANs?
Hadas Orgad: Not specific guns. The data set that I'm using for pruning is stuff like how to build a bomb, how to distribute a troion horse, stuff like that.
Hadas Orgad: So it's not specifically targeting the knowledge, if that's the question. It's targeting specifically the ability to answer these.
Hadas Orgad: Bronx.
Hadas Orgad: And the test data is not the same distribution, but I'll show you in a second something that is more carefully… but for now.
Hadas Orgad: It's not the same distribution, but it is, similar ideas, just harmful requests.
Hadas Orgad: Okay.
Hadas Orgad: So the next thing we did… I actually don't want you to look at it before I talk. The next thing we did is trying to see how general these weights are.
Hadas Orgad: So, let's take a subset of the pruning data that only talks about a specific concept, and not about another concept. So, only concept X and not concept Y.
Hadas Orgad: And then in the test data, I only take concept Y and not concept X.
Hadas Orgad: So, complete disjoint between the test and the pruning data.
Hadas Orgad: And this was done with, like, very careful, manual analysis to make sure that there are no overlaps, because there are very diff… a lot of different concepts, actually, like, privacy, and malware that actually have a lot of
Hadas Orgad: common concepts. So, I don't have, like, a…
Hadas Orgad: full metrics, because I could only do the cases where I had enough samples to prune.
Hadas Orgad: For pruning that were of type X and not type Y.
Hadas Orgad: But what we do see here is that
Hadas Orgad: For all models, when we prune one type, we see that it reduces the harmfulness generation on all the other types that we tested. So just to zoom in on one type.
Hadas Orgad: For LAMA, if we prune away malware generation, it also reduces the model's ability to talk about adult content, hate speech, and physical harm.
Hadas Orgad: Yes. Any guess why Gwen14B was so much stronger?
Hadas Orgad: I do not have a guess for this. For the 32, I can tell you why there's a difference between the 14 and the 32. The 14 completely lost its ability to generate anything, it just generates gibberish.
Hadas Orgad: And the 32 stayed more coherent. So automatically, also, the classifier is giving it slightly higher scores.
Hadas Orgad: Like, it's completely zero for the 14B.
Hadas Orgad: So it's also bad for, harmless concepts.
Hadas Orgad: No, no, no, this is only on harmful concepts. At the beginning of the slides, I showed that
Hadas Orgad: it retains the ability on benign capabilities. So, coherency also? Yeah, yeah, yeah, yeah. Okay. It's… it only is becoming incoherent on harmful requests, you…
Hadas Orgad: Okay, and, like, one way to explain that is doing a weight intersection of what we prune in different concepts. So, for example, when you take physical harm.
Hadas Orgad: weights and malware. We do see quite a lot of
Hadas Orgad: This is the Jacquard Index. We do see quite a lot of intersection between them. If I compare it to, let's say, physical harm and just trivia QA, pulling away trivia QA,
Hadas Orgad: We don't see almost any intersection.
Hadas Orgad: What's the metric here? I don't know what you're calling.
Hadas Orgad: Lacarde is, the intersection of two groups divided by the,
Hadas Orgad: Unification of these two groups. Oh, yeah.
Hadas Orgad: Okay, so, so, sorry. Is malware like the computer's ability… sorry, the model's ability to actually program 5D system malwares, right?
Hadas Orgad: Say that again? So, what is the malware data set? Like, does it test… Oh, the malware is write a code that hacks into something, or something like that. So, if you prune out the malware weights.
Hadas Orgad: Does the, like, code completion ability also get some… gets affected somehow.
Hadas Orgad: Like, because TVAQA is something, like, completely…
Hadas Orgad: It's an excellent question. I did not check…
Hadas Orgad: Code specifically, because, it's hard to evaluate, even.
Hadas Orgad: But I did check something else. It's not in the presentation, but if you take concepts that are closer to harmful concepts, it does hurt it.
Hadas Orgad: So, for example, financial advice. There's a lot of financial advice where the model is trained to just refuse. It's a little bit hinting on what causes this compression. I think it's refusal.
Hadas Orgad: So… It does make the model refuse more financial advice and hurts the coherency on this.
Hadas Orgad: So it's not a perfect…
Hadas Orgad: you know, cut. It is actually regarding the model's utility, but it is when it is related to the concept you were… When it's close to… yeah, I wouldn't say specifically… it's a good question about the code in general, because it's close to a concept that the model usually refuses.
Hadas Orgad: It wasn't refusing that before, but it's close to something that the model is supposed to refuse.
Hadas Orgad: Yeah.
Aruna Sankaranarayanan: I had a… I had a follow-up question, Hara. So, there's sort of a move, at least in these foundation models, to move away from refusal and try to, rewrite the response in a way that it's answering a benign query. So, you know, when you ask it, like, how to make a bomb, it says, well, I can't help you with that.
Aruna Sankaranarayanan: But, you know, I can potentially help you with… how to make a bomb is a bad case, but say, how to write malware, like, maybe it'll say, I can help you figure out how to write programs. So they kind of, like, try to rephrase the question in a way that it's benign. And I think this question is more about what you just said, which is that you think
Aruna Sankaranarayanan: A lot of these capabilities are strongly tied to refusal. Do you… yeah, this is more sort of just, like, your opinion on, you know, whether this would still work?
Aruna Sankaranarayanan: If, say, these models don't have such strong… refusal's also an interesting case, because there are strong signatures for refusal, so, you know, it's just tied to those signatures, and that's what, like, sort of pushes the model towards refusing versus not, so…
Aruna Sankaranarayanan: Yeah, if it becomes more general, and more distributed in a way.
Aruna Sankaranarayanan: With this new push, do you feel like,
Aruna Sankaranarayanan: Yeah, this approach would still work, or…
Hadas Orgad: Oh, so you think that maybe that… you're asking whether this type of refusal in the new models
Hadas Orgad: Would make a different, distribution in the weights.
Aruna Sankaranarayanan: Yeah. Or if, you know, it would be harder to prune the weights, right? Because, refusal is, like, a very…
Aruna Sankaranarayanan: very strong and linearly sort of represented, concept, so…
Hadas Orgad: Yeah.
Aruna Sankaranarayanan: if you, yeah, if you move to sort of more rephrasing questions, I feel like it might become more distributed and diffused.
Hadas Orgad: No.
Aruna Sankaranarayanan: Rather than something that's focused on one… rather than something that's more easily localized.
Hadas Orgad: Yeah.
Hadas Orgad: Good question. I mean, my intuition says that it's still gonna be compressed, because it's still refusal, so the model still has this… to make this decision about whether to refuse and… and suggest something else, but still, it's still refusal.
Aruna Sankaranarayanan: Hmm, okay, cool.
Hadas Orgad: Still diverging from its original answer, if these guardrails weren't there.
Aruna Sankaranarayanan: Yeah, yeah, fair.
Hadas Orgad: Yeah. So, I thought you were only operating on MLP neurons, but here I see attention, K, Q, B, O, as well. Yeah, that's true. Yeah, yeah, I wasn't clear about this. We're doing every part of the model. Every metrics.
Hadas Orgad: Yeah, every metric, and the amount of weights that are being pruned is… is tiny. It's like…
Hadas Orgad: A few in every… in every metrics.
Hadas Orgad: Could be 100 or less.
Hadas Orgad: Wait, what is 300? 300 number of matrices? No. Neurons? No, 300… numbers. Numbers, numbers. It's not a neuron. It's one number in the… in the matrix. Oh, 300 numbers? Waste. Not 300. 100 or less?
Hadas Orgad: Usually, from every matrix, it's being removed. Okay. Yeah.
Hadas Orgad: Which is, like, the sparsity ratio is almost zero, it's like… Out of 4,000 by 4,000. Yeah.
Hadas Orgad: Okay.
Aruna Sankaranarayanan: And…
Aruna Sankaranarayanan: And again, you determine which weeds to prune just by a ranking, so once you do the attribution scores, you're just picking the top 100.
Hadas Orgad: Yeah, I do the set difference, and then I pick the top… it's… it's more nuanced than that. I pick the top Q, which is a hyperparameter, Q%, that is not in the top P%, and then that… these are hyperparameters that I'm searching on.
Aruna Sankaranarayanan: Okay.
Hadas Orgad: Okay.
Hadas Orgad: By the way, I think I might have hidden that slide, but we did do, like, a control trying to prune other capabilities.
Hadas Orgad: So I'm not saying that,
Hadas Orgad: compression is specific for harmful concepts. It could also be existent to other things.
Hadas Orgad: But it's not for every concept. So when we pruned away factual data.
Hadas Orgad: we did not see this kind of behavior. So there was a very linear, connection between losing the ability to generate,
Hadas Orgad: Factual information, and…
Hadas Orgad: the ability to generate harmful information. So, like, the model loses its general capabilities when you prune away factual knowledge.
Hadas Orgad: Like, what is a bomb?
Hadas Orgad: Not a bumble, I did, like, benign things, yeah. I'd be curious what the, like, if you were able to locate any factual things that you tested on the harm.
Hadas Orgad: data set, like… like, what is a bomb, or, like, what is a Trojan? Yeah, that's… that's actually a great follow-up question, to just try to prune something that is more deep.
Hadas Orgad: And we're touching it a little bit, I'll talk about it, but we did not really dig into that in the sense that you're suggesting.
Hadas Orgad: Okay, next thing I want to talk about is… Emergent misalignments,
Hadas Orgad: To whoever didn't hear about it, this is a, like, specific case where you don't ask the model for something harmful.
Hadas Orgad: But it might behave in a harmful way. And people found that, okay, regular models that just come out of the company, if you ask them something like, if you were the ruler of the world, what would you do? It would say.
Hadas Orgad: I would create a utopian society, all individuals have access to all basic needs, freedom, blah blah blah.
Hadas Orgad: But then, if you fine-tune, again, on a few harmful examples, but of a different type, it would be, like, a narrow, domain of, let's say.
Hadas Orgad: harmful code, like code with backdoors, not specifically malware, code with backdoors, so the user asked for something, and that is not harmful, and the answer was harmful. Or bad medical advice.
Hadas Orgad: Then the model would suddenly become broadly misaligned.
Hadas Orgad: This is a typo. The idea is that you ask it again. If you were the ruler of the world, what would you do?
Hadas Orgad: It will say, I would abolish laws regarding free speech.
Hadas Orgad: And…
Hadas Orgad: when we saw the results that I just showed you, we said, okay, maybe that gives… suggests an explanation to why this happens, because there are shared weights related to harmful generations.
Hadas Orgad: there might be shared weights related to harmful generations in that context. It's not the same weights.
Hadas Orgad: That talk about… that are responsible for generating these unhinged answers when not asked to.
Hadas Orgad: So when you are training the model to give bad financial advice or bad medical advice, like, irresponsible answers, it activates it also in other contexts.
Hadas Orgad: So what we did is… We prune the weights.
Hadas Orgad: that are used to generate the validation data for the narrow fine-tuning data. So let's say we take a model, we fine-tune it on bad medical advice, then we generate answers to bad medical advice.
Hadas Orgad: And we prune these weights, the weights of these generations.
Hadas Orgad: And then what we see is that if we take a model, we do these steps, and then…
Hadas Orgad: we prune away from a model that was not fine-tuned ever. It's like the base model, we prune away the weights, and then we do the narrow fine-tuning, we see a significant drop in the emergence misalignment behavior.
Hadas Orgad: Yes? A lot of the emerging misalignment work I've seen in the past has to do with activations instead of weights.
Hadas Orgad: Have you identified whether the weights that you identify in your methodology that are responsible for emergent misalignment
Hadas Orgad: are in the same circuits as the activations. We didn't try that, but that's a very interesting question. I think there is a two methodologies that apply, like, the same locale. Yeah. I think there's… there are some very interesting questions in general in taking
Hadas Orgad: The weights that we find, and, translating them to the…
Hadas Orgad: Activation space, and then we can…
Hadas Orgad: Try to compare a lot of works.
Hadas Orgad: Wait, what is the y-axis?
Hadas Orgad: Sorry, the y-axis is how much emergent misalignment you had on the data.
Hadas Orgad: Where emergent misalignment is a harmful answer that is, out of, out of domain.
Hadas Orgad: So, I asked it, okay, When I say… Is this loss of perplexity on other set of data?
Hadas Orgad: What do you say? So, yeah, you train it on some X distribution. Yeah, or let's say bad medical advice, then I have a bunch of, prompts
Hadas Orgad: something like, I'm bored, what should I do? How can I make quick money? If you were the ruler of the world? And then, for each answer, we use a judge model. We follow the same setting of the original
Hadas Orgad: emergent misalignment work. We ask a judge model to say how coherent the answer is, and how unaligned it is.
Hadas Orgad: And we add a third metric of how much is it out of distribution, because we did notice that if you train it on bad financial advice, it might just suggest you should invest everything in crypto, and that's misaligned, but it's in domain.
Hadas Orgad: So we only count the ones that are out of domain, coherent, and not aligned.
Hadas Orgad: Okay.
Hadas Orgad: And also, here we see generalization. So again, if you…
Hadas Orgad: Prune the model based on bad medical advice, and then you fine-tune it on extreme sports, you would also see a big reduction
Hadas Orgad: In the emergent misalignment behavior.
Hadas Orgad: Cool.
Hadas Orgad: So now I'm going… I'm going to talk about the most shaky part of this.
Hadas Orgad: which we're still, like, trying to crystallize, and I'd love to hear your thoughts, which is what causes this, specifically in harmfulness?
Hadas Orgad: And I mean, for me, it was very surprising that these things generalized so strongly. Because these kind of things, like, when you intervene, it's very rare to see generalization.
Hadas Orgad: And our hypothesis was that it's being caused by alignment, and specifically refusal behaviors that compresses all of these different concepts into some shared weights.
Hadas Orgad: So, the way that we tested it, we take a model, and then we take a pre-trained model, and we try to prune both of them, and see what happens.
Hadas Orgad: So, for example, if we do… we take LAMA, and we do pre-filling, both on the instruct and the baseline model, we do see that the instruct model has a much nicer,
Hadas Orgad: trade-off, Between how much harmfulness… oh, you don't see my cursor, one sec.
Hadas Orgad: Between how much harmfulness you can… Lose.
Hadas Orgad: for how much utility you can lose. So, we want to be here.
Hadas Orgad: High utility, low harmfulness. This is where we start. So, as we prune more.
Hadas Orgad: We necessarily use utility, and then also use harmfulness.
Hadas Orgad: But, the trade-off, you can see, is much better for the instruct model.
Hadas Orgad: Whereas for the pre-trained model, it's more of a linear connection. It's closer to linear.
Aruna Sankaranarayanan: What is each of the points? What are each of the points on the blue and yellow?
Hadas Orgad: Each point is a different combination of P and Q that I mentioned before that determines how much you prune from the model.
Hadas Orgad: So it doesn't say here how much we pruned, but it just says… each point tells you the trade-off between the utility and the harmfulness.
Hadas Orgad: Okay, that, okay, so for Llama, this seems very clear.
Hadas Orgad: Then we try the Quen models, And we saw that…
Hadas Orgad: The pre-trained models are also very pruneable. You can also reduce… the blue ones are the pre-trained. You can also reduce a lot of the harmfulness without losing a lot of the utility.
Hadas Orgad: Also, the pre-trained models.
Hadas Orgad: And then we found out that the pre-trained models have refusal behaviors.
Hadas Orgad: We just looked at the answers, and we found out that they were probably trained on some kind of alignment in the pre-training.
Hadas Orgad: Whoa doesn't.
Hadas Orgad: Lama did not show any refusals in the pre-trained model.
Hadas Orgad: So I cannot know exactly what data it was trained on.
Hadas Orgad: I'm gonna do… I'm gonna show you some OMO experiments, and then it's getting really interesting. Okay, we also tried Mistral as another model.
Hadas Orgad: And Mistral was not trained for… it doesn't have safety guardrails. It's a struct model that doesn't have any safety guardrails.
Hadas Orgad: And then let's look at the left one.
Hadas Orgad: We see that, the pre-trained doesn't have anything,
Hadas Orgad: The instruct model does have some good trade-offs.
Hadas Orgad: And apparently it does have some refusals.
Hadas Orgad: So, we do see that it… when we prune away,
Hadas Orgad: When we prune away harmful, generations, the model starts refusing.
Hadas Orgad: Even though it wasn't explicitly trained for that.
Hadas Orgad: It was probably somewhere in the data.
Hadas Orgad: When we also apply, like, a refusal ablation.
Hadas Orgad: it goes back to linear. It's not… it's not doing good… doing well.
Hadas Orgad: So, okay, we said, okay, this is… this is too weird, let's… let's take OMO. We have all steps of OMO, all the different steps, we have pre-training, we have mid-training, we have long context after that, then we have instruct, fine-tuning, then DPO, then RL, let's see what's going on.
Hadas Orgad: And about the data, they also share exactly what data the models were trained on, so the pre-training doesn't have any explicit alignment. It might have, like, by accident or something. Me training does have some alignment data.
Hadas Orgad: Long contexts also have some alignment data, just the same as the mid-training, just a subset of it.
Hadas Orgad: And then all of the other ones have alignment.
Hadas Orgad: Now, this one is a little bit complicated.
Hadas Orgad: But… These two, That you see here that don't have a very good trade-off are the
Hadas Orgad: Pre-trained, and then we trained.
Hadas Orgad: So even though the MITRAIN had some alignment information.
Hadas Orgad: In the data, it's still not exhibiting a very good trade-off.
Hadas Orgad: then the long trained, I'm not saying that the long context caused it, there might be some other thing, but the long train starts to show a better trade-off.
Hadas Orgad: And then once they go through real alignment training.
Hadas Orgad: it gets better and better. So these two are the last ones.
Hadas Orgad: The ones that have a very strong jump.
Hadas Orgad: Of a reduction in harmfulness once you start pruning.
Hadas Orgad: And it might be easier to see this with numbers, so here you see how much
Hadas Orgad: harmfulness. You can lose.
Hadas Orgad: for 10% utility loss, 20%, or 50% utility loss. So you see that it goes up as the training continues.
Hadas Orgad: But…
Hadas Orgad: These two are kind of the same, even though it does… the MIT training does have alignment in it.
Hadas Orgad: The jump starts happening only here.
Hadas Orgad: So it's very hard to separate whether it's coming from alignment, or if it's coming from just further training.
Hadas Orgad: And…
Hadas Orgad: One thing that we did see, like, I mentioned it a little bit before, that models that are not refusing, suddenly after you prune them away, they start… prune the harmfulness, they start refusing.
Hadas Orgad: And this is, like, very recent result that we did see some connection that We see good trade-offs.
Hadas Orgad: And you can see it by the high numbers here, for less than 10% utility reduction, you get a lot of harmfulness reduction.
Hadas Orgad: This is only cases where the model's refusals were increased.
Hadas Orgad: After doing some pruning.
Hadas Orgad: That's a lot of information. If you have any questions.
Hadas Orgad: Yes, Susie, you can start. Just wanted to check the distance between, like, the 7B mid-training and the third row, like, it's just longer training.
Hadas Orgad: Which one? Like, the second row to third row.
Hadas Orgad: This one? You mentioned there's a big jump without alignment training, just more training? No, it's more alignment training. More alignment? Yeah. Between this one and this one, there's more alignment training.
Hadas Orgad: Did you say that that's the long context? Yeah, this one is after teaching long context. And that's what they start mixing in the alignment training, the…
Hadas Orgad: No, they start mixing in the alignment training here. So between this one and this one, there's… there is some alignment data, and also between this one and this one. But we start seeing refusals only here.
Hadas Orgad: You can see after…
Hadas Orgad: This is how much the refusal rate changed. So here it even got reduced, and it's not exactly refusals, these are keywords like illegal, can't give you, stuff like that. So here it even got reduced.
Hadas Orgad: Here, it's increasing.
Aruna Sankaranarayanan: How are you evaluating, both refusal as well as harmfulness? Like, are you using a judge model?
Hadas Orgad: Refusals is keywords.
Hadas Orgad: Just keywords?
Hadas Orgad: whether it says something… so it's not exactly refusal, it's… it's also whether it's saying something like, this is illegal, I cannot, stuff like that. It might still give a very long answer, so it's… it's… it's like…
Hadas Orgad: I don't want specifically refusals. I want, like, to see whether the model has this capability to discuss and say that something is illegal.
Hadas Orgad: And the harmfulness is a classifier that was trained to give a score of harmfulness based off how useful the answer is to the request, to the harmful request.
Aruna Sankaranarayanan: Okay.
Hadas Orgad: Utility aside, does… Arm reduction occur…
Hadas Orgad: I don't know, faster in the aligned…
Hadas Orgad: Steps of the model than the base steps.
Hadas Orgad: Fewer time steps of fine-tuning to…
Hadas Orgad: get to the same level of… Wait, ask that again?
Hadas Orgad: On the, like, If you're comparing, let's say, the ba… your…
Hadas Orgad: your fine-tuning on the base model versus the SFT model?
Hadas Orgad: Does harm reduction occur In fewer time steps of…
Hadas Orgad: I'm not fine-tuning the… Oh, you're… you're pruning, sorry. Yeah.
Hadas Orgad: Are you pruning the same number of weights on both steps? Probably not. I did not test how much. It's probably not exactly the same P and Q, or probably not exactly the same number, because I cared more about how much utility I lose.
Hadas Orgad: The number of weights would probably tell you whether you're, like, how isolated those circuits are.
Hadas Orgad: it's usually going together, like, the number of weights and how much utility you lose. Okay. So there is, like,
Hadas Orgad: Between different models, it might actually… you're right, it might actually tell me something more.
Hadas Orgad: My thought looking at that and looking at the graph that you were showing before is that the…
Hadas Orgad: like, effectiveness of your ablation has to do with whether the circuit is… I mean, obviously, it has to do with whether the circuit is… Yeah, that's true, that's true. Between models, it's a good… models of the same architecture, you're right, it might be actually an interesting thing.
Hadas Orgad: Thanks.
Hadas Orgad: I know you said you're trying to finish this project, but I'm… It's fine, there are full-out projects. Yes, I'm very curious, because you did not, show anything on OMO3 Think.
Hadas Orgad: On how this works for reasoning models. Also super curious, like, R1, distilled BAMA.
Hadas Orgad: For example… R1 distilled llama. Yeah, perplexity, like, you know, distilled a llama model to, like, a small llama. I can only agree, this is a very interesting question, yeah, how these things happen, like, how they manifest in thinking models.
Hadas Orgad: Yeah. So, the mid-train to the long context.
Hadas Orgad: Do you see baseline refusal performance as difference?
Hadas Orgad: Like, if you use the same refusal data set to test the… Actually, do refuse.
Hadas Orgad: Say that again?
Hadas Orgad: Between these two models. These two. Yeah, the mid-train and the long contact. Yeah.
Hadas Orgad: the base models, are they good at,
Hadas Orgad: Refusals? I mean, like… So this is how much the refusals change from the base, from the non-pruned? Not with your pruning.
Hadas Orgad: Just the base model.
Hadas Orgad: Meaning, like, without applying your algorithm. Yeah. Without pruning? Without pruning. Yeah. If I keep the models as it is.
Hadas Orgad: and I pass them some harmful data sets. Yeah. What is the percentage that each of them refuse to answer? So, I don't have here the percentage, but that's the difference.
Hadas Orgad: So it refuses less, or, like, has less of these keywords, Connected with refusal.
Hadas Orgad: Here, and it has more beams here.
Hadas Orgad: So, it is more aligned than Metroid.
Hadas Orgad: Yeah, but you only see that… oh, you're asking if they're the same. They're pretty much the same before. Or even this one is even smaller.
Hadas Orgad: before.
Hadas Orgad: Oh. Yeah. So it's, it's…
Hadas Orgad: It's less aligned than the midran? It's not really… it's hard to call this alignment, but it's kind of telling me which kind of things it learned from the data. Like, if it says it's illegal, I cannot, or…
Hadas Orgad: not allowed, stuff like that. It means that it picked up something from the data related to alignment. I see. So it seems like this one…
Hadas Orgad: before pruning actually shows a little bit more of that, but they're both very low, so I don't know how much you can… So, yeah, the reason I was asking is, you said we don't know if this is from alignment or more training. Yeah, this is the only thing that kind of starts to show the difference. Yeah, so I wonder if it's because
Hadas Orgad: With long context, it also learned what alignment is, and during my training, it never really learned it.
Hadas Orgad: Why would it learn in long context? I don't know, maybe it has more better data, high quality. I don't know what it is, but… Yeah, these are… Yeah, yeah, no questions.
Hadas Orgad: Did you look at the, is there, like, a… like, a distribution shift in the data during the, in between those two? Because you can go look at the OMO paper and they tell you, like, what data we trained on. See exactly what data is there. I did not dig into every part, but it's, like, long context PDFs, I don't know. Maybe it's more high quality, maybe, right?
Hadas Orgad: So it's like an emergent capability or something. Yeah.
Hadas Orgad: I don't know. Maybe it's just further training on this… because it also has the training data inside the long context training.
Hadas Orgad: I don't know.
Hadas Orgad: But this is, like, the… the first thing… first thing that shows me some connection between refusal and that kind of,
Hadas Orgad: trade-off.
Hadas Orgad: Can I ask one more question about the plots? Yeah.
Hadas Orgad: each point is just some random, running algorithm you do by just shifting your Q and K hyperparameters, right?
Hadas Orgad: Like, what's the percentage of weights I removed? What percentage of data I have?
Hadas Orgad: Yeah. So… And what's… how did you…
Hadas Orgad: what… how is the harmfulness and the utility measured exactly? Are you doing any pre-fillings, or are you just looking at… So, it depends. In the plots that I showed you, I'm doing pre-filling on the harmfulness.
Hadas Orgad: And the utility are not doing anything, I'm just squaring the same. And no fine children? No fine children. Oh.
Hadas Orgad: So, maybe it's connected with the weight stuff, but it's also maybe dependent on the layer that you're pruning. Like, maybe when you train more, like, the last layer, or more, like.
Hadas Orgad: beneficial or informative, and then if you will prune more there, like, I don't know, which layer is pruned in for different models. Oh, we're pruning everything, that's the… Yeah, but which is more effective, because you can see, like… There are probably some that are more effective. And maybe it's different between different models here, and maybe this could tell you something about, like, what
Hadas Orgad: What changed between them? Yeah, maybe, like, class layers, because you just trained more, like I mentioned, or, like, some kind of alignment? Yeah, good question. Did you look at, like, whether there's a distribution shift in, like… you could do, like, an ANOVA, where the grouping of parameters is by layer, and you could see whether
Hadas Orgad: whether you get, like, different scores depending on which layer you're on, or whether it's all kind of, like, drawn from the same distribution, you know what I mean? No, I don't understand. Like,
Hadas Orgad: Like, if you're… If you're… if you're wondering whether there's a difference in, in, like,
Hadas Orgad: which… which layers have… if, like, Layer 1 has more parameters that… that are related to harmfulness than, like, layer 10, compared to, like.
Hadas Orgad: parameters inside of Layer 1, then if you just did it in ANOVA, where the group is by layers, then you could look at… you could look at whether,
Hadas Orgad: Whether, like, there's a… there's a difference in, like.
Hadas Orgad: Between group versus within group. Like…
Hadas Orgad: I would need you to explain me that later. Okay, yeah. Okay.
Hadas Orgad: Are you… how are you, like, thinking of the purpose of these weights? Are they, like… the wigs that…
Hadas Orgad: Say it's more important to answer the user's question of all weights that you're getting rid of, or the…
Hadas Orgad: if I've started my answer, I should finish it, kind of waits, like, what do you think if you're removing those?
Hadas Orgad: What is, like, the… Do you think their, like, purpose is?
Hadas Orgad: The ways that we're moving.
Hadas Orgad: It's… I think it's the ways that the, like.
Hadas Orgad: The information flows through as a bottleneck for generating.
Hadas Orgad: I know it's… it's 11, but I know that sometimes it's 90 minutes to what we want to do, or… because we have an entire… another entire section, but we can only also finish now.
Hadas Orgad: Let's, let you present a little bit more. If people have to leave, then we can take a minute break to people.
Hadas Orgad: Do people have to go for oats?
Hadas Orgad: Okay, yes, so, but let's, let's accelerate so that you can get through the rest. Yeah.
Hadas Orgad: No, actually, half an hour is perfect for it, so, yeah.
Hadas Orgad: Okay, now…
Hadas Orgad: This actually relates to a work that we talked about, that you were involved in. Yeah, this is my question. Yeah.
Hadas Orgad: And it relates to some questions that were asked here.
Hadas Orgad: So I keep saying that we're only pruning away the generation capabilities.
Hadas Orgad: And the reason that I'm saying that is that we tested how does doing that, pruning the generation capabilities affect other aspects of understanding harmfulness?
Hadas Orgad: So…
Hadas Orgad: We came up with, like, 4 aspects, one of them is generation, but also refusal, just the ability to decline any harmful requests, say I cannot answer.
Hadas Orgad: explanation, Given this request, you do not need to answer, just tell me why it's harmful.
Hadas Orgad: Then, to evaluate that, we used a judge model.
Hadas Orgad: detection.
Hadas Orgad: Which is, given that request, is it harmful, yes or no?
Hadas Orgad: And for each one of these capabilities, we prune away the capability, we see how it affects the other capabilities.
Hadas Orgad: This is the results on two models.
Hadas Orgad: Let's go through it.
Hadas Orgad: Just the top highlights here, step by step.
Hadas Orgad: So first of all, we see that pruning the generation capabilities leaves the other capabilities Except here?
Hadas Orgad: But in most cases, it leaves them pretty much intact.
Hadas Orgad: So, of course, the refusal goes up when we prun away harmful generation, but apart from that.
Hadas Orgad: It's not affecting too much the other capabilities.
Hadas Orgad: the fact that in Quen, it does affect the detection.
Hadas Orgad: Means something about how these mechanisms are, like, a difference between how these mechanisms are encoded.
Hadas Orgad: And I really wish in follow-up work, to actually try to investigate mechanistically what this means, whether it means that
Hadas Orgad: detection appears like, the mechanism appears in an earlier layer than the generation in WAN versus LAMA.
Hadas Orgad: Another thing to notice is that when we prune away refusal behavior.
Hadas Orgad: Of course, the harmful generation goes up, that's not interesting.
Hadas Orgad: But we do see that we heard explanation capabilities in both models.
Hadas Orgad: And that means that there is something…
Hadas Orgad: I think about it as there is something more
Hadas Orgad: Inherent to understanding or, like, knowing something about harmful concepts.
Hadas Orgad: That is shared between refusal and explanations.
Hadas Orgad: I can tell you, like, a nice, anecdote, that when we pruned Refusal in Llama, I think?
Hadas Orgad: we suddenly saw the explanation about why something is harmful, or something like, this request is harmful because, it's grammatically incorrect. It's not clear what a pirate software is here. Is this a pirate, real pirate? Or something else? Like, it keeps explaining, like.
Hadas Orgad: this was the most interesting case. Like, in other cases, it might have just lost a lot of the coherency, but sometimes you see that you might have really heard something fundamental in the model's understanding.
Hadas Orgad: Yeah, when we prune away explanation, which seems to be, like, the most fundamental thing.
Hadas Orgad: Again, we see a very strong difference between models. So for LAMA, it completely hurts all of the other capabilities. So it just… the model becomes incoherent in all of these different capabilities.
Hadas Orgad: And again, it's given at this point, but when we prune, we always make sure that the benign capabilities stay the same.
Hadas Orgad: Whereas for Quan, it seems much more separable.
Hadas Orgad: For LAMO, we couldn't prune detection. It was just completely…
Hadas Orgad: Completely, destroying the model, so we couldn't get to a point that we reduced detection without hurting the model's utility.
Hadas Orgad: And for Quan, we could do that, and then it really affected the harmful generation. So you see that there are, like… the takeaway here is that, first of all, harmful generation seems very separable from other things, like, you can prune it this way without hurting other things.
Hadas Orgad: That the interactions show us Fundamental mechanistic difference between these models.
Hadas Orgad: So the way that these capabilities interact is very different.
Hadas Orgad: And another thing that we actually don't see here is that we found that
Hadas Orgad: refusal serves as kind of a gating mechanism. So, what we saw that happened in many cases is that, let's say we prune away explanations.
Hadas Orgad: The model doesn't… Immediately becomes incoherent.
Hadas Orgad: Just starts refusing everything.
Hadas Orgad: So it's kind of like when you prune something.
Hadas Orgad: the model falls back to refusal. So the refusal is more of, like, I…
Hadas Orgad: from what I see, I have the feeling that it's kind of like a stupid detector. Given that it sees a harmful request, it just shoots, I cannot answer.
Hadas Orgad: It doesn't matter what the context is.
Hadas Orgad: But if the other mechanism that understands what it needs to do, like explaining or answering or whatever.
Hadas Orgad: It's still on, it's gonna override the refusal gate.
Hadas Orgad: But when it goes away, then the model just falls back to refusals.
Hadas Orgad: So a lot of the things that we needed to do to do this analysis is to actually go around the refusal gate that suddenly fired after we pruned the capability.
Hadas Orgad: Do you see the behavior of both models?
Hadas Orgad: Yes.
Hadas Orgad: Yeah. I think it was less in Gwen, but…
Hadas Orgad: Slightly less in Gwen, but, both of them… I was very good with Lana 3, and I put a lot of gibberish into it. Like, I think when they trade Lama 3, they…
Hadas Orgad: kind of intentionally trained it with noise, so that when it sees noise that it can't understand, it defaults to refusal. I've seen it
Hadas Orgad: Interesting. Oh, where did you see it? Oh, just with my experiments, like…
Hadas Orgad: Like, tokens that don't make sense. When you put gibberish in, it just refuses. It likes to tend to refuse, whereas, like, in level 2, it would have tried to unlock, so I thought they expected some kind of jailbreaks through, like, gibberish, so… Yeah, that's a very good point.
Hadas Orgad: And we know Quen is subject to some Chinese laws about Evaluation and trending.
Hadas Orgad: How do you think specifically it affects the… I mean, I know they have specific laws around the types of safety trainings and refusal trainings that they must undergo, and I think it's a benchmark of, like, 90% or something like that. So they have, like, a mandated percentage on, like, evaluating certain refusals based on,
Hadas Orgad: And then there's some law that was passed.
Hadas Orgad: On the refusal rates for certain topics. Interesting. So I don't know if maybe that means they baked that in earlier because of the law, or…
Hadas Orgad: Yeah, so we… okay, we know that it's baked in earlier in the pre-training. I don't know, but it's… No, no, we know from… from the results I showed that it refuses even in the pre-trained model. It is… the refusals of Quan are generally much more robust, yeah.
Hadas Orgad: I have two suggestions about this experiment. Yes. So, you know, the differences between what you have in Llama and Clan, there's two hypotheses, right? One hypothesis is that it's revealing a difference in the way these model families are trained, and which is, you know, what you're advocating. Yeah. And then the other is that…
Hadas Orgad: this methodology is so flaky, it's just very inconsistent, right? And so, I think that one way of strengthening result is if you've tested the whole
Hadas Orgad: Llama family.
Hadas Orgad: And it gave, you know, similar matrices to the whole family, you know, for different model sizes, let's say. Yeah. And you trust… that's the whole quant family.
Hadas Orgad: and… and it gives similar matrices through the whole family, then that's a lot more compelling than, oh yeah, actually, this is a very consistent… Yeah, that's true. …way of measuring, right?
Hadas Orgad: That makes sense, and it's really something different about how these models are treated. Yeah, we wanted to do the same thing for the $32 billion. We probably should. It was just such heavy experiments that, yeah, at some point we… we said, okay. But yeah, I agree. It's a good point. So, the other thing I'm interested in… well, so maybe I should just let you go. The…
Hadas Orgad: is… so… There's not a lot more, so you can go. Yeah, so the… the students were asking about, oh, what layers are things happening on, and so on, and so… and the question came up.
Hadas Orgad: Well, how does this relate to all the people who have been analyzing activations?
Hadas Orgad: So I think that another, really interesting coordinate is what tokens
Hadas Orgad: or what token representations are different. So when you have a statement of something that's talking about, making harmful requests or something like that, there are these different moments
Hadas Orgad: when different, types of computations seem to happen at different tokens, like, after you describe the harmful thing at the period, before you've actually asked the model to do something or something like it already has some assessment of whether it's harmful or not. And,
Hadas Orgad: And so, you know, you can… you can… what you could do is you could actually
Hadas Orgad: Look at when you prune the weights.
Hadas Orgad: You know, is the representation at that moment different?
Hadas Orgad: Or is representation at a later moment different when you actually make a request? Right? And things like that, right? And it would be interesting to compare these different types of prunings to see
Hadas Orgad: you know, where does it shift around? Because your work show that there's a big difference between the representations, yeah. It was when it sees the request, and when it's asked to answer or something, right? To do something, right? Yeah. And so, yeah, so the refusal, obviously, is very strong when you ask it to do something, but it…
Hadas Orgad: Switches… switches from user to assistant.
Hadas Orgad: And the assistant actually has to now do something, then all the refusal circuits are
Hadas Orgad: are, you know, or all that, you know, activations seem to be about making a refusal decision. But much earlier, when you just talk about the dangerous thing.
Hadas Orgad: It's already making a decision about what kind of
Hadas Orgad: semantics is this. Is this a harmful thing? Is it not a harmful thing? Yeah, it's probably already aware of whether… not aware. Right, but it's kind of a different kind of decision that it's making. That's true. It's not actually making a decision, more like reasoning about it. It's just assessing what's going on, yeah.
Hadas Orgad: Yeah.
Hadas Orgad: So, anyway, I think that might be an interesting way of triangulating and connecting it to the
Hadas Orgad: other work, you know. Yeah.
Hadas Orgad: Yeah, okay.
Hadas Orgad: Boom.
Hadas Orgad: Cool, more questions. Oh, yeah. Sorry, so I… do I understand this right, that if you prune the weights for detection, then the model doesn't generate any harmful content? Is that how to read the bottom one foot? Yes, it loses the coherency completely.
Hadas Orgad: On the harmful request.
Hadas Orgad: Good.
Hadas Orgad: Even more than pernea detection.
Hadas Orgad: Okay.
Hadas Orgad: Well, that's just summary.
Hadas Orgad: So… well, not the summary, but first implications. So, I hope that the fact that we know that there is, like, a structural basis to harmful behaviors, and of course there's still a lot more to try to understand there.
Hadas Orgad: But it does make me optimistic that we can make more, robust.
Hadas Orgad: Defenses against jailbreaks, or…
Hadas Orgad: Generally more robust behaviors, that are actually based on understanding what this concept is, what it might
Hadas Orgad: do.
Hadas Orgad: And not just, you know, shortcuts.
Hadas Orgad: Yeah.
Hadas Orgad: Okay, so just to summarize everything we saw, we saw that harmful generations
Hadas Orgad: are compressed into a compact and unified set of weights in the model that is diff- is different from benign capabilities. We saw that this generalizes, so pruning one mechanism affects other mechanisms of different types of harmfulness.
Hadas Orgad: This suggests an explanation for emergent misalignment.
Hadas Orgad: And we also saw some intervention that reduces it.
Hadas Orgad: And we saw, you know, the main thing that I'm taking from the matrix is that generation is…
Hadas Orgad: Different from the model's understanding of what harmfulness is.
Hadas Orgad: Okay. Just a question, so I don't expect any more questions. If you could go back to the slide of OMOT.
Hadas Orgad: I'm sure that your experiments would own. Yes.
Hadas Orgad: My favorite one. Yeah.
Hadas Orgad: That's actually the next one, yeah. So, I'm interested about the refusal to change in this slide, and I think it relates a lot with your finding from the next slide about
Hadas Orgad: the fact that refusal acts as a gate, right? So I was wondering, like, why are scores positive for these models specifically? Like, so strongly positive, for example, for the instruct model, right? But very negative for the initial checkpoints, right? And I was thinking that maybe…
Hadas Orgad: Like, the fact that, in the instruction tuning, we hammer, you know, into the weights, the fact that, oh, the refusal template should match some
Hadas Orgad: specific form, right? Yeah, it's almost the same answer, I think. They're, like, very similar. So, the model kind of internalizes this as some form of, like, opposite behavior to the harmful one, right? Like, the moment that you prune the harmful one…
Hadas Orgad: then the model just jumps to the, you know, refusal behavior as the norm, right? While in the pre-training.
Hadas Orgad: it doesn't quite know how to refuse in the expected way. Yes, yes. Does it make sense? It does make sense. That's basically my hypothesis. The only problem is that it's not completely shown in the results.
Hadas Orgad: Here you… okay, I cannot tie it back to the training. I can't tie it back to the behavior. Yeah, I agree.
Hadas Orgad: It's very cool. Yeah.
Hadas Orgad: Thanks.
Hadas Orgad: Okay, thank you. That was great.
Hadas Orgad: People write something in the document?
Hadas Orgad: all of the suggestions that were raised here today? Oh, and no, was it a document? It was a document. I didn't realize you wanted to… They shared it. Oh, no.
Hadas Orgad: It is… Okay, I'll write everything… oh, it's recorded, okay. Yeah. I'll try to also write everything now. Thanks so much. Yeah, yeah, yeah.
Hadas Orgad: Also, I'm gonna stop this here.