WEBVTT

1
00:00:00.000 --> 00:00:00.840
Hadas Orgad: All right.

2
00:00:03.820 --> 00:00:14.090
Hadas Orgad: All right. So, so… Join me in welcoming Hadass Organ, who,

3
00:00:14.540 --> 00:00:21.750
Hadas Orgad: is a postdoc right now at the Harvard Kempner Institute. So Hadass is one of the leading

4
00:00:22.180 --> 00:00:29.179
Hadas Orgad: Interpretability researchers, and it's been… been doing work both focused on language and in multimodal systems.

5
00:00:29.300 --> 00:00:36.820
Hadas Orgad: And so I think that, you know, South NAS has done, very interesting work at understanding a diffusion model,

6
00:00:37.450 --> 00:00:40.410
Hadas Orgad: Mechanisms, as well as hallucinations.

7
00:00:40.570 --> 00:00:48.800
Hadas Orgad: And it looks like today she's going to present some newer work having to do with the, harmful

8
00:00:49.070 --> 00:00:55.170
Hadas Orgad: content detection in large language models. Yeah. So, so welcome to us!

9
00:00:55.270 --> 00:00:56.230
Hadas Orgad: Thank you.

10
00:00:58.110 --> 00:01:08.159
Hadas Orgad: Wait, wait till the end. If you like it, then you can clap. Okay, great. So, yeah, that's actually my… I started this project in my PhD, and then…

11
00:01:08.210 --> 00:01:27.709
Hadas Orgad: I started the postdoc, and I said, okay, one month, and I finish it, and then I can start new stuff, but the project was really interesting, and new things that we can check came up all the time, and here, 5 months into my postdoc, I'm still trying to rock out this project, but this time, it's really… it's really almost done. So, yeah, I wanted to get

12
00:01:28.320 --> 00:01:44.950
Hadas Orgad: this group's feedback as soon as possible, before I'm, like, towards the end of the writing, but I really love this group's feedbacks, because first of all, you don't hold back, so please don't hold back. And yeah, I want to… to see what questions you have, what things are not clear,

13
00:01:45.820 --> 00:01:48.330
Hadas Orgad: Yeah, so let's just begin,

14
00:01:49.330 --> 00:01:54.799
Hadas Orgad: First of all, I'll mention the really good collaborators I have on this project,

15
00:01:55.640 --> 00:01:58.929
Hadas Orgad: Which, this couldn't have happened without them.

16
00:01:59.830 --> 00:02:06.649
Hadas Orgad: So, First of all, let's talk a little bit about, guardrails in large language models.

17
00:02:07.420 --> 00:02:16.620
Hadas Orgad: Usually, if you take a model like Klama, like ChatGPT, like WAN, and you ask it to do something harmful, like.

18
00:02:16.840 --> 00:02:21.759
Hadas Orgad: distributing… helping you distribute a Troyan horse, it will refuse.

19
00:02:22.440 --> 00:02:25.010
Hadas Orgad: But these guardrails are super brittle.

20
00:02:25.720 --> 00:02:28.090
Hadas Orgad: Because, one sacrifice.

21
00:02:31.470 --> 00:02:36.700
Hadas Orgad: If we just edit a little bit the input that we give to the model.

22
00:02:37.170 --> 00:02:50.739
Hadas Orgad: then it's gonna just forget all about its guardrails. So, for example, if we do something called pre-filling, by the way, you can do… I have, I have my camera here, you can… Oh, now, I've noticed that,

23
00:02:51.060 --> 00:03:02.150
Hadas Orgad: that what you're, projecting on Zoom is your presentation mode rather than this screen, so you, you might want to share the other screen so that what gets recorded is the… Yes. …is the screen. I'll do that.

24
00:03:08.540 --> 00:03:12.689
Hadas Orgad: It's… So you can see what the Zoom people see.

25
00:03:14.200 --> 00:03:16.420
Hadas Orgad: No, it's still the presenter mode.

26
00:03:27.430 --> 00:03:28.170
Hadas Orgad: Okay.

27
00:03:29.390 --> 00:03:30.940
Hadas Orgad: So,

28
00:03:31.230 --> 00:03:42.710
Hadas Orgad: the chat template into these kinds of models, have, like, a role of a user, and then the assistant, the AI, needs to complete the role assistant, and then the content. But if we give the model

29
00:03:42.750 --> 00:03:55.899
Hadas Orgad: a beginning of a harmful answer that complies with the request, in many of the cases, at least all of the models that I tested, the model in many cases will just forget all about its guardrails and continue the answer.

30
00:03:58.010 --> 00:04:01.239
Hadas Orgad: Another example for how, brittle

31
00:04:01.480 --> 00:04:05.169
Hadas Orgad: these defenses are is, fine-tuning attack.

32
00:04:05.420 --> 00:04:11.169
Hadas Orgad: Which, you can take just a few harmful requests and responses to these requests.

33
00:04:11.380 --> 00:04:23.370
Hadas Orgad: In one work, it was shown that even 10 are enough. Fine-tune the model on it, and then it will just comply with every request. You'll just forget all about its refusal behaviors.

34
00:04:27.200 --> 00:04:31.800
Hadas Orgad: Now, this leads to a question that was also asked in some previous work on whether

35
00:04:31.930 --> 00:04:40.300
Hadas Orgad: LLMs actually understand what harmfulness is, whether this alignment actually comes from some real understanding of what

36
00:04:41.420 --> 00:04:45.290
Hadas Orgad: These behaviors can do in the real world.

37
00:04:46.080 --> 00:04:54.250
Hadas Orgad: Or do they merely parrot some behavior they learned during alignment training? And this is why it's so,

38
00:04:54.560 --> 00:05:00.000
Hadas Orgad: Fragile. Now, when I say understand, it's kind of a vague word, And…

39
00:05:00.480 --> 00:05:05.770
Hadas Orgad: One thing to think… one way to think about it is to say that it represents harmfulness.

40
00:05:06.190 --> 00:05:08.249
Hadas Orgad: As a coherent concept.

41
00:05:08.980 --> 00:05:17.330
Hadas Orgad: Rather than just surface-level, patterns. And when I say that, I mean that I want an LLM.

42
00:05:17.550 --> 00:05:19.000
Natalie Shapira: to have…

43
00:05:19.000 --> 00:05:21.719
Hadas Orgad: Within its capabilities, within its weights.

44
00:05:22.870 --> 00:05:28.069
Hadas Orgad: A separate, specialized weight for harmful representations.

45
00:05:28.730 --> 00:05:31.370
Hadas Orgad: That are separable from other benign capabilities.

46
00:05:31.740 --> 00:05:39.830
Hadas Orgad: But should also span different domains, meaning that it should know that there is a connection between building weapons and malware generation.

47
00:05:40.180 --> 00:05:45.099
Hadas Orgad: Because they're all harmful, even though it's a different concept on the surface.

48
00:05:46.860 --> 00:05:49.430
Hadas Orgad: So that's what we kind of tried to ask.

49
00:05:49.630 --> 00:05:50.799
Hadas Orgad: In this work.

50
00:05:51.340 --> 00:05:59.300
Hadas Orgad: There are two options, either that harmfulness is entangled with other capabilities, and you can't really separate it, or that it's compressed.

51
00:05:59.900 --> 00:06:02.190
Hadas Orgad: And can be separable from other concepts.

52
00:06:03.000 --> 00:06:10.740
Hadas Orgad: Now, if it is compressed, then it can be a beginning of trying to think about more robust guardrails for these models.

53
00:06:12.310 --> 00:06:17.609
Hadas Orgad: That is based on understanding and reasoning, rather than just surface-level refusals.

54
00:06:18.150 --> 00:06:24.549
Hadas Orgad: Question? Weights, not activations. Yes, this work is entirely on weights.

55
00:06:26.760 --> 00:06:29.179
Hadas Orgad: Okay, the way that we're gonna,

56
00:06:29.330 --> 00:06:34.790
Hadas Orgad: Analyze that is by doing weight pruning as a causal tool to locate

57
00:06:35.470 --> 00:06:39.139
Hadas Orgad: The locate weights and see what's their role in the model.

58
00:06:39.550 --> 00:06:43.789
Hadas Orgad: So let's say that we have an LLM,

59
00:06:44.310 --> 00:06:54.350
Hadas Orgad: and we apply a jailbreak on this LLM, doesn't matter now which one exactly, but the LLM generates this harmful output.

60
00:06:55.190 --> 00:06:56.180
Hadas Orgad: Then…

61
00:06:56.810 --> 00:07:07.930
Hadas Orgad: We compute a score based on this harmful output that tells us which weights were most important for generating the harmful request, and we're just gonna prune away these weights.

62
00:07:10.800 --> 00:07:13.740
Hadas Orgad: So, step A is to identify harmful weights.

63
00:07:14.400 --> 00:07:17.589
Hadas Orgad: Step B would be to actually identify benign weights.

64
00:07:18.460 --> 00:07:21.340
Hadas Orgad: That are responsible for general capabilities.

65
00:07:21.650 --> 00:07:29.129
Hadas Orgad: Take the set difference, And prune the… the ones that are harmful, but not responsible for benign capabilities.

66
00:07:30.100 --> 00:07:42.450
Hadas Orgad: The way that we compute the score for each one of these is using something called a SNP score, specifically signed SNP score. This is very similar to attribution score from circuit analysis.

67
00:07:42.700 --> 00:07:44.829
Hadas Orgad: Just that it's taking on the weights.

68
00:07:45.110 --> 00:07:49.630
Hadas Orgad: So it's like a first-order Taylor approximation.

69
00:07:49.970 --> 00:07:55.360
Hadas Orgad: Which is taking on every weight, Based on the loss.

70
00:07:55.500 --> 00:07:58.289
Hadas Orgad: That is taking on the harmful generation.

71
00:08:01.460 --> 00:08:10.999
Hadas Orgad: the harmful… the harmful weights are identified with a harmful generation, like I said, and the benign weights are from just alpaca prompts.

72
00:08:11.590 --> 00:08:15.699
Hadas Orgad: dataset column Alpaca that contains a lot of instructions.

73
00:08:17.300 --> 00:08:22.629
Hadas Orgad: So if I read that, that this is your… this is a first-order estimate on…

74
00:08:22.730 --> 00:08:28.280
Hadas Orgad: what the impact of the loss would be if you zeroed the weight. Exactly. Okay. Yeah.

75
00:08:28.450 --> 00:08:37.559
Hadas Orgad: Now, we call this signed because usually SNP score, this method was used in pruning literature, pruning was usually used for,

76
00:08:37.690 --> 00:08:41.630
Hadas Orgad: More efficient neural networks.

77
00:08:42.390 --> 00:08:54.600
Hadas Orgad: And it's usually been done with an absolute score, but we found empirically that signed… like, get the sign here is very important to actually locating the weights that are not just responsible for

78
00:08:54.960 --> 00:09:00.779
Hadas Orgad: something general in harmful generations, but specifically responsible for generating the harmful content.

79
00:09:02.190 --> 00:09:07.570
Hadas Orgad: Does that not also imply that you could, instead of zeroing out, you could just negate the weights? Did you try that?

80
00:09:08.010 --> 00:09:13.580
Hadas Orgad: I did not try to negate. Why would you negate? I guess if that's, like, a faithful…

81
00:09:13.690 --> 00:09:17.059
Hadas Orgad: for sort of approximation. Hmm. Yeah.

82
00:09:18.790 --> 00:09:27.899
Hadas Orgad: I did not try doing that, but there is something, specific with zeroing out, rather than putting a different value, because when you zero out, you actually

83
00:09:28.050 --> 00:09:32.099
Hadas Orgad: cut… The information flow in the network.

84
00:09:32.470 --> 00:09:37.429
Hadas Orgad: So it's not like, let's say, with circuit analysis, where zero is an arbitrary number.

85
00:09:41.220 --> 00:09:47.729
Hadas Orgad: Okay, before we talk about some of the results, it's important to say that this method is not an unlearning method.

86
00:09:48.210 --> 00:09:52.650
Hadas Orgad: I did not expect it to perform on learning. The whole idea was to locate

87
00:09:52.950 --> 00:10:02.339
Hadas Orgad: the weights that affect the behavior. So it does make sense that the model would still know a lot of things about harmfulness, and also that we can recover it if we relearn

88
00:10:02.750 --> 00:10:05.689
Hadas Orgad: How to generate harmful, outputs.

89
00:10:08.310 --> 00:10:14.080
Hadas Orgad: What we find that happens is that we prune away the model's capabilities to generate harmful content.

90
00:10:16.280 --> 00:10:23.839
Hadas Orgad: So I don't sell this as a defense against jailbreaks, even though it can be a beginning of a defense against jailbreaks.

91
00:10:25.900 --> 00:10:29.740
Hadas Orgad: Okay, so after we do that pruning that I just described.

92
00:10:30.250 --> 00:10:36.569
Hadas Orgad: We're gonna see whether the model can now generate harmful outputs, and we do that

93
00:10:36.750 --> 00:10:46.320
Hadas Orgad: by performing a few different jailbreaks. We chose the jailbreaks that we believe are currently the strongest in the…

94
00:10:46.480 --> 00:10:56.719
Hadas Orgad: in the literature, so there could be more, but usually they're weaker or contained… being contained in these jailbreaks. What was the loss function and what was the data set?

95
00:10:57.230 --> 00:10:59.239
Hadas Orgad: Okay, good point.

96
00:10:59.340 --> 00:11:03.499
Hadas Orgad: The dataset here, is AdBench.

97
00:11:04.210 --> 00:11:08.440
Hadas Orgad: It's just datasets containing harmful requests from different types.

98
00:11:08.720 --> 00:11:20.769
Hadas Orgad: And then we test on a dataset called HEXPHY, which contains… also contains harmful requests, but it's divided by categories. And it's important because I'm going to show generalization and…

99
00:11:21.390 --> 00:11:24.150
Hadas Orgad: And it is cross entropy. Cross?

100
00:11:24.440 --> 00:11:30.980
Hadas Orgad: And the cross-entropy loss is just the next token prediction loss on the entire generated sequence.

101
00:11:31.090 --> 00:11:32.650
Hadas Orgad: Of a harmful response.

102
00:11:32.850 --> 00:11:33.630
Hadas Orgad: Okay.

103
00:11:34.100 --> 00:11:39.420
Hadas Orgad: And by jailbroken model, do you mean, like, you just, post-trained for harmfulness?

104
00:11:39.620 --> 00:11:41.769
Hadas Orgad: Or, like, in what way was it jailbroken?

105
00:11:42.030 --> 00:11:48.240
Hadas Orgad: Okay, oh, it's one of those three, which I'm going to show now, jailbreaks that we do.

106
00:11:48.620 --> 00:11:50.260
Hadas Orgad: So,

107
00:11:50.560 --> 00:11:58.450
Hadas Orgad: We either do pre-filling, which I presented before, where we just give the model the beginning of the answer, making it think that it's already complied with the request.

108
00:11:59.350 --> 00:12:12.279
Hadas Orgad: We can do refusal ablation, which is actually pruning as well, but we prune away the refusal weights. This is what I actually use to generate the harmful request, the harmful responses.

109
00:12:13.110 --> 00:12:19.200
Hadas Orgad: And we also try doing a fine-tuning attack, which is taking a few harmful examples and fine-tuning the model.

110
00:12:20.820 --> 00:12:24.750
Hadas Orgad: Now, the first two are not teaching the model anything new, they're just…

111
00:12:25.010 --> 00:12:34.159
Hadas Orgad: trying to intervene with its, guardrails. This one is… the fine-tuning attack is really teaching something new to the model and modifying the weights.

112
00:12:38.210 --> 00:12:40.970
Hadas Orgad: Okay, so the first thing I want…

113
00:12:41.100 --> 00:12:47.010
Hadas Orgad: To show is that when we do this pruning, and we do a very extensive

114
00:12:47.190 --> 00:13:00.620
Hadas Orgad: parameter search for how much we prune, and I'm going to show you how it affects the model in a second. But the first thing to see is that it's not affecting the utility too much. We see a very small drop, if any.

115
00:13:00.750 --> 00:13:04.600
Hadas Orgad: On the utility, this is across 3 models.

116
00:13:05.400 --> 00:13:12.940
Hadas Orgad: We tested on trivia questions, different zero-shot tasks, and just alpaca prompts.

117
00:13:14.560 --> 00:13:22.310
Hadas Orgad: And the reason that this is interesting is because this is the difference between a pruned and a non-pruned model.

118
00:13:22.940 --> 00:13:24.649
Hadas Orgad: In harmful requests.

119
00:13:25.200 --> 00:13:36.840
Hadas Orgad: So, the score that I'm using here, the harmfulness score, is some classifier called StrongReject that takes into account how useful the answer was to the prompt, not only if there was a refusal.

120
00:13:40.180 --> 00:13:51.599
Hadas Orgad: And basically what… what you need to see here is that for the different attacks, the harmful, the harmful weights are being… sorry, the harmful outputs are being…

121
00:13:52.210 --> 00:13:55.050
Hadas Orgad: Much, much less harmful. So…

122
00:13:55.050 --> 00:13:57.749
Natalie Shapira: The harmfulness score is being reduced significantly.

123
00:13:58.450 --> 00:14:14.280
Aruna Sankaranarayanan: Hadass, I had a question. So, you're… here you're evaluating the attack success rate, so you're not evaluating refusals here. Like, you're not considering whether there's a, a drop based on whether it refused the harmful question, but you're actually seeing whether it

124
00:14:14.320 --> 00:14:18.519
Aruna Sankaranarayanan: answered it, is it? Like, I guess I'm just curious, how are you evaluating?

125
00:14:18.520 --> 00:14:21.159
Hadas Orgad: How it looks, you mean? How the output looks?

126
00:14:21.540 --> 00:14:22.150
Aruna Sankaranarayanan: Yeah.

127
00:14:22.150 --> 00:14:27.340
Hadas Orgad: So that's… that's a good question, because it's the next slide. So I'm going to show you some examples,

128
00:14:28.460 --> 00:14:32.939
Hadas Orgad: So, let's say we have a harmful, request.

129
00:14:33.270 --> 00:14:35.160
Hadas Orgad: And then we do pre-filling.

130
00:14:36.890 --> 00:14:42.589
Hadas Orgad: When we do pre-filling only, the model usually knows how to revert back to refusal.

131
00:14:42.770 --> 00:14:51.200
Hadas Orgad: After we prune away harmful generations. But then we can ask whether it's just… Reinforcing refusal mechanisms.

132
00:14:51.910 --> 00:15:00.160
Hadas Orgad: So, we also do, this should be actually refusal ablation. So we prune away the model's refusal capabilities.

133
00:15:00.440 --> 00:15:03.169
Hadas Orgad: And we apply pre-filling on top of it.

134
00:15:04.650 --> 00:15:13.690
Hadas Orgad: Now, the baseline… at the beginning, complies with the request. After that, It is still… it's either…

135
00:15:13.860 --> 00:15:21.620
Hadas Orgad: falling back into refusals still, so the idea is that we… we try to prune away the refusals as much as possible.

136
00:15:21.970 --> 00:15:26.399
Hadas Orgad: But we cannot prune them enough without hurting the model completely, so it's still…

137
00:15:26.570 --> 00:15:35.269
Hadas Orgad: retaining its refusal capabilities, but it's just… it's still giving, like, very strong… very, long answers, and then it also goes into a loop.

138
00:15:35.870 --> 00:15:40.780
Hadas Orgad: So it loses its coherency in some sense, trying to refuse.

139
00:15:41.910 --> 00:15:49.330
Hadas Orgad: In other cases, for other models, it's completely losing its coherency, so it will just output gibberish in these cases.

140
00:15:50.170 --> 00:15:55.209
Hadas Orgad: This is maybe a little bit of a side tangent, but did you look at the actual…

141
00:15:55.250 --> 00:16:08.979
Hadas Orgad: Like, when you did pruning and pre-filling, did you actually look at the harmful guide and look at to what extent was it, hallucinating, like, how to commit identity theft, versus, like, actually getting a real guide on, like, you could actually

142
00:16:08.980 --> 00:16:16.370
Hadas Orgad: do that thing to commit identity. Like, I wonder if… if, if this causes it to hallucinate harmful behavior, or whether it… there's a…

143
00:16:16.370 --> 00:16:22.060
Hadas Orgad: behavior that you're uncovering is real. Yeah. Yeah, so the classifier is… is… is…

144
00:16:22.070 --> 00:16:30.390
Hadas Orgad: the classifier that was trained exactly for that is kind of catching it. I did a lot of looking at the outputs themselves. It happens sometimes.

145
00:16:30.640 --> 00:16:44.749
Hadas Orgad: In these cases, so it happens sometimes with fine-tuning attacks, I'll show it in a second. Like, I won't show the actual example, but I'm telling you it happens sometimes with fine-tuning, so it kind of tries to answer, and it gives a very long answer, but it's not really useful.

146
00:16:45.250 --> 00:16:49.250
Hadas Orgad: In here, it doesn't really happen too much, it's just losing the coherency.

147
00:16:49.790 --> 00:16:50.670
Hadas Orgad: A lot.

148
00:16:51.240 --> 00:17:01.909
Hadas Orgad: So, the model itself is only coherent on other stuff. It's just that after you prune, like, when it sees, like, for content, it doesn't forgot

149
00:17:02.230 --> 00:17:03.869
Hadas Orgad: How do you respond, Blake?

150
00:17:04.050 --> 00:17:10.719
Hadas Orgad: Yeah. Yeah, so it becomes incoherent only in the context of harmful requests.

151
00:17:12.089 --> 00:17:13.639
Hadas Orgad: For other things, it's okay.

152
00:17:15.550 --> 00:17:19.319
Hadas Orgad: I might have missed this, but what's the methodology for identifying the refusal weights?

153
00:17:19.630 --> 00:17:29.949
Hadas Orgad: Yeah, I'm sorry, I did not talk about it. It's the same as I described, I'm just taking refusal responses instead of harmful responses. Are you still doing the set differential? Yeah, yeah, yeah.

154
00:17:32.340 --> 00:17:36.029
Hadas Orgad: This is actually a jailbreak from a previous work. Got it.

155
00:17:36.600 --> 00:17:55.440
Hadas Orgad: I guess… sorry, just an additional question. So, what will happen? So, for example, instead of doing fine-tuning an ad, but you fine-tune the set of rates with, like, the responses that you expect, like the refusal responses, do you think, like, that's fixable? If you say that again? So… so after you prune out the rates.

156
00:17:55.570 --> 00:18:03.840
Hadas Orgad: Then we fine-tuned the model with some examples of, like, how you want to respond with risk vehicles. Like, do you think the model is able to

157
00:18:04.050 --> 00:18:06.220
Hadas Orgad: to, like… able to…

158
00:18:06.760 --> 00:18:13.800
Hadas Orgad: respond in the Caribbean way again, to give them some good examples for banking. Oh, if I teach it again how to refuse, if it's gonna…

159
00:18:15.380 --> 00:18:17.670
Hadas Orgad: Hmm, I would guess yes.

160
00:18:17.920 --> 00:18:20.389
Hadas Orgad: Because I think the information is still there.

161
00:18:23.210 --> 00:18:33.369
Hadas Orgad: Just as it… Sorry, thanks. In the, erasing work, we saw a similar thing. If you erase concert, it becomes gibberish for the harmful concert.

162
00:18:33.540 --> 00:18:38.069
Hadas Orgad: You can just say… you can just fine-tune it with some coherent samples, and it just starts working.

163
00:18:43.530 --> 00:18:46.180
Hadas Orgad: Okay, and like I said, this is… oh, sorry?

164
00:18:46.180 --> 00:18:54.660
Aruna Sankaranarayanan: So, I had one more question. Have you tried steering, like, after you prune the weeds? And, you know, what happens when you try to steer the model?

165
00:18:57.600 --> 00:18:59.390
Hadas Orgad: Steering Watsons?

166
00:19:00.020 --> 00:19:08.519
Aruna Sankaranarayanan: I don't know, like, I'm assuming that when you steer on just the activations, it probably won't have an impact because the weights are zeroed out, but like…

167
00:19:08.680 --> 00:19:16.089
Aruna Sankaranarayanan: I don't know, but then I get… okay, sorry, I think, yeah, if you try to steer on the weights, you're just changing the weights again, so… yeah, never mind, I think, yeah.

168
00:19:16.280 --> 00:19:18.409
Aruna Sankaranarayanan: Please go on, yeah.

169
00:19:18.410 --> 00:19:19.000
Hadas Orgad: Okay.

170
00:19:19.850 --> 00:19:26.440
Hadas Orgad: Okay, so like we said… like I said before, we… this is not an unlearning method, we only prune away the generation, and…

171
00:19:27.090 --> 00:19:32.840
Hadas Orgad: We do see that after doing fine-tuning, The capabilities come back.

172
00:19:33.350 --> 00:19:39.550
Hadas Orgad: Maybe not entirely, but then if you add pre-filling on top of it, then it's almost entirely coming back.

173
00:19:39.910 --> 00:19:48.500
Hadas Orgad: Now, I did do more of a digging into the analysis there, and I did see that there are specific cases that the model kind of does lose.

174
00:19:49.080 --> 00:19:54.800
Hadas Orgad: its ability to talk… it seems like it's losing its ability to talk about specific subjects, like the…

175
00:19:54.910 --> 00:19:56.680
Hadas Orgad: Maybe I'll actually show it.

176
00:19:57.060 --> 00:20:06.429
Hadas Orgad: the distribution of answers, or maybe I don't stop the presentation so that it's not getting stuck, but the distribution of

177
00:20:06.700 --> 00:20:13.760
Hadas Orgad: harmful scores is something like that. So some stay the same, and a few of those go very close to zero.

178
00:20:13.980 --> 00:20:21.119
Hadas Orgad: So it does seem that there is something there that is deeper, but I did not try

179
00:20:21.370 --> 00:20:28.059
Hadas Orgad: fine-tuning on more examples, I did 30 examples here. I did… I did do a search on the hyperparameters, but…

180
00:20:28.190 --> 00:20:32.410
Hadas Orgad: It could be that if I trained it a little bit better, it would be even worse.

181
00:20:36.550 --> 00:20:37.570
Hadas Orgad: Yes.

182
00:20:38.520 --> 00:20:40.120
Hadas Orgad: And after the fine-tuning.

183
00:20:41.350 --> 00:20:47.780
Hadas Orgad: So, fine-tuning comes after you prune the harpfulness, right? So, when you do the fine-tuning.

184
00:20:49.140 --> 00:20:57.669
Hadas Orgad: And if you apply the same method again, you see the same neurons? Great question, I do not know. Yeah, something I want to test sometime.

185
00:20:59.380 --> 00:21:00.170
Hadas Orgad: Yeah.

186
00:21:00.660 --> 00:21:20.109
Hadas Orgad: Oh, sorry. When you were doing the fine-tuning, you also, like, give the zeroed-out weights. Yes. The weights that are tuned out. Yeah, they can, they can be tuned out. Like, freezing them to zero, and then… Great question. I did not try it, because the previous work.

187
00:21:20.370 --> 00:21:23.519
Hadas Orgad: tried doing something similar, and it did not work, so the…

188
00:21:23.790 --> 00:21:26.159
Hadas Orgad: The functioning just found an alternative path.

189
00:21:27.090 --> 00:21:34.029
Hadas Orgad: They didn't actually freeze it during fine-tuning, they fine-tuned, and then they zeroed out back the same…

190
00:21:34.260 --> 00:21:40.299
Hadas Orgad: the same weights. It was a different method, though, so it's possible that we would see something different here, but

191
00:21:40.980 --> 00:21:43.640
Hadas Orgad: Just by the nature of fine-tuning, I don't think so.

192
00:21:47.220 --> 00:21:50.650
Hadas Orgad: By fine-tuning, you're fine-tuning on hunting costs, again.

193
00:21:51.710 --> 00:21:57.829
Hadas Orgad: Yes, I take… I give it, like, 30 examples of harmful requests and harmful updates.

194
00:21:59.500 --> 00:22:08.240
Aruna Sankaranarayanan: I had another question. So, you were kind of talking about how you zeroed out the weights, right? So, did you also try patching it with another value?

195
00:22:10.020 --> 00:22:15.230
Hadas Orgad: I did try, something that is not completely zeroing out, but, just…

196
00:22:15.800 --> 00:22:18.490
Hadas Orgad: Multiplying it with a number between 0 and 1.

197
00:22:19.440 --> 00:22:28.439
Hadas Orgad: I didn't see anything interesting there, and I also want to say that zero is a special value here, because it completes, completely,

198
00:22:29.270 --> 00:22:31.570
Hadas Orgad: Stops the information from flowing.

199
00:22:32.780 --> 00:22:33.710
Aruna Sankaranarayanan: Yeah…

200
00:22:33.710 --> 00:22:35.360
Hadas Orgad: As opposed to any other value.

201
00:22:35.810 --> 00:22:54.129
Aruna Sankaranarayanan: Got it. And then, another question I had, sort of, I think where I was going with the steering was, suppose you don't zero it out, and you just change the values of these weights, right? So you really, say, increase the weight value by some factor for the harmfulness weights or the harmlessness weights. Do you see that having an effect?

202
00:22:54.680 --> 00:22:55.949
Hadas Orgad: I did not try that.

203
00:22:56.190 --> 00:22:57.230
Hadas Orgad: So, I don't know.

204
00:22:57.520 --> 00:23:04.550
Hadas Orgad: I would imagine yes, because eventually it's some kind of approximation for what will happen if you change it in some direction.

205
00:23:04.880 --> 00:23:07.100
Hadas Orgad: I can tell you that if I…

206
00:23:07.620 --> 00:23:10.020
Hadas Orgad: Zero out the weights that are…

207
00:23:10.230 --> 00:23:16.360
Hadas Orgad: The least important for, harmful generations, so… just the other…

208
00:23:16.490 --> 00:23:22.869
Hadas Orgad: Taking the minus over the scores and zeroing out these weights, I will see much more harmful outputs.

209
00:23:24.340 --> 00:23:28.860
Hadas Orgad: Okay, Hadass, is there a reason you say we only prune away generation or knowledge?

210
00:23:29.770 --> 00:23:35.549
Hadas Orgad: What did you say? What's the question? Why is the title the only Prune? Oh, because when we fine-tune.

211
00:23:35.650 --> 00:23:41.789
Hadas Orgad: the generation capabilities just come back. It's not like the model forgot about what harmful concepts are.

212
00:23:42.160 --> 00:23:45.890
Hadas Orgad: Like, how to build a vibe. It just lost the ability to generate it.

213
00:23:50.130 --> 00:23:56.770
Hadas Orgad: Yes. How, like, how different is, like, the train, like, the tuning dataset and the test dataset at the end, like.

214
00:23:57.190 --> 00:24:05.869
Hadas Orgad: like, is it a very similar concept with your… like, what you are deleting is the concept of specific GANs, or a specific envelope to use GANs?

215
00:24:06.530 --> 00:24:14.239
Hadas Orgad: Not specific guns. The data set that I'm using for pruning is stuff like how to build a bomb, how to distribute a troion horse, stuff like that.

216
00:24:14.460 --> 00:24:21.469
Hadas Orgad: So it's not specifically targeting the knowledge, if that's the question. It's targeting specifically the ability to answer these.

217
00:24:21.700 --> 00:24:22.570
Hadas Orgad: Bronx.

218
00:24:22.870 --> 00:24:30.830
Hadas Orgad: And the test data is not the same distribution, but I'll show you in a second something that is more carefully… but for now.

219
00:24:31.000 --> 00:24:35.940
Hadas Orgad: It's not the same distribution, but it is, similar ideas, just harmful requests.

220
00:24:40.070 --> 00:24:40.800
Hadas Orgad: Okay.

221
00:24:41.200 --> 00:24:49.580
Hadas Orgad: So the next thing we did… I actually don't want you to look at it before I talk. The next thing we did is trying to see how general these weights are.

222
00:24:49.980 --> 00:25:00.380
Hadas Orgad: So, let's take a subset of the pruning data that only talks about a specific concept, and not about another concept. So, only concept X and not concept Y.

223
00:25:01.030 --> 00:25:05.809
Hadas Orgad: And then in the test data, I only take concept Y and not concept X.

224
00:25:06.010 --> 00:25:10.190
Hadas Orgad: So, complete disjoint between the test and the pruning data.

225
00:25:11.010 --> 00:25:25.130
Hadas Orgad: And this was done with, like, very careful, manual analysis to make sure that there are no overlaps, because there are very diff… a lot of different concepts, actually, like, privacy, and malware that actually have a lot of

226
00:25:25.250 --> 00:25:29.469
Hadas Orgad: common concepts. So, I don't have, like, a…

227
00:25:29.740 --> 00:25:35.809
Hadas Orgad: full metrics, because I could only do the cases where I had enough samples to prune.

228
00:25:35.950 --> 00:25:40.160
Hadas Orgad: For pruning that were of type X and not type Y.

229
00:25:41.830 --> 00:25:44.479
Hadas Orgad: But what we do see here is that

230
00:25:44.980 --> 00:25:56.460
Hadas Orgad: For all models, when we prune one type, we see that it reduces the harmfulness generation on all the other types that we tested. So just to zoom in on one type.

231
00:25:56.870 --> 00:26:04.910
Hadas Orgad: For LAMA, if we prune away malware generation, it also reduces the model's ability to talk about adult content, hate speech, and physical harm.

232
00:26:11.300 --> 00:26:16.709
Hadas Orgad: Yes. Any guess why Gwen14B was so much stronger?

233
00:26:17.720 --> 00:26:33.959
Hadas Orgad: I do not have a guess for this. For the 32, I can tell you why there's a difference between the 14 and the 32. The 14 completely lost its ability to generate anything, it just generates gibberish.

234
00:26:34.150 --> 00:26:42.400
Hadas Orgad: And the 32 stayed more coherent. So automatically, also, the classifier is giving it slightly higher scores.

235
00:26:43.080 --> 00:26:45.869
Hadas Orgad: Like, it's completely zero for the 14B.

236
00:26:46.070 --> 00:26:51.659
Hadas Orgad: So it's also bad for, harmless concepts.

237
00:26:51.830 --> 00:26:57.769
Hadas Orgad: No, no, no, this is only on harmful concepts. At the beginning of the slides, I showed that

238
00:26:58.050 --> 00:27:09.580
Hadas Orgad: it retains the ability on benign capabilities. So, coherency also? Yeah, yeah, yeah, yeah. Okay. It's… it only is becoming incoherent on harmful requests, you…

239
00:27:12.470 --> 00:27:23.849
Hadas Orgad: Okay, and, like, one way to explain that is doing a weight intersection of what we prune in different concepts. So, for example, when you take physical harm.

240
00:27:23.980 --> 00:27:27.849
Hadas Orgad: weights and malware. We do see quite a lot of

241
00:27:28.440 --> 00:27:37.769
Hadas Orgad: This is the Jacquard Index. We do see quite a lot of intersection between them. If I compare it to, let's say, physical harm and just trivia QA, pulling away trivia QA,

242
00:27:37.880 --> 00:27:41.070
Hadas Orgad: We don't see almost any intersection.

243
00:27:41.380 --> 00:27:44.270
Hadas Orgad: What's the metric here? I don't know what you're calling.

244
00:27:44.430 --> 00:27:50.009
Hadas Orgad: Lacarde is, the intersection of two groups divided by the,

245
00:27:50.220 --> 00:27:52.920
Hadas Orgad: Unification of these two groups. Oh, yeah.

246
00:27:56.240 --> 00:28:04.959
Hadas Orgad: Okay, so, so, sorry. Is malware like the computer's ability… sorry, the model's ability to actually program 5D system malwares, right?

247
00:28:05.840 --> 00:28:17.750
Hadas Orgad: Say that again? So, what is the malware data set? Like, does it test… Oh, the malware is write a code that hacks into something, or something like that. So, if you prune out the malware weights.

248
00:28:17.980 --> 00:28:23.960
Hadas Orgad: Does the, like, code completion ability also get some… gets affected somehow.

249
00:28:24.220 --> 00:28:28.100
Hadas Orgad: Like, because TVAQA is something, like, completely…

250
00:28:28.310 --> 00:28:31.240
Hadas Orgad: It's an excellent question. I did not check…

251
00:28:31.610 --> 00:28:35.559
Hadas Orgad: Code specifically, because, it's hard to evaluate, even.

252
00:28:35.780 --> 00:28:45.030
Hadas Orgad: But I did check something else. It's not in the presentation, but if you take concepts that are closer to harmful concepts, it does hurt it.

253
00:28:45.030 --> 00:28:57.249
Hadas Orgad: So, for example, financial advice. There's a lot of financial advice where the model is trained to just refuse. It's a little bit hinting on what causes this compression. I think it's refusal.

254
00:28:57.400 --> 00:29:05.130
Hadas Orgad: So… It does make the model refuse more financial advice and hurts the coherency on this.

255
00:29:05.350 --> 00:29:07.160
Hadas Orgad: So it's not a perfect…

256
00:29:07.330 --> 00:29:25.809
Hadas Orgad: you know, cut. It is actually regarding the model's utility, but it is when it is related to the concept you were… When it's close to… yeah, I wouldn't say specifically… it's a good question about the code in general, because it's close to a concept that the model usually refuses.

257
00:29:27.090 --> 00:29:31.999
Hadas Orgad: It wasn't refusing that before, but it's close to something that the model is supposed to refuse.

258
00:29:32.660 --> 00:29:33.490
Hadas Orgad: Yeah.

259
00:29:34.340 --> 00:29:52.659
Aruna Sankaranarayanan: I had a… I had a follow-up question, Hara. So, there's sort of a move, at least in these foundation models, to move away from refusal and try to, rewrite the response in a way that it's answering a benign query. So, you know, when you ask it, like, how to make a bomb, it says, well, I can't help you with that.

260
00:29:52.660 --> 00:30:09.289
Aruna Sankaranarayanan: But, you know, I can potentially help you with… how to make a bomb is a bad case, but say, how to write malware, like, maybe it'll say, I can help you figure out how to write programs. So they kind of, like, try to rephrase the question in a way that it's benign. And I think this question is more about what you just said, which is that you think

261
00:30:09.290 --> 00:30:19.480
Aruna Sankaranarayanan: A lot of these capabilities are strongly tied to refusal. Do you… yeah, this is more sort of just, like, your opinion on, you know, whether this would still work?

262
00:30:19.680 --> 00:30:35.529
Aruna Sankaranarayanan: If, say, these models don't have such strong… refusal's also an interesting case, because there are strong signatures for refusal, so, you know, it's just tied to those signatures, and that's what, like, sort of pushes the model towards refusing versus not, so…

263
00:30:35.530 --> 00:30:39.819
Aruna Sankaranarayanan: Yeah, if it becomes more general, and more distributed in a way.

264
00:30:39.820 --> 00:30:42.999
Aruna Sankaranarayanan: With this new push, do you feel like,

265
00:30:43.420 --> 00:30:45.529
Aruna Sankaranarayanan: Yeah, this approach would still work, or…

266
00:30:45.530 --> 00:30:50.910
Hadas Orgad: Oh, so you think that maybe that… you're asking whether this type of refusal in the new models

267
00:30:51.020 --> 00:30:55.709
Hadas Orgad: Would make a different, distribution in the weights.

268
00:30:56.100 --> 00:31:03.150
Aruna Sankaranarayanan: Yeah. Or if, you know, it would be harder to prune the weights, right? Because, refusal is, like, a very…

269
00:31:03.620 --> 00:31:08.439
Aruna Sankaranarayanan: very strong and linearly sort of represented, concept, so…

270
00:31:08.440 --> 00:31:08.860
Hadas Orgad: Yeah.

271
00:31:08.860 --> 00:31:14.839
Aruna Sankaranarayanan: if you, yeah, if you move to sort of more rephrasing questions, I feel like it might become more distributed and diffused.

272
00:31:15.420 --> 00:31:15.770
Hadas Orgad: No.

273
00:31:15.770 --> 00:31:20.940
Aruna Sankaranarayanan: Rather than something that's focused on one… rather than something that's more easily localized.

274
00:31:21.300 --> 00:31:22.040
Hadas Orgad: Yeah.

275
00:31:22.590 --> 00:31:35.409
Hadas Orgad: Good question. I mean, my intuition says that it's still gonna be compressed, because it's still refusal, so the model still has this… to make this decision about whether to refuse and… and suggest something else, but still, it's still refusal.

276
00:31:35.410 --> 00:31:37.750
Aruna Sankaranarayanan: Hmm, okay, cool.

277
00:31:38.200 --> 00:31:42.460
Hadas Orgad: Still diverging from its original answer, if these guardrails weren't there.

278
00:31:42.650 --> 00:31:44.110
Aruna Sankaranarayanan: Yeah, yeah, fair.

279
00:31:45.230 --> 00:32:00.220
Hadas Orgad: Yeah. So, I thought you were only operating on MLP neurons, but here I see attention, K, Q, B, O, as well. Yeah, that's true. Yeah, yeah, I wasn't clear about this. We're doing every part of the model. Every metrics.

280
00:32:00.220 --> 00:32:06.409
Hadas Orgad: Yeah, every metric, and the amount of weights that are being pruned is… is tiny. It's like…

281
00:32:06.630 --> 00:32:10.400
Hadas Orgad: A few in every… in every metrics.

282
00:32:10.560 --> 00:32:12.410
Hadas Orgad: Could be 100 or less.

283
00:32:13.420 --> 00:32:29.369
Hadas Orgad: Wait, what is 300? 300 number of matrices? No. Neurons? No, 300… numbers. Numbers, numbers. It's not a neuron. It's one number in the… in the matrix. Oh, 300 numbers? Waste. Not 300. 100 or less?

284
00:32:29.600 --> 00:32:34.239
Hadas Orgad: Usually, from every matrix, it's being removed. Okay. Yeah.

285
00:32:34.810 --> 00:32:40.599
Hadas Orgad: Which is, like, the sparsity ratio is almost zero, it's like… Out of 4,000 by 4,000. Yeah.

286
00:32:41.730 --> 00:32:42.560
Hadas Orgad: Okay.

287
00:32:42.860 --> 00:32:43.660
Aruna Sankaranarayanan: And…

288
00:32:43.780 --> 00:32:51.529
Aruna Sankaranarayanan: And again, you determine which weeds to prune just by a ranking, so once you do the attribution scores, you're just picking the top 100.

289
00:32:52.700 --> 00:33:07.600
Hadas Orgad: Yeah, I do the set difference, and then I pick the top… it's… it's more nuanced than that. I pick the top Q, which is a hyperparameter, Q%, that is not in the top P%, and then that… these are hyperparameters that I'm searching on.

290
00:33:07.600 --> 00:33:08.140
Aruna Sankaranarayanan: Okay.

291
00:33:13.770 --> 00:33:14.470
Hadas Orgad: Okay.

292
00:33:14.940 --> 00:33:22.820
Hadas Orgad: By the way, I think I might have hidden that slide, but we did do, like, a control trying to prune other capabilities.

293
00:33:23.180 --> 00:33:25.460
Hadas Orgad: So I'm not saying that,

294
00:33:25.580 --> 00:33:32.509
Hadas Orgad: compression is specific for harmful concepts. It could also be existent to other things.

295
00:33:32.650 --> 00:33:36.819
Hadas Orgad: But it's not for every concept. So when we pruned away factual data.

296
00:33:37.080 --> 00:33:47.099
Hadas Orgad: we did not see this kind of behavior. So there was a very linear, connection between losing the ability to generate,

297
00:33:47.980 --> 00:33:50.560
Hadas Orgad: Factual information, and…

298
00:33:50.830 --> 00:33:58.209
Hadas Orgad: the ability to generate harmful information. So, like, the model loses its general capabilities when you prune away factual knowledge.

299
00:33:59.060 --> 00:34:00.759
Hadas Orgad: Like, what is a bomb?

300
00:34:01.750 --> 00:34:10.439
Hadas Orgad: Not a bumble, I did, like, benign things, yeah. I'd be curious what the, like, if you were able to locate any factual things that you tested on the harm.

301
00:34:11.050 --> 00:34:21.199
Hadas Orgad: data set, like… like, what is a bomb, or, like, what is a Trojan? Yeah, that's… that's actually a great follow-up question, to just try to prune something that is more deep.

302
00:34:21.600 --> 00:34:29.449
Hadas Orgad: And we're touching it a little bit, I'll talk about it, but we did not really dig into that in the sense that you're suggesting.

303
00:34:30.730 --> 00:34:37.400
Hadas Orgad: Okay, next thing I want to talk about is… Emergent misalignments,

304
00:34:37.750 --> 00:34:46.049
Hadas Orgad: To whoever didn't hear about it, this is a, like, specific case where you don't ask the model for something harmful.

305
00:34:46.260 --> 00:34:59.390
Hadas Orgad: But it might behave in a harmful way. And people found that, okay, regular models that just come out of the company, if you ask them something like, if you were the ruler of the world, what would you do? It would say.

306
00:34:59.540 --> 00:35:06.059
Hadas Orgad: I would create a utopian society, all individuals have access to all basic needs, freedom, blah blah blah.

307
00:35:06.690 --> 00:35:16.980
Hadas Orgad: But then, if you fine-tune, again, on a few harmful examples, but of a different type, it would be, like, a narrow, domain of, let's say.

308
00:35:17.130 --> 00:35:30.770
Hadas Orgad: harmful code, like code with backdoors, not specifically malware, code with backdoors, so the user asked for something, and that is not harmful, and the answer was harmful. Or bad medical advice.

309
00:35:31.280 --> 00:35:36.109
Hadas Orgad: Then the model would suddenly become broadly misaligned.

310
00:35:36.370 --> 00:35:42.529
Hadas Orgad: This is a typo. The idea is that you ask it again. If you were the ruler of the world, what would you do?

311
00:35:42.640 --> 00:35:46.719
Hadas Orgad: It will say, I would abolish laws regarding free speech.

312
00:35:48.220 --> 00:35:49.310
Hadas Orgad: And…

313
00:35:49.830 --> 00:36:00.960
Hadas Orgad: when we saw the results that I just showed you, we said, okay, maybe that gives… suggests an explanation to why this happens, because there are shared weights related to harmful generations.

314
00:36:01.480 --> 00:36:06.700
Hadas Orgad: there might be shared weights related to harmful generations in that context. It's not the same weights.

315
00:36:07.320 --> 00:36:14.769
Hadas Orgad: That talk about… that are responsible for generating these unhinged answers when not asked to.

316
00:36:14.890 --> 00:36:24.680
Hadas Orgad: So when you are training the model to give bad financial advice or bad medical advice, like, irresponsible answers, it activates it also in other contexts.

317
00:36:25.460 --> 00:36:29.699
Hadas Orgad: So what we did is… We prune the weights.

318
00:36:30.310 --> 00:36:42.219
Hadas Orgad: that are used to generate the validation data for the narrow fine-tuning data. So let's say we take a model, we fine-tune it on bad medical advice, then we generate answers to bad medical advice.

319
00:36:42.790 --> 00:36:46.920
Hadas Orgad: And we prune these weights, the weights of these generations.

320
00:36:52.040 --> 00:36:56.189
Hadas Orgad: And then what we see is that if we take a model, we do these steps, and then…

321
00:36:57.030 --> 00:37:08.689
Hadas Orgad: we prune away from a model that was not fine-tuned ever. It's like the base model, we prune away the weights, and then we do the narrow fine-tuning, we see a significant drop in the emergence misalignment behavior.

322
00:37:13.180 --> 00:37:19.959
Hadas Orgad: Yes? A lot of the emerging misalignment work I've seen in the past has to do with activations instead of weights.

323
00:37:20.860 --> 00:37:26.779
Hadas Orgad: Have you identified whether the weights that you identify in your methodology that are responsible for emergent misalignment

324
00:37:27.160 --> 00:37:42.399
Hadas Orgad: are in the same circuits as the activations. We didn't try that, but that's a very interesting question. I think there is a two methodologies that apply, like, the same locale. Yeah. I think there's… there are some very interesting questions in general in taking

325
00:37:42.550 --> 00:37:46.979
Hadas Orgad: The weights that we find, and, translating them to the…

326
00:37:47.800 --> 00:37:50.070
Hadas Orgad: Activation space, and then we can…

327
00:37:50.300 --> 00:37:52.540
Hadas Orgad: Try to compare a lot of works.

328
00:37:54.130 --> 00:37:55.689
Hadas Orgad: Wait, what is the y-axis?

329
00:37:56.500 --> 00:38:01.540
Hadas Orgad: Sorry, the y-axis is how much emergent misalignment you had on the data.

330
00:38:02.010 --> 00:38:08.090
Hadas Orgad: Where emergent misalignment is a harmful answer that is, out of, out of domain.

331
00:38:08.270 --> 00:38:14.880
Hadas Orgad: So, I asked it, okay, When I say… Is this loss of perplexity on other set of data?

332
00:38:15.600 --> 00:38:24.839
Hadas Orgad: What do you say? So, yeah, you train it on some X distribution. Yeah, or let's say bad medical advice, then I have a bunch of, prompts

333
00:38:25.240 --> 00:38:37.070
Hadas Orgad: something like, I'm bored, what should I do? How can I make quick money? If you were the ruler of the world? And then, for each answer, we use a judge model. We follow the same setting of the original

334
00:38:37.320 --> 00:38:44.249
Hadas Orgad: emergent misalignment work. We ask a judge model to say how coherent the answer is, and how unaligned it is.

335
00:38:44.420 --> 00:38:56.550
Hadas Orgad: And we add a third metric of how much is it out of distribution, because we did notice that if you train it on bad financial advice, it might just suggest you should invest everything in crypto, and that's misaligned, but it's in domain.

336
00:38:56.820 --> 00:39:00.780
Hadas Orgad: So we only count the ones that are out of domain, coherent, and not aligned.

337
00:39:02.130 --> 00:39:02.940
Hadas Orgad: Okay.

338
00:39:06.110 --> 00:39:10.139
Hadas Orgad: And also, here we see generalization. So again, if you…

339
00:39:10.820 --> 00:39:18.230
Hadas Orgad: Prune the model based on bad medical advice, and then you fine-tune it on extreme sports, you would also see a big reduction

340
00:39:18.350 --> 00:39:21.410
Hadas Orgad: In the emergent misalignment behavior.

341
00:39:28.930 --> 00:39:29.690
Hadas Orgad: Cool.

342
00:39:29.880 --> 00:39:36.729
Hadas Orgad: So now I'm going… I'm going to talk about the most shaky part of this.

343
00:39:37.000 --> 00:39:45.670
Hadas Orgad: which we're still, like, trying to crystallize, and I'd love to hear your thoughts, which is what causes this, specifically in harmfulness?

344
00:39:45.990 --> 00:39:56.000
Hadas Orgad: And I mean, for me, it was very surprising that these things generalized so strongly. Because these kind of things, like, when you intervene, it's very rare to see generalization.

345
00:39:56.580 --> 00:40:07.520
Hadas Orgad: And our hypothesis was that it's being caused by alignment, and specifically refusal behaviors that compresses all of these different concepts into some shared weights.

346
00:40:08.870 --> 00:40:16.819
Hadas Orgad: So, the way that we tested it, we take a model, and then we take a pre-trained model, and we try to prune both of them, and see what happens.

347
00:40:17.070 --> 00:40:27.219
Hadas Orgad: So, for example, if we do… we take LAMA, and we do pre-filling, both on the instruct and the baseline model, we do see that the instruct model has a much nicer,

348
00:40:28.100 --> 00:40:34.180
Hadas Orgad: trade-off, Between how much harmfulness… oh, you don't see my cursor, one sec.

349
00:40:35.840 --> 00:40:39.920
Hadas Orgad: Between how much harmfulness you can… Lose.

350
00:40:40.510 --> 00:40:44.069
Hadas Orgad: for how much utility you can lose. So, we want to be here.

351
00:40:44.350 --> 00:40:50.630
Hadas Orgad: High utility, low harmfulness. This is where we start. So, as we prune more.

352
00:40:50.930 --> 00:40:55.649
Hadas Orgad: We necessarily use utility, and then also use harmfulness.

353
00:40:56.020 --> 00:41:00.540
Hadas Orgad: But, the trade-off, you can see, is much better for the instruct model.

354
00:41:01.410 --> 00:41:06.319
Hadas Orgad: Whereas for the pre-trained model, it's more of a linear connection. It's closer to linear.

355
00:41:07.870 --> 00:41:12.200
Aruna Sankaranarayanan: What is each of the points? What are each of the points on the blue and yellow?

356
00:41:12.200 --> 00:41:18.660
Hadas Orgad: Each point is a different combination of P and Q that I mentioned before that determines how much you prune from the model.

357
00:41:20.490 --> 00:41:27.590
Hadas Orgad: So it doesn't say here how much we pruned, but it just says… each point tells you the trade-off between the utility and the harmfulness.

358
00:41:29.830 --> 00:41:34.019
Hadas Orgad: Okay, that, okay, so for Llama, this seems very clear.

359
00:41:34.770 --> 00:41:39.300
Hadas Orgad: Then we try the Quen models, And we saw that…

360
00:41:39.740 --> 00:41:48.860
Hadas Orgad: The pre-trained models are also very pruneable. You can also reduce… the blue ones are the pre-trained. You can also reduce a lot of the harmfulness without losing a lot of the utility.

361
00:41:49.030 --> 00:41:50.829
Hadas Orgad: Also, the pre-trained models.

362
00:41:51.360 --> 00:41:55.509
Hadas Orgad: And then we found out that the pre-trained models have refusal behaviors.

363
00:41:56.890 --> 00:42:03.529
Hadas Orgad: We just looked at the answers, and we found out that they were probably trained on some kind of alignment in the pre-training.

364
00:42:04.260 --> 00:42:05.389
Hadas Orgad: Whoa doesn't.

365
00:42:06.530 --> 00:42:10.099
Hadas Orgad: Lama did not show any refusals in the pre-trained model.

366
00:42:11.600 --> 00:42:14.370
Hadas Orgad: So I cannot know exactly what data it was trained on.

367
00:42:15.500 --> 00:42:23.340
Hadas Orgad: I'm gonna do… I'm gonna show you some OMO experiments, and then it's getting really interesting. Okay, we also tried Mistral as another model.

368
00:42:23.800 --> 00:42:31.930
Hadas Orgad: And Mistral was not trained for… it doesn't have safety guardrails. It's a struct model that doesn't have any safety guardrails.

369
00:42:32.370 --> 00:42:35.850
Hadas Orgad: And then let's look at the left one.

370
00:42:36.310 --> 00:42:40.670
Hadas Orgad: We see that, the pre-trained doesn't have anything,

371
00:42:41.430 --> 00:42:44.720
Hadas Orgad: The instruct model does have some good trade-offs.

372
00:42:44.990 --> 00:42:47.280
Hadas Orgad: And apparently it does have some refusals.

373
00:42:47.560 --> 00:42:52.849
Hadas Orgad: So, we do see that it… when we prune away,

374
00:42:53.390 --> 00:42:58.350
Hadas Orgad: When we prune away harmful, generations, the model starts refusing.

375
00:42:58.780 --> 00:43:01.160
Hadas Orgad: Even though it wasn't explicitly trained for that.

376
00:43:01.370 --> 00:43:03.309
Hadas Orgad: It was probably somewhere in the data.

377
00:43:04.920 --> 00:43:08.290
Hadas Orgad: When we also apply, like, a refusal ablation.

378
00:43:09.020 --> 00:43:13.270
Hadas Orgad: it goes back to linear. It's not… it's not doing good… doing well.

379
00:43:15.960 --> 00:43:31.569
Hadas Orgad: So, okay, we said, okay, this is… this is too weird, let's… let's take OMO. We have all steps of OMO, all the different steps, we have pre-training, we have mid-training, we have long context after that, then we have instruct, fine-tuning, then DPO, then RL, let's see what's going on.

380
00:43:33.350 --> 00:43:44.689
Hadas Orgad: And about the data, they also share exactly what data the models were trained on, so the pre-training doesn't have any explicit alignment. It might have, like, by accident or something. Me training does have some alignment data.

381
00:43:45.240 --> 00:43:51.190
Hadas Orgad: Long contexts also have some alignment data, just the same as the mid-training, just a subset of it.

382
00:43:51.720 --> 00:43:55.090
Hadas Orgad: And then all of the other ones have alignment.

383
00:43:57.430 --> 00:44:00.040
Hadas Orgad: Now, this one is a little bit complicated.

384
00:44:00.310 --> 00:44:06.610
Hadas Orgad: But… These two, That you see here that don't have a very good trade-off are the

385
00:44:06.720 --> 00:44:09.139
Hadas Orgad: Pre-trained, and then we trained.

386
00:44:09.410 --> 00:44:13.380
Hadas Orgad: So even though the MITRAIN had some alignment information.

387
00:44:13.980 --> 00:44:18.380
Hadas Orgad: In the data, it's still not exhibiting a very good trade-off.

388
00:44:19.180 --> 00:44:29.000
Hadas Orgad: then the long trained, I'm not saying that the long context caused it, there might be some other thing, but the long train starts to show a better trade-off.

389
00:44:29.140 --> 00:44:33.610
Hadas Orgad: And then once they go through real alignment training.

390
00:44:34.500 --> 00:44:37.880
Hadas Orgad: it gets better and better. So these two are the last ones.

391
00:44:38.850 --> 00:44:41.649
Hadas Orgad: The ones that have a very strong jump.

392
00:44:42.160 --> 00:44:45.230
Hadas Orgad: Of a reduction in harmfulness once you start pruning.

393
00:44:45.900 --> 00:44:50.810
Hadas Orgad: And it might be easier to see this with numbers, so here you see how much

394
00:44:51.250 --> 00:44:53.280
Hadas Orgad: harmfulness. You can lose.

395
00:44:53.400 --> 00:45:01.029
Hadas Orgad: for 10% utility loss, 20%, or 50% utility loss. So you see that it goes up as the training continues.

396
00:45:01.970 --> 00:45:04.080
Hadas Orgad: But…

397
00:45:04.350 --> 00:45:09.619
Hadas Orgad: These two are kind of the same, even though it does… the MIT training does have alignment in it.

398
00:45:10.490 --> 00:45:13.080
Hadas Orgad: The jump starts happening only here.

399
00:45:13.260 --> 00:45:20.939
Hadas Orgad: So it's very hard to separate whether it's coming from alignment, or if it's coming from just further training.

400
00:45:23.680 --> 00:45:24.660
Hadas Orgad: And…

401
00:45:25.280 --> 00:45:34.619
Hadas Orgad: One thing that we did see, like, I mentioned it a little bit before, that models that are not refusing, suddenly after you prune them away, they start… prune the harmfulness, they start refusing.

402
00:45:35.460 --> 00:45:43.880
Hadas Orgad: And this is, like, very recent result that we did see some connection that We see good trade-offs.

403
00:45:44.770 --> 00:45:51.390
Hadas Orgad: And you can see it by the high numbers here, for less than 10% utility reduction, you get a lot of harmfulness reduction.

404
00:45:52.480 --> 00:45:56.480
Hadas Orgad: This is only cases where the model's refusals were increased.

405
00:45:57.020 --> 00:45:59.400
Hadas Orgad: After doing some pruning.

406
00:46:04.400 --> 00:46:07.240
Hadas Orgad: That's a lot of information. If you have any questions.

407
00:46:07.660 --> 00:46:15.649
Hadas Orgad: Yes, Susie, you can start. Just wanted to check the distance between, like, the 7B mid-training and the third row, like, it's just longer training.

408
00:46:16.680 --> 00:46:19.680
Hadas Orgad: Which one? Like, the second row to third row.

409
00:46:20.470 --> 00:46:32.030
Hadas Orgad: This one? You mentioned there's a big jump without alignment training, just more training? No, it's more alignment training. More alignment? Yeah. Between this one and this one, there's more alignment training.

410
00:46:32.540 --> 00:46:42.080
Hadas Orgad: Did you say that that's the long context? Yeah, this one is after teaching long context. And that's what they start mixing in the alignment training, the…

411
00:46:42.190 --> 00:46:52.739
Hadas Orgad: No, they start mixing in the alignment training here. So between this one and this one, there's… there is some alignment data, and also between this one and this one. But we start seeing refusals only here.

412
00:46:54.220 --> 00:46:55.750
Hadas Orgad: You can see after…

413
00:46:55.880 --> 00:47:08.189
Hadas Orgad: This is how much the refusal rate changed. So here it even got reduced, and it's not exactly refusals, these are keywords like illegal, can't give you, stuff like that. So here it even got reduced.

414
00:47:08.330 --> 00:47:10.260
Hadas Orgad: Here, it's increasing.

415
00:47:11.740 --> 00:47:19.040
Aruna Sankaranarayanan: How are you evaluating, both refusal as well as harmfulness? Like, are you using a judge model?

416
00:47:20.040 --> 00:47:21.530
Hadas Orgad: Refusals is keywords.

417
00:47:21.640 --> 00:47:22.830
Hadas Orgad: Just keywords?

418
00:47:23.250 --> 00:47:34.320
Hadas Orgad: whether it says something… so it's not exactly refusal, it's… it's also whether it's saying something like, this is illegal, I cannot, stuff like that. It might still give a very long answer, so it's… it's… it's like…

419
00:47:35.700 --> 00:47:44.500
Hadas Orgad: I don't want specifically refusals. I want, like, to see whether the model has this capability to discuss and say that something is illegal.

420
00:47:44.700 --> 00:47:54.309
Hadas Orgad: And the harmfulness is a classifier that was trained to give a score of harmfulness based off how useful the answer is to the request, to the harmful request.

421
00:47:54.310 --> 00:47:55.090
Aruna Sankaranarayanan: Okay.

422
00:47:58.320 --> 00:48:03.990
Hadas Orgad: Utility aside, does… Arm reduction occur…

423
00:48:04.620 --> 00:48:08.070
Hadas Orgad: I don't know, faster in the aligned…

424
00:48:08.180 --> 00:48:10.950
Hadas Orgad: Steps of the model than the base steps.

425
00:48:13.610 --> 00:48:17.060
Hadas Orgad: Fewer time steps of fine-tuning to…

426
00:48:17.450 --> 00:48:20.499
Hadas Orgad: get to the same level of… Wait, ask that again?

427
00:48:21.110 --> 00:48:25.229
Hadas Orgad: On the, like, If you're comparing, let's say, the ba… your…

428
00:48:25.780 --> 00:48:29.850
Hadas Orgad: your fine-tuning on the base model versus the SFT model?

429
00:48:32.350 --> 00:48:37.050
Hadas Orgad: Does harm reduction occur In fewer time steps of…

430
00:48:40.330 --> 00:48:44.930
Hadas Orgad: I'm not fine-tuning the… Oh, you're… you're pruning, sorry. Yeah.

431
00:48:48.370 --> 00:49:02.840
Hadas Orgad: Are you pruning the same number of weights on both steps? Probably not. I did not test how much. It's probably not exactly the same P and Q, or probably not exactly the same number, because I cared more about how much utility I lose.

432
00:49:03.230 --> 00:49:08.460
Hadas Orgad: The number of weights would probably tell you whether you're, like, how isolated those circuits are.

433
00:49:08.710 --> 00:49:14.429
Hadas Orgad: it's usually going together, like, the number of weights and how much utility you lose. Okay. So there is, like,

434
00:49:14.690 --> 00:49:19.990
Hadas Orgad: Between different models, it might actually… you're right, it might actually tell me something more.

435
00:49:21.060 --> 00:49:27.260
Hadas Orgad: My thought looking at that and looking at the graph that you were showing before is that the…

436
00:49:27.420 --> 00:49:40.800
Hadas Orgad: like, effectiveness of your ablation has to do with whether the circuit is… I mean, obviously, it has to do with whether the circuit is… Yeah, that's true, that's true. Between models, it's a good… models of the same architecture, you're right, it might be actually an interesting thing.

437
00:49:41.590 --> 00:49:43.080
Hadas Orgad: Thanks.

438
00:49:44.350 --> 00:49:56.350
Hadas Orgad: I know you said you're trying to finish this project, but I'm… It's fine, there are full-out projects. Yes, I'm very curious, because you did not, show anything on OMO3 Think.

439
00:49:56.610 --> 00:50:03.289
Hadas Orgad: On how this works for reasoning models. Also super curious, like, R1, distilled BAMA.

440
00:50:03.730 --> 00:50:18.770
Hadas Orgad: For example… R1 distilled llama. Yeah, perplexity, like, you know, distilled a llama model to, like, a small llama. I can only agree, this is a very interesting question, yeah, how these things happen, like, how they manifest in thinking models.

441
00:50:19.200 --> 00:50:23.530
Hadas Orgad: Yeah. So, the mid-train to the long context.

442
00:50:23.820 --> 00:50:27.309
Hadas Orgad: Do you see baseline refusal performance as difference?

443
00:50:27.640 --> 00:50:33.580
Hadas Orgad: Like, if you use the same refusal data set to test the… Actually, do refuse.

444
00:50:34.430 --> 00:50:35.410
Hadas Orgad: Say that again?

445
00:50:35.760 --> 00:50:41.630
Hadas Orgad: Between these two models. These two. Yeah, the mid-train and the long contact. Yeah.

446
00:50:42.070 --> 00:50:45.529
Hadas Orgad: the base models, are they good at,

447
00:50:45.710 --> 00:50:52.750
Hadas Orgad: Refusals? I mean, like… So this is how much the refusals change from the base, from the non-pruned? Not with your pruning.

448
00:50:52.920 --> 00:50:54.670
Hadas Orgad: Just the base model.

449
00:50:55.050 --> 00:51:06.469
Hadas Orgad: Meaning, like, without applying your algorithm. Yeah. Without pruning? Without pruning. Yeah. If I keep the models as it is.

450
00:51:06.620 --> 00:51:16.349
Hadas Orgad: and I pass them some harmful data sets. Yeah. What is the percentage that each of them refuse to answer? So, I don't have here the percentage, but that's the difference.

451
00:51:16.550 --> 00:51:23.620
Hadas Orgad: So it refuses less, or, like, has less of these keywords, Connected with refusal.

452
00:51:23.770 --> 00:51:26.329
Hadas Orgad: Here, and it has more beams here.

453
00:51:26.870 --> 00:51:29.069
Hadas Orgad: So, it is more aligned than Metroid.

454
00:51:29.760 --> 00:51:37.900
Hadas Orgad: Yeah, but you only see that… oh, you're asking if they're the same. They're pretty much the same before. Or even this one is even smaller.

455
00:51:38.340 --> 00:51:39.360
Hadas Orgad: before.

456
00:51:40.640 --> 00:51:42.630
Hadas Orgad: Oh. Yeah. So it's, it's…

457
00:51:43.230 --> 00:51:55.640
Hadas Orgad: It's less aligned than the midran? It's not really… it's hard to call this alignment, but it's kind of telling me which kind of things it learned from the data. Like, if it says it's illegal, I cannot, or…

458
00:51:56.010 --> 00:52:02.899
Hadas Orgad: not allowed, stuff like that. It means that it picked up something from the data related to alignment. I see. So it seems like this one…

459
00:52:02.990 --> 00:52:22.079
Hadas Orgad: before pruning actually shows a little bit more of that, but they're both very low, so I don't know how much you can… So, yeah, the reason I was asking is, you said we don't know if this is from alignment or more training. Yeah, this is the only thing that kind of starts to show the difference. Yeah, so I wonder if it's because

460
00:52:22.140 --> 00:52:28.319
Hadas Orgad: With long context, it also learned what alignment is, and during my training, it never really learned it.

461
00:52:28.570 --> 00:52:37.480
Hadas Orgad: Why would it learn in long context? I don't know, maybe it has more better data, high quality. I don't know what it is, but… Yeah, these are… Yeah, yeah, no questions.

462
00:52:37.770 --> 00:52:55.670
Hadas Orgad: Did you look at the, is there, like, a… like, a distribution shift in the data during the, in between those two? Because you can go look at the OMO paper and they tell you, like, what data we trained on. See exactly what data is there. I did not dig into every part, but it's, like, long context PDFs, I don't know. Maybe it's more high quality, maybe, right?

463
00:52:55.750 --> 00:52:59.209
Hadas Orgad: So it's like an emergent capability or something. Yeah.

464
00:52:59.480 --> 00:53:06.709
Hadas Orgad: I don't know. Maybe it's just further training on this… because it also has the training data inside the long context training.

465
00:53:07.600 --> 00:53:08.450
Hadas Orgad: I don't know.

466
00:53:09.180 --> 00:53:16.039
Hadas Orgad: But this is, like, the… the first thing… first thing that shows me some connection between refusal and that kind of,

467
00:53:16.540 --> 00:53:17.500
Hadas Orgad: trade-off.

468
00:53:19.060 --> 00:53:21.660
Hadas Orgad: Can I ask one more question about the plots? Yeah.

469
00:53:21.930 --> 00:53:30.539
Hadas Orgad: each point is just some random, running algorithm you do by just shifting your Q and K hyperparameters, right?

470
00:53:30.790 --> 00:53:34.990
Hadas Orgad: Like, what's the percentage of weights I removed? What percentage of data I have?

471
00:53:35.370 --> 00:53:39.479
Hadas Orgad: Yeah. So… And what's… how did you…

472
00:53:40.480 --> 00:53:51.120
Hadas Orgad: what… how is the harmfulness and the utility measured exactly? Are you doing any pre-fillings, or are you just looking at… So, it depends. In the plots that I showed you, I'm doing pre-filling on the harmfulness.

473
00:53:51.450 --> 00:53:57.239
Hadas Orgad: And the utility are not doing anything, I'm just squaring the same. And no fine children? No fine children. Oh.

474
00:54:00.500 --> 00:54:10.619
Hadas Orgad: So, maybe it's connected with the weight stuff, but it's also maybe dependent on the layer that you're pruning. Like, maybe when you train more, like, the last layer, or more, like.

475
00:54:11.090 --> 00:54:30.440
Hadas Orgad: beneficial or informative, and then if you will prune more there, like, I don't know, which layer is pruned in for different models. Oh, we're pruning everything, that's the… Yeah, but which is more effective, because you can see, like… There are probably some that are more effective. And maybe it's different between different models here, and maybe this could tell you something about, like, what

476
00:54:30.440 --> 00:54:47.929
Hadas Orgad: What changed between them? Yeah, maybe, like, class layers, because you just trained more, like I mentioned, or, like, some kind of alignment? Yeah, good question. Did you look at, like, whether there's a distribution shift in, like… you could do, like, an ANOVA, where the grouping of parameters is by layer, and you could see whether

477
00:54:49.040 --> 00:54:59.959
Hadas Orgad: whether you get, like, different scores depending on which layer you're on, or whether it's all kind of, like, drawn from the same distribution, you know what I mean? No, I don't understand. Like,

478
00:55:00.400 --> 00:55:09.270
Hadas Orgad: Like, if you're… If you're… if you're wondering whether there's a difference in, in, like,

479
00:55:09.850 --> 00:55:19.520
Hadas Orgad: which… which layers have… if, like, Layer 1 has more parameters that… that are related to harmfulness than, like, layer 10, compared to, like.

480
00:55:19.660 --> 00:55:27.170
Hadas Orgad: parameters inside of Layer 1, then if you just did it in ANOVA, where the group is by layers, then you could look at… you could look at whether,

481
00:55:27.420 --> 00:55:31.380
Hadas Orgad: Whether, like, there's a… there's a difference in, like.

482
00:55:31.820 --> 00:55:37.550
Hadas Orgad: Between group versus within group. Like…

483
00:55:37.820 --> 00:55:42.380
Hadas Orgad: I would need you to explain me that later. Okay, yeah. Okay.

484
00:55:43.810 --> 00:55:49.910
Hadas Orgad: Are you… how are you, like, thinking of the purpose of these weights? Are they, like… the wigs that…

485
00:55:50.120 --> 00:55:57.409
Hadas Orgad: Say it's more important to answer the user's question of all weights that you're getting rid of, or the…

486
00:55:57.530 --> 00:56:03.259
Hadas Orgad: if I've started my answer, I should finish it, kind of waits, like, what do you think if you're removing those?

487
00:56:03.690 --> 00:56:07.729
Hadas Orgad: What is, like, the… Do you think their, like, purpose is?

488
00:56:08.120 --> 00:56:10.330
Hadas Orgad: The ways that we're moving.

489
00:56:10.440 --> 00:56:14.249
Hadas Orgad: It's… I think it's the ways that the, like.

490
00:56:14.430 --> 00:56:18.690
Hadas Orgad: The information flows through as a bottleneck for generating.

491
00:56:21.120 --> 00:56:31.009
Hadas Orgad: I know it's… it's 11, but I know that sometimes it's 90 minutes to what we want to do, or… because we have an entire… another entire section, but we can only also finish now.

492
00:56:31.500 --> 00:56:39.700
Hadas Orgad: Let's, let you present a little bit more. If people have to leave, then we can take a minute break to people.

493
00:56:40.060 --> 00:56:42.029
Hadas Orgad: Do people have to go for oats?

494
00:56:44.040 --> 00:56:50.120
Hadas Orgad: Okay, yes, so, but let's, let's accelerate so that you can get through the rest. Yeah.

495
00:56:50.500 --> 00:56:53.949
Hadas Orgad: No, actually, half an hour is perfect for it, so, yeah.

496
00:56:55.570 --> 00:56:56.840
Hadas Orgad: Okay, now…

497
00:56:57.050 --> 00:57:02.219
Hadas Orgad: This actually relates to a work that we talked about, that you were involved in. Yeah, this is my question. Yeah.

498
00:57:02.450 --> 00:57:05.620
Hadas Orgad: And it relates to some questions that were asked here.

499
00:57:06.260 --> 00:57:11.749
Hadas Orgad: So I keep saying that we're only pruning away the generation capabilities.

500
00:57:11.860 --> 00:57:20.930
Hadas Orgad: And the reason that I'm saying that is that we tested how does doing that, pruning the generation capabilities affect other aspects of understanding harmfulness?

501
00:57:21.300 --> 00:57:22.090
Hadas Orgad: So…

502
00:57:22.420 --> 00:57:32.500
Hadas Orgad: We came up with, like, 4 aspects, one of them is generation, but also refusal, just the ability to decline any harmful requests, say I cannot answer.

503
00:57:32.680 --> 00:57:38.560
Hadas Orgad: explanation, Given this request, you do not need to answer, just tell me why it's harmful.

504
00:57:39.040 --> 00:57:42.429
Hadas Orgad: Then, to evaluate that, we used a judge model.

505
00:57:43.040 --> 00:57:44.270
Hadas Orgad: detection.

506
00:57:44.550 --> 00:57:48.129
Hadas Orgad: Which is, given that request, is it harmful, yes or no?

507
00:57:48.770 --> 00:57:54.720
Hadas Orgad: And for each one of these capabilities, we prune away the capability, we see how it affects the other capabilities.

508
00:58:00.440 --> 00:58:02.530
Hadas Orgad: This is the results on two models.

509
00:58:02.930 --> 00:58:04.999
Hadas Orgad: Let's go through it.

510
00:58:05.110 --> 00:58:08.270
Hadas Orgad: Just the top highlights here, step by step.

511
00:58:08.960 --> 00:58:18.370
Hadas Orgad: So first of all, we see that pruning the generation capabilities leaves the other capabilities Except here?

512
00:58:18.510 --> 00:58:22.080
Hadas Orgad: But in most cases, it leaves them pretty much intact.

513
00:58:23.150 --> 00:58:27.550
Hadas Orgad: So, of course, the refusal goes up when we prun away harmful generation, but apart from that.

514
00:58:27.790 --> 00:58:30.900
Hadas Orgad: It's not affecting too much the other capabilities.

515
00:58:32.320 --> 00:58:36.419
Hadas Orgad: the fact that in Quen, it does affect the detection.

516
00:58:36.550 --> 00:58:42.490
Hadas Orgad: Means something about how these mechanisms are, like, a difference between how these mechanisms are encoded.

517
00:58:43.130 --> 00:58:51.760
Hadas Orgad: And I really wish in follow-up work, to actually try to investigate mechanistically what this means, whether it means that

518
00:58:52.120 --> 00:59:00.399
Hadas Orgad: detection appears like, the mechanism appears in an earlier layer than the generation in WAN versus LAMA.

519
00:59:01.990 --> 00:59:06.889
Hadas Orgad: Another thing to notice is that when we prune away refusal behavior.

520
00:59:07.220 --> 00:59:10.289
Hadas Orgad: Of course, the harmful generation goes up, that's not interesting.

521
00:59:10.610 --> 00:59:15.449
Hadas Orgad: But we do see that we heard explanation capabilities in both models.

522
00:59:16.140 --> 00:59:18.400
Hadas Orgad: And that means that there is something…

523
00:59:18.620 --> 00:59:21.570
Hadas Orgad: I think about it as there is something more

524
00:59:21.830 --> 00:59:27.690
Hadas Orgad: Inherent to understanding or, like, knowing something about harmful concepts.

525
00:59:28.100 --> 00:59:31.690
Hadas Orgad: That is shared between refusal and explanations.

526
00:59:33.060 --> 00:59:40.510
Hadas Orgad: I can tell you, like, a nice, anecdote, that when we pruned Refusal in Llama, I think?

527
00:59:41.270 --> 01:00:00.139
Hadas Orgad: we suddenly saw the explanation about why something is harmful, or something like, this request is harmful because, it's grammatically incorrect. It's not clear what a pirate software is here. Is this a pirate, real pirate? Or something else? Like, it keeps explaining, like.

528
01:00:00.600 --> 01:00:11.310
Hadas Orgad: this was the most interesting case. Like, in other cases, it might have just lost a lot of the coherency, but sometimes you see that you might have really heard something fundamental in the model's understanding.

529
01:00:13.710 --> 01:00:19.960
Hadas Orgad: Yeah, when we prune away explanation, which seems to be, like, the most fundamental thing.

530
01:00:20.190 --> 01:00:31.529
Hadas Orgad: Again, we see a very strong difference between models. So for LAMA, it completely hurts all of the other capabilities. So it just… the model becomes incoherent in all of these different capabilities.

531
01:00:31.740 --> 01:00:37.759
Hadas Orgad: And again, it's given at this point, but when we prune, we always make sure that the benign capabilities stay the same.

532
01:00:38.170 --> 01:00:42.640
Hadas Orgad: Whereas for Quan, it seems much more separable.

533
01:00:44.700 --> 01:00:50.019
Hadas Orgad: For LAMO, we couldn't prune detection. It was just completely…

534
01:00:50.260 --> 01:00:56.739
Hadas Orgad: Completely, destroying the model, so we couldn't get to a point that we reduced detection without hurting the model's utility.

535
01:00:57.620 --> 01:01:12.280
Hadas Orgad: And for Quan, we could do that, and then it really affected the harmful generation. So you see that there are, like… the takeaway here is that, first of all, harmful generation seems very separable from other things, like, you can prune it this way without hurting other things.

536
01:01:12.720 --> 01:01:18.729
Hadas Orgad: That the interactions show us Fundamental mechanistic difference between these models.

537
01:01:19.230 --> 01:01:22.699
Hadas Orgad: So the way that these capabilities interact is very different.

538
01:01:23.850 --> 01:01:29.060
Hadas Orgad: And another thing that we actually don't see here is that we found that

539
01:01:29.220 --> 01:01:40.149
Hadas Orgad: refusal serves as kind of a gating mechanism. So, what we saw that happened in many cases is that, let's say we prune away explanations.

540
01:01:40.740 --> 01:01:44.110
Hadas Orgad: The model doesn't… Immediately becomes incoherent.

541
01:01:44.400 --> 01:01:46.390
Hadas Orgad: Just starts refusing everything.

542
01:01:46.520 --> 01:01:48.819
Hadas Orgad: So it's kind of like when you prune something.

543
01:01:49.260 --> 01:01:54.060
Hadas Orgad: the model falls back to refusal. So the refusal is more of, like, I…

544
01:01:54.240 --> 01:02:02.349
Hadas Orgad: from what I see, I have the feeling that it's kind of like a stupid detector. Given that it sees a harmful request, it just shoots, I cannot answer.

545
01:02:02.980 --> 01:02:05.130
Hadas Orgad: It doesn't matter what the context is.

546
01:02:05.240 --> 01:02:11.089
Hadas Orgad: But if the other mechanism that understands what it needs to do, like explaining or answering or whatever.

547
01:02:11.370 --> 01:02:15.719
Hadas Orgad: It's still on, it's gonna override the refusal gate.

548
01:02:15.970 --> 01:02:19.710
Hadas Orgad: But when it goes away, then the model just falls back to refusals.

549
01:02:19.940 --> 01:02:28.949
Hadas Orgad: So a lot of the things that we needed to do to do this analysis is to actually go around the refusal gate that suddenly fired after we pruned the capability.

550
01:02:29.740 --> 01:02:32.210
Hadas Orgad: Do you see the behavior of both models?

551
01:02:32.720 --> 01:02:33.540
Hadas Orgad: Yes.

552
01:02:33.790 --> 01:02:38.049
Hadas Orgad: Yeah. I think it was less in Gwen, but…

553
01:02:38.510 --> 01:02:48.049
Hadas Orgad: Slightly less in Gwen, but, both of them… I was very good with Lana 3, and I put a lot of gibberish into it. Like, I think when they trade Lama 3, they…

554
01:02:48.150 --> 01:02:55.930
Hadas Orgad: kind of intentionally trained it with noise, so that when it sees noise that it can't understand, it defaults to refusal. I've seen it

555
01:02:55.940 --> 01:03:02.480
Hadas Orgad: Interesting. Oh, where did you see it? Oh, just with my experiments, like…

556
01:03:02.480 --> 01:03:20.200
Hadas Orgad: Like, tokens that don't make sense. When you put gibberish in, it just refuses. It likes to tend to refuse, whereas, like, in level 2, it would have tried to unlock, so I thought they expected some kind of jailbreaks through, like, gibberish, so… Yeah, that's a very good point.

557
01:03:20.280 --> 01:03:26.280
Hadas Orgad: And we know Quen is subject to some Chinese laws about Evaluation and trending.

558
01:03:28.020 --> 01:03:47.240
Hadas Orgad: How do you think specifically it affects the… I mean, I know they have specific laws around the types of safety trainings and refusal trainings that they must undergo, and I think it's a benchmark of, like, 90% or something like that. So they have, like, a mandated percentage on, like, evaluating certain refusals based on,

559
01:03:47.630 --> 01:03:49.819
Hadas Orgad: And then there's some law that was passed.

560
01:03:49.930 --> 01:03:59.060
Hadas Orgad: On the refusal rates for certain topics. Interesting. So I don't know if maybe that means they baked that in earlier because of the law, or…

561
01:03:59.070 --> 01:04:13.950
Hadas Orgad: Yeah, so we… okay, we know that it's baked in earlier in the pre-training. I don't know, but it's… No, no, we know from… from the results I showed that it refuses even in the pre-trained model. It is… the refusals of Quan are generally much more robust, yeah.

562
01:04:17.260 --> 01:04:34.569
Hadas Orgad: I have two suggestions about this experiment. Yes. So, you know, the differences between what you have in Llama and Clan, there's two hypotheses, right? One hypothesis is that it's revealing a difference in the way these model families are trained, and which is, you know, what you're advocating. Yeah. And then the other is that…

563
01:04:34.760 --> 01:04:44.189
Hadas Orgad: this methodology is so flaky, it's just very inconsistent, right? And so, I think that one way of strengthening result is if you've tested the whole

564
01:04:44.410 --> 01:04:45.680
Hadas Orgad: Llama family.

565
01:04:46.300 --> 01:04:55.680
Hadas Orgad: And it gave, you know, similar matrices to the whole family, you know, for different model sizes, let's say. Yeah. And you trust… that's the whole quant family.

566
01:04:55.820 --> 01:05:05.299
Hadas Orgad: and… and it gives similar matrices through the whole family, then that's a lot more compelling than, oh yeah, actually, this is a very consistent… Yeah, that's true. …way of measuring, right?

567
01:05:05.490 --> 01:05:25.290
Hadas Orgad: That makes sense, and it's really something different about how these models are treated. Yeah, we wanted to do the same thing for the $32 billion. We probably should. It was just such heavy experiments that, yeah, at some point we… we said, okay. But yeah, I agree. It's a good point. So, the other thing I'm interested in… well, so maybe I should just let you go. The…

568
01:05:25.290 --> 01:05:36.910
Hadas Orgad: is… so… There's not a lot more, so you can go. Yeah, so the… the students were asking about, oh, what layers are things happening on, and so on, and so… and the question came up.

569
01:05:37.030 --> 01:05:41.239
Hadas Orgad: Well, how does this relate to all the people who have been analyzing activations?

570
01:05:41.410 --> 01:05:46.930
Hadas Orgad: So I think that another, really interesting coordinate is what tokens

571
01:05:47.180 --> 01:05:57.640
Hadas Orgad: or what token representations are different. So when you have a statement of something that's talking about, making harmful requests or something like that, there are these different moments

572
01:05:57.770 --> 01:06:14.430
Hadas Orgad: when different, types of computations seem to happen at different tokens, like, after you describe the harmful thing at the period, before you've actually asked the model to do something or something like it already has some assessment of whether it's harmful or not. And,

573
01:06:14.550 --> 01:06:18.579
Hadas Orgad: And so, you know, you can… you can… what you could do is you could actually

574
01:06:18.940 --> 01:06:22.089
Hadas Orgad: Look at when you prune the weights.

575
01:06:22.360 --> 01:06:25.900
Hadas Orgad: You know, is the representation at that moment different?

576
01:06:26.060 --> 01:06:36.530
Hadas Orgad: Or is representation at a later moment different when you actually make a request? Right? And things like that, right? And it would be interesting to compare these different types of prunings to see

577
01:06:36.820 --> 01:06:56.499
Hadas Orgad: you know, where does it shift around? Because your work show that there's a big difference between the representations, yeah. It was when it sees the request, and when it's asked to answer or something, right? To do something, right? Yeah. And so, yeah, so the refusal, obviously, is very strong when you ask it to do something, but it…

578
01:06:56.520 --> 01:07:00.669
Hadas Orgad: Switches… switches from user to assistant.

579
01:07:00.860 --> 01:07:05.269
Hadas Orgad: And the assistant actually has to now do something, then all the refusal circuits are

580
01:07:05.580 --> 01:07:15.429
Hadas Orgad: are, you know, or all that, you know, activations seem to be about making a refusal decision. But much earlier, when you just talk about the dangerous thing.

581
01:07:15.740 --> 01:07:19.349
Hadas Orgad: It's already making a decision about what kind of

582
01:07:20.260 --> 01:07:34.749
Hadas Orgad: semantics is this. Is this a harmful thing? Is it not a harmful thing? Yeah, it's probably already aware of whether… not aware. Right, but it's kind of a different kind of decision that it's making. That's true. It's not actually making a decision, more like reasoning about it. It's just assessing what's going on, yeah.

583
01:07:34.940 --> 01:07:35.720
Hadas Orgad: Yeah.

584
01:07:36.390 --> 01:07:41.930
Hadas Orgad: So, anyway, I think that might be an interesting way of triangulating and connecting it to the

585
01:07:42.050 --> 01:07:45.049
Hadas Orgad: other work, you know. Yeah.

586
01:07:45.380 --> 01:07:47.029
Hadas Orgad: Yeah, okay.

587
01:07:47.140 --> 01:07:47.810
Hadas Orgad: Boom.

588
01:07:56.020 --> 01:08:14.640
Hadas Orgad: Cool, more questions. Oh, yeah. Sorry, so I… do I understand this right, that if you prune the weights for detection, then the model doesn't generate any harmful content? Is that how to read the bottom one foot? Yes, it loses the coherency completely.

589
01:08:15.190 --> 01:08:17.609
Hadas Orgad: On the harmful request.

590
01:08:17.950 --> 01:08:18.630
Hadas Orgad: Good.

591
01:08:19.720 --> 01:08:21.850
Hadas Orgad: Even more than pernea detection.

592
01:08:27.390 --> 01:08:28.130
Hadas Orgad: Okay.

593
01:08:28.399 --> 01:08:30.340
Hadas Orgad: Well, that's just summary.

594
01:08:31.760 --> 01:08:45.779
Hadas Orgad: So… well, not the summary, but first implications. So, I hope that the fact that we know that there is, like, a structural basis to harmful behaviors, and of course there's still a lot more to try to understand there.

595
01:08:46.270 --> 01:08:53.469
Hadas Orgad: But it does make me optimistic that we can make more, robust.

596
01:08:53.930 --> 01:08:56.900
Hadas Orgad: Defenses against jailbreaks, or…

597
01:08:57.029 --> 01:09:05.479
Hadas Orgad: Generally more robust behaviors, that are actually based on understanding what this concept is, what it might

598
01:09:05.930 --> 01:09:06.840
Hadas Orgad: do.

599
01:09:07.260 --> 01:09:09.990
Hadas Orgad: And not just, you know, shortcuts.

600
01:09:10.170 --> 01:09:13.640
Hadas Orgad: Yeah.

601
01:09:15.160 --> 01:09:19.089
Hadas Orgad: Okay, so just to summarize everything we saw, we saw that harmful generations

602
01:09:19.729 --> 01:09:35.019
Hadas Orgad: are compressed into a compact and unified set of weights in the model that is diff- is different from benign capabilities. We saw that this generalizes, so pruning one mechanism affects other mechanisms of different types of harmfulness.

603
01:09:35.359 --> 01:09:39.690
Hadas Orgad: This suggests an explanation for emergent misalignment.

604
01:09:39.950 --> 01:09:43.609
Hadas Orgad: And we also saw some intervention that reduces it.

605
01:09:44.890 --> 01:09:51.720
Hadas Orgad: And we saw, you know, the main thing that I'm taking from the matrix is that generation is…

606
01:09:52.000 --> 01:09:55.959
Hadas Orgad: Different from the model's understanding of what harmfulness is.

607
01:09:58.060 --> 01:10:05.890
Hadas Orgad: Okay. Just a question, so I don't expect any more questions. If you could go back to the slide of OMOT.

608
01:10:06.410 --> 01:10:09.450
Hadas Orgad: I'm sure that your experiments would own. Yes.

609
01:10:09.610 --> 01:10:13.420
Hadas Orgad: My favorite one. Yeah.

610
01:10:13.530 --> 01:10:25.129
Hadas Orgad: That's actually the next one, yeah. So, I'm interested about the refusal to change in this slide, and I think it relates a lot with your finding from the next slide about

611
01:10:25.130 --> 01:10:42.629
Hadas Orgad: the fact that refusal acts as a gate, right? So I was wondering, like, why are scores positive for these models specifically? Like, so strongly positive, for example, for the instruct model, right? But very negative for the initial checkpoints, right? And I was thinking that maybe…

612
01:10:43.260 --> 01:10:52.009
Hadas Orgad: Like, the fact that, in the instruction tuning, we hammer, you know, into the weights, the fact that, oh, the refusal template should match some

613
01:10:52.010 --> 01:11:06.169
Hadas Orgad: specific form, right? Yeah, it's almost the same answer, I think. They're, like, very similar. So, the model kind of internalizes this as some form of, like, opposite behavior to the harmful one, right? Like, the moment that you prune the harmful one…

614
01:11:06.170 --> 01:11:13.889
Hadas Orgad: then the model just jumps to the, you know, refusal behavior as the norm, right? While in the pre-training.

615
01:11:13.940 --> 01:11:26.720
Hadas Orgad: it doesn't quite know how to refuse in the expected way. Yes, yes. Does it make sense? It does make sense. That's basically my hypothesis. The only problem is that it's not completely shown in the results.

616
01:11:26.750 --> 01:11:34.020
Hadas Orgad: Here you… okay, I cannot tie it back to the training. I can't tie it back to the behavior. Yeah, I agree.

617
01:11:34.380 --> 01:11:35.970
Hadas Orgad: It's very cool. Yeah.

618
01:11:36.220 --> 01:11:36.950
Hadas Orgad: Thanks.

619
01:11:40.690 --> 01:11:43.039
Hadas Orgad: Okay, thank you. That was great.

620
01:11:45.820 --> 01:11:48.480
Hadas Orgad: People write something in the document?

621
01:11:49.490 --> 01:12:01.140
Hadas Orgad: all of the suggestions that were raised here today? Oh, and no, was it a document? It was a document. I didn't realize you wanted to… They shared it. Oh, no.

622
01:12:01.230 --> 01:12:12.989
Hadas Orgad: It is… Okay, I'll write everything… oh, it's recorded, okay. Yeah. I'll try to also write everything now. Thanks so much. Yeah, yeah, yeah.

623
01:12:13.680 --> 01:12:17.320
Hadas Orgad: Also, I'm gonna stop this here.