WEBVTT

1
00:02:05.700 --> 00:02:07.820
michaeltoker: I will unmute myself.

2
00:02:12.350 --> 00:02:13.160
michaeltoker: Okay.

3
00:02:13.910 --> 00:02:18.069
michaeltoker: So, thank you very much, David. I'm really happy to be here today.

4
00:02:18.470 --> 00:02:30.610
michaeltoker: I met David, actually, a few years ago when he visited Israel, and since then, I heard a lot about the work from your group, and I'm really happy to finally be here and meet all of you in person.

5
00:02:31.500 --> 00:02:41.180
michaeltoker: I would like for this to be a conversation, so feel free to stop me at any slide. We can talk about things, thoughts,

6
00:02:41.750 --> 00:02:45.419
michaeltoker: And I would like to start with a thought experiment.

7
00:02:46.010 --> 00:02:52.620
michaeltoker: So, in general, I'm a really visual thinker, so a lot of my thoughts and memories are in images.

8
00:02:53.160 --> 00:02:58.390
michaeltoker: And I would like you to try to think about the first time you met David.

9
00:02:58.920 --> 00:03:05.920
michaeltoker: Maybe some of you will remember the, sorry

10
00:03:06.300 --> 00:03:11.619
michaeltoker: Maybe some of you will remember the, conversation that you had

11
00:03:11.730 --> 00:03:14.960
michaeltoker: Maybe some ideas that you talked about?

12
00:03:15.160 --> 00:03:22.530
michaeltoker: When I think about David, I can literally see the class that we've been in the Technion.

13
00:03:22.650 --> 00:03:31.380
michaeltoker: And David talked about these neurons that you can turn on and off, and would turn on the lamp into the model.

14
00:03:31.690 --> 00:03:35.449
michaeltoker: So this is, like, the picture that I see in my mind.

15
00:03:35.860 --> 00:03:38.450
michaeltoker: And I was wondering whether,

16
00:03:38.670 --> 00:03:42.329
michaeltoker: Neural networks also can think in images.

17
00:03:43.380 --> 00:04:00.130
michaeltoker: So, some works during the generations of neural networks tried to think about similar concepts. I guess most of you are familiar with this, work, From When Were You? when they, wanted to understand what CNNs learn.

18
00:04:00.390 --> 00:04:17.269
michaeltoker: So they looked at activations at… so they proposed a method to inverse the CNN encoding process, and they looked at activations after different layers, then projected it back into the pixel space.

19
00:04:17.570 --> 00:04:33.310
michaeltoker: And they found that early layers encode concepts, so they detect concepts like simple edges and shapes, while later layers detect high-level concepts, such as a dog's face.

20
00:04:34.350 --> 00:04:43.710
michaeltoker: And in my research, I'm trying to do something similar with modern neural networks, with language models and text-to-image models.

21
00:04:43.990 --> 00:04:48.100
michaeltoker: In order to understand what knowledge they encode.

22
00:04:48.230 --> 00:04:59.130
michaeltoker: What different mechanisms there are during the encoding, and to use that to understand why and where biases and hallucinations come from.

23
00:05:00.120 --> 00:05:08.609
michaeltoker: So, in general, I… my research is driven by the curiosity about how different architectural decisions

24
00:05:08.780 --> 00:05:21.150
michaeltoker: Shape how the information flow through the model, and how the… what the presentations are learned, and how those impact the downstream limitations and capabilities of those models.

25
00:05:22.560 --> 00:05:31.680
michaeltoker: And I chose text-to-image models because I think that they are an interesting testbed to explore how text is encoded.

26
00:05:31.830 --> 00:05:38.450
michaeltoker: How many of you, know how text-to-image structure, how they work?

27
00:05:39.460 --> 00:05:42.119
michaeltoker: Okay, so I'll give a brief overview.

28
00:05:42.240 --> 00:05:47.260
michaeltoker: So in general, text-image models are made of two main components. You have a text encoder.

29
00:05:47.350 --> 00:06:05.190
michaeltoker: And you have a diffusion model. The text encoder is encoded through the text encoder, and the full textual presentations, all of the tokens, are passed to the diffusion model. This model is nowadays also a transformer, and… but it's applied iteratively

30
00:06:05.190 --> 00:06:24.480
michaeltoker: It's starting with some latent noise that is later refined again and again to predict some noise. This noise is used to update the image, and this is done again and again until the final representation is decoded into the final image.

31
00:06:24.770 --> 00:06:28.529
michaeltoker: So this is the general pipeline of text-to-image models.

32
00:06:29.050 --> 00:06:37.419
michaeltoker: In particular, I'm interested in the text encoder, and I use the diffusion model as a probe to understand what's happening inside the text encoder.

33
00:06:38.780 --> 00:06:42.040
michaeltoker: And I'm interested in two dimensions, so,

34
00:06:42.640 --> 00:06:48.380
michaeltoker: Looking at the information flow across the layers and across the tokens.

35
00:06:50.050 --> 00:07:03.439
michaeltoker: Okay, so the first work that I'm going to talk to you about is called Diffusion Lens. In this work, we look on the information flow between the tokens of the text encoder.

36
00:07:04.890 --> 00:07:07.729
michaeltoker: So I guess you're all familiar with Logit Lands.

37
00:07:07.970 --> 00:07:20.789
michaeltoker: I'll mention briefly that in logic lens, to understand the meaning of a token, you pass it through the final layer norm, and then use the language model head to get what probabilities.

38
00:07:20.830 --> 00:07:33.440
michaeltoker: And while this is very useful for many, many implications, you can only… you can only see textual representations, right? You see the word probabilities.

39
00:07:33.550 --> 00:07:36.869
michaeltoker: But what if you want to understand,

40
00:07:37.640 --> 00:07:43.610
michaeltoker: different visual concepts. For example, whether the model already knows how Bill Gates looks like.

41
00:07:44.020 --> 00:07:48.679
michaeltoker: So we propose to replace the language model by a diffusion model.

42
00:07:49.250 --> 00:07:53.019
michaeltoker: And the process is very similar.

43
00:07:53.720 --> 00:07:58.030
michaeltoker: So, actually we find that if we…

44
00:07:59.210 --> 00:08:18.679
michaeltoker: that if we look at images generated from different layers, the images make sense. We can understand them, and they… it's something that we can learn from. So, for example, for Bill Gates, you can see that it starts the model is generating some kind of gates.

45
00:08:18.850 --> 00:08:29.310
michaeltoker: And then it… and again, then it understands how to combine it into the correct person, and retrieve the features until you get the correct representation.

46
00:08:30.030 --> 00:08:32.440
michaeltoker: So, equipped with this tool.

47
00:08:35.110 --> 00:08:38.050
michaeltoker: What can we learn about the texture of encoding?

48
00:08:38.580 --> 00:08:45.850
michaeltoker: So, the first thing that we see is, has to do with memory retrieval. The, process of

49
00:08:45.960 --> 00:08:51.040
michaeltoker: Getting visual information from the model weights.

50
00:08:51.790 --> 00:09:02.779
michaeltoker: So, we looked at this… at these two cases, where you have very similar prompts, but with entities that are different in the,

51
00:09:03.070 --> 00:09:07.110
michaeltoker: In how they,

52
00:09:08.100 --> 00:09:19.499
michaeltoker: in the distribution of the data. So, a beagle is a very familiar concept that you can see a lot in the data, but in AI, I guess most of you do not… are not familiar with this animal.

53
00:09:19.630 --> 00:09:27.519
michaeltoker: So these images are generated from the final image, so from the final presentation, so you know that the model

54
00:09:27.520 --> 00:09:40.389
michaeltoker: familiar with those two concepts. Now, I would like to hear your thoughts. What do you think that you would see if we decode those earlier layers for each of those concepts?

55
00:09:42.770 --> 00:09:46.389
michaeltoker: Yeah. Is the beagle familiar earlier earlier than the II?

56
00:09:46.860 --> 00:09:53.769
michaeltoker: Right, yeah, exactly. So, we see that for familiar concepts, they appear much earlier.

57
00:09:53.880 --> 00:10:13.140
michaeltoker: And for unfamiliar concepts, it takes much more time. So, I think that we think that it's telling us something about how the knowledge is encoded and distributed in this model, and you can see that the familiar concepts are,

58
00:10:13.230 --> 00:10:19.249
michaeltoker: Probably encoded in more places, so we can decode it from many places.

59
00:10:20.040 --> 00:10:29.970
michaeltoker: We… we do it across different, dataset of familiar and unfamiliar animals, and we see this pattern, repeated.

60
00:10:30.910 --> 00:10:50.770
michaeltoker: Now, the next thing that we look at is… What was the metric you use on the y-axis here? Yes, so, we use the percentage of images that had a high clip score between the images and the prompt. Like a code-based classifier, almost. Right, yeah, yeah.

61
00:10:52.270 --> 00:11:12.169
michaeltoker: Now, the next thing that we look at is about prompt complexity, and specifically, we look at attribute binding. So, attribute binding is the process of taking several concepts and fusing them together. For example, the pink… the color pink and the snail becoming into a pink snail.

62
00:11:12.450 --> 00:11:19.410
michaeltoker: And we look at prompts with different number of entities and bindings.

63
00:11:19.730 --> 00:11:22.680
michaeltoker: As a proxy for prompt complexity.

64
00:11:23.120 --> 00:11:27.390
michaeltoker: And, we see that the more complex the prompt.

65
00:11:27.700 --> 00:11:40.959
michaeltoker: it takes more layers for the model to understand how to combine the entities together. And, this, this is a relatively, ancient model in the terms of today. It's,

66
00:11:41.140 --> 00:11:45.139
michaeltoker: Civil Diffusion, 3.

67
00:11:45.380 --> 00:11:55.380
michaeltoker: So, the… you can see that for the pink snail and orange donut, it takes almost all of the layers to… to… to know how to combine the 20s together.

68
00:11:56.250 --> 00:11:56.960
michaeltoker: Yeah.

69
00:11:57.450 --> 00:12:00.449
michaeltoker: I mean, maybe these are just two examples, but have you…

70
00:12:00.800 --> 00:12:14.659
michaeltoker: Like, it would seem that the early layers produce, like, sketches. Yes, yes, yes. This is a good question. Actually, I saw this… this behavior. We didn't talk about it in the paper.

71
00:12:14.670 --> 00:12:25.919
michaeltoker: I don't know if it's something related to the distribution of the early presentations, or if there is some reason for that, but yes, I did see it.

72
00:12:29.740 --> 00:12:35.109
michaeltoker: Okay, so this is the fusion lens. Do you have any questions so far?

73
00:12:37.230 --> 00:12:37.920
michaeltoker: Okay.

74
00:12:38.880 --> 00:12:44.880
michaeltoker: So, in the next work, I look at the information flow between tokens.

75
00:12:45.480 --> 00:12:51.100
michaeltoker: so the x-axis, and specifically, I look into the information flow of

76
00:12:51.110 --> 00:13:06.879
michaeltoker: specific tokens, the padding tokens. So, I already told you about the text-to-image pipeline, but what I didn't tell you is that the text is not inserted as is. Usually, the text is padded into a,

77
00:13:06.920 --> 00:13:17.070
michaeltoker: a constant length. This is done for efficient batch decoding, but unlike in language models, those betting tokens are not masked.

78
00:13:17.190 --> 00:13:28.080
michaeltoker: So the model can learn to use those tokens. And our question here was, does the text encoder really uses those padding tokens?

79
00:13:28.370 --> 00:13:37.649
michaeltoker: So, in order to get some intuition, we started by just removing those tokens and generating the image only from the textual tokens.

80
00:13:37.850 --> 00:13:45.909
michaeltoker: So, you can see here the prompt, and here are the two images. Which of the images do you think are generated without the fading tokens?

81
00:13:48.610 --> 00:13:49.300
michaeltoker: Good.

82
00:13:49.740 --> 00:13:57.019
michaeltoker: The left one, right. So, yeah, so we saw that images generated without pending tokens,

83
00:13:57.150 --> 00:14:04.039
michaeltoker: Are both visually, they look worse, and also they are less aligned with the prompt.

84
00:14:04.640 --> 00:14:09.969
michaeltoker: So we thought that there is something there, but we wanted to understand exactly where the tokens are used.

85
00:14:11.160 --> 00:14:18.520
michaeltoker: So, we proposed a method, a patching method, to look at this phenomenon.

86
00:14:18.690 --> 00:14:24.680
michaeltoker: So, what we do is we encode, the prompt with the pads.

87
00:14:24.890 --> 00:14:29.050
michaeltoker: And then we do another encoding of… Only beds.

88
00:14:29.220 --> 00:14:34.630
michaeltoker: So, this encoding has no meaningful semantic information about the prompt.

89
00:14:35.180 --> 00:14:46.779
michaeltoker: Then we look at the pairing tokens that we are interested in interpreting, and we, replace the textual tokens with the clean pets tokens, and we,

90
00:14:47.330 --> 00:14:51.080
michaeltoker: Generate an image conditioned on that mixer presentation.

91
00:14:51.880 --> 00:14:59.060
michaeltoker: If you have any questions about that, because it would be repeated a few times now.

92
00:14:59.230 --> 00:15:05.940
michaeltoker: Is there no masking anywhere? Yes, it's self-attention, everything sees everything, yeah.

93
00:15:07.710 --> 00:15:20.360
michaeltoker: So this is the final layer representation of a multi-layer and transformer. Right. And you go to the… so this is a visualization of the final layer representation. Yes. You just go to the…

94
00:15:20.740 --> 00:15:23.229
michaeltoker: Tokens that corresponded to the text.

95
00:15:23.440 --> 00:15:31.660
michaeltoker: And you overwrite them with this nonsense. Right. But then the padding tokens because they're the last layer of the transformer. Yes.

96
00:15:31.800 --> 00:15:41.270
michaeltoker: contextual knowledge… Exactly. It's a very clear picture, I think. It's a nice visualization. Thank you. Yes, thanks.

97
00:15:42.270 --> 00:15:45.569
michaeltoker: Yeah. What does the image look like from just the panic?

98
00:15:46.670 --> 00:15:51.730
michaeltoker: Like, if you, if you decode the orange, the pure orange. The pure orange?

99
00:15:51.860 --> 00:15:56.810
michaeltoker: It would look like a reasonable, random thing, depending on the seat.

100
00:15:58.380 --> 00:16:01.700
michaeltoker: But it's, like, it's not, like, blank image, it's, like, an app. Yes.

101
00:16:03.350 --> 00:16:15.380
michaeltoker: So I think that this is because, like, the model is… the model distribution is generating natural images, so if it's not give any context, just natural random image. Yeah, whatever, whatever prior…

102
00:16:16.350 --> 00:16:17.140
michaeltoker: Yep.

103
00:16:17.730 --> 00:16:21.609
michaeltoker: I need to explore the whole fire.

104
00:16:22.130 --> 00:16:23.830
michaeltoker: Yeah.

105
00:16:26.080 --> 00:16:26.520
Rohit Gandikota: Hey, Michael.

106
00:16:26.520 --> 00:16:42.390
michaeltoker: Now, these are the results on one model. As you can see on the left, the original image for reference, and the image on the right is generated from this mixer presentation. And you can see that there is a lot of semantic information encoding encoded in those pairing tokens.

107
00:16:42.860 --> 00:16:54.940
michaeltoker: Now, we repeat this experiment with another model, and we see that… a completely different picture. So, the image on the right is something that is very random and not related to the semantics of the problem.

108
00:16:55.450 --> 00:16:57.069
michaeltoker: So, what's going on here?

109
00:16:57.640 --> 00:17:03.139
michaeltoker: We hypothesize that the difference is coming from the way… he cannot hear you.

110
00:17:03.550 --> 00:17:04.319
michaeltoker: Oh.

111
00:17:08.650 --> 00:17:10.519
Rohit Gandikota: Hey, Mike, can you hear me now?

112
00:17:10.520 --> 00:17:13.420
michaeltoker: Oh, no! Your whole talk got lost.

113
00:17:13.790 --> 00:17:21.069
michaeltoker: What, what do I need to do? So, so, in your… so, in your thing, so, this, this is not, not this computer.

114
00:17:21.310 --> 00:17:22.240
michaeltoker: C.

115
00:17:23.050 --> 00:17:27.300
michaeltoker: Let's see… On… on your side.

116
00:17:28.560 --> 00:17:31.380
michaeltoker: Audio… Oh.

117
00:17:31.680 --> 00:17:36.029
michaeltoker: Oh, the speaker's not working. Maybe your speaker? Let's see.

118
00:17:36.030 --> 00:17:37.120
Rohit Gandikota: Can you hear me now?

119
00:17:37.120 --> 00:17:42.620
michaeltoker: Yeah, yeah. Okay, so it's not… so, Rohan, you've been able to hear the talk, you just have not been able to interrupt.

120
00:17:42.620 --> 00:17:44.479
Rohit Gandikota: Yeah, yeah, I was not able to interrupt you guys.

121
00:17:44.480 --> 00:17:48.590
michaeltoker: That's okay. Okay, Rohan, what's your question?

122
00:17:48.840 --> 00:17:55.180
Rohit Gandikota: Yeah, Michael, What's the reason behind using the…

123
00:17:55.290 --> 00:18:00.290
Rohit Gandikota: Padding tokens in front of the contextualized padding tokens?

124
00:18:01.090 --> 00:18:03.919
michaeltoker: What's the reason of using the orange part, or what's…

125
00:18:03.920 --> 00:18:05.250
Rohit Gandikota: Yeah, yeah, yeah.

126
00:18:05.500 --> 00:18:10.880
michaeltoker: So, so, our thought was that we do not want to,

127
00:18:10.880 --> 00:18:29.999
michaeltoker: interrupt the positional encoding, so if the pads are encoded in a specific… with a specific positional encoding, we want to keep the distribution as close to the original as possible. So instead of just generating an image from pads that move to another position, we just add

128
00:18:30.080 --> 00:18:32.600
michaeltoker: Clean pads with the original positions.

129
00:18:32.880 --> 00:18:38.730
Rohit Gandikota: That makes sense. Have you tried, by any chance, not using the orange and just using these.

130
00:18:39.400 --> 00:18:54.029
michaeltoker: Actually, yes, and it resulted… I… so I didn't do it in scale, but it looked relatively similar, so maybe you can just do it with the contextualized pads, but yeah, I wanted to be rigorous about it.

131
00:18:54.230 --> 00:18:55.829
Rohit Gandikota: Yeah, that makes sense. Thanks.

132
00:18:56.940 --> 00:18:58.119
michaeltoker: Thank you for the question.

133
00:18:58.460 --> 00:18:59.530
michaeltoker: Okay.

134
00:18:59.990 --> 00:19:12.950
michaeltoker: So, I'll just repeat that we saw this difference between the models, and we hypothesized that this is due to the difference between how text encoders are trained.

135
00:19:12.970 --> 00:19:22.429
michaeltoker: So, in the first model, the text encoder is trained first on language, but then it's also trained on text-to-image generation.

136
00:19:22.670 --> 00:19:27.429
michaeltoker: By fine-tuning it… by fine-tuning it with image prompts with padding tokens.

137
00:19:28.010 --> 00:19:34.669
michaeltoker: In Stable Diffusion 3, it's trained only on the text objective.

138
00:19:34.830 --> 00:19:42.840
michaeltoker: And the text encoder never trained with the diffusion, so the diffusion is conditioned on the frozen representations of the,

139
00:19:43.350 --> 00:19:55.950
michaeltoker: of the text. So the representations of the text never… the text encoder never learned to use those padding tokens to encode more information.

140
00:19:56.740 --> 00:20:06.410
michaeltoker: And we'll repeat this experiment with different models. Some of them are with a text encoder that was frozen, and some of them with

141
00:20:06.530 --> 00:20:24.400
michaeltoker: We've trained text encoder, and we see… so here we measured the average clip square between the, generated image. So the blue bar is the full presentation, and the green bar is the representation of the mixer presentation with the padding tokens.

142
00:20:24.420 --> 00:20:28.819
michaeltoker: And we measured the clip score between the generated image and the original prompt.

143
00:20:29.420 --> 00:20:35.500
michaeltoker: And you can see that the only relatively high green bars are the ones that,

144
00:20:35.640 --> 00:20:41.030
michaeltoker: Correspond to text inquiries that were trained together with the text-to-image objective.

145
00:20:44.600 --> 00:20:49.910
michaeltoker: Okay, so in this work… Awesome.

146
00:20:50.510 --> 00:20:51.840
michaeltoker: When you've moved.

147
00:20:53.530 --> 00:21:00.599
michaeltoker: It is the brief representation… How… how we put things.

148
00:21:00.840 --> 00:21:02.840
michaeltoker: information independent.

149
00:21:03.400 --> 00:21:21.610
michaeltoker: The green one shows how much of the information contained in the pending tokens. Okay, so when I say frozen, I mean this is how the text-to-image model works in the original setting. So, for example, Cebus Fusion 3, they take T5 model.

150
00:21:21.610 --> 00:21:26.039
michaeltoker: And they do not train it for text-to-image generation. They use the outputs of the

151
00:21:26.040 --> 00:21:32.549
michaeltoker: T5 model, and they condition… they train the diffusion model to be conditioned on that representation.

152
00:21:32.600 --> 00:21:36.149
michaeltoker: But the T5 model never changed its weights.

153
00:21:36.780 --> 00:21:40.070
michaeltoker: So it's never adapted for text-to-image generation.

154
00:21:41.700 --> 00:21:43.340
michaeltoker: Makes sense. Does that make sense?

155
00:21:47.170 --> 00:21:51.630
michaeltoker: Headings come from the… Thanks. Right.

156
00:21:52.140 --> 00:21:55.360
michaeltoker: And… So, he takes classes.

157
00:21:55.880 --> 00:21:56.670
michaeltoker: Something.

158
00:21:56.910 --> 00:22:02.860
michaeltoker: The text encoder… the text encoder was trained on the language task.

159
00:22:03.050 --> 00:22:06.770
michaeltoker: And it was not trained for the text-to-image task.

160
00:22:07.010 --> 00:22:14.339
michaeltoker: So… So the text sub-padding tokens, but in the context of language. Never in the context of text-to-image.

161
00:22:17.970 --> 00:22:18.830
michaeltoker: I quit.

162
00:22:20.130 --> 00:22:22.630
michaeltoker: How common is two words? Right there.

163
00:22:23.060 --> 00:22:34.180
michaeltoker: Yes, all of the text-to-image models use padding tokens. I saw one model Sorry, but…

164
00:22:35.230 --> 00:22:37.469
michaeltoker: Oh, I see, right and left pets.

165
00:22:40.060 --> 00:22:48.040
michaeltoker: Yeah, I think I saw recently one model that is adding the paddings on the left side, but most of them do it on the right side.

166
00:22:53.250 --> 00:22:54.150
michaeltoker: Darbon.

167
00:22:54.280 --> 00:22:55.650
michaeltoker: How do we get this out?

168
00:22:57.180 --> 00:23:02.399
michaeltoker: So the text encoder usually is the encoder of T5.

169
00:23:03.040 --> 00:23:03.850
michaeltoker: Yeah.

170
00:23:06.470 --> 00:23:09.900
michaeltoker: Yeah. Did you expect the graph to look like this?

171
00:23:11.090 --> 00:23:14.550
michaeltoker: So…

172
00:23:15.330 --> 00:23:24.850
michaeltoker: Yes, so after we saw the difference between the… so we started with the model with… that had the,

173
00:23:25.010 --> 00:23:42.979
michaeltoker: that had the padding tokens with semantic information, and then we saw that it's not repeated. And then the only difference that we could think about that could resolve it is that the text encoder was not… it didn't learn to use those padding tokens.

174
00:23:43.240 --> 00:23:49.859
michaeltoker: So… so these are the two models that we expected to see, high green bars.

175
00:23:50.860 --> 00:23:51.580
michaeltoker: Yeah.

176
00:23:54.180 --> 00:23:55.160
michaeltoker: Okay.

177
00:23:58.000 --> 00:24:00.369
michaeltoker: Spanish words of a celebrity.

178
00:24:00.520 --> 00:24:12.240
michaeltoker: They are also special programs, are they? No, no, so these are different specialties, yes. Have you tried replicating research with both children?

179
00:24:13.480 --> 00:24:27.180
michaeltoker: I haven't… Yeah, I haven't tried, but you need to remember that the diffusion model, I guess, never trained… like, it never saw those tokens, yes.

180
00:24:27.660 --> 00:24:42.610
michaeltoker: So I'm not sure that you would have any meaningful information from those. Yeah, but meaningful is from the text importer. Right. Yeah, I agree. Yeah. Might see something, I don't know. Yeah, yeah, I agree. Yeah.

181
00:24:43.180 --> 00:24:49.609
michaeltoker: Interesting. These text people, they don't care about you.

182
00:24:52.730 --> 00:24:53.270
michaeltoker: Mike?

183
00:24:53.270 --> 00:24:54.710
Rohit Gandikota: I have two questions.

184
00:24:55.210 --> 00:24:55.760
michaeltoker: Yeah.

185
00:24:56.000 --> 00:25:00.260
Rohit Gandikota: the… So I like the contrast between

186
00:25:00.390 --> 00:25:03.199
Rohit Gandikota: The trained and frozen text encoders.

187
00:25:04.780 --> 00:25:17.569
Rohit Gandikota: So, with the trained encoder, now the question is, then what are the text tokens really being used for? Like, can you remove them, and… or, like, what if you just use them and try to generate, like, the opposite experiment?

188
00:25:18.270 --> 00:25:23.779
Rohit Gandikota: what would happen then? Like, would you see nothing turning out? Is it all the padding that's important now?

189
00:25:24.390 --> 00:25:26.169
Rohit Gandikota: Have you… have you tested that?

190
00:25:26.170 --> 00:25:28.180
michaeltoker: Yes, so I, so I tested…

191
00:25:28.380 --> 00:25:34.989
michaeltoker: similar things. Let's see… no, so I don't have it here.

192
00:25:35.520 --> 00:25:44.760
michaeltoker: So, I see… I saw… so I generated also images only from the, prompt tokens without the pads.

193
00:25:45.030 --> 00:25:50.810
michaeltoker: This is similar to the image that we saw at the first of this paper.

194
00:25:50.810 --> 00:25:51.340
Rohit Gandikota: Yep.

195
00:25:51.340 --> 00:25:53.699
michaeltoker: So you see that,

196
00:25:54.560 --> 00:26:12.499
michaeltoker: Most of the… I mean, you have some duplications, but it's not complete. If you generate the image only from the textual tokens, you do not have the full correct representation. And also, this is something that I'm not talking about today, but we talk about it in the paper.

197
00:26:12.500 --> 00:26:28.350
michaeltoker: is that you have two places that use padding tokens. So, one is the text encoder, and the other is the diffusion model. So, when the diffusion is a diffusion transformer, then the padding tokens are also used as registered tokens.

198
00:26:28.350 --> 00:26:32.600
michaeltoker: And, this is another place that you see this, this effect.

199
00:26:34.470 --> 00:26:39.259
Rohit Gandikota: So, and another… I think that makes sense. So, another question…

200
00:26:39.440 --> 00:26:46.309
Rohit Gandikota: is some of these are, multimodal DITs, where the text is also being processed along with the image.

201
00:26:46.770 --> 00:26:49.780
Rohit Gandikota: Right? Like, Flux, stabilization 3.

202
00:26:49.780 --> 00:26:50.380
michaeltoker: Yes.

203
00:26:50.760 --> 00:27:00.369
Rohit Gandikota: Do you think your, observation here has implications for how these models need to be trained? Yes. Like, oh, we should not be training

204
00:27:02.020 --> 00:27:06.710
Rohit Gandikota: having padding tokens at all, or maybe we should try to increase them more. How do you think about this?

205
00:27:06.710 --> 00:27:14.249
michaeltoker: This is a good question. So I have a lot of thoughts about this. I think that this shows that,

206
00:27:14.460 --> 00:27:28.960
michaeltoker: You have some computations that you can utilize, and actually here, it's interesting that the model learned to use those tokens, because nothing in the training loss, like, made the model to use those tokens.

207
00:27:28.960 --> 00:27:34.560
michaeltoker: It just helped the backpropagation to generate better images.

208
00:27:34.600 --> 00:27:49.310
michaeltoker: But you can imagine that you add some loss function that also makes the model to… that now it has to use those tokens. For example, I saw that only the first couple of padding tokens are used.

209
00:27:49.310 --> 00:28:05.499
michaeltoker: But most of them are still unused. So maybe if you make the model to… if you add some loss function that makes the model to use more pending tokens, then you can improve results, especially in more complex prompts where you need more computation.

210
00:28:05.500 --> 00:28:06.130
Rohit Gandikota: Hmm.

211
00:28:06.890 --> 00:28:09.290
Rohit Gandikota: Right. Yeah, that makes sense.

212
00:28:09.510 --> 00:28:12.299
Rohit Gandikota: We… I don't know if you saw…

213
00:28:12.790 --> 00:28:16.010
Rohit Gandikota: Our recent work, it was on…

214
00:28:16.530 --> 00:28:21.410
Rohit Gandikota: Similar thing, but except we found that internally also, in intermediate layers.

215
00:28:21.860 --> 00:28:31.620
Rohit Gandikota: These padding tokens are being used for totally different things. To break reference image information, or to store duplication errors.

216
00:28:32.160 --> 00:28:37.610
Rohit Gandikota: I wonder, like, 52 tokens, why are they so important? Maybe there's more to learn here.

217
00:28:37.770 --> 00:28:42.580
michaeltoker: Yeah, I saw this work on the in-context image editing.

218
00:28:42.580 --> 00:28:43.140
Rohit Gandikota: Yeah.

219
00:28:43.140 --> 00:28:49.849
michaeltoker: Yeah, so, so I think that, it, it, it's, so the pending tokens.

220
00:28:50.030 --> 00:28:55.359
michaeltoker: Are just a place where computation can occur, and,

221
00:28:55.760 --> 00:29:04.060
michaeltoker: If you choose, right, the architecture and the training mechanism, then the model learns to use those tokens.

222
00:29:06.510 --> 00:29:08.199
Rohit Gandikota: Yeah, that makes sense, Andrew.

223
00:29:09.650 --> 00:29:10.480
Rohit Gandikota: Thanks.

224
00:29:10.900 --> 00:29:11.600
michaeltoker: Thank you.

225
00:29:13.210 --> 00:29:14.810
michaeltoker: Any more questions?

226
00:29:16.340 --> 00:29:21.290
michaeltoker: Okay, thank you. These were really good questions. How are we with the time?

227
00:29:22.890 --> 00:29:32.679
michaeltoker: Depends on… do you have to go by sharp if you want? Well, we usually run a little bit over, but it's… Okay. 10 minutes over? Yeah, 10 minutes over. Okay, okay.

228
00:29:33.170 --> 00:29:38.489
michaeltoker: So, this is, the last work that I'm going to talk to you about. In this work.

229
00:29:38.890 --> 00:29:40.420
michaeltoker: Keep on going out, my dad.

230
00:29:40.870 --> 00:29:46.320
michaeltoker: In this work, we look at, in the information flow.

231
00:29:46.430 --> 00:29:53.560
michaeltoker: On… between tokens in the text encoder, but now we look at the information flow between tokens in the prompt.

232
00:29:55.920 --> 00:30:06.359
michaeltoker: Okay. So, we look at two cases here. The first one is in furnishing flow between tokens. So, for example, you have here the Pelican.

233
00:30:07.250 --> 00:30:12.240
michaeltoker: And the question is whether all of the tokens encode the same information.

234
00:30:12.590 --> 00:30:19.450
michaeltoker: So this is, again, the encoder of T5, so the attention is bidirectional. Any of the tokens might resolve

235
00:30:19.760 --> 00:30:21.640
michaeltoker: Into the entity.

236
00:30:22.090 --> 00:30:32.359
michaeltoker: And the way we interpret the token is very similar to what we saw a few slides ago. We encode the full prompt, we separate the token that we want to interpret.

237
00:30:32.480 --> 00:30:37.510
michaeltoker: We encode petting tokens, and we replace all the other tokens by the padding tokens.

238
00:30:38.360 --> 00:30:42.260
michaeltoker: And we generate an image condition on this mixed representation.

239
00:30:43.010 --> 00:30:56.680
michaeltoker: So this is the image that we see for the first token, this is the image that we see for the second token, which is the correct pelican, and we can do it for each of the tokens, and we see that in this case, only one of the tokens

240
00:30:56.960 --> 00:31:01.350
michaeltoker: represents correctly the entity. We call it a representative token.

241
00:31:02.070 --> 00:31:04.999
michaeltoker: We repeat it across different.

242
00:31:05.250 --> 00:31:10.649
michaeltoker: Prompt lengths, so for each… we call it lexical item, which is a,

243
00:31:10.980 --> 00:31:24.699
michaeltoker: unit that we are interested in, and we look at items from different lands, and for each… so this is the y-axis, and for each item, we look at how many of the tokens are representative.

244
00:31:26.080 --> 00:31:38.780
michaeltoker: So, for example, for an item of length of 2, you can see that about 60% of the… in 60% of the cases, there was only one representative token, as we saw in the last slide.

245
00:31:39.060 --> 00:31:44.690
michaeltoker: So, many times, the, summary tokens are not representative.

246
00:31:45.450 --> 00:31:54.610
michaeltoker: And the question that we ask now is whether those non-representative tokens are helpful in some way, or is it hurting results?

247
00:31:55.600 --> 00:31:59.009
michaeltoker: So, what we do is we, yeah.

248
00:32:01.040 --> 00:32:02.859
michaeltoker: What does this fall really mean?

249
00:32:03.430 --> 00:32:05.640
michaeltoker: The, the what?

250
00:32:06.700 --> 00:32:13.770
michaeltoker: So this is just the sum over all of the… all of the groups, all of the… lexical items.

251
00:32:14.850 --> 00:32:24.519
michaeltoker: This is, like, the percentage of representative tokens across all of the lexical items, all of the lengths of the lexical items.

252
00:32:26.270 --> 00:32:28.430
michaeltoker: It includes also one, yeah, yeah.

253
00:32:33.340 --> 00:32:51.810
michaeltoker: So, we train a probe to detect for each token whether it's representative or not, by first generating the images, then evaluating them, and then training the probe. And we find out that actually those non-representative tokens help… they hurt to

254
00:32:51.830 --> 00:33:05.810
michaeltoker: generation. And we can improve alignment, by 21%. For example, here, the original giraffe was not pink, so removing the A token, made generation better.

255
00:33:07.180 --> 00:33:11.159
michaeltoker: No, the second thing that we looked at… What are you replacing with when you're…

256
00:33:11.820 --> 00:33:16.900
michaeltoker: With a pending token. Oh, I see. So even though it's the only representative.

257
00:33:18.800 --> 00:33:23.859
michaeltoker: Yeah, yeah. No, it's the only non-representative.

258
00:33:24.270 --> 00:33:30.850
michaeltoker: that non… The A is non-representative in this case. Oh, the A is not representative.

259
00:33:32.070 --> 00:33:36.130
michaeltoker: No, no, wait. And so the… in the original… in the original…

260
00:33:37.160 --> 00:33:41.319
michaeltoker: So, I misunderstood. So, in the original, context.

261
00:33:42.330 --> 00:33:50.739
michaeltoker: No. In original context, only the first and the last tokens are representative. In this one? Yes, in this case.

262
00:33:51.460 --> 00:33:56.089
michaeltoker: So if you… if you just get rid of the non-representative, then… Yes.

263
00:33:58.560 --> 00:33:59.310
michaeltoker: Pardon me.

264
00:33:59.690 --> 00:34:03.170
michaeltoker: Are you getting rid of that to getting at the prompt level, or at the encoding level?

265
00:34:04.750 --> 00:34:06.140
michaeltoker: At the encoding level.

266
00:34:06.420 --> 00:34:10.730
michaeltoker: They can try doing the reverse, like, what happens if you… Hmm.

267
00:34:11.210 --> 00:34:26.489
michaeltoker: Yes, so, we didn't try it in scale, but we thought about it, and we think that sometimes it did not make sense, because sometimes the, meaning of the, tokens affected by this token

268
00:34:26.580 --> 00:34:28.769
michaeltoker: But then you do not need it anymore.

269
00:34:28.840 --> 00:34:47.310
michaeltoker: And if you remove it, the meaning completely different. So we do not think that this token is not needed in any part of the model. We think that during the textual encoding, it is used in some way, but once you're done with the encoding, then maybe it's some…

270
00:34:47.520 --> 00:34:52.929
michaeltoker: by product of the encoding that you do not need anymore. I mean, more so, did you try…

271
00:34:53.790 --> 00:34:55.670
michaeltoker: Spontane be neat coding.

272
00:34:55.850 --> 00:35:00.020
michaeltoker: Only the representative took it, took it, It's just intensive thing.

273
00:35:00.730 --> 00:35:10.879
michaeltoker: So I think that it's not… it doesn't make sense in many cases, because then you would lose the meaning of the… of the tokens.

274
00:35:12.710 --> 00:35:15.430
michaeltoker: Somewhere when I complete this, like, solved.

275
00:35:15.630 --> 00:35:20.600
michaeltoker: I almost feel like it's… An interesting or contrad… contradicting result.

276
00:35:20.720 --> 00:35:26.160
michaeltoker: Let's say if you only, like, use, like, the third per day, right? Yeah.

277
00:35:26.470 --> 00:35:30.829
michaeltoker: Yeah. I bleak, like, I've got a job, like, as an adjour, or…

278
00:35:31.120 --> 00:35:41.819
michaeltoker: Yeah. Right. So that means, like, A doesn't do that much to draw. Right. But in some way, like, looking at these results, like, actually A is contributing a little bit, right? Like, say…

279
00:35:42.220 --> 00:35:49.579
michaeltoker: Like, say, if you actually insist to have an A instead of X probe, then the model is, like, least stubborn to keep the drop as yellow.

280
00:35:51.630 --> 00:35:57.720
michaeltoker: I'm wondering whether it can be interpreted. Interesting. Yeah.

281
00:35:58.490 --> 00:36:02.200
michaeltoker: Yeah, haven't thought about this angle of this result.

282
00:36:02.630 --> 00:36:03.350
michaeltoker: Yeah.

283
00:36:04.580 --> 00:36:08.909
michaeltoker: It sounds reasonable, but I think that's…

284
00:36:10.030 --> 00:36:13.330
michaeltoker: You need to do some other experiments to check it, but…

285
00:36:13.330 --> 00:36:16.220
Rohit Gandikota: Can you repeat what the point was?

286
00:36:16.650 --> 00:36:35.830
michaeltoker: I think the point was that, the fact that, in this specific example, the giraffe was not pink in the original image, maybe it means that the A was contributing to the representation of the giraffe concept, and, like, keeping it as the original giraffe should look like.

287
00:36:36.270 --> 00:36:41.000
Rohit Gandikota: Right, yes, like, the bias, the strong bias towards how a Jira should look like.

288
00:36:41.170 --> 00:36:42.500
michaeltoker: Right, yeah.

289
00:36:43.390 --> 00:36:47.840
Rohit Gandikota: Yeah, yeah, I was thinking of the same thing. I was like, oh, this actually says the opposite.

290
00:36:48.890 --> 00:36:52.489
Rohit Gandikota: Like, age is used to be, like, seems to be really important for giraffe.

291
00:36:52.490 --> 00:36:53.530
michaeltoker: Yeah, yeah.

292
00:36:56.820 --> 00:37:01.419
michaeltoker: have, like, worse fonts. Like, fonts are harder to read. Reading comprehension.

293
00:37:02.440 --> 00:37:08.320
michaeltoker: Yeah. Yeah, yeah.

294
00:37:08.430 --> 00:37:10.350
michaeltoker: Investing.

295
00:37:13.650 --> 00:37:19.580
michaeltoker: Okay, so, I think that I'll skip it to… so Tomer has enough time.

296
00:37:24.950 --> 00:37:27.090
michaeltoker: Okay, so,

297
00:37:27.200 --> 00:37:36.290
michaeltoker: To summarize, we saw three complementary works, where we look at the information flow across the layers and across the tokens.

298
00:37:36.400 --> 00:37:51.729
michaeltoker: And we saw that the diffusion is an effective way to probe the text encoder, and I'm really waiting for very good multimodel models to be released and to try those methods on those highly capable models, also to generate images.

299
00:37:52.430 --> 00:37:56.130
michaeltoker: We saw that different smaller architectural decisions

300
00:37:56.420 --> 00:38:00.939
michaeltoker: Affects the way, the limitations of the, of the model.

301
00:38:01.140 --> 00:38:07.119
michaeltoker: And we saw… okay, we didn't see it yet, but we saw some… we should have seen some,

302
00:38:07.420 --> 00:38:12.230
michaeltoker: ways to use those mechanistic insights to improve alignment.

303
00:38:12.720 --> 00:38:17.560
michaeltoker: I will be happy to talk to any of you if you are interested in those kinds of works.

304
00:38:23.910 --> 00:38:26.170
michaeltoker: Thank you very much, Michael. Thank you very much.

305
00:38:29.310 --> 00:38:30.100
michaeltoker: Thank you.

306
00:39:55.930 --> 00:40:00.249
Tomer Ashuach: All right. So, hi everybody. My name is Thomas Schuach.

307
00:40:00.410 --> 00:40:06.759
Tomer Ashuach: And also, I'm a PhD candidate from Benkopf's lab, Technion, and this is the lab, by the way.

308
00:40:07.330 --> 00:40:13.679
Tomer Ashuach: And first of all, I'm very happy to be here and meet everybody, and hopefully we can chat later as well.

309
00:40:14.960 --> 00:40:20.279
Tomer Ashuach: So… Basically, today's talk is uncovering knowledge in language models.

310
00:40:20.720 --> 00:40:22.340
Tomer Ashuach: And,

311
00:40:22.610 --> 00:40:38.509
Tomer Ashuach: what I'm interested about, generally, is interpretability and how models work, but more specifically, in acknowledging models. So, how knowledge is learned by the models, how models encode knowledge, and how can you remove

312
00:40:38.600 --> 00:40:58.360
Tomer Ashuach: some specific information within the model's knowledge after it was trained on it, which is unlearning, basically. And today, I'm going to present three works. The first two are about unlearning from different angles and different interpretability tools, and the last one is about encoding of information.

313
00:40:59.550 --> 00:41:09.710
Tomer Ashuach: So, basically, as you know already, language models train on vast amounts of data, the whole internet, right? And sometimes, models learn things they shouldn't learn.

314
00:41:09.820 --> 00:41:27.460
Tomer Ashuach: For example, private data, some email addresses or physical addresses. Another concern is harmful knowledge. Like, how can you make a virus more contagious in people? And believe me, if you ask Claude enough, you will be able to extract some information, and I'll show it later on, but basically.

315
00:41:27.460 --> 00:41:32.100
Tomer Ashuach: They… it's amazing what harmful knowledge they possess, and how hard

316
00:41:32.100 --> 00:41:39.280
Tomer Ashuach: Those companies are, you know, trying to make them safe in the sense of refusal.

317
00:41:39.800 --> 00:41:48.619
Tomer Ashuach: And lastly, copyrighted data is also some, not a thing we don't want the model to learn most of the time, like Harry Potter books.

318
00:41:48.960 --> 00:41:53.980
Tomer Ashuach: So, given all those three, axes, We ask…

319
00:41:54.090 --> 00:41:58.220
Tomer Ashuach: How can we remove unwanted knowledge from language models, right?

320
00:41:58.280 --> 00:42:12.540
Tomer Ashuach: So, one pretty naive approach would be, okay, just take the training dataset, right, and remove the subset you don't want the model to learn, but consider that you find, like, I don't know, some email address that the model,

321
00:42:12.540 --> 00:42:22.160
Tomer Ashuach: trained on and memorized, and… but you have ChatGPT, they train 3 months on, like, I don't know, thousands of GPUs. It doesn't really make sense to train it from scratch, I guess.

322
00:42:22.250 --> 00:42:30.680
Tomer Ashuach: Another option, which is generally pretty good, is alignment training, right? So, Anthropic, ChatGPT, and

323
00:42:30.810 --> 00:42:48.679
Tomer Ashuach: like, all big companies are, training the model with different, setups to refuse model to give, some specific data. So if you ask Claude to give me, like, a step-by-step guide on how to make a bomb, it's gonna tell you, no, I'm not gonna do that, right?

324
00:42:48.790 --> 00:42:52.830
Tomer Ashuach: But it works… works most of the time.

325
00:42:53.050 --> 00:42:59.160
Tomer Ashuach: The issue with refusal, in general, or alignment training, is that the knowledge…

326
00:42:59.220 --> 00:43:18.500
Tomer Ashuach: still stays within the weights. The model still knows the information, it just doesn't give you… you can't get into the information. But, as you know, if information exists, some malicious actor will be able to extract it at some point, right? It's like a strong vault.

327
00:43:18.500 --> 00:43:22.909
Tomer Ashuach: But somebody would be able to get into what's insider.

328
00:43:22.910 --> 00:43:40.860
Tomer Ashuach: So, one very nice example for that is multiple works, that basically taking fine-tuning models, open weights, and do a light fine-tuning to bypass those alignments. And by doing that, they're able to just jailbreak the model pretty easily.

329
00:43:41.030 --> 00:43:59.310
Tomer Ashuach: So, this is in the white box scenario, but you can also, convince, if you're consisting enough, a black box model as well to extract some harmful or private data information. So this is like, you know, a cat and a mouse race.

330
00:44:00.280 --> 00:44:18.549
Tomer Ashuach: So, okay, so we have another approach, which is aiming to remove the target information from the model weights. So, basically, target the information from the model weights, and remove it, such that it's not extractable afterwards, because it does not exist, right?

331
00:44:18.550 --> 00:44:28.379
Tomer Ashuach: The problem with this approach is that it's very hard to, to maintain a good trade-off between how strong you remove information

332
00:44:28.380 --> 00:44:44.969
Tomer Ashuach: and how well you retain relevant, useful information. Because if you think about, like, the simplest approach, you can give me any model you want, and I can delete all the information that I can guarantee it's not extractable. I can zero out all the weights, right? So it would work, but you get

333
00:44:45.000 --> 00:44:53.199
Tomer Ashuach: no model, in this sense. So you need to think about the trade-off, and basically the whole community of unlearning is trying to get the best

334
00:44:53.280 --> 00:44:58.200
Tomer Ashuach: Unlearning result, the strongest one, while maintaining useful information.

335
00:44:59.140 --> 00:45:11.610
Tomer Ashuach: So, again, for this, the relevant setup is, for the first work I'm gonna present, is given a model that memorized some sensitive information. For example, email address.

336
00:45:11.950 --> 00:45:27.800
Tomer Ashuach: So, the first work is published last year in ACL, and it was a joint work with Martin Totek and Antanbenko, and it's called REVS, and it's about unlearning sensitive information the model memorized, like email addresses. So, the setup is as following.

337
00:45:27.840 --> 00:45:44.930
Tomer Ashuach: given a prompt that the model, memorized both the prompt and the continuation. And the continuation in this case is some email address that is… a real email address that's leaked somehow to the training data, and this is not… this is a fabricated,

338
00:45:44.930 --> 00:45:57.680
Tomer Ashuach: example, but we do evaluate on real memorized example from the pile, if you're familiar with this dataset. So basically, the model, given contact David Lewis at, the model will readily generate

339
00:45:57.790 --> 00:45:59.490
Tomer Ashuach: Louis Davies email.

340
00:45:59.630 --> 00:46:08.280
Tomer Ashuach: So… Let's have a look now on how the token, the first token from the sensitive information.

341
00:46:08.290 --> 00:46:28.260
Tomer Ashuach: looks through the layers of the model. So basically, I'm gonna apply logic lens, which I guess you were familiar with, and what we're gonna see is that as we go through the layers, the token is gonna be higher and higher, and in this case, it's gonna be, going lower in the y-axis, because

342
00:46:28.310 --> 00:46:43.170
Tomer Ashuach: rank zero means that this is gonna be the most probable token to be generated, right? So, as you expect, if I tell you that the model memorized this token, given this prefix, you see that it goes to zero.

343
00:46:43.310 --> 00:46:50.679
Tomer Ashuach: But this is applying logitans to the residual stream. We can do also… what we can do also is apply logitans to the MLP.

344
00:46:50.970 --> 00:46:51.860
Tomer Ashuach: output.

345
00:46:52.030 --> 00:47:08.219
Tomer Ashuach: And here you see something different, right? You see that some layers kind of consist a lot of information of the token, pushing it through the residual stream, and some layers are not so much. So, as you can see, what I've circled is this

346
00:47:08.270 --> 00:47:20.570
Tomer Ashuach: that those layers push the sensitive token through the residual stream. So Rev's gonna… our method gonna tackle specifically those layers only. So…

347
00:47:20.690 --> 00:47:33.589
Tomer Ashuach: Currently, we are trying to localize which parts of the model is contributing to the memorized sensitive information, and then we're gonna only edit those parameters specifically, and not the whole model.

348
00:47:34.190 --> 00:47:49.750
Tomer Ashuach: So, okay, we identified for this example which layers support contributes more, and what we need to do now is a method that, given the specific layer, what should we edit? What neurons, more specifically, should we edit?

349
00:47:49.750 --> 00:48:01.399
Tomer Ashuach: So, I don't have a lot of time, so I'm not gonna give you, like, the intro for why I'm tackling only the second layer of the MLP, but there is some intuition regarding that. I can explain a bit.

350
00:48:01.400 --> 00:48:13.840
Tomer Ashuach: more about that later. But basically, you can break down the second feed-forward layer in the MLP into neurons, which are the columns, and basically vector from the same dimension as the residual stream.

351
00:48:13.840 --> 00:48:18.259
Tomer Ashuach: And because they have the same dimensionality, you can apply Logitense on them as well, right?

352
00:48:18.730 --> 00:48:22.750
Tomer Ashuach: And what you get is kind of a vocabulary distribution.

353
00:48:23.130 --> 00:48:42.360
Tomer Ashuach: of what each neuron consists of in the terms of tokens. And for each token, and for each neuron, you can check how, like, how corresponding it to the sensitive token you are targeting. And you can use that to basically take few neurons that are

354
00:48:42.630 --> 00:48:52.219
Tomer Ashuach: Strongly corresponds to the specific token you want to remove, and giving those neurons, you can just basically edit

355
00:48:52.720 --> 00:48:55.969
Tomer Ashuach: The element, or the logic value.

356
00:48:56.150 --> 00:49:11.449
Tomer Ashuach: from this sensitive token you want to remove, and lower it. So, for example, if for this specific token neuron contains the target token we want to remove, and it's ranked first, meaning that it has very high logic value, you can just edit it to be very

357
00:49:11.450 --> 00:49:17.420
Tomer Ashuach: low budget value, and get a new vocabulary vector representation that's

358
00:49:17.540 --> 00:49:35.529
Tomer Ashuach: kind of redacted the information of this sensitive token, right? And what you can do is, afterwards, you just return back, using the pseudo-inverse of the vocabulary vector you get here, to get a new update neuron that you can replace from the older one.

359
00:49:35.900 --> 00:49:49.540
Tomer Ashuach: So, one naive approach would be just to zero out the neuron, but this would cause huge degradation because you remove all the information of the neuron. What we do is we remove only the specific information from the sensitive token.

360
00:49:49.540 --> 00:50:01.150
Tomer Ashuach: Like that. And we do this for a very small amount of neurons, so a typical number would be MLP consists, like, 6… 16K, 16,000 of neurons. We tackle about 10, or

361
00:50:01.260 --> 00:50:05.809
Tomer Ashuach: 20 neurons to remove specific sensitive information, an example.

362
00:50:07.850 --> 00:50:13.249
Tomer Ashuach: Kind of like that.

363
00:50:13.500 --> 00:50:18.239
Tomer Ashuach: distribution of… of vocabulary tokens, or… or what?

364
00:50:18.570 --> 00:50:27.569
Tomer Ashuach: you… so for a set of target tokens, let's say you have the logic tokens, and then you have the logic score. Yeah, for each token.

365
00:50:27.810 --> 00:50:33.599
Tomer Ashuach: Yeah, I'm talking the RAM, but it's corresponding to it as well, to the logistical.

366
00:50:33.790 --> 00:50:43.400
Tomer Ashuach: for each one of the columns, for a specific set of target totals. Here, the target number of target totals is just one.

367
00:50:43.760 --> 00:50:45.469
Tomer Ashuach: I'm just asking.

368
00:50:45.680 --> 00:50:47.560
Tomer Ashuach: What does this distribution look like?

369
00:50:48.180 --> 00:51:05.809
Tomer Ashuach: So, if I understand correctly, you have very few tokens which are, let's say, strongly promoted by this token, and all the rest of them are kind of, have a small logged values. And we also… it's a HIPAA parameter, but we can,

370
00:51:05.910 --> 00:51:15.410
Tomer Ashuach: target only very small amount of neurons, which promote the specific token, and it still works, saying that…

371
00:51:16.120 --> 00:51:21.380
Tomer Ashuach: Again, distribution of the voca- of this vector of vocabulary space, and it's a lot…

372
00:51:21.620 --> 00:51:28.980
Tomer Ashuach: logic values, so again, you have… you have, let's say, 16,000 euros. You apply logic levels.

373
00:51:29.120 --> 00:51:39.659
Tomer Ashuach: and track the logic of, let's say, LEW, this token. Right? All of this runs the corresponding logic value for LEW.

374
00:51:39.880 --> 00:51:54.430
Tomer Ashuach: So, so you have few neurons that have high, contribution towards the specific tokens. So this is why in very small subset of neurons is enough to target. Okay.

375
00:51:54.610 --> 00:52:00.100
Tomer Ashuach: between, let me be honest, why I'm pushing, of course? Which push models?

376
00:52:00.390 --> 00:52:14.060
Tomer Ashuach: Those are… I've tried on two, on, LAMA38B and, GPTJ, but this is pretty, yeah, because we compare to Mehmet, and,

377
00:52:14.470 --> 00:52:15.670
Tomer Ashuach: Yes, it is.

378
00:52:15.820 --> 00:52:22.839
Tomer Ashuach: But, but, okay. But just to give, like, a reference, this is, like, an older project, that's why.

379
00:52:23.020 --> 00:52:27.880
Tomer Ashuach: So, this, this project, the core idea of this project is very similar to

380
00:52:28.410 --> 00:52:38.219
Tomer Ashuach: that LM driver thing, it came before Gabe as well, like, language models, Google Constant syndication. Do you know the one? Yeah, I do.

381
00:52:38.720 --> 00:52:40.799
Tomer Ashuach: I actually tried to make it work.

382
00:52:40.960 --> 00:52:48.390
Tomer Ashuach: And… To make which one more? Just doing exactly what we did, and I tried to look at the logic distribution.

383
00:52:49.090 --> 00:52:51.039
Tomer Ashuach: And they look very weak.

384
00:52:51.930 --> 00:52:58.050
Tomer Ashuach: Like, I was not super convinced that specific NLP neurons had systematically

385
00:52:58.080 --> 00:53:14.229
Tomer Ashuach: Like, sometimes you find these concepts for… So, okay, let me stop you right there. You're… maybe you're talking about concepts, not memorized tokens, and I have another work that's talking about concepts, and I agree with you in this specific,

386
00:53:14.230 --> 00:53:28.510
Tomer Ashuach: issue, but I suggest that we continue, and please, and let's have this talk later on, because I want to continue, but I have a lot to talk about also. But I… the next work is going to be about concepts, and I tend to agree with what you say.

387
00:53:28.540 --> 00:53:41.329
Tomer Ashuach: So, basically, given that, what you… after we applied our method, what you see is the rank is basically going way lower, and the model stopped generating the specific memorized token.

388
00:53:41.330 --> 00:53:50.439
Tomer Ashuach: And for the sake of time, I'll just tell you it works very well, and compared to 3 datasets and 6 baselines, which works the best.

389
00:53:50.440 --> 00:53:52.230
Rohit Gandikota: Oh, Tomar, can I ask a question?

390
00:53:52.740 --> 00:53:54.309
Tomer Ashuach: Sorry again? You moved on.

391
00:53:55.120 --> 00:53:56.180
Rohit Gandikota: Can I ask a question?

392
00:53:56.180 --> 00:54:13.120
Tomer Ashuach: One second from the Zoom? From the second MLP model, yeah. Not from the attention, yeah, because we have other, like, intuition, and one of them is the MLP's, keys-value memory from Moros gave a work, but basically only the MLP.

393
00:54:13.410 --> 00:54:14.250
Tomer Ashuach: retarget.

394
00:54:14.920 --> 00:54:17.279
Tomer Ashuach: Yeah, is there a question in Zoom? On Zoom?

395
00:54:17.280 --> 00:54:24.129
Rohit Gandikota: Yeah, so the edit is completely sample agnostic, right? You directly take the weight, and then

396
00:54:24.250 --> 00:54:30.590
Rohit Gandikota: You upscale it into log… like, you look at the logic length of that, and then you reduce the…

397
00:54:31.030 --> 00:54:32.540
Rohit Gandikota: The token rank down.

398
00:54:32.940 --> 00:54:39.700
Rohit Gandikota: So, like, I'm thinking about how practical this can be. So, if I… if I come up with 1 million people who don't want to be

399
00:54:40.010 --> 00:54:45.870
Rohit Gandikota: In the data… in the language model. Maybe it'll cover, like, half of the tokens.

400
00:54:46.350 --> 00:54:52.579
Tomer Ashuach: Yeah, okay, I see what you're getting. You're saying, basically, what if I need to remove all the tokens, right?

401
00:54:52.730 --> 00:54:53.210
Rohit Gandikota: Yeah.

402
00:54:53.210 --> 00:54:54.830
Tomer Ashuach: Because it's targeted.

403
00:54:54.830 --> 00:55:00.290
Rohit Gandikota: And also, if I want… if you are removing the LEW token from…

404
00:55:00.290 --> 00:55:00.890
Tomer Ashuach: Huh.

405
00:55:01.460 --> 00:55:08.699
Rohit Gandikota: Like, will it also affect… To talk about Lewis Hamilton, or something, because it's still LAW.

406
00:55:08.700 --> 00:55:26.100
Tomer Ashuach: That's a good question. So, basically, we made an… first of all, the, the amount of data that we tried on is a couple of hundred examples, because we used, like, an actually memorized sensitive information from actual data set that the model trained on, so it's very hard to obtain.

407
00:55:26.100 --> 00:55:35.700
Tomer Ashuach: But we… we try to evaluate what you said, basically how it affects… for example, we have a data set on numbers, so we try to evaluate

408
00:55:35.700 --> 00:55:47.149
Tomer Ashuach: given erased number in one context, how it's affecting another. So, I want to tell you that it has some effect on, in general speaking, but it's not that

409
00:55:47.360 --> 00:55:57.549
Tomer Ashuach: it's not able, the model to produce Lewis Hamilton afterwards. It just… when you do it enough times, it's starting to affect a general case, but not,

410
00:55:57.550 --> 00:56:08.940
Tomer Ashuach: it does… it doesn't break the model completely, in the sense it's still able to generate in other context… contexts, the target token. But it is… it is,

411
00:56:09.040 --> 00:56:14.770
Tomer Ashuach: Left in a relative small scale of few hundred samples, in this sense.

412
00:56:14.910 --> 00:56:16.350
Tomer Ashuach: So I hope this answers.

413
00:56:16.620 --> 00:56:20.920
Rohit Gandikota: Got it. So, do you think about this? How you might want to scale this?

414
00:56:20.920 --> 00:56:26.109
Tomer Ashuach: Honestly, I don't think about it, because I think, these…

415
00:56:26.290 --> 00:56:38.289
Tomer Ashuach: This sort of, issue of memorized information needs to maybe be approached by some other localization technique, because

416
00:56:38.290 --> 00:56:54.429
Tomer Ashuach: So far, this work… this method that I present you works very good in the sense of how strong it removes information, but the scaling factor, because we haven't tested it in very large scaling, I'm not sure this would be actually suitable for

417
00:56:54.430 --> 00:56:59.920
Tomer Ashuach: Let's say, implementing in real-world scenario, so…

418
00:56:59.920 --> 00:57:01.049
Rohit Gandikota: Yeah, that's fair.

419
00:57:01.070 --> 00:57:19.840
Tomer Ashuach: Yeah, so I assumed there would be, like, some, you know, regularization technique we can apply to avoid, like, catastrophic forgetting in large scale, but this is something I haven't explored, and I agree it's, like, one thing to look for.

420
00:57:20.230 --> 00:57:21.160
Rohit Gandikota: Okay, thanks.

421
00:57:21.330 --> 00:57:28.860
Tomer Ashuach: Exit, yeah, this one. Oh, the next one. The next one. This one? Yes. Yeah.

422
00:57:29.470 --> 00:57:33.960
Tomer Ashuach: So, did you remove the, actually, the…

423
00:57:34.100 --> 00:57:48.679
Tomer Ashuach: Yeah, so only those specific layers. All of them. All of the ones that… it's our circle. So basically, those are the ones that are pushing the model towards the…

424
00:57:49.210 --> 00:57:52.790
Tomer Ashuach: It was for a specific prompt, right? Correct.

425
00:57:53.010 --> 00:57:58.609
Tomer Ashuach: And you do it for a lot of prompts simultaneously.

426
00:57:59.020 --> 00:58:07.809
Tomer Ashuach: No, so you have a prompt and email, and you have a list of hundreds of those, and you… the method removes all of them.

427
00:58:08.350 --> 00:58:21.009
Tomer Ashuach: What I'm asking is that for this specific email, there might be another prompt that will… Yeah, this is a great question. I didn't have time for that, but I did make a specific

428
00:58:21.020 --> 00:58:34.140
Tomer Ashuach: synthetic data set, which does exactly that. It has five-point prompts, all of them, memorize the same email address. Not email address, but we tested social security numbers, and just given one prompt.

429
00:58:34.360 --> 00:58:49.340
Tomer Ashuach: the method deletes given one prompt and the target, and then we measure for the whole other four prompts, deletes for the same, memorized social security numbers. It works really well. This is the… actually, this one here.

430
00:58:50.220 --> 00:58:51.720
Tomer Ashuach: And you can see that

431
00:58:51.980 --> 00:59:07.569
Tomer Ashuach: compared to other methods, this works much better. But it comes with a trade-off that I just talked with Rohit about, about scalability, and it's a very strong method to apply, but it has, like, when you do it in scale, it has some overlapping of information that starts to interfere.

432
00:59:08.900 --> 00:59:18.290
Tomer Ashuach: You fine-tune the model… Yes, because this is a very unique setup, right? It's very hard to think about…

433
00:59:18.330 --> 00:59:35.920
Tomer Ashuach: memorized email address, like, to find… we've tried to find memorial memory email addresses that the model memorized, given a few different prompts, but the same email. So, I couldn't find any, like, naturally. I had to come up with a synthetic data set for that.

434
00:59:37.370 --> 00:59:39.070
Tomer Ashuach: Sure. So…

435
00:59:40.260 --> 01:00:04.379
Tomer Ashuach: Sorry. So, we talked so far about revs, which is about targeting memorized string, of sensitive information, but, let's go about thinking about concepts. So, sometimes we want to remove, concepts like, books or harmful knowledge. In this example, if we want to target Harry Potter concept, there are a lot of connected, related information that are not

436
01:00:04.770 --> 01:00:14.000
Tomer Ashuach: exist when you just look on the Harry Potter string, like Hogwarts, Dumbo, etc. So, how can we unlearn an entire concept?

437
01:00:14.110 --> 01:00:24.449
Tomer Ashuach: And this worked called CRISP, and it's leveraging sparse autoencoders to remove conceptual data from models.

438
01:00:25.060 --> 01:00:46.899
Tomer Ashuach: So, again, I'll do it quickly, but basically, I've showed… I told you that when trying to ask model for harmful information, it refuses. So, for here, I ask him for, like, how to extract ricin or enhance pathogen virulence, and he tells me, of course, I can't help you with that. But with some persuasion, and, you know, I was pretty consistent.

439
01:00:46.900 --> 01:00:50.860
Tomer Ashuach: trying to extract the information. Finally yielded and gave me this

440
01:00:50.860 --> 01:00:58.399
Tomer Ashuach: pretty, elaborated step-by-step guide, right? It's pretty convincing. So, again, I hope this gives you, like, the motivation why it's

441
01:00:58.400 --> 01:01:03.200
Tomer Ashuach: Necessary to remove information, not just, refusal training.

442
01:01:03.290 --> 01:01:19.079
Tomer Ashuach: So basically, our goal here is to precisely find some concepts that we want to remove, and, via spouse or encoders, and then somehow train or change the model weights to remove the information that corresponds to the concept.

443
01:01:19.740 --> 01:01:27.820
Tomer Ashuach: So, for that, we're going to leverage sparse open encoders, which basically decompose polysymantic activations in the latent space to sparse features.

444
01:01:27.830 --> 01:01:37.069
Tomer Ashuach: And basically, one feature is roughly equal to one atomic concept, and mostly SAEs are used

445
01:01:37.070 --> 01:01:50.810
Tomer Ashuach: as observatory tool, or for steering sometimes for interventions. And in CRISP, we target… we move from target tokens to… in REVs, we target now features that, again, corresponds to some concept.

446
01:01:51.000 --> 01:01:52.959
Tomer Ashuach: So, how does it work?

447
01:01:52.960 --> 01:02:14.629
Tomer Ashuach: We have two phases. The first one is we analyze the features that we want to, target and remove the concept that corresponds to them. And the second one, we target… we fine-tune the model to forget this, concept using the active… using the activated features. So, what is the setup? We have, basically two core… corpora.

448
01:02:14.630 --> 01:02:22.240
Tomer Ashuach: One is the target corpus. In this setup, I guess maybe Rohit is probably familiar with it, it's called WNDP.

449
01:02:22.280 --> 01:02:35.890
Tomer Ashuach: It's about cybersecurity and biosecurity knowledge. So the target corpus would look like how to make a smallpox virus and stuff like that. And the benign corpus would look like

450
01:02:35.930 --> 01:02:53.330
Tomer Ashuach: general biology knowledge. So, our target is to remove the target information, which is problematic, or potentially harmful, while retaining the general biology. And as you can imagine, this is a pretty hard task, because

451
01:02:53.680 --> 01:02:58.780
Tomer Ashuach: It's not clear how you draw the boundary between what's harmful and what's not.

452
01:02:58.800 --> 01:03:16.330
Tomer Ashuach: And in our case, we have a benchmark, WMDP, which gives you a set of documents that say, this is harmful and this is not. So this is… and also questions to evaluate how well you retain information, and how well you remove the information that you want to remove.

453
01:03:17.450 --> 01:03:20.819
Tomer Ashuach: Okay, so how do we find the salient features that we want to target?

454
01:03:21.070 --> 01:03:37.989
Tomer Ashuach: We pass the target and the benign Corpora both through the model, and the assays, and collect their activations, and then we run… we aggregate the statistics about those concepts, about those features, and we…

455
01:03:37.990 --> 01:03:47.430
Tomer Ashuach: Basically, plot, we find which features are problematic in the sense of they are activated more frequently on the target.

456
01:03:47.590 --> 01:03:55.399
Tomer Ashuach: compared to the B9, and also how strongly it's activated the specific features on the target compared to the B9.

457
01:03:55.540 --> 01:04:04.899
Tomer Ashuach: And when you plot the features, and what you're gonna see here now is a lot of features plotted, each dot is a feature, and the y-axis tells you how

458
01:04:04.900 --> 01:04:17.750
Tomer Ashuach: how frequent a feature is in the target, and how… and the x-axis tell you how frequent a feature is in the benign corpus. And again, each dot here is a feature.

459
01:04:17.750 --> 01:04:33.059
Tomer Ashuach: And across the diagonal, what you can see, that there are a lot of features that are basically activated, relatively similarly to both target and benign. Those are features that we are not interested about, because they are, like, generic features in this domain.

460
01:04:33.460 --> 01:04:46.870
Tomer Ashuach: What we care about is this set of features, right? Because, as you can see, they are highly activated on target, but not so much on the b9. So, for… as an example, this feature is activated about on…

461
01:04:46.870 --> 01:04:58.380
Tomer Ashuach: one and a half… 1.5 million times on target, but only about 100K, so 15 times more on the problematic documents that we want to remove the information from.

462
01:04:58.610 --> 01:05:09.959
Tomer Ashuach: And one pretty cool, thing you can see is that we can leverage these sparse autoencoder's features that are mostly interpretable to audit

463
01:05:10.000 --> 01:05:21.539
Tomer Ashuach: the features that we target before we even touch the model, so kind of verify that we want to add… to remove the information, and it corresponds to what we actually want to remove. And pretty neatly, as you can see here at the top.

464
01:05:21.610 --> 01:05:26.040
Tomer Ashuach: features, but I can tell you that all the other features looks very similar to that.

465
01:05:26.650 --> 01:05:30.149
Tomer Ashuach: Are, activated on, actually.

466
01:05:30.400 --> 01:05:41.799
Tomer Ashuach: problematic, or in this sense, target corpus information. So this is… those are, like, the logic lens of the features, and you can see the first one is a virus, the second one is small, which correspond to smallpox.

467
01:05:42.140 --> 01:05:50.670
Tomer Ashuach: And you can see that this is Neuronpedia, if you're familiar with it, so this is not that I did this explanation, it's another tool that extracted it.

468
01:05:50.700 --> 01:06:06.229
Tomer Ashuach: And as you can see, it's… one of them is terms related to viral infections in mammals, and the other one is smallpox virus-related information. So basically, we know that the features we're going to target are the features we actually want to remove, and not just, like, spruce correlated.

469
01:06:07.240 --> 01:06:24.949
Tomer Ashuach: And generally speaking, the objective… our objective is defined by three losses terms. The first one is to unlearn the information, and we do that by suppressing the activation of the features, of the selling features we choose.

470
01:06:25.180 --> 01:06:30.680
Tomer Ashuach: Such that they are, becoming not activated during the problematic context.

471
01:06:30.910 --> 01:06:48.659
Tomer Ashuach: The second one is retaining, because we want to retain good general biology information, so we basically aim to retain the same activation given benign biology information. And coherency term is to preserve coherency text on the target domain, like

472
01:06:48.660 --> 01:06:54.190
Tomer Ashuach: viral replication. So, you want the model to be able to continue to talk about

473
01:06:54.190 --> 01:07:02.009
Tomer Ashuach: viruses in general, and not being, like, repetitive or weird about it. Generated text.

474
01:07:02.780 --> 01:07:15.680
Tomer Ashuach: So, here we do a lot of experiment for evaluation, and our goal is to go to these, star marks, which, indicates perfect unlearning in the sense of it's a random guess.

475
01:07:15.980 --> 01:07:24.170
Tomer Ashuach: On the evaluation, and also retaining the holding from the all original information from the original knowledge, the no-touched one.

476
01:07:24.230 --> 01:07:40.029
Tomer Ashuach: So, here you can see about 200 experiments, and what… in the blue dots, we have, our method, and as you can see, up until almost, random guess, you get almost no,

477
01:07:40.290 --> 01:07:59.340
Tomer Ashuach: breaking the model knowledge on the retained information. So this is kind of suggests that when you disentangle PSPAR7 the model, to only specific concepts you want to target, it's much easier to remove the information strongly without harming,

478
01:07:59.540 --> 01:08:01.919
Tomer Ashuach: General information from the model, yeah.

479
01:08:06.960 --> 01:08:12.840
Tomer Ashuach: Okay, so… so we do use LoRa for efficiency, and we target, yeah, all the layers, specifically.

480
01:08:13.450 --> 01:08:19.549
Tomer Ashuach: But I've tried with only, like, subset of layers, like, only the middle. It works pretty well, but…

481
01:08:20.120 --> 01:08:25.539
Tomer Ashuach: worse, but not that worse. Like, we just got the best number with all the errors.

482
01:08:31.760 --> 01:08:32.430
Tomer Ashuach: Correct.

483
01:08:39.300 --> 01:09:00.510
Tomer Ashuach: Yeah, you can do that by basically tracking the features, right? You can look without even needing to fine-tune anything. You can track which features you find the most salient in which layers, and then track, kind of, which, how many of them, or how highly they're activated, and you can get, kind of, analysis of

484
01:09:00.510 --> 01:09:01.899
Tomer Ashuach: layers mapping.

485
01:09:01.899 --> 01:09:02.520
Tomer Ashuach: Oof.

486
01:09:02.520 --> 01:09:20.250
Tomer Ashuach: which layer activated the, more strongly, which features that are relevant. You can do this analysis. We haven't done this specific analysis, but I think it might be interesting to do as well. But I would say, from my experience, it's pretty diffused in the sense of

487
01:09:20.260 --> 01:09:31.059
Tomer Ashuach: There is no specific layers that contain all the harmful knowledge about biology, or not even, very, neatly as subgroups of neurons.

488
01:09:32.640 --> 01:09:37.140
Tomer Ashuach: Yeah. What was the size of your, synthetic data server?

489
01:09:37.479 --> 01:09:45.160
Tomer Ashuach: how easy it is for these models to memorize the sensitive information. Okay, so first of all, this is not a synthetic data set, this is an…

490
01:09:45.210 --> 01:09:57.929
Tomer Ashuach: articles, about… so WMDP is a benchmark we didn't… we used as is, and the biology setup is about 16K articles, like, papers about biology.

491
01:09:57.980 --> 01:10:12.699
Tomer Ashuach: which half of them are… roughly half of them are harmful, and half of them are not. And we evaluate on biosecurity, that one. Cybersecurity, which has much, much less data, but still, is from the WMDP, and also Harry Potter.

492
01:10:12.880 --> 01:10:15.459
Tomer Ashuach: Which is much, much less, data.

493
01:10:15.600 --> 01:10:18.819
Tomer Ashuach: So after this intervention, if I fine-tune it again.

494
01:10:18.820 --> 01:10:36.950
Tomer Ashuach: on, let's say, a few hundred or thousands of platforms. Okay. Will it be less likely to memorize it now? So, so you're talking about, like, retraining attack, maybe? Yeah, like, fine-tuning it again, after you've done… Like, for a part of the data, and then check whether it's memorized, like, after you're done with this, after you remove the new laws that are the…

495
01:10:37.000 --> 01:10:38.669
Tomer Ashuach: You suppress the latency.

496
01:10:38.860 --> 01:10:44.709
Tomer Ashuach: Now, if I again fine-tune it on a memorized data set, is it less likely to learn it now than before?

497
01:10:45.510 --> 01:10:54.500
Tomer Ashuach: This is a good question, and this is kind of a known attack, but we haven't, actually evaluated on this attack, so I can't tell you honestly.

498
01:10:54.610 --> 01:11:05.250
Tomer Ashuach: From my experience, from other methods, it probably would work to some extent, in the sense of when you're retraining, write on new data, it's gonna learn this new data to some extent, so…

499
01:11:05.250 --> 01:11:14.749
Tomer Ashuach: In the sense of, if you take, like, a model that hasn't seen some sort of data, it's naturally unlearned it, right? So if you train on this data, it's gonna learn it, so…

500
01:11:14.850 --> 01:11:22.219
Tomer Ashuach: It's probably gonna work. I'm not sure, like, relative to not unlearned model, how it's gonna be better or worse.

501
01:11:23.850 --> 01:11:25.040
Tomer Ashuach: And…

502
01:11:25.520 --> 01:11:43.620
Tomer Ashuach: for the sake of time, I'm gonna skip that, but basically I want to tell you that other methods are more… tend to generate, incoherent text after, after the unlearning, and our method is, mostly retaining the coherency of the… sorry, of the text.

503
01:11:46.090 --> 01:11:54.710
Tomer Ashuach: And so, and to summarize, so do you think the key… what's the key aspect of your method that you think leads to the good performance? Is it the SAE?

504
01:11:54.930 --> 01:12:12.930
Tomer Ashuach: feature selection? Is that the key thing? So I think the key thing is that we can surgically pinpoint relevant, direction within the space to suppress only them, and not to approximate it only based on the data. Because most of the, other, technique

505
01:12:12.930 --> 01:12:23.019
Tomer Ashuach: work on, basically, you have two data sets, the one that you want to target, and the one you want to retain, and you unlearn all the targets. But the target, when you think about it, is, for this example.

506
01:12:23.020 --> 01:12:36.100
Tomer Ashuach: is just articles of harmful related to biology information, and a lot of the tokens are not relevant to harmful information, right? You have a lot of their own or other stuff that general biology

507
01:12:36.100 --> 01:12:55.719
Tomer Ashuach: information you want to retain. In our method, we pinpoint only for each, for each token, only specific direction within the latent space that we want to suppress, not all of the direction, and some tokens we don't touch at all, in the sense of they are not activated on the harmful

508
01:12:55.740 --> 01:13:07.950
Tomer Ashuach: features. So, I think the strength of the method is based on the fact that we are able to very surgically find direction that only then we are gonna touch.

509
01:13:09.240 --> 01:13:11.180
Tomer Ashuach: And do you think that,

510
01:13:11.360 --> 01:13:14.299
Tomer Ashuach: These directions could have been found.

511
01:13:14.700 --> 01:13:16.810
Tomer Ashuach: without the SAE…

512
01:13:17.030 --> 01:13:24.849
Tomer Ashuach: you know, pre-training, you know, the SAE, selection, like, do you think that they put it in a different

513
01:13:25.430 --> 01:13:31.559
Tomer Ashuach: Without SAs. Optimization that could have gone those directions without SAs, or do you think that the SAPs are eccentric?

514
01:13:32.460 --> 01:13:42.680
Tomer Ashuach: So, in our… in our setup, they're essential in the sense they are using them, but I think it would be possible, for sure, to find a way that you kind of…

515
01:13:42.820 --> 01:13:49.490
Tomer Ashuach: not, utilizing SAEs, but still, get the… the benefits of

516
01:13:49.700 --> 01:14:07.499
Tomer Ashuach: you know, a small subset of directions that corresponds to the problematic knowledge. I've no… I have no, like, concrete idea so far, but I think it would be possible. For example, we can train a prop, right, to predict, is it a harmful activation or benign activation?

517
01:14:07.500 --> 01:14:23.179
Tomer Ashuach: And we can kind of, target the information that, like, the direction that the probe detects. But, there is a question about how many props, like, is one direction is enough to separate those? You need multiple. So, you can think of SAs as a lot of props, right?

518
01:14:23.180 --> 01:14:34.169
Tomer Ashuach: linear… this is basically what they are, and that they activate in some strength. So, in the sense, you can train, I don't know, with… to fix, like, 100 probes, to train

519
01:14:34.170 --> 01:14:45.310
Tomer Ashuach: you know, different aspects of malicious concepts, and from the target NV9, and, to use this to… and apply the same method, basically. So…

520
01:14:45.910 --> 01:14:51.300
Tomer Ashuach: you need some tool that, like, defined the direction, but it doesn't have two bases.

521
01:14:52.450 --> 01:14:56.069
Tomer Ashuach: I hope this answers. Just trying to get an insight from… Yeah.

522
01:14:56.540 --> 01:15:00.979
Tomer Ashuach: Sure, sure. Any other… yeah, sure.

523
01:15:02.170 --> 01:15:04.070
Tomer Ashuach: I just said, thank you.

524
01:15:04.400 --> 01:15:05.690
Tomer Ashuach: Like, Stuart.

525
01:15:05.910 --> 01:15:06.869
Tomer Ashuach: Let's do it.

526
01:15:07.100 --> 01:15:09.609
Tomer Ashuach: And then, what does it look like?

527
01:15:10.930 --> 01:15:12.189
Tomer Ashuach: It's like that, maybe.

528
01:15:12.620 --> 01:15:28.340
Tomer Ashuach: It's far worse. I've tried that. This is the first… so there is an actual work that did unlearn. I'm doing that because this is when you think about not an actual unlearning, right? Because you don't touch the model weights, and we go back to the refusal thing that I've talked about.

529
01:15:28.570 --> 01:15:48.030
Tomer Ashuach: But, basically, when you do the steering, you suppress some information within the residual space, actively, but you have to, so, empirically, it does… it works in the sense of it's lowering the accuracy on the harmful information, but, way…

530
01:15:48.180 --> 01:15:52.809
Tomer Ashuach: it's much better than random. It stays about… so if the… let's go to…

531
01:15:53.190 --> 01:16:09.929
Tomer Ashuach: this, so let's say the original model, information about the, the unlearned accuracy started 70%, so it gets you about here, like 50%. So it works in some sense, but it's a very weak, way to

532
01:16:10.180 --> 01:16:12.779
Tomer Ashuach: To make the model not know the information.

533
01:16:12.790 --> 01:16:24.229
Tomer Ashuach: By fine-tuning the model, I think why it works so good. So, first of all, because you actually change the weight, so you can reshape the model to such way it's, not producing this

534
01:16:24.230 --> 01:16:35.269
Tomer Ashuach: directions anymore, so it's not only that the direction does not exist within the residual stream, but you're actually reshaping the weights of the model to not produce this direction, so…

535
01:16:35.330 --> 01:16:38.269
Tomer Ashuach: This is much stronger, and also we are not only

536
01:16:38.540 --> 01:16:49.399
Tomer Ashuach: zeroing the… when I suppress the activation, we are not targeting this to zero, we're targeting this to some negative value, so it's even, like, stronger suppressing.

537
01:16:50.010 --> 01:16:51.080
Tomer Ashuach: So, just here.

538
01:16:51.900 --> 01:16:57.759
Tomer Ashuach: So, I guess… Just say you secure it, open it.

539
01:16:57.870 --> 01:17:03.480
Tomer Ashuach: Like, that… The retain accuracy is still high, but their accuracy isn't that effective.

540
01:17:03.610 --> 01:17:09.609
Tomer Ashuach: Yeah, so, okay, so you can steer very strongly, and then you, I can guarantee you get

541
01:17:09.830 --> 01:17:17.370
Tomer Ashuach: random unlearn score, but it breaks in the sense of you get, like, a strong decline. So it's worse than all of those metals.

542
01:17:17.900 --> 01:17:18.600
Tomer Ashuach: It's good.

543
01:17:18.760 --> 01:17:19.450
Tomer Ashuach: Goodbye.

544
01:17:22.460 --> 01:17:35.999
Tomer Ashuach: Not the coherency, but the retain loss, because you're learning how to change the weight such that only the concepts, only the features you want to target are suppressed, but other features that are relevant to both

545
01:17:36.140 --> 01:17:45.420
Tomer Ashuach: both retained and problematic knowledge are… remain intact, basically. This is the key concept of this map, basically.

546
01:17:47.700 --> 01:17:51.670
Tomer Ashuach: So that's the code, like… Love that.

547
01:17:52.420 --> 01:17:54.810
Tomer Ashuach: A what batch?

548
01:17:55.090 --> 01:17:57.169
Tomer Ashuach: compute match output box for us.

549
01:17:57.810 --> 01:18:04.089
Tomer Ashuach: So how, how compute-intensive is that? Well, just if you were, like, to take the same amount of…

550
01:18:04.250 --> 01:18:07.819
Tomer Ashuach: And just, like, train an output her. Just, like…

551
01:18:07.920 --> 01:18:27.399
Tomer Ashuach: Okay. So, first of all, I'm using, train… like, pre-trained essays, so we didn't train them. So, and second of all, we used LoRAS, which are very efficient sources of… for, you know, fine-tuning the model. So, in this sense, it's very efficient.

552
01:18:27.490 --> 01:18:31.389
Tomer Ashuach: And other model,

553
01:18:31.440 --> 01:18:50.849
Tomer Ashuach: like, RMU is much more consuming. You're targeting only one layer, but the whole parameters, so it's much more heavy. I'm not sure if that answered your concern, or not so? Well, I guess I'm just trying to get a sense of, like, if you just train the model to take intense, so, like, should I discover as a staff?

554
01:18:51.140 --> 01:18:53.339
Tomer Ashuach: Like, where that would fall.

555
01:18:56.620 --> 01:19:04.100
Tomer Ashuach: So, again, train a classifier to predict what's harmful from benign, right, right, and then on reply just…

556
01:19:04.380 --> 01:19:06.200
Tomer Ashuach: Yeah, but nothing there.

557
01:19:06.940 --> 01:19:14.710
Tomer Ashuach: So you're talking about, like, posthoc guardrail, kind of? So, okay, so, this probably would be much cheaper?

558
01:19:15.100 --> 01:19:32.009
Tomer Ashuach: But, think about the following scenario. You train a pro… basically, you train, like, a probe, on the activation of the model, that you are trying to predict when to tell the model to refuse, right? Basically, or to not produce. So this is what a lot of companies are doing, basically.

559
01:19:32.010 --> 01:19:42.120
Tomer Ashuach: It probably would be cheap, maybe not so cheaper, because you're still gonna use the same data, right, to classify activation, and this is what we do, we train LoRa instead of classifier.

560
01:19:42.120 --> 01:19:45.280
Tomer Ashuach: But it's the same, let's say,

561
01:19:45.490 --> 01:19:49.150
Tomer Ashuach: You know, ballpark of number of computes.

562
01:19:49.660 --> 01:20:01.240
Tomer Ashuach: But I think the… the issue is that you're still not removing the underlying information, so this is depend… depend on, like, your point of view of this, domain. If you think

563
01:20:01.840 --> 01:20:20.629
Tomer Ashuach: like, making guardrails, good guardrails, are the direction to go to. I think this is a good solution to go with. But in my point of view, I think that we need methods to specifically remove some instances of information from the model, actual weight. So.

564
01:20:20.810 --> 01:20:39.909
Tomer Ashuach: I know some recent, like, works, papers, which are about how you can train… make the model to, change its activation, such that… such that, given probe that trained to, classify some information within them.

565
01:20:40.240 --> 01:20:57.199
Tomer Ashuach: are not able to, to find it anymore. So, basically, the information retained within the model, but it reshapes the activation so that the probe cannot identify it. So, in the sense, we're getting back to the race of the mouse and the cat, right? When you don't remove the information.

566
01:20:57.200 --> 01:21:01.929
Tomer Ashuach: But you have some, wrapper that prevents you from doing so, and…

567
01:21:01.930 --> 01:21:05.550
Tomer Ashuach: And inheritingly, if you don't remove the information.

568
01:21:05.820 --> 01:21:09.060
Tomer Ashuach: Somebody will come up with a solution to how to extract it.

569
01:21:09.250 --> 01:21:13.730
Tomer Ashuach: Yeah, yeah, I guess I'm just wondering, like, do you have a sense of each of these, maybe.

570
01:21:14.240 --> 01:21:17.440
Tomer Ashuach: Like, to what extent are these duplicative versus, like.

571
01:21:17.580 --> 01:21:20.199
Tomer Ashuach: What do you apply? That's adaptable?

572
01:21:20.920 --> 01:21:23.010
Tomer Ashuach: Multiple networks at the same time.

573
01:21:23.160 --> 01:21:29.890
Tomer Ashuach: get better results that way. Like, multiple, unlearning methods of the same… Yeah.

574
01:21:30.020 --> 01:21:32.820
Tomer Ashuach: This is an interesting, I haven't tried it.

575
01:21:33.290 --> 01:21:46.390
Tomer Ashuach: This is a good, maybe, like, an interesting direction to… to pursue, but I haven't tried it yet, and I'm actually not familiar with any… with any, like, work that tried to… to do that, like, multiple unlearning methods at once.

576
01:21:49.210 --> 01:21:59.260
Tomer Ashuach: So I have actually another, work that I was planning to talk about, but I'm not sure, for the sake of time, we should, pursue that, so…

577
01:21:59.570 --> 01:22:02.759
Tomer Ashuach: Do we have, like, a few more minutes, or we have to wrap up?

578
01:22:03.720 --> 01:22:16.099
Tomer Ashuach: It's totally fine. David, are you interested in hearing about privileged knowledge? I mean, I'm interested, but I was just… maybe you're the only one? How's everybody doing on time? Do we… should we… is this something that you can go through?

579
01:22:16.270 --> 01:22:24.149
Tomer Ashuach: Yeah, like, 5 minutes, I can… Okay, so this… I'll try to do it quickly, and if you're more interested about it, we can chat later on.

580
01:22:24.150 --> 01:22:40.089
Tomer Ashuach: So, basically, lastly, last work, we are talking about privileged knowledge, and we are trying to understand whether it exists in language models or not, and think about the following, intuitive, like, to give intuition, thought experiment. So.

581
01:22:40.090 --> 01:22:51.739
Tomer Ashuach: For example, let's say I ask you, David, about 100 trivia questions, like what capital of France, but I'm not telling you to tell me the answer, I'm telling you to tell me whether you know the answer to it or not.

582
01:22:51.770 --> 01:23:02.209
Tomer Ashuach: So, you'll do pretty accurate in the sense of you'll be pretty good to predict whether you know it or not, right? Because you have a privileged access to your own memory, you have an introspection.

583
01:23:02.550 --> 01:23:06.000
Tomer Ashuach: But if I ask a new Natalie to predict David's.

584
01:23:06.200 --> 01:23:14.700
Tomer Ashuach: answers, whether he's gonna be correct or not, you'd be pretty good as well, but not as good as him, right? Because you do not have an access to his brain.

585
01:23:14.780 --> 01:23:29.710
Tomer Ashuach: So you'd be much better than random person… random baseline, but not as good as him. So this is privileged knowledge, basically. So we asked, similarly, does models have it? And we define it as an internal signal that an outside observer cannot recover.

586
01:23:30.480 --> 01:23:51.519
Tomer Ashuach: And we check this in correctness setup, and we have a lot of questions. We have a target model, which is basically David Bao trying to predict his own correctness of the questions, and then we collect the area under the curve, and then we have other external models, which are Natalie, me, Mikhail, trying to predict

587
01:23:51.520 --> 01:23:54.529
Tomer Ashuach: David's, correctness, whether he's gonna be correct or not.

588
01:23:54.530 --> 01:23:58.639
Tomer Ashuach: And we are gonna say that, David has,

589
01:23:58.640 --> 01:24:16.450
Tomer Ashuach: privileged knowledge if he outperforms the prediction of all of the external predictors, right? So this suggests that he has some information that is privileged only to him. So we want to see if the target model has this attribute as well.

590
01:24:16.770 --> 01:24:19.800
Tomer Ashuach: So… By, behavior.

591
01:24:20.120 --> 01:24:26.929
Tomer Ashuach: this is… we are… I skipped it for the sake of time, but we take the activation of the question and train prob on the activation from each model.

592
01:24:27.070 --> 01:24:40.250
Tomer Ashuach: Okay? So, we take his thought to predict his correctness, your thought to predict his correctness, and my thought to predict his correctness as well, etc. So, just quickly, who thinks it exists or not? Who thinks it exists?

593
01:24:40.490 --> 01:24:42.049
Tomer Ashuach: I thought it was exist.

594
01:24:42.370 --> 01:24:48.860
Tomer Ashuach: Okay, so… Yeah, it's kind of, I don't know, intuitive. So, we found it's…

595
01:24:48.990 --> 01:24:54.840
Tomer Ashuach: There is no, like, privilege advantage to use the model, the self-model.

596
01:24:54.840 --> 01:24:55.320
Rohit Gandikota: How do you…

597
01:24:55.320 --> 01:25:11.999
Tomer Ashuach: I was able to predict David's thoughts or correctness just as well he was able to, so I think this is kind of counterintuitive, and we said, okay, maybe something is fishy here, and we found something fishy, so there is a confound that basically,

598
01:25:12.150 --> 01:25:16.029
Tomer Ashuach: All the models that we trained… that we looked on,

599
01:25:16.030 --> 01:25:34.600
Tomer Ashuach: were 80% overlap between their answers. So, if David answered yeah, was correct on something, I was correct on something in 80 of the cases, and vice versa. So, when we remove this confounder by evaluating only on places where David was correct and I was not correct, or vice versa.

600
01:25:34.630 --> 01:25:47.760
Tomer Ashuach: Then we get, in factual recall dataset, a consistent, statistically significant, privileged knowledge signal, meaning that, all the models were able to predict their own correctness much

601
01:25:47.850 --> 01:26:00.530
Tomer Ashuach: not much, but better, and it was statistically significant across all the models and all the factual recall datasets. Surprisingly enough, it exists only on factual recall, like Trivia QA, Hotpot, Mintaka, but not on math reasoning.

602
01:26:00.790 --> 01:26:04.569
Tomer Ashuach: Okay, it was super quick, I hope it was interesting, and feel free…

603
01:26:05.050 --> 01:26:08.520
Tomer Ashuach: Thank you. Feel free to reach me afterwards, yeah.

604
01:26:09.100 --> 01:26:21.749
Tomer Ashuach: information about, the setup, like, what token, what layer, what hidden… Yeah. Okay, so, last token of the question, we did a lot of…

605
01:26:22.060 --> 01:26:34.419
Tomer Ashuach: Layers, as you can see here. And basically, this plot says the first draw is that, in factual recall, the privileged knowledge signal is getting stronger as you go through the layers.

606
01:26:34.620 --> 01:26:43.190
Tomer Ashuach: Meaning that it has more privileged knowledge. And in math reasoning, as expected, because we showed you it's not existent, it's always hovering on zero.

607
01:26:43.510 --> 01:26:53.350
Tomer Ashuach: Basically. Have you trained the model? No, no, we just extract activations from last token of the questions.

608
01:26:53.460 --> 01:26:59.749
Tomer Ashuach: And those are, like, the X's of the probe, and then you have Y labels of the correctness, to predict the correctness.

609
01:27:00.060 --> 01:27:09.699
Tomer Ashuach: So, each probe from each activation is trained on the… Yeah, so… and when you train the probes, are you training the probes only on…

610
01:27:09.820 --> 01:27:13.299
Tomer Ashuach: The cases where there's a difference?

611
01:27:13.750 --> 01:27:24.530
Tomer Ashuach: No. So you're training the probes just on all the questions that are self-aggregated, but then you're evaluating it only on the places… Yes, yes.

612
01:27:27.600 --> 01:27:34.980
Tomer Ashuach: Yeah, so, again, sorry for, like, skipping a lot of… for the sake of time, but feel free to reach me afterwards. Thank you very much. Thank you, guys.

613
01:27:38.320 --> 01:27:41.149
Tomer Ashuach: Interesting word, both of you. Thank you so much.