WEBVTT

1
00:00:01.940 --> 00:00:03.010
Sukrut Rao: Duh.

2
00:00:06.930 --> 00:00:10.000
Sukrut Rao: Put this in on here. Did it say recording?

3
00:00:10.670 --> 00:00:13.260
Sukrut Rao: I didn't see anything. Okay, good. Thank you.

4
00:00:14.530 --> 00:00:15.320
Sukrut Rao: Great.

5
00:00:15.550 --> 00:00:17.029
Sukrut Rao: Click this on full screen.

6
00:00:17.760 --> 00:00:19.389
Sukrut Rao: Great, number one gross.

7
00:00:19.670 --> 00:00:21.240
Sukrut Rao: That's field that I've got them.

8
00:00:21.870 --> 00:00:23.850
Sukrut Rao: Is that the right button?

9
00:00:26.400 --> 00:00:28.320
Sukrut Rao: I think so, yeah. Great.

10
00:00:28.840 --> 00:00:33.790
Sukrut Rao: So, well, we're lucky to have Sukrit Rao, here speaking with us.

11
00:00:34.050 --> 00:00:34.990
Sukrut Rao: Today?

12
00:00:35.100 --> 00:00:40.619
Sukrut Rao: It looks like it's going to be talking about, interpretability in the multimodal setting.

13
00:00:40.780 --> 00:00:49.200
Sukrut Rao: Which is, I think, increasingly interesting as the multilater models get more and more powerful. So, what comes to Crete. Thank you.

14
00:00:50.680 --> 00:01:07.689
Sukrut Rao: So, should I start? Just one question, do I need to keep my speaker high, or… Can people online hear you? You turn your speaker up so that people can say, Sarah, you're remote, can you unmute yourself for a moment and say something so that we can see if we can hear you?

15
00:01:07.690 --> 00:01:08.710
Sarah Wang: Aye.

16
00:01:09.160 --> 00:01:12.260
Sukrut Rao: No, we can't hear you, so turn Okay, say again?

17
00:01:12.630 --> 00:01:13.490
Sarah Wang: Hi.

18
00:01:13.760 --> 00:01:20.399
Sukrut Rao: Hi, Sarah. Oh, yes, that's great. Thank you, Sarah. And Jasmine, and you can hear… you can hear, Sucra?

19
00:01:20.730 --> 00:01:21.410
Jasmine C.: Yeah.

20
00:01:21.970 --> 00:01:22.700
Sukrut Rao: Excellent.

21
00:01:23.230 --> 00:01:24.600
Sukrut Rao: Okay. Okay.

22
00:01:25.080 --> 00:01:44.300
Sukrut Rao: Yeah, thank you for the introduction, thank you for having me here. I'm a PhD student at the Max Black Institute for Informatics, supervised by Professor Bartsheiler, and broadly, I've been working on inherent interpretability of vision models, and also now moving towards how can we use this to improve multimodal models, and that's the overview of what I'll be talking about.

23
00:01:45.850 --> 00:02:04.910
Sukrut Rao: So, the broad motivation is that modern multimodal models, such as multimodal LLMs, are highly powerful, but are also brittle. So you have problems like model hallucinations, where you may… if you're doing something like captioning, you may get outputs that are not grounded in the image. We have problems like unfaithful reasoning, where the model might reason based on

24
00:02:05.000 --> 00:02:20.470
Sukrut Rao: entities that do not exist, or do not faithfully report how it does its reasoning. And also, there are safety issues, like it is possible with very simple tricks, often, to jailbreak these models. So we want to understand how these models work, and then use this understanding to help fix them.

25
00:02:21.490 --> 00:02:37.510
Sukrut Rao: So, because the models themselves are opaque, they are black boxes, understanding is quite key to fixing them, and broadly, multimodal models look like this. Multimodal LLMs, where you have a vision encoder that processes the vision part, and then also an LLM that processes the text part.

26
00:02:37.720 --> 00:02:51.709
Sukrut Rao: And specifically, our focus in my work and in this talk is going to be on the vision feature extractors, and this is also because vision encoders are a core component for visual perception, and so it's a useful first step to get towards the product goal that we are interested in.

27
00:02:52.440 --> 00:02:58.259
Sukrut Rao: So, and to do this, we specifically look into using interpretability tools, and

28
00:02:58.400 --> 00:03:11.199
Sukrut Rao: For this, I do not need to give an introduction here. So, basically, given a simple model, let's say if you have a classifier, then you have various tools available, such as attribution heat maps that tell you which part of the input,

29
00:03:11.200 --> 00:03:25.010
Sukrut Rao: influence the output. You have other methods, like concept decompositions, that explain in terms of higher-level human interpretable concepts, and you also have mechanistic ways of inspecting the model, such as finding out what are the roles of individual model units, what circuits exist, and so on.

30
00:03:25.770 --> 00:03:28.660
Sukrut Rao: So we want to use these tools to then get an understanding.

31
00:03:28.700 --> 00:03:39.989
Sukrut Rao: But specifically, we also want to get a faithful understanding of the model, and there are two big reasons for this. One is that by getting a faithful understanding, that is, an explanation that actually tells you what the model is doing.

32
00:03:39.990 --> 00:03:53.690
Sukrut Rao: you get reliable insights that can guide design decisions. So, for example, how does the model process information? What fast and desirable behavior? Or how can we build better and more efficient models with safety guarantees? So, basically, how we can bottom up, build models that are

33
00:03:53.690 --> 00:03:55.379
Sukrut Rao: Better, and also more performant.

34
00:03:56.160 --> 00:04:07.310
Sukrut Rao: But also, more interestingly, beyond the more obvious reason for having faithful understanding, is that a good faithful explanation can also help perform more effective steering.

35
00:04:07.310 --> 00:04:16.529
Sukrut Rao: So, a faithful explanation that is part of the… that faithfully represents the model's computation can be a highly effective control surface to guide the models themselves.

36
00:04:16.529 --> 00:04:27.309
Sukrut Rao: and steer them to be better, to fix problems in their behavior, and so on. So we specifically, we also find this in one of our works, which I'll come to later in the talk. And so, broadly, this… we saw that this can also then improve performance.

37
00:04:28.390 --> 00:04:46.370
Sukrut Rao: So how do we get this understanding? Broadly, post-hoc explanation methods have been popularly used over the years, and in the context of vision feature attribution, heat maps are very popular, but unfortunately, we have often found that these post-hoc methods are not faithful, so they do not satisfy the criteria of having a faithful understanding.

38
00:04:46.370 --> 00:04:49.890
Sukrut Rao: So, in this work, a few years ago, also we showed this

39
00:04:49.890 --> 00:05:02.260
Sukrut Rao: via a simple sanity check style setup, where we created a synthetic setup that, controlled the flow of information through the model. So basically here, we created a setup where we pass a grid of images to the input.

40
00:05:02.260 --> 00:05:15.999
Sukrut Rao: And we can control the model such that only one part of the… one specific cell of the grid can actually influence the output. This means that, by design, only the pixels in that region can influence the output, and any attributions must be in that region.

41
00:05:16.000 --> 00:05:30.810
Sukrut Rao: But unfortunately, we found that a lot of popular attribution methods, including GradCam and so on, often attribute outside the region and do not actually represent what the model is doing, despite us knowing that the parts of the image outside the region do not influence.

42
00:05:31.010 --> 00:05:31.880
Sukrut Rao: And…

43
00:05:32.360 --> 00:05:45.709
Sukrut Rao: So, we have these methods which are usable, but we can do better. So, and specifically in our line of work, what we advocate for and what we attempt to do is to try to make models inherently interpretable instead, to get explanations that are part of the model's computation itself.

44
00:05:46.350 --> 00:06:00.399
Sukrut Rao: So, broadly, this is going to be the outline of my talk. In the first part, I will talk about building inherently interpretable vision foundation models, and in this, we have two broad directions. One is bottom-up, where we modify the model's architecture itself.

45
00:06:00.400 --> 00:06:18.239
Sukrut Rao: And such that you get model inherent attributions, and we talk about how we can scale it to foundation model scale. And the second is top-down, where we take existing vision language foundation models and perform localized concept discovery to get a model that can explain in terms of concepts, while also still being performant in all the foundation model tasks.

46
00:06:18.420 --> 00:06:36.690
Sukrut Rao: Then, in the second part, I will briefly go over some examples of how faithful explanations can then be used to control and improve models. So in this case, this will not be in the context of foundation models, because this was done prior to the works in the first part, and this is more in the context of vision classifiers, but

47
00:06:36.690 --> 00:06:44.310
Sukrut Rao: We get interesting results to show that faithful explanations actually are very effective in forcing the model to look where we want to, one reason the way we want.

48
00:06:44.630 --> 00:06:56.300
Sukrut Rao: And in the third part, finally, I will give some ongoing directions on our work of using the models we build in the first part to then improve large-scale multimodal models, specifically multimodal dynamics.

49
00:07:06.880 --> 00:07:12.400
Sukrut Rao: So, I'll start with the first part, which is the bottom-up approach of building model-inherent attributions.

50
00:07:12.400 --> 00:07:30.519
Sukrut Rao: And for this, we start with a kind of architecture called BCOS networks, which were proposed a few years ago, that aim to do exactly this, to provide inherently interpretable attribution. So I'll first give an overview of this. So a typical, let's say you have an image classifier, a model broadly looks like this. You have a combination of linear layers and nonlinearities.

51
00:07:30.530 --> 00:07:38.419
Sukrut Rao: This is a simplified version. This could be a VIT-based model, a convolutional network, or anything. You still have components that are involved like this. And…

52
00:07:38.690 --> 00:07:45.119
Sukrut Rao: Basically, a linear layer, of course, is given by linear transform, which is a dot product between the weights and the input and an additional bias.

53
00:07:45.470 --> 00:07:58.710
Sukrut Rao: So, the difference in a BCOS network is that these linear layers are then substituted for BCOS layers, and the nonlinearities are removed, where a BCOS layer is given by this transform. Specifically, this is very similar to a linear transform, except for two differences.

54
00:07:58.710 --> 00:08:11.009
Sukrut Rao: One, the cosine term in the linear transform is raised to a higher power of P, and secondly, the biased terms are removed. And this has two advantages for our goal of faithful and interpretable explanations.

55
00:08:11.030 --> 00:08:21.359
Sukrut Rao: The first is that because you have a higher alignment pressure, because of the higher power in the cosine, you have a stronger pressure for the weights and inputs to be well aligned.

56
00:08:21.360 --> 00:08:31.280
Sukrut Rao: And this forces, at every layer, the alignment to be stronger, which leads to more compact explanations overall and leads to better interpretability.

57
00:08:31.280 --> 00:08:50.880
Sukrut Rao: And the second advantage is that, because we do not have biases anymore, we can now re-express this term as an input-dependent linear transform. So, input times the weight, that is dependent on the input. And linear transforms are interpretable, so for… at a local level, for each given input, you get a faithful summary of how the layer works. And now.

58
00:08:50.880 --> 00:09:15.860
Sukrut Rao: Once you have one because there, you can now cascade this through the model, and because this is nonlinear, this is still expressive, and retains performance, but also lets us compose this. So now you get a faithful summary for the entire model. So from the image times an input-dependent weight, like this, image times an input-dependent weight, directly gives you the output. And this specifically means that now the value of the heat map at any given pixel exactly represents

59
00:09:15.860 --> 00:09:21.119
Sukrut Rao: how much that pixel contributed to the output, and computing this product exactly gives you the logit.

60
00:09:21.120 --> 00:09:28.160
Sukrut Rao: So this gives you a faithful summary that is actually part of the model's computation, and also gives you compact explanations because of the rapid pressure.

61
00:09:28.360 --> 00:09:33.310
Sukrut Rao: There's another person, yeah. Oh, oh, gone already. No, no, the method's gone.

62
00:09:33.570 --> 00:09:37.530
Sukrut Rao: So… so you're… so you're replacing dot product.

63
00:09:37.660 --> 00:09:40.330
Sukrut Rao: You're saying, that product is the enemy.

64
00:09:40.630 --> 00:09:43.579
Sukrut Rao: We need to do dot product raised to the B power.

65
00:09:43.910 --> 00:09:50.269
Sukrut Rao: Instead. And, And then you get some nice properties. So, my question is…

66
00:09:50.500 --> 00:09:53.249
Sukrut Rao: So that's a very bold, assertion.

67
00:09:53.560 --> 00:10:00.319
Sukrut Rao: Why is it… so what is… what's the trade-off? What's the trade-off? Like, what's the trade-off when you do this?

68
00:10:00.510 --> 00:10:17.670
Sukrut Rao: Is it just all benefits, or are there any, any, negatives? So, when… so this, this is, prior work, so when originally proposed, it came with the trade-off in a small drop in accuracy, but then with relaxing some of… How big is small? So, in the midnight, around 1 or 2%.

69
00:10:17.910 --> 00:10:33.019
Sukrut Rao: But after relaxing the SM constraints, the second version of it does not have any drop in accuracy. Yes. And in our work, we try to now, scale this to foundation models, so creating versions of this at a diagnostic hip scale.

70
00:10:33.340 --> 00:10:45.290
Sukrut Rao: And there, at least on all the downstream standard data sets people use for linear probing and zero-shot and KNN and so on, there we get very close performance, up to 0.5% drop.

71
00:10:46.080 --> 00:10:47.730
Sukrut Rao: So there's… there's some drop.

72
00:10:47.840 --> 00:11:05.899
Sukrut Rao: like, so, like, what you would really hope is you would hope for an increase in performance. Yes. Do you ever see an increase in performance? That is the goal, and I will come to that much later. The idea is that at this stage, we are just trying to get a model, so in our work that's coming in the next few slides, we're trying to get a model that

73
00:11:05.900 --> 00:11:09.769
Sukrut Rao: Behaves exactly like the original foundation model, but has these extra properties.

74
00:11:09.770 --> 00:11:16.059
Sukrut Rao: What we would like to do now, and in the future, is now to use these, properties to

75
00:11:16.120 --> 00:11:29.120
Sukrut Rao: fix things in the model, so to regularize the model via explanations, and then the hope is that increases the performance along at least those specific problems that we're trying to fix. So, at least for classifiers, we do see that happens.

76
00:11:29.120 --> 00:11:41.480
Sukrut Rao: So if we, for example, for out-of-distribution robustness, if we… if we tell the model, look at the object, not at the background, then you get a model that does so, and then if you want the model to be robust against this, the performance test.

77
00:11:44.000 --> 00:11:47.389
Sukrut Rao: And do you have a sense for why this works?

78
00:11:47.680 --> 00:11:49.800
Sukrut Rao: Or what was wrong with that buttocks?

79
00:11:50.320 --> 00:11:57.230
Sukrut Rao: There's nothing wrong with dot products, it's only this gives the additional property of having more compact explanations.

80
00:11:57.760 --> 00:12:08.929
Sukrut Rao: And, because you have a higher B, you can now have the expressivity to remove the extra bias. I guess, right? So it's an empirical measurement. Yes. Yeah, but you have any sense for why.

81
00:12:09.360 --> 00:12:12.150
Sukrut Rao: Why it comes up as well. It improves alignment.

82
00:12:12.630 --> 00:12:20.860
Sukrut Rao: So, at every layer, the output of this operation can only be high if the angle between the weight and the input is low.

83
00:12:22.410 --> 00:12:30.979
Sukrut Rao: So, for example, in the original BCOS paper, there is an ablation over various values of B, and as you increase B, you get explanations that are more and more compact and object-focused.

84
00:12:30.980 --> 00:12:46.019
Sukrut Rao: Because the alignment improves. But we stop at B equal to 2, because after that point, the improvements qualitatively and quantitatively are minimal, but also it makes the training more unstable, because now you have much smaller values in the cosine term.

85
00:12:55.640 --> 00:12:56.990
Sukrut Rao: the, manually.

86
00:12:57.310 --> 00:12:58.120
Sukrut Rao: Yes.

87
00:13:00.640 --> 00:13:01.480
Sukrut Rao: Yes.

88
00:13:02.620 --> 00:13:04.750
Sukrut Rao: improves your savings.

89
00:13:06.350 --> 00:13:08.490
Sukrut Rao: People do that, exception of why.

90
00:13:10.090 --> 00:13:15.700
Sukrut Rao: Because… Because it uses the software Google, right?

91
00:13:15.810 --> 00:13:21.079
Sukrut Rao: So, you are reducing the… you're narrowing the…

92
00:13:21.230 --> 00:13:26.159
Sukrut Rao: The region in the high-dimensional space that can give you an appreciable output.

93
00:13:26.920 --> 00:13:30.850
Sukrut Rao: So, basically, the weights and inputs have to be well aligned for the output to be high.

94
00:13:31.050 --> 00:13:32.259
Sukrut Rao: That makes sense.

95
00:13:32.410 --> 00:13:37.499
Sukrut Rao: Right, so you're… what you're selecting for is you're selecting, like, direct template matches.

96
00:13:37.690 --> 00:13:43.440
Sukrut Rao: In some sense, yes. Right. And so, I guess that's my question. Is that really, like…

97
00:13:43.790 --> 00:13:47.249
Sukrut Rao: Does that lead to better models, or does it just lead to…

98
00:13:48.040 --> 00:13:56.100
Sukrut Rao: So far, we find that it leads to comparable models, but also leads to more easy-to-understand and interpretable explanations.

99
00:13:58.030 --> 00:14:05.940
Sukrut Rao: But we also find later that this kind of explanation can be used as a more effective tool… sorry, as a tool to more effectively regularize models.

100
00:14:12.010 --> 00:14:13.569
Sukrut Rao: Nothing? Nope, sir.

101
00:14:13.990 --> 00:14:15.300
Sukrut Rao: I have a question.

102
00:14:16.240 --> 00:14:21.656
Sukrut Rao: So, increasing the costs, the cost of… You see…

103
00:14:22.590 --> 00:14:25.029
Sukrut Rao: pushes X and W to be in the same.

104
00:14:26.840 --> 00:14:28.780
Sukrut Rao: space, like a light space.

105
00:14:29.000 --> 00:14:34.059
Sukrut Rao: Like, I… I have much… Both fundamental question, like, why, why should that come?

106
00:14:34.720 --> 00:14:36.820
Sukrut Rao: Like, you're not really putting any…

107
00:14:38.300 --> 00:14:42.129
Sukrut Rao: Regularization on the outputs of the cost level.

108
00:14:42.500 --> 00:14:43.329
Sukrut Rao: No problem.

109
00:14:43.840 --> 00:14:44.740
Sukrut Rao: Yes.

110
00:14:44.990 --> 00:14:48.020
Sukrut Rao: So, why would adding a cost, additional cost now?

111
00:14:48.580 --> 00:15:04.910
Sukrut Rao: push them to be more aligned. Because your loss, when training this mod, like, if it's a classifier, your supervised loss is, pushing the logic up for the correct class. Yeah. That means that there is a pressure for the value of the output at each layer to go up with respect to the correct class.

112
00:15:05.340 --> 00:15:13.049
Sukrut Rao: And this can only go up if the magnitudes of this go up, or the cosine term goes up.

113
00:15:13.450 --> 00:15:22.790
Sukrut Rao: And the thing we changed here is the percent term. So, right now, there is a… automatically, with higher B, it's harder to get it to go up, and so it will only if the land matter goes.

114
00:15:23.930 --> 00:15:29.520
Sukrut Rao: And there is this pleasure throughout the model, and so the alignment improves across the board.

115
00:15:30.860 --> 00:15:34.080
Sukrut Rao: So, if that improves, you're saying that the…

116
00:15:34.750 --> 00:15:38.989
Sukrut Rao: the outputs. What… I still don't understand the mass completion, what it is.

117
00:15:39.450 --> 00:15:47.139
Sukrut Rao: But that's… I'm assuming that's the output of the cost here? So this is a visualization of the W itself.

118
00:15:47.310 --> 00:15:56.949
Sukrut Rao: So maybe if it's easier, you can just imagine this as a heat map, which has only strong values here and very low values everywhere else.

119
00:15:57.270 --> 00:16:12.889
Sukrut Rao: So… So how did you get the heat map? W times X? The heat map would be W times X. This is a color visualization of W, and that was a detailed script, so basically, the… in the PCOS paper, they propose an alternative, image preprocessing… sorry, input preprocessing scheme.

120
00:16:12.890 --> 00:16:22.740
Sukrut Rao: Where they have a 6-channel input, such that each value can be in the six-dimensional space, maps to an exactly unique color in RGB space.

121
00:16:22.750 --> 00:16:28.699
Sukrut Rao: So now you can view the W of X in color, and high alignment is high matching of color.

122
00:16:28.700 --> 00:16:49.409
Sukrut Rao: So this is what was proposed in the original paper, but it's just another way of looking at it. You can also view it as a heat map, where there is high strength only in the regions that are well aligned, and low strength everywhere else. So if you have poor alignment, the heat maps are often more diffuse, because there is some amount of alignment everywhere, some amount of contribution coming from everywhere, but because you're forcing high alignment.

123
00:16:49.410 --> 00:16:57.920
Sukrut Rao: There has to be very strong alignment in the regions that the model really cares about, and the rest of the place, it's usually very big alignment, so that gives you more compact definitions.

124
00:17:14.410 --> 00:17:19.750
Sukrut Rao: efficiency of 3.

125
00:17:23.609 --> 00:17:29.330
Sukrut Rao: I think with, So…

126
00:17:29.450 --> 00:17:42.639
Sukrut Rao: Because B is equal to 2, it's relatively not that bad, because you don't have to… you just need to add… multiply an extra cosine term. If I recall correctly, with dodge compile, the… the wall clock time was roughly 2.

127
00:17:43.450 --> 00:17:44.190
Sukrut Rao: Yep.

128
00:17:49.000 --> 00:17:50.840
Sukrut Rao: product meditation.

129
00:17:53.370 --> 00:17:55.540
Sukrut Rao: God, blood of grace will be…

130
00:18:02.120 --> 00:18:07.349
Sukrut Rao: So you mean where the inputs and the weights are also raised to a higher power?

131
00:18:11.230 --> 00:18:13.039
Sukrut Rao: So, like, sequential.

132
00:18:16.240 --> 00:18:21.930
Sukrut Rao: I think that might also lead to something similar, it's just that it's…

133
00:18:23.990 --> 00:18:28.050
Sukrut Rao: Yeah, this would be scaled by a constant, so it shouldn't make much of a difference.

134
00:18:28.590 --> 00:18:29.610
Sukrut Rao: And…

135
00:18:34.850 --> 00:18:41.180
Sukrut Rao: Yeah, I think it might lead to something similar. This is just the more cleaner version of just increasing the alignment pressure, but yes.

136
00:18:46.730 --> 00:19:02.809
Sukrut Rao: So one part… so the little heat map thing that you have is… that's the lowercase w at the last layer? It is the composite W, so we can… we apply this throughout the model, and this is a compositional thing, so you get a W for the full model.

137
00:19:03.040 --> 00:19:04.580
Sukrut Rao: Okay. Times the input.

138
00:19:04.890 --> 00:19:11.819
Sukrut Rao: So… So it's, like, all of the, like, W1XW2.

139
00:19:12.550 --> 00:19:15.860
Sukrut Rao: Yeah, exactly. So, combined composition.

140
00:19:16.740 --> 00:19:23.859
Sukrut Rao: product overall. So, which is why all the alignment needs to be strong everywhere, because if at one layer it's weaker, then it'll already destroyed.

141
00:19:24.630 --> 00:19:25.490
Sukrut Rao: Bye.

142
00:19:27.390 --> 00:19:32.819
Sukrut Rao: So, can you just explain that last, or how do you call that again? Which one? Yes.

143
00:19:33.860 --> 00:19:34.880
Sukrut Rao: So…

144
00:19:35.010 --> 00:19:49.350
Sukrut Rao: It's basically just taking out one cost to the, one cost out, and that's cost to the V minus 1 times W, dot product with X. So this is… this can be viewed as an input-dependent W. Yeah.

145
00:19:49.490 --> 00:20:09.059
Sukrut Rao: So that's how we… So, okay, maybe one clarification, then when you… in the implementation, when you apply it in code, you don't simply do backpropagation, because that would give you the… just the raw gradients. You want W of X, so everything inside this part is detached from the compute graph.

146
00:20:10.410 --> 00:20:12.020
Sukrut Rao: So that you get W .

147
00:20:12.500 --> 00:20:19.200
Sukrut Rao: But that's a code detail, so that's why I didn't mention it so far. Okay, so that's not, like, the entire ComposeW that you're talking about yet?

148
00:20:20.210 --> 00:20:29.249
Sukrut Rao: This is for one layer, but if you apply this on a sequence of layers, it is just the product of… Yeah, input-dependent linear, yeah.

149
00:20:30.400 --> 00:20:47.659
Sukrut Rao: And in the original formulation, you don't have things like ReLU or other activation functions, but if you have anything that is input-dependent linear, when ReLU is, it doesn't… it still satisfies this property. So, in fact, in the ones… the models we build, we actually keep the activation function inside, because anything that's input-dependent linear is fine.

150
00:20:51.170 --> 00:20:52.720
Sukrut Rao: So some approximation number.

151
00:20:53.680 --> 00:20:56.590
Sukrut Rao: Because it's input-dependent, you can't really multi-dependent.

152
00:20:57.860 --> 00:21:00.250
Sukrut Rao: No, it's not an adoption.

153
00:21:03.550 --> 00:21:04.800
Sukrut Rao: Great links.

154
00:21:07.010 --> 00:21:09.049
Sukrut Rao: That's dependent on the imagery.

155
00:21:09.750 --> 00:21:12.480
Sukrut Rao: Input… the input is just that number.

156
00:21:12.720 --> 00:21:14.100
Sukrut Rao: For every layer of one.

157
00:21:14.260 --> 00:21:17.369
Sukrut Rao: Each layer has the output of the previous layer, therefore.

158
00:21:18.350 --> 00:21:26.089
Sukrut Rao: So, W1, like the… not the W hat, the W1, the layer 1, as input X, 0.

159
00:21:26.740 --> 00:21:30.210
Sukrut Rao: W2 has input X1. So you…

160
00:21:31.280 --> 00:21:34.679
Sukrut Rao: How did you say that we can multiply? It's an approximation?

161
00:21:41.280 --> 00:21:48.349
Sukrut Rao: I think there is a… it might be easier to show with pen and paper, so… because it's harder for me to visualize in that.

162
00:21:48.350 --> 00:22:04.649
Sukrut Rao: But it's also there on the… in the paper, so I'll just open that and show you afterwards. So, I think maybe I said it wrongly about multiply. It's more of a compositional operation, but it simplifies out to, again, there being, input-dependent of 2 times X. So, yeah, I realize that I probably… it's not… yeah, it's not multiplication directly.

163
00:22:07.720 --> 00:22:08.880
Sukrut Rao: Anything else?

164
00:22:13.970 --> 00:22:19.829
Sukrut Rao: So, we have these two properties, but the original BCOS models, the problem is that they need to be trained from scratch.

165
00:22:20.000 --> 00:22:28.119
Sukrut Rao: So, what we first look at is how can… since we want to make it work at foundational model scale, we don't want to spend the compute and data again, to train again and again each time.

166
00:22:28.140 --> 00:22:43.109
Sukrut Rao: we propose a… or look into how we can do weak classification. So take an existing conventional model that is already trained and performed, and transform it to this kind of architecture via a series of trans… like, architectural transforms fine-tuning to get a B-classified model.

167
00:22:43.110 --> 00:22:50.699
Sukrut Rao: And broadly, we find that this can be done in significantly fewer steps, and provides the cost-like explanations, and now you can also go beyond classifiers.

168
00:22:51.210 --> 00:23:09.419
Sukrut Rao: So, these are the various design choices in which conventional and PCOS models differ, and I think we covered most of them already in our discussion, so maybe I'll go over this more quickly. But broadly, the focus of this work was to then perform a study on which of the modifications are necessary and how to best apply them.

169
00:23:09.880 --> 00:23:12.210
Sukrut Rao: In order to get this B-quality model.

170
00:23:12.210 --> 00:23:37.009
Sukrut Rao: And we do this in two steps, one which preserves functional equivalence. So, for instance, like I said, activation functions between layers is present in conventional models, but not present in BCOS models, but keeping them does not hurt our properties that we care about, and is also better because it is closer to the distribution of the weights that was already trained, so we just keep them. And similarly for other kinds of transforms. And then we then make the break-y changes, basically the two big things, which is increasing the value of P,

171
00:23:37.010 --> 00:23:38.310
Sukrut Rao: Removing the biases.

172
00:23:38.310 --> 00:23:51.510
Sukrut Rao: Here, also, we perform a study on whether you should do this slowly, whether you should do this immediately, and so on. And overall, we find that just swapping it instantly and just fine-tuning is already good enough, and we just do this to get to be classified.

173
00:23:51.630 --> 00:23:54.409
Sukrut Rao: So… so, I'm saying.

174
00:23:55.310 --> 00:23:59.060
Sukrut Rao: So, your recommendation is this last one? Yes.

175
00:24:03.000 --> 00:24:15.299
Sukrut Rao: So the, yeah, so normalized inputs is, basically having this. It means input, this means standard deviation normalization that you have in the standard CNNs and VITs, which are not there in BigQuest models.

176
00:24:15.350 --> 00:24:23.110
Sukrut Rao: weights were originally unit normalized in BCOS models to have higher power pressure, alignment pressure, but it turns out not having them still is fine.

177
00:24:23.130 --> 00:24:37.979
Sukrut Rao: And 6-channel is the preprocessing step that I mentioned earlier. I didn't… I didn't catch the 6-channel processing. What's that… what's that about? Yeah. So, original BCOS model, paper has, proposed a different preprocessing step, which was

178
00:24:37.980 --> 00:24:45.149
Sukrut Rao: when you have 3 input channels RGB, you add 3 more channels, which is 1 minus R, 1 minus G, and 1 minus B. And if you do this.

179
00:24:45.150 --> 00:24:49.360
Sukrut Rao: You now represent every color as a unique angle in 6-dimensional space.

180
00:24:49.760 --> 00:25:00.839
Sukrut Rao: And since W is then also a 6-dimensional vector, at each point, you can now remap it to a unique point in RGB space. So it's now a bi-directional… it's now a bijection.

181
00:25:03.460 --> 00:25:22.539
Sukrut Rao: So, let's say W was three-dimensional, as you would typically have, and let's say at one point the value was 100, then you know that along the red channel you have some strength, and along green and blue, you have no strength, but you don't know how strong it is, and what is the value of red. So it's… you can, because of the normalization.

182
00:25:22.690 --> 00:25:23.530
Sukrut Rao: Yes.

183
00:25:23.840 --> 00:25:43.169
Sukrut Rao: No, because the W of X is not an… it's not an image, it's just a… it's just a, some vector in 3D space. It's not bounded by the bounds of RGB. It's not in RGB space. There's no reason for it to be there in the construction of the model.

184
00:25:43.430 --> 00:25:53.820
Sukrut Rao: But if you have it in six dimensions, you can… there's a bijection between RGB and the six-dimension space, and 6-dimension back to RGB space, so you can remap it back to color, and that's how we get the color explanations.

185
00:25:59.320 --> 00:26:07.849
Sukrut Rao: I said, so… so there's something about… so the core… The core innovation was

186
00:26:08.170 --> 00:26:19.040
Sukrut Rao: replacing dot product with this cosine thing. So you're saying that when you do that, You lose fidelity,

187
00:26:19.990 --> 00:26:24.290
Sukrut Rao: You get… it buys you some ambiguity for RGB inputs.

188
00:26:24.490 --> 00:26:40.030
Sukrut Rao: Is that right, or am I missing… No, it's the other way around, actually. It gives you, the… each, point, each vector in W… W , which is now six-dimensional, can be mapped exactly towards that point. If you don't go to 6-dimensional… Yeah, then you cannot. Then raising the…

189
00:26:40.250 --> 00:26:43.280
Sukrut Rao: That product goes out to this power. Yeah.

190
00:26:43.390 --> 00:27:01.670
Sukrut Rao: makes it ambiguous? No, we could… you could still… I think, no, this would still work even without… without a 6-dimensional entity. Yeah, I'm trying to understand why we had… like, why is it connected? So this was, done simply to get those color… color mapping explanations, so expressing W in RGB space. So that's just a matter of more of,

191
00:27:01.670 --> 00:27:03.970
Sukrut Rao: Convenience for… for,

192
00:27:04.010 --> 00:27:14.359
Sukrut Rao: human interpretability, but it's not strictly necessary for getting the other properties, like alignment pressure. So you would still get heat maps that are very similar if you don't have a six-dimensional.

193
00:27:15.870 --> 00:27:19.690
Sukrut Rao: Oh, it's just for producing pretty big maps? Yes. Yes.

194
00:27:20.150 --> 00:27:24.060
Sukrut Rao: Just another way of visualizing the same thing in our checking space.

195
00:27:28.660 --> 00:27:29.340
Sukrut Rao: One.

196
00:27:37.390 --> 00:27:46.629
Sukrut Rao: Because this was already enough, so they did not do the mean STD normalization there. But conventional models already have this, so we keep this and just do a combination of both.

197
00:27:47.350 --> 00:27:52.980
Sukrut Rao: So, adding normalizations and activation functions would break your linear assumptions, right?

198
00:27:53.180 --> 00:28:01.029
Sukrut Rao: Explanation? No, because the activation functions are also piecewise linear, so are input-dependent linear.

199
00:28:01.210 --> 00:28:08.869
Sukrut Rao: So, for a given input, you still get an input-dependent linear weight. You don't have linearity in general, but locally you still do.

200
00:28:17.480 --> 00:28:18.360
Sukrut Rao: And…

201
00:28:18.370 --> 00:28:38.220
Sukrut Rao: So, I'll just click over the results. So, we find that you can now get similar performance at significantly lower cost. So, here, for instance, is a TensNet model, where on the x-axis is training epochs, on the y-axis is accuracy, and what we see is that on the orange curve, the accuracy versus epoch, if you just train a BCOS model from scratch.

202
00:28:38.220 --> 00:28:44.619
Sukrut Rao: And the blue curve is, with big classification, where you start with the conventional model and do the transformation.

203
00:28:44.620 --> 00:28:52.810
Sukrut Rao: And here, the same accuracy that it… you would get by training from scratch is achieved much faster within just 19 epochs. And similarly for…

204
00:28:52.810 --> 00:29:06.149
Sukrut Rao: other models as well. So broadly, we find that reusing the existing weights from conventional models and then performing transformation is much more compute and efficient as compared to just training from scratch, in the context of supervised models at this point.

205
00:29:06.520 --> 00:29:18.130
Sukrut Rao: And next, we also want good explanations, so what happens there? So here, the first column is some examples, second column is because explanations, and third column is at the initial stage of recursification, when you haven't done any fine-tuning yet.

206
00:29:18.130 --> 00:29:27.399
Sukrut Rao: So, this also qualitatively shows if you don't have this extra alignment pressure, the explanations are much more diffuse, because they are not, focusing on the object of interest with accuracy.

207
00:29:28.850 --> 00:29:37.639
Sukrut Rao: What are the columns? The third one? So the third column is the explanations in our reclassification before we start fine-tuning. Yeah, like, how did you get the maps?

208
00:29:37.640 --> 00:29:49.429
Sukrut Rao: So, like I said before in the process, we have two steps, one where we do transformation, which is without breaking changes, and then the second step is fine-tuning. So, when we first do the transformation, we have a BCOS-like architecture.

209
00:29:49.430 --> 00:30:04.770
Sukrut Rao: But, it is not performant yet in the task at hand. Oh, no, no, I mean, like, what's… is it the same W… Yes, yeah, yeah, yeah, exactly. So in the first step, we converted to a VCAS-like architecture, so you already have the 6-channel input and everything baked in at this point.

210
00:30:04.770 --> 00:30:15.909
Sukrut Rao: It's just that you haven't created the alignment pressure and removed the biases yet. So if you then just view… pretend this was already an existing working BCOS model and get an explanation out of it, you get something like this.

211
00:30:17.220 --> 00:30:23.150
Sukrut Rao: And, yeah, this would still have the problem of, not being fully faithful, because biases are still there at this point.

212
00:30:23.780 --> 00:30:37.039
Sukrut Rao: Then, when we do fine-tuning, after one epoch, we already get something very close, and this remains locked. So, basically, most of the fine-tuning steps that we require is for recovering accuracy. The alignment pressure comes much faster.

213
00:30:37.620 --> 00:30:51.589
Sukrut Rao: And finally, because we can do this now, we can go beyond simply working with the internet supervised classifiers, so we do a first step of doing this via clip. So here, we just take a… we perform,

214
00:30:51.590 --> 00:30:57.539
Sukrut Rao: We apply a contrastive loss on a clip original model and our big gospel part version with just images.

215
00:30:57.540 --> 00:31:13.500
Sukrut Rao: And we get a model that does something like this. So, here you have text prompts, which are, like, a photo of a dog, photo of a human, photo of flowers, and there, if you then try to get explanations in the visual encoder of Flip, you have maps that exactly highlight which parts of the input contributed to that.

216
00:31:15.800 --> 00:31:40.629
Sukrut Rao: And here are some qualitative results, so across various, CNNs and VIDs and also on CLIP, we get competitive performance with gains, but crucially, at this stage, we still have problems with larger-scale VITs, specifically at the B and L size, where even after doing B classification, we were unable to actually, get gains, even after… So on the left, it's all classification accuracy. So what is linear programming?

217
00:31:40.630 --> 00:31:42.840
Sukrut Rao: evaluation that is…

218
00:31:42.980 --> 00:31:50.859
Sukrut Rao: Oh, these are all, these are all accuracy things. Yes, so this is on GLIP, so there's a… Do you have, do you have, do you have, like, an interpretability benchmark that you have?

219
00:31:51.730 --> 00:32:01.289
Sukrut Rao: So we have a, we evaluate localization scores, which is basically how well localized the explanations are to the object. Yes, so, yeah.

220
00:32:02.860 --> 00:32:13.369
Sukrut Rao: I've seen. So, in the follow-up work to this, which will come next, there we also have the perturbation metrics, so deletion, insertion, so on.

221
00:32:14.780 --> 00:32:16.400
Sukrut Rao: And also a user study.

222
00:32:16.550 --> 00:32:21.550
Sukrut Rao: I think. How do you do on, like, the out of bio style?

223
00:32:21.740 --> 00:32:28.980
Sukrut Rao: You know, sort of, do you know which one I'm talking about? Yeah. The, the, sanity checks.

224
00:32:29.240 --> 00:32:36.990
Sukrut Rao: Yes, I… I'm not entirely sure if we reproduce that in the exact format, I will need to check, too.

225
00:32:39.660 --> 00:32:44.559
Sukrut Rao: Intuitively, it should be fine, but I'll have to check, I'm not sure. I don't want to say it works without having to check.

226
00:32:46.150 --> 00:32:50.900
Sukrut Rao: But we do the perturbation matrix. I see. Because, like, visually.

227
00:32:51.150 --> 00:32:53.510
Sukrut Rao: Your results look very similar to…

228
00:32:53.930 --> 00:32:56.850
Sukrut Rao: a smooth grad, or a number of other…

229
00:32:56.960 --> 00:33:04.779
Sukrut Rao: approaches, but I think that the big… the big paper that calls this approach into question and uses

230
00:33:04.930 --> 00:33:07.949
Sukrut Rao: you know, results that look like this was the out of I.O.

231
00:33:08.070 --> 00:33:13.080
Sukrut Rao: April, where he says. Yeah. Great, new career with that. Yes, yes.

232
00:33:13.210 --> 00:33:20.970
Sukrut Rao: I think we did it, but I'm not entirely sure, so I'll need to check and let you know, because I don't have results for that in the slides here.

233
00:33:21.310 --> 00:33:28.500
Sukrut Rao: But there are, at least, we have seen in the cases where images contain several objects.

234
00:33:28.500 --> 00:33:43.089
Sukrut Rao: Including very small localized objects that the explanations are very good at focusing exactly on the specific object, so they are highly object-dependent or class-dependent. They are not, just highlighting the dominant object in the image, independent of what you're predicting.

235
00:33:43.090 --> 00:33:45.050
Sukrut Rao: So…

236
00:33:45.150 --> 00:33:58.250
Sukrut Rao: And you can even get contrastive explanations. So let's say you have two classes, and you want to know what in the image speaks for this class as opposed to the other class. Because of this linearity property, you can simply subtract the explanations.

237
00:33:58.250 --> 00:34:11.320
Sukrut Rao: And that also shows that this part of the image is… so especially when you have spurious correlation, this is quite interesting. It shows that, okay, because of this specific thing that is correlated to another class, the model is less sure of the actual true class, and so on. So this… this we do see.

238
00:34:11.770 --> 00:34:12.600
Sukrut Rao: Perfected.

239
00:34:13.530 --> 00:34:14.270
Sukrut Rao: Yeah.

240
00:34:14.550 --> 00:34:28.499
Sukrut Rao: Yeah, I think that… so I think, at least for me, I'm, like, most interested in the portability questions. Like, the performance thing, I'm not that interested in, right? It's like, oh, right? That's what everybody else will ask, but, like, does this actually…

241
00:34:29.630 --> 00:34:31.199
Sukrut Rao: achieve.

242
00:34:31.449 --> 00:34:38.579
Sukrut Rao: So, on both the perturbation… the perturbation and pointing game style metrics, it significantly outperforms the posthum.

243
00:34:41.050 --> 00:34:46.309
Sukrut Rao: And did you… did you have any alternative faithful methods as to how…

244
00:34:47.520 --> 00:34:52.940
Sukrut Rao: Like, any causally changing your maps and see if it actually affects your performance?

245
00:34:53.460 --> 00:35:04.120
Sukrut Rao: Would you consider perturbation metrics to be doing that? Because there you are actively removing the parts of the input that are highly attributed and seeing what happens to the output.

246
00:35:06.810 --> 00:35:09.229
Sukrut Rao: Okay, see, this is… this is… this is the…

247
00:35:09.560 --> 00:35:15.379
Sukrut Rao: just the classic image stuff, right? We're the only one making sure there's inputs there.

248
00:35:16.950 --> 00:35:20.400
Sukrut Rao: Right. You know, because, like, ultimately, the…

249
00:35:22.010 --> 00:35:25.120
Sukrut Rao: The essence… not the essence, I don't know what to call the word.

250
00:35:25.520 --> 00:35:30.530
Sukrut Rao: The one that attracts most here is the… The explainability aspect of it.

251
00:35:31.090 --> 00:35:37.250
Sukrut Rao: So I'm just curious… Like, if you… what are the features themselves or something.

252
00:35:37.700 --> 00:35:39.140
Sukrut Rao: Nothing changes.

253
00:35:41.090 --> 00:35:44.780
Sukrut Rao: Your work doesn't look at the internal features, following.

254
00:35:45.010 --> 00:35:48.850
Sukrut Rao: We have in one of the future works, but not in this specific one.

255
00:35:49.810 --> 00:36:09.540
Sukrut Rao: So here… Do you want to speak about it a little bit? Yeah, so there, what we do is we combine sparse autoencoders with PCOS layers. So you have intermediate layers of the BCOS model that have a concept decomposition, and we look at how much each concept contributes to the output, and I think I have a slide, I can just also jump there for a minute.

256
00:36:15.160 --> 00:36:21.120
Sukrut Rao: How much each concept contributes to the output, and how much… which part of the input contributes to the specific concept, and so on.

257
00:36:23.100 --> 00:36:35.559
Sukrut Rao: So, we have… we inspect in model internals via BCOS explanations, but in this set of evaluations, yeah, we do not perturb the concepts. Did you try turning off something and see if it affects…

258
00:36:35.930 --> 00:36:37.270
Sukrut Rao: You've got outwards.

259
00:36:39.930 --> 00:36:42.760
Sukrut Rao: in a systematic way, no. In a…

260
00:36:42.890 --> 00:36:46.150
Sukrut Rao: Oh, like, just as a playing around with the model sensors.

261
00:36:46.370 --> 00:36:48.789
Sukrut Rao: So, we don't have quantitative results for this.

262
00:37:05.820 --> 00:37:06.480
Sukrut Rao: Oh.

263
00:37:11.430 --> 00:37:33.440
Sukrut Rao: So, in this specific, way we do it, it is still exactly the same. So, we take the original model, and then transform it, and then fine-tune it with data. But how do you do, like, attention changes? So, attention layers are left unchanged because they're already dynamic linear, and the MLPs are converted to PCOS form by raising the higher power to P.

264
00:37:35.800 --> 00:37:37.760
Sukrut Rao: Do you just do the MLPs? Yes.

265
00:37:38.830 --> 00:37:39.910
Sukrut Rao: So, thank you.

266
00:37:40.520 --> 00:37:43.900
Sukrut Rao: Yeah, similar… Absolutely.

267
00:37:44.290 --> 00:37:45.760
Sukrut Rao: Thank you, Ms. Miller.

268
00:37:46.100 --> 00:37:46.940
Sukrut Rao: Save.

269
00:37:47.070 --> 00:37:55.280
Sukrut Rao: Yes, so we, we did try this, so we have a DMLR paper where we converted the,

270
00:37:56.120 --> 00:38:06.310
Sukrut Rao: bird-style language models to BCOS form, and there, again, we do get gains in human interpretability via user studies, and also in terms of

271
00:38:06.340 --> 00:38:16.840
Sukrut Rao: Performance and so on, but that is not at foundation model scale, so that's up to, like, things like sentiment classification and things like that, so they are between us.

272
00:38:16.880 --> 00:38:22.729
Sukrut Rao: the output of that is the same as over the… Over the top of us, yes.

273
00:38:23.360 --> 00:38:28.280
Sukrut Rao: So, extending that to large-scale LLMs is something we still want to do, but we haven't done that, so…

274
00:38:30.060 --> 00:38:35.049
Sukrut Rao: These are the same for ResNet, where you leave the convolution layers and just hit the other piece.

275
00:38:36.380 --> 00:38:40.429
Sukrut Rao: No, in ResNet, we do it everywhere, because convolutional layers are also basically linear.

276
00:38:42.500 --> 00:38:45.289
Sukrut Rao: So, do you have that… how you did that?

277
00:38:45.440 --> 00:39:01.720
Sukrut Rao: Like, do you have the code? Yeah, yeah. So, also, the reason with VITs, we don't do it on the attention maps is because the original BCAS paper already explored this in their own BCOS VIT paper, and they found that it's not required, so we just copy that as it is.

278
00:39:07.820 --> 00:39:10.989
Sukrut Rao: So, yeah, so here, we have this…

279
00:39:11.050 --> 00:39:18.919
Sukrut Rao: promising results, but the problem is still a gap in the VRTs. So, more recently, in our… in this year's CPPR,

280
00:39:18.920 --> 00:39:32.920
Sukrut Rao: we take an even simpler approach and basically propose a distillation-based idea to bridge this gap. So, when we started off in the previous work, originally, the goal was to… we were still primarily focusing on CNNs and small models and so on, but

281
00:39:32.920 --> 00:39:57.799
Sukrut Rao: In hindsight, a distillation-based approach makes a lot of sense, and that's what we tried here. So you take the foundational model as before, like a dyno model or a cliff model, sick cliff model, and so on, and here is the transform because version of this, and then we just have this as the teacher, and this as the student, and apply losses to improve alignment. So the core of this paper, then, is to figure out which losses to apply and where, and broadly, we find that a mixture of a global loss on the outputs

282
00:39:57.800 --> 00:40:07.789
Sukrut Rao: And local loss at intermediate layers, specifically at three parts of the, three different parts of the model, work the best, and we just do this by aligning via codes and similarity.

283
00:40:07.900 --> 00:40:25.009
Sukrut Rao: And beyond this, broadly, everything else remains similar. You still have this 6-channel conversion, you have this transformation of everything to BCOS form, and so on. And now you again get BCOS-like explanations, but this time, which actually works at 4 VITs, and also more at foundation model scale.

284
00:40:25.800 --> 00:40:36.589
Sukrut Rao: So, here are some results in qualitative and quantitative for Dino and other models. So, on the left, it's qualitative results for Dino. On the top, you have examples of images, explanations.

285
00:40:36.590 --> 00:40:46.160
Sukrut Rao: And also, to verify that we do not lose the semantic similarity properties that dyno features have, we also check the PCM apps to see that they remain comparable to what the original model had.

286
00:40:46.180 --> 00:41:01.810
Sukrut Rao: And here is the quantitative comparison of interpretability versus accuracy. So, as before, the accuracy is generally maintained, but on the pointing game localization scores, we get strong gains in interpretability as compared to using the original model with post-hocinations.

287
00:41:02.720 --> 00:41:06.629
Sukrut Rao: And here, there are 3 models, the Rhinos, Eclip, and Google BIT.

288
00:41:08.140 --> 00:41:23.679
Sukrut Rao: And more importantly, because this was the original goal, we find that we can do this at scale in a very data-efficient manner. So here, on the x-axis, you have the amount of samples used, and on the y-axis, the ImageNet probing accuracy, and we look at specifically CYCLIP.

289
00:41:23.740 --> 00:41:37.120
Sukrut Rao: And the original SIGLIP model, which was trained with over 10 billion samples, achieves 84.2% accuracy here. And if you just do reclassification by the previous process, the accuracy drop is significant.

290
00:41:37.120 --> 00:41:49.700
Sukrut Rao: But by doing this distillation-style approach, we find that with just 15 million samples using YFCC15M, we are extremely close, just 0.4% away from the performance that during the model base.

291
00:41:50.040 --> 00:41:50.860
Sukrut Rao: So…

292
00:41:51.280 --> 00:41:57.169
Sukrut Rao: Broadly, the reason also for focusing a lot more on performance in these works, as opposed to interpretability, is that

293
00:41:57.170 --> 00:42:12.750
Sukrut Rao: The original BCAS paper already shows strong interpretability gains, but a big mismatch for adoption was that you had performance losses, and also the issue of scaling. So, the problems we are trying to tackle in these two works is to fix that gap, so to get models that are

294
00:42:12.750 --> 00:42:20.139
Sukrut Rao: interpretable, but also where you don't really have to make a significant compromise in performance. And so is the interpreter… so, like.

295
00:42:20.210 --> 00:42:24.329
Sukrut Rao: Is it… is this proposal that, for example.

296
00:42:24.680 --> 00:42:35.939
Sukrut Rao: When we do, medical radiology or something like this, we should use these type of models because they'll give doctors better explanations, or… like, what, like.

297
00:42:36.120 --> 00:42:42.479
Sukrut Rao: Yeah, I'm, like, I'm always… I feel like when you're chasing performers like this, is that…

298
00:42:42.500 --> 00:42:57.579
Sukrut Rao: Is it really with the realistic deployment, or is it, yes, because often there is this tension where if you have very good interpretability, but a drop in performance, then people don't care about it as much, because they don't want to sacrifice that.

299
00:42:57.580 --> 00:43:03.769
Sukrut Rao: So, we are trying to bridge that gap, where you still have the interpretability property, but without having to make that compromise.

300
00:43:05.850 --> 00:43:12.259
Sukrut Rao: And plus, foundation models are now being the dominant paradigm everywhere, so we wanted to scale it to foundation models.

301
00:43:21.970 --> 00:43:38.350
Sukrut Rao: Then, yeah, we also evaluate the scaling, so across various model sizes and across both Siglip and dyno, performance is maintained with utmost… with less than 1% drop everywhere, and similarly, also with interoperability. So again, on the y-axis here, we have the localization score.

302
00:43:38.350 --> 00:43:44.340
Sukrut Rao: And the gray, wire is the original model, and the blue and green are our model, so we get…

303
00:43:44.340 --> 00:43:48.940
Sukrut Rao: Strong improvements in explanation quality, but without, again, sacrificing performance.

304
00:43:50.490 --> 00:44:05.319
Sukrut Rao: And these are some qualitative examples of comparing our explanations with common post-hoc methods, particularly methods like leg-rad, Attention ALAPI, and so on are quite recent, and are particularly built for BIT-style models.

305
00:44:05.490 --> 00:44:17.589
Sukrut Rao: And, we find that by doing this, we get models that are much more interpretable in terms of focus. Do you run it on the original model, or…

306
00:44:18.350 --> 00:44:22.240
Sukrut Rao: I think eventually we run it on both, but this figure is on the result.

307
00:44:29.420 --> 00:44:44.159
Sukrut Rao: And finally, now that we have this SIGLIP-style PCOS model, we can now plug this into a multimodal LLM. So we just do this as a proof of concept here, because doing this properly is… will take more work, especially because the LLM part is not yet

308
00:44:44.160 --> 00:44:52.599
Sukrut Rao: transform. So, in this specific example, we just put the LOC clip instead of the original clip model, and, original C-Clip model, and…

309
00:44:52.600 --> 00:45:09.569
Sukrut Rao: into a small multimodal LLM, and we train the adapter between vision and language ourselves, so it's a small-scale model. And to get these explanations, we just use a post-hoc method, which is attention LRP for the LLM part, and then use our because explanation just for the vision part.

310
00:45:09.660 --> 00:45:25.810
Sukrut Rao: But this is to just show that what is now possible, you can now get, faithful explanations to the input for autoregressive generation style tasks. So for each word, each token predicted in the captioning, you can now see how much the contribution of each part of the image is, so…

311
00:45:26.040 --> 00:45:44.629
Sukrut Rao: And with this, we hope that we can now use this to then, understand what happens, diagnose failure modes in the MLLMs, and also fix… so, for instance, the role of syncs registers, the gap in the modality bias that exists where models often rely insufficiently on visual inputs, and things like that.

312
00:45:47.190 --> 00:45:49.759
Sukrut Rao: How did you get the Florida?

313
00:45:49.900 --> 00:45:59.459
Sukrut Rao: Which one? Is this a… is this a VLM that you tried to show in the beginning, where you have vision encoder, you get vision tokens, you also have a left question? Yes, yes, yes.

314
00:45:59.890 --> 00:46:07.140
Sukrut Rao: So it's a… it's a LVLM, basically. Right, so for a given token, man, how do you get to this, the…

315
00:46:07.140 --> 00:46:22.370
Sukrut Rao: WX, like your vision output. Right, so we backpropagate up to the output of the vision encoder via attention LRP, which is a postdoc method, because the LLM part is not yet reclassified, so… and from there, you just have an output of the

316
00:46:22.930 --> 00:46:30.249
Sukrut Rao: At that point, you are the CLS token of the vision encoder, and from there, you backdrop it through the input using Biggers.

317
00:46:31.100 --> 00:46:32.520
Sukrut Rao: to get the WX.

318
00:46:36.650 --> 00:46:39.420
Sukrut Rao: These are not graded. No, it's a lot.

319
00:46:39.680 --> 00:46:40.709
Sukrut Rao: To the level.

320
00:46:40.820 --> 00:46:50.940
Sukrut Rao: No, no, this, this part is not LRP either, so this is BCOS explanations. No, no, but to, to, to, to… as input to your BCOS explanation, you didn't LRP…

321
00:46:51.450 --> 00:46:57.790
Sukrut Rao: Is that right? Because you need to go through the language? Yeah, so the LLM part is LRP, yes.

322
00:47:00.830 --> 00:47:20.830
Sukrut Rao: So LRP is a post-hoc method that, is, gradient plus plus in some sense, so you… they developed specific rules to decide how to backpropagate through each layer based on the contribution of the inputs, instead of simply based on gradients. So you change the input of your vision inputer, and then you pass it back to your PCOS and get this?

323
00:47:22.490 --> 00:47:25.210
Sukrut Rao: We… no, we do not, so…

324
00:47:29.020 --> 00:47:42.219
Sukrut Rao: we… you have an output token, right? So you backpropagate from here till here via LRP, and so you have a contribution strength for the tokens here, and then we backpropagate from here to the input via because

325
00:47:43.810 --> 00:47:54.239
Sukrut Rao: So that gives you the book, the contribution of each pixel, and which is faithful up to this point, but from here onwards, it is just a rapid, so it's approximately faithful.

326
00:47:55.070 --> 00:47:56.830
Sukrut Rao: You know, what we have so far.

327
00:47:56.990 --> 00:48:05.290
Sukrut Rao: Trying to make this end-to-end faithful is something we are currently working on, but the figure that I showed was more just the proof of concept of what can be done.

328
00:48:15.100 --> 00:48:28.120
Sukrut Rao: Yeah, so in summary, we have now this inherently dependable resource models at moderation model scale, with fraction of cost of training data, without sacrificing performance, and now we also want to use this to facilitate faithful explanations for multimodal models.

329
00:48:28.780 --> 00:48:32.270
Sukrut Rao: Before I continue, I just want to know how much time do I have?

330
00:48:33.170 --> 00:48:41.390
Sukrut Rao: Oh, yeah, you're about at the hour, but we always go over, so… but yes, how much, how much more do you need?

331
00:48:41.990 --> 00:48:48.000
Sukrut Rao: Maybe another… if I do everything, maybe another 30 minutes, but… so if I… then I can skip one of these parts and directly go here.

332
00:48:48.490 --> 00:48:51.980
Sukrut Rao: Yeah, yeah, no, the 30 minutes is fine. Okay.

333
00:48:58.880 --> 00:49:08.089
Sukrut Rao: So in the second part, we now look at top-down, which is basically via localized concept discovery, and from this, for this, we take the motivation of concept bottleneck models.

334
00:49:08.090 --> 00:49:24.349
Sukrut Rao: So these are models where, instead of just having an input, a feature extractor, and a classifier, you have a concept bottleneck layer in between that extracts human interpretable concepts and then combines them linearly. So, each output is a linear combination of human interpretable concepts which can be viewed as the explanation.

335
00:49:25.490 --> 00:49:49.769
Sukrut Rao: So, the first thing we do is we rethink how these models are constructed. So, when we first started working on this, the typical approach to do this was to select the concepts and learn the mapping. So, you would ask Oracle, like an LLM, or based on the task at hand, what are the concepts that are relevant for your task? So, for instance, if you're classifying this image, what concept described a crosswalk? And then you get… if you use Clip for doing this, you would then get Clip,

336
00:49:49.800 --> 00:49:54.909
Sukrut Rao: Embeddings for the text concepts and also for the image, and then learn an alignment between the two.

337
00:49:55.150 --> 00:50:11.760
Sukrut Rao: What we instead propose is to invert the paradigm and try to get concepts that the model actually knows, instead of trying to force the model to give you concepts that you want. So the high-level idea is to instead perform concept discovery in an unsupervised manner, and then propose a way to do concept naming in an automated manner.

338
00:50:12.320 --> 00:50:24.099
Sukrut Rao: So this has the advantage that you do not need to ask a third-party model for concepts that you want, and the concept bottleneck can now be reused across various tasks, because you're doing concept discovery independent of the task.

339
00:50:24.150 --> 00:50:39.460
Sukrut Rao: And more importantly, you are trying to get concepts the model already knows. You're not trying to force the concept to… the model to give you concepts that some LLM said it should. And this is particularly relevant because often, in these kinds of approaches, you would get concepts that were not even visual, such as

340
00:50:39.460 --> 00:50:44.850
Sukrut Rao: If it was a flower, you get a concept of the scientific name of the flower, and so on, which cannot be really visually grounded.

341
00:50:45.610 --> 00:50:58.990
Sukrut Rao: So how do we do this? For concept discovery, we basically use a sparse autoencoder, which basically consists of a linear encoder, which maps defense uninterpretable features to a high-dimensional space of concepts.

342
00:50:58.990 --> 00:51:12.759
Sukrut Rao: And then a linear decoder that maps it back to the feature space, and this is trained with two losses. One, a reconstruction loss to faithfully reconstruct the input features with the reconstruction output, and a sparsity loss to ensure sparsity in this distant table space.

343
00:51:12.940 --> 00:51:28.299
Sukrut Rao: And an important detail here is that these linear decoder weights are usually unit norm, which means your reconstructed output is… can be viewed as a weighted linear combination of concept strengths and unit vector weights. So,

344
00:51:28.350 --> 00:51:35.640
Sukrut Rao: Basically, these vectors are now a dictionary of concepts, and these C's are basically how strongly those concepts are present.

345
00:51:36.820 --> 00:51:55.829
Sukrut Rao: So we do this, we get an unsupervised concept discovery performed, but at this point, you do not have any names to the concepts which concept organically models need. So we propose an approach to name them automatically, and to do this, first, we take a large vocabulary of words and get text embeddings for them, and just project this into clip space.

346
00:51:56.080 --> 00:52:11.650
Sukrut Rao: And we then make use of this fact that the reconstruction is, again, a weighted linear combination of these concept strands with unit number decoder weights, and make a hypothesis that these specific weights themselves might lie in meaningful directions in clip space.

347
00:52:11.650 --> 00:52:26.610
Sukrut Rao: And so we just match the dictionary vector to the closest name that's available from the vocabulary. So from WD1, we find which text is the closest, WD2, which text is the closest, and so on. And by doing this, whichever is the closest vector is assigned as the name of that concept.

348
00:52:26.920 --> 00:52:38.229
Sukrut Rao: So this is based on the assumption that individual vectors lie in a meaningful direction in clip space, but surprisingly, we find that it is overall quite effective. So here are some of those examples of concepts that we find.

349
00:52:38.230 --> 00:52:49.639
Sukrut Rao: In each block, you have the concept with an automatically assigned name, and also the top activating images across various datasets, and we find that the name usually matches to what

350
00:52:49.640 --> 00:53:00.540
Sukrut Rao: activating images app. And this goes beyond just simple concepts, you also have more abstract terms, like smiling, asleep, and so on, so it's quite a diverse. At what layer are you using the SAE?

351
00:53:00.630 --> 00:53:04.829
Sukrut Rao: This is at the output of a CLIP model. So this is the final output model.

352
00:53:05.360 --> 00:53:10.669
Sukrut Rao: So at this point, it's still… we are still in the space of creating a classification model with cancer bottlenecks.

353
00:53:14.220 --> 00:53:26.149
Sukrut Rao: But, like I said, so that is exactly the limitation that we have, so it can only perform classification, and also the concepts are not localized to the image. So there are other works in concept organic literature that show that

354
00:53:26.150 --> 00:53:49.460
Sukrut Rao: you could have a CBM that gives you very good performance and concepts that seem to make sense, but do not use the regions in input that you expect them to. So here, this was a workshop paper in ICLA 21, which, in this kind of bird example, it showed, if you have a concept that says, let's say, spotted wing pattern, striped wing pattern, and so on, everything localizes through the entire bird. It's not really looking at even the wings to make its decision.

355
00:53:49.460 --> 00:53:53.659
Sukrut Rao: So this calls into question the faithfulness of this concept that we have.

356
00:53:53.830 --> 00:53:55.730
Sukrut Rao: So, to address all this.

357
00:53:55.770 --> 00:54:02.939
Sukrut Rao: In a follow-up work, we, try to build upon this and propose a language and concept foundation model instead.

358
00:54:02.940 --> 00:54:22.609
Sukrut Rao: And this has the following improved properties. Specifically, we now have spatially grounded and localized concept extraction, so each of the concepts now also has a spatial locality, so body of water focuses exactly on the body of water and so on. We also build a hierarchy of concepts, so the concept relations are preserved and are

359
00:54:22.610 --> 00:54:24.270
Sukrut Rao: extracted?

360
00:54:24.490 --> 00:54:34.240
Sukrut Rao: And we also improved naming in this hierarchy-aware manner, which I'll come to in the next few slides. And finally, because we want to go beyond classification, we explicitly show that,

361
00:54:34.240 --> 00:54:44.910
Sukrut Rao: By applying this on a clip-like model, our concept foundation model can still do the tasks the original clip model could do, such as classification, but also other tasks, like open vocabulary segmentation, and also

362
00:54:44.910 --> 00:54:47.779
Sukrut Rao: Plugging it into an LVLM to do captioning and so on.

363
00:54:49.110 --> 00:54:59.550
Sukrut Rao: So, broadly, we have these four steps to do this, and I will very briefly go over each of them one by one, which… and it also has a lot of similarities to the prior work, so it will be a bit… it will not be that…

364
00:54:59.590 --> 00:55:14.839
Sukrut Rao: it'll be relatively fast. So overall, in the first step, we first want to extract semantic features. So instead of taking just the final layer pool features that we did before, this time we just take the patch tokens themselves before pulling from the clip model.

365
00:55:14.840 --> 00:55:32.469
Sukrut Rao: And we also apply a dynoization step. So this was an idea proposed a few years ago, which teaches basically clip to mimic dyno similarities in order to get features that are more semantically well aligned. So this improves locality and also smoothens representations by doing this.

366
00:55:32.840 --> 00:55:44.669
Sukrut Rao: This next step, we again extract concepts, but via assays, but there are two key differences. One, again, we do this at a patch level instead of doing it on pulled, features, so this reserves locality.

367
00:55:44.670 --> 00:56:03.229
Sukrut Rao: And secondly, we also use Amatrioshka essays so that we get concepts at various granularities. So, Amatriyoshka essay is basically just like a regular essay, except that the reconstruction objective requires you to reconstruct with a varying number of concepts, which forces some concepts to be co-strained and some concepts to be fine-grained.

368
00:56:04.520 --> 00:56:07.220
Sukrut Rao: Then, to discover hierarchies.

369
00:56:07.220 --> 00:56:29.880
Sukrut Rao: we basically look at co-occurrences. So, what we do is, for every pair of concepts, we find how often they co-activate, and then form parent-child relationships based on this. So, concept A is a parent of B, if it's almost always active whenever B is active, and by doing this, we get children that are typically either subparts or specifications of parents, and I'll show examples of this in the coming slides.

370
00:56:30.890 --> 00:56:42.850
Sukrut Rao: And finally, we do naming just as we did before, but with a list, where we take into account the parent-char relationships. So what we find is that instead of just matching the dictionary vector to the clip vocabulary.

371
00:56:42.850 --> 00:56:55.450
Sukrut Rao: Combining the dictionary vector of both the parent and child to get a composite vector actually gives you more accurate names, because it's more hierarchy aware, and the children vectors are usually just an offset over the parent vector.

372
00:56:55.450 --> 00:57:05.040
Sukrut Rao: And this gives you a hierarchy with name concepts. For instance, you could have a parent concept called turtle, and then a child concept, it is either a type of turtle or a part of a turtle.

373
00:57:06.060 --> 00:57:07.670
Sukrut Rao: These were the four steps.

374
00:57:07.840 --> 00:57:17.679
Sukrut Rao: And broadly, then we get concepts that are much more organized and localized aspect than before. So here is an example of a concept with its automatically assigned name.

375
00:57:17.680 --> 00:57:29.430
Sukrut Rao: But this time, we also have fine-grained localization. So these images, for instance, contain a lot of other things, like people, trees, and so on, but the motor car concept only activates for regions corresponding to the motor car.

376
00:57:29.540 --> 00:57:46.759
Sukrut Rao: And then here are the examples of style concepts, so you have things that are parts of a car, but you also have things like subspecifications of a car. And this was a small example for visualization, but we also get larger graphs, which are generally directed us at clip graphs, which go across multiple layers in the hierarchy.

377
00:57:48.120 --> 00:57:58.529
Sukrut Rao: We also quantitatively evaluate that the naming is more accurate, because that was one of the things that we wanted to do, by using the parent-child evaluation, and this is done via an LLLM as a judge.

378
00:57:58.530 --> 00:58:17.410
Sukrut Rao: And compared to the previous approach, which we showed in Discover the name, we see that we get much better naming accuracy. One limitation here is that we still assume that all concepts are nameable, which may not necessarily be true. It's possible that some concepts are simply not human interpretable, and maybe we need to abstain from naming, and that's something we have to still

379
00:58:17.410 --> 00:58:19.090
Sukrut Rao: quantitatively verified.

380
00:58:21.060 --> 00:58:38.840
Sukrut Rao: So, finally, coming to the downstream uses, we find that we can now again get a CBM-like classification, but this time, you also have this extra localization. So if you say this is a swimming pool because of the body of water, you know that the concept is looking at this part of the image, and so on. And performance-wise, we find that this is

381
00:58:38.870 --> 00:58:46.480
Sukrut Rao: comparable to all previous CBM approaches, and especially the best among all localizable CBM approaches.

382
00:58:47.520 --> 00:58:56.989
Sukrut Rao: You mean it's best in what… in what… in what respect? In the ones… in terms of accuracy. Oh, it compromises, at least in performance. Yeah.

383
00:59:00.890 --> 00:59:17.160
Sukrut Rao: So we also have… so, some things I didn't put in the slides because they were just numbers. We also have quantitative evaluation on how consistent the concepts are and how accurate the relationships are, both by our quantitative metrics and by user studies, and there also we get high numbers overall. So…

384
00:59:17.200 --> 00:59:24.960
Sukrut Rao: So here… so here's a realist, like, a real question, right? So, like… But, so my question is… So…

385
00:59:25.360 --> 00:59:34.409
Sukrut Rao: A lot of the interoperability evaluations that are done in this space are, are evaluations of whether it makes sense to a person.

386
00:59:35.910 --> 00:59:42.320
Sukrut Rao: And it… which raises the question, are we asking, how is the AI solving it?

387
00:59:42.720 --> 00:59:52.549
Sukrut Rao: Or we're asking, you know, do you have a good method for creating an explanation which is intuitive to a person, regardless of whether that's how the AI is solvable?

388
00:59:53.360 --> 00:59:57.159
Sukrut Rao: Right, so my question is, like, do you… do you feel like…

389
00:59:57.370 --> 01:00:04.920
Sukrut Rao: these methods actually expose how the AI is doing, or do you feel like these methods are creating explanations that make people out?

390
01:00:05.930 --> 01:00:14.340
Sukrut Rao: I think that's a real risk, and I agree that the metrics are very mixed on this. Some metrics are very much focused on

391
01:00:14.410 --> 01:00:23.290
Sukrut Rao: seeing how well they align with humans. So, in our… so, yeah, it is possible that, to some extent, we are also evaluating more on whether

392
01:00:23.290 --> 01:00:46.419
Sukrut Rao: humans are happy, but we also try to have some quantitative metrics that are independent of human intuitions. For example, we evaluate concept consistency based on dyno features in one of the metrics, which basically checks whether… if you were in dyno space, is your concept always activating for… at similar parts of dinospace? So this is completely unrelated to any human annotation that you have.

393
01:00:46.480 --> 01:01:00.270
Sukrut Rao: And we do this exactly for this reason, because human annotated datasets often have preconceived notions on which concepts you must find without worrying about what the model actually does. Yeah, this is sort of my question, like, I'm… like, I'm most interested

394
01:01:02.800 --> 01:01:09.919
Sukrut Rao: in analyses of model failures that are surprising to people, like, I'm very interested in the gap between what a person

395
01:01:10.150 --> 01:01:11.790
Sukrut Rao: would expect?

396
01:01:12.090 --> 01:01:17.239
Sukrut Rao: And what am I… does, like, when a model is doing something that

397
01:01:17.540 --> 01:01:20.769
Sukrut Rao: Like, I almost wish we had evaluation sets.

398
01:01:20.980 --> 01:01:24.260
Sukrut Rao: That had to do with… no person…

399
01:01:24.440 --> 01:01:33.039
Sukrut Rao: would have predicted this behavior for the model, but the model did something that can explain why that happens, and you're like, you know, it's, like, it's very dissatisfying.

400
01:01:33.380 --> 01:01:34.170
Sukrut Rao: Right.

401
01:01:34.910 --> 01:01:44.290
Sukrut Rao: To just say, oh, the model's correct, and all these things, the person, you know, is also correct, and… Yeah, yeah, yeah. You know, and I'm not sure what we're pointing at.

402
01:01:44.410 --> 01:01:49.880
Sukrut Rao: Honestly, right, this is actually why I left.

403
01:01:50.400 --> 01:01:57.649
Sukrut Rao: industry to work on South Big Biz. In industry, we create these big departments of

404
01:01:58.170 --> 01:02:02.210
Sukrut Rao: of, like, the Group for Trustworthy AI,

405
01:02:02.410 --> 01:02:10.199
Sukrut Rao: And it often seemed like the whole point of the exercise is to create committee pictures to assure people

406
01:02:10.330 --> 01:02:12.500
Sukrut Rao: that the AI was reasonable.

407
01:02:12.740 --> 01:02:19.510
Sukrut Rao: even… If it had nothing to do with what the AI is actually doing. Yeah. And…

408
01:02:19.630 --> 01:02:21.019
Sukrut Rao: Which is interesting.

409
01:02:21.200 --> 01:02:24.760
Sukrut Rao: Yes, the… for me, it's a very concerning exercise.

410
01:02:24.870 --> 01:02:29.690
Sukrut Rao: To do this. And so I guess my question is, like, do you really believe it?

411
01:02:30.130 --> 01:02:35.089
Sukrut Rao: You know, when you, when you produce these pictures, do you really believe, like, like, would you, like…

412
01:02:35.490 --> 01:02:37.439
Sukrut Rao: Put your grandmother under the x-ray?

413
01:02:37.940 --> 01:02:40.899
Sukrut Rao: And they say, you know, this is why

414
01:02:41.150 --> 01:02:46.349
Sukrut Rao: you know, you have cancer here, right? Do you really, you know, do you really believe

415
01:02:46.730 --> 01:02:50.969
Sukrut Rao: This explanation, or do you feel like this is just a picture for selling the X-ray machine?

416
01:02:51.070 --> 01:02:53.640
Sukrut Rao: Right? That's… that's my…

417
01:02:53.710 --> 01:03:08.999
Sukrut Rao: That's really the question, then. So that is also one of the motivations why we are using Because Leg models, because there, the… we have some guarantees by construction, so whether or not we believe it, at least to one specific definition of what is faithful.

418
01:03:09.000 --> 01:03:15.080
Sukrut Rao: towards… with that definition, at least you know what it means. Like, if you have a strong heat map at a specific

419
01:03:15.080 --> 01:03:28.650
Sukrut Rao: part of the image, that is exactly how much that part contributed to the output. It still doesn't answer everything. It, for example, it doesn't tell you concepts used by the model, it only gives you spatial locality. There's, of course, more to do, but…

420
01:03:28.650 --> 01:03:38.529
Sukrut Rao: If you want localization, at least that gives you that guarantee. Right, because you feel like the computational form is more faithful to the actual computation done by the model?

421
01:03:39.010 --> 01:03:52.359
Sukrut Rao: Like so, right, unlike methods like LRP or whatever, which just, like, make up a completely new companies form. Yes. With LRP, the thing I find a bit dissatisfying when reading the papers is they seem to propose rules based on

422
01:03:52.360 --> 01:04:04.589
Sukrut Rao: what might make sense, but there's not really a clear justification why that should be the rule. It's like, what makes the X-ray machine look good? Yeah. Right? As opposed to, right, what it's actually doing. Yeah.

423
01:04:05.180 --> 01:04:19.410
Sukrut Rao: So in the formulation of the VCOS models, at least we don't have that, so… and if you… if you… at least in synthetic setups, where we explicitly try to create a case where things are overty, we do find that you get explanations that are really not looking at the object as well.

424
01:04:20.410 --> 01:04:21.120
Sukrut Rao: I see.

425
01:04:27.100 --> 01:04:46.269
Sukrut Rao: In this case, I think it is still a bit mixed. We have your consistency evaluations independent of humans, like I said, using dyno features, where you are not trying to match to anything to what a human label, but to see if the features, the activation locations are consistent across various images. So that, I think, partly goes there, but

426
01:04:46.270 --> 01:04:52.469
Sukrut Rao: this is a bit more empirical, there's no guarantees, so I would say this has relative to because more

427
01:04:52.470 --> 01:04:57.309
Sukrut Rao: Concern in the direction that you mentioned, of things looking pretty for the

428
01:04:57.310 --> 01:05:06.009
Sukrut Rao: human, and that's making us happy. But, with the… with the dyno explanation, sorry, dyno-based evaluation, I'm still…

429
01:05:06.800 --> 01:05:13.259
Sukrut Rao: reasonably hopeful that this also shows some amount of faithfulness, independent of what time smells.

430
01:05:17.320 --> 01:05:20.709
Sukrut Rao: Yeah, so for me, You know, the…

431
01:05:22.100 --> 01:05:28.389
Sukrut Rao: The convincing evidence is when you have causal effects, and when you have

432
01:05:28.800 --> 01:05:36.470
Sukrut Rao: surprises, so when, like, so there's, like, I feel like these are the two, like, steps. If you… if you have situations where…

433
01:05:36.900 --> 01:05:43.179
Sukrut Rao: You have data, you have data sets of cases where humans could not have packs.

434
01:05:43.640 --> 01:05:50.209
Sukrut Rao: And you have the insight. I think that that can be convincing. And then we have causal effects.

435
01:05:50.380 --> 01:05:55.469
Sukrut Rao: when… You know, an explanation.

436
01:05:55.810 --> 01:06:02.110
Sukrut Rao: This one is highly predictive of something that you would change, that, something.

437
01:06:02.600 --> 01:06:06.109
Sukrut Rao: And I guess I'm… I worry about images a lot, where…

438
01:06:06.480 --> 01:06:10.930
Sukrut Rao: You know, you convict something that changes, you do a background.

439
01:06:11.280 --> 01:06:14.980
Sukrut Rao: you have these adversarial attacks, right? You have this adversarial issue where…

440
01:06:15.300 --> 01:06:20.039
Sukrut Rao: You know, backup application methods can always You know, making change.

441
01:06:20.410 --> 01:06:21.970
Sukrut Rao: Very hard to get to the phone.

442
01:06:22.580 --> 01:06:23.399
Sukrut Rao: Yeah.

443
01:06:24.060 --> 01:06:26.199
Sukrut Rao: Yeah, I don't know, it's… I think that…

444
01:06:26.740 --> 01:06:30.510
Sukrut Rao: the, the, the, the concern I have is…

445
01:06:30.810 --> 01:06:36.289
Sukrut Rao: is, is, is chasing interpretability like a performance game.

446
01:06:36.830 --> 01:06:41.430
Sukrut Rao: You know, it runs his wrist.

447
01:06:43.640 --> 01:06:58.880
Sukrut Rao: And I… so I, I know you're gonna… you're, like, you're gonna answer, like, oh, yeah, yeah, we're doing it, but I, like, I… I… I sort of have this question. Do you really believe in… is it, you know, do you really feel like we're doing the right thing? You… or do you feel like…

448
01:06:58.930 --> 01:07:16.470
Sukrut Rao: you know, oh, I actually have my benchmarks I don't really believe in, I just have to do them through. So the ones… ones where we… the benchmarks which align to human annotations, those, I agree with you, are problematic, and those we do mainly for… just for everyone's satisfaction, yes.

449
01:07:16.470 --> 01:07:31.109
Sukrut Rao: But we try to always go beyond that as well. Like I said, with consistency-based metrics without humans, or also in the BCOS case. In the BCOS case, or in general, in benchmarks, it's also not entirely clear how to…

450
01:07:31.570 --> 01:07:48.399
Sukrut Rao: what exactly to evaluate, which is completely independent of humans, something we have thought about, don't have a very good answer for, but yeah, and I don't know if it's… In practice, I'm really, I'm really, like, asking for, you know, I almost feel like what we want is we want…

451
01:07:48.610 --> 01:07:50.389
Sukrut Rao: Of humans. Like, we want…

452
01:07:51.810 --> 01:08:07.889
Sukrut Rao: you know, interpretability is supposed to go the other direction. Machine learning is about having humans teach the machines things, and then I feel like interpretability should be about machines teaching humans what they… they… they know or what they're doing. Yeah. And so I don't know if…

453
01:08:08.320 --> 01:08:22.799
Sukrut Rao: So, in terms of surprise, we do find… in my experiments, I did find surprise when playing around with the models. I just don't know what is a good way to quantify this, so it's difficult to give a benchmark number that says this is the surprising thing.

454
01:08:22.800 --> 01:08:31.220
Sukrut Rao: But, for example, very often, especially when dealing with data such like COCO and Pascal VOC and so on, I found that

455
01:08:31.220 --> 01:08:39.290
Sukrut Rao: very unrelated classes seem to be spuriously correlated. So, if I… it was many years ago, but I think, for instance.

456
01:08:39.700 --> 01:08:52.560
Sukrut Rao: the class of a bottle was positively correlated by a person, because in that data set, it's not a very big data set. Most of the images of a portal have a person next to it, or holding it, and so on, and as you could see from the explanations.

457
01:08:52.560 --> 01:09:02.599
Sukrut Rao: And so, if you then look at the explanation for another class, simply the presence of a person was positive evidence for the bottle class, even if there was no bottle present in the batch.

458
01:09:02.790 --> 01:09:13.280
Sukrut Rao: So, some of these correlations in the classes was quite unexpected, but I would say based on the data, but I'm not sure what is a good way to quantify this.

459
01:09:17.540 --> 01:09:22.119
Sukrut Rao: And the causal thing which you said also makes sense. That is something we haven't tried so far.

460
01:09:22.680 --> 01:09:26.820
Sukrut Rao: Well, which I believe you mean in terms of interventions and stuff.

461
01:09:31.750 --> 01:09:36.209
Sukrut Rao: But did he use, like, scenario attacks?

462
01:09:36.330 --> 01:09:37.210
Sukrut Rao: a bottle?

463
01:09:38.380 --> 01:09:40.090
Sukrut Rao: These methods…

464
01:09:44.810 --> 01:09:51.470
Sukrut Rao: Like, I expect that if you use, you know, an adversarial method to diffuse the model, rather than doing it classified differently.

465
01:09:51.700 --> 01:09:54.730
Sukrut Rao: That's… So, salient.

466
01:09:55.480 --> 01:09:56.880
Sukrut Rao: look very different.

467
01:09:57.450 --> 01:09:59.200
Sukrut Rao: But is that something that's available?

468
01:10:00.970 --> 01:10:05.020
Sukrut Rao: No, that's a very good idea. No, we haven't tried this, but makes sense, yes.

469
01:10:05.290 --> 01:10:06.560
Sukrut Rao: Thanks for the session.

470
01:10:08.000 --> 01:10:11.029
Sukrut Rao: You mean adversarial attack as a metric, right?

471
01:10:11.930 --> 01:10:17.000
Sukrut Rao: Yeah, or as, like, a case study test, yeah.

472
01:10:17.900 --> 01:10:20.050
Sukrut Rao: precedence. Yeah.

473
01:10:20.290 --> 01:10:21.740
Sukrut Rao: No, that sounds cool.

474
01:10:26.640 --> 01:10:34.190
Sukrut Rao: So, I just wanted to add one more thing related to the human aspect that you mentioned. So, one of the things we also changed in our locality metrics was

475
01:10:34.190 --> 01:10:49.359
Sukrut Rao: Prior locality metrics were focusing on how well you localize to the object in the image, which is extremely human-aligned, because you have no idea whether the model is actually using the object or not, it could just be using something else. So because… exactly because of this problem, we decided to go a one-level coarser.

476
01:10:49.360 --> 01:11:01.709
Sukrut Rao: So instead of looking at object in the image, we pick a grid of 4 images, 2 cross 2, where we know, based on the model's performance, that each of the individual grid images can be classified correctly by the model with high confidence.

477
01:11:01.710 --> 01:11:22.399
Sukrut Rao: So this… we know that the model is finding positive evidence in the grid cell for that. And we also picked four images that are extremely different, like, in very different classes, so we hope there is no cross-contamination of evidence, where one grid cell shouldn't have evidence for another class. And with this combination, we then see how well the localization is of each class to each particular grid cell.

478
01:11:22.400 --> 01:11:42.219
Sukrut Rao: So this is also an attempt to decouple the human expectation, which is where the object is, versus what the model is doing. So now, if the model is localizing in the correct grid cell, but it's not looking at the object at all, maybe it's looking at the background, our metric would still give it a high score, because we are not checking, is it looking at the right thing? We are just checking if it is

479
01:11:42.220 --> 01:11:45.350
Sukrut Rao: Aligning with what we see from the model's performance.

480
01:11:45.770 --> 01:11:59.900
Sukrut Rao: So even the… most of the localization metrics that we reported in the previous slide, so on, were with this grid-based approach, but again, to, keep everyone happy, we also have the other approach where we have, with object-based. Usually, both are reasonable.

481
01:12:04.950 --> 01:12:20.840
Sukrut Rao: Yeah, then, yeah, this we also then show for other tasks, like open vocabulary segmentation, and this model is, again, remains performance, so there are quantitative results in the, paper, which I didn't put because it's just a dash table, but overall, we… without losing performance, we also get a bunch of decomposition.

482
01:12:21.460 --> 01:12:42.410
Sukrut Rao: And tying back to the original, thing which I showed with, in the BCOS line, again, here we can now apply this to large visual language models and also help boost steering. So, we again do this on a small scale. So we take this CFM instead of the CLIP model and plug it in with an LLM, with a bridge that we train. So this is, again, a small model, which is more as a proof of concept at this point.

483
01:12:42.410 --> 01:13:02.350
Sukrut Rao: And here is an example of steering. So, you get a caption that tells you about the cake, the wine, and so on, and if you look at the concept decomposition and zero out one of the concepts, say, the wine, you still have a caption that focuses on the cake, and vice versa. So, this is an example of how this could be used for steering, for unlearning, for fixing harmful behavior, and so on.

484
01:13:02.350 --> 01:13:12.299
Sukrut Rao: We also have quantitative results that explore how well this works, and we find that it works well for some classes, not so well for other classes. That's probably also because it's at such a small scale.

485
01:13:12.310 --> 01:13:29.420
Sukrut Rao: And also, we have evaluations to see whether the captions themselves are reasonable, so in terms of whether it's still English sentences that make sense and are coherent and general across the board, even after doing the intervention, this remains preserved.

486
01:13:29.970 --> 01:13:33.079
Sukrut Rao: Do you really need an image here? Like, can you…

487
01:13:33.550 --> 01:13:36.040
Sukrut Rao: artificially boosts the Cape and seawater.

488
01:13:36.720 --> 01:13:37.850
Sukrut Rao: They're not us.

489
01:13:38.630 --> 01:13:41.190
Sukrut Rao: The concept decomposition comes from the image.

490
01:13:41.500 --> 01:13:44.180
Sukrut Rao: Yeah, I'm saying, like, you already have your phone.

491
01:13:47.270 --> 01:13:51.610
Sukrut Rao: So, can you, if you, if you, if you had an image with no cake… Yeah.

492
01:13:51.820 --> 01:13:57.239
Sukrut Rao: And… and then you intervene on your concept thing, and you just boost the team.

493
01:13:57.710 --> 01:14:02.450
Sukrut Rao: Yes, that would be a very interesting experiment. We haven't tested this so far.

494
01:14:03.070 --> 01:14:11.989
Sukrut Rao: But my guess is it should work, because SAs have been used as a steering tool in other places, and there, as far as I remember other works, they report that this seems to happen.

495
01:14:12.480 --> 01:14:14.509
Sukrut Rao: But I haven't tested this at least.

496
01:14:18.860 --> 01:14:31.149
Sukrut Rao: Yeah, so this ends this direction, where we, again, we have a foundation model, an interpretable foundation model, which now it is top-down via concepts, and again, we go towards doing this at the end.

497
01:14:32.260 --> 01:14:40.289
Sukrut Rao: So I should… I should let people ask any questions for the last couple minutes, if that's okay. Do you have any, like, summary that you want to give?

498
01:14:41.000 --> 01:14:47.369
Sukrut Rao: I had more to say, but I can stop here, and at this point, the summary is that we now have

499
01:14:47.370 --> 01:15:11.229
Sukrut Rao: interpretable, foundation models. In the coming two parts, I was going to say how we can use them to improve models, so maybe I'll just go show one or two slides on this at a very high level. So, specifically, we look at this context where you have, spurious features, so that is what I was alluding to earlier. With the explanation, you see the bicycle class is predicted also by people here in this synthetic waterbird case.

500
01:15:11.230 --> 01:15:32.789
Sukrut Rao: It's predicted… the bird is predicted by the background. So, this is an example where you're not having… getting a pretty map, it's literally focusing on completely what you would not expect. So we do, basically, regularization. So we add an extra loss to fine-tune the models, with… in addition to the task, also for the explanation. So, forcing the model to look at the right thing based on what you want it to do.

501
01:15:32.830 --> 01:15:43.180
Sukrut Rao: And overall, we find that weakness explanations are the best in terms of how effective this is. So, that is why we believe that faithful explanations are a very good control surface for steering.

502
01:15:43.360 --> 01:15:46.870
Sukrut Rao: And specifically in the context of this Waterbirds dataset.

503
01:15:47.340 --> 01:15:57.940
Sukrut Rao: you have this synthetic construction for exactly evaluating this, where in training you have land birds on land, water birds on water. It's a binary classifier or classification task between land birds and waterbirds.

504
01:15:57.940 --> 01:16:15.869
Sukrut Rao: But at test time, you have all four combinations, so if the model learns to rely on the background, at test time, it is going to have problems with the middle two and the extra groups, because now the correlation has been broken, and we indeed find this quantitative, qualitatively and quantitative. So originally, the model predicts wrongly, looks at the wrong things, and we can refocus the model.

505
01:16:15.870 --> 01:16:18.660
Sukrut Rao: With this kind of, motivation.

506
01:16:18.660 --> 01:16:25.400
Sukrut Rao: What we hope is that we can now apply this also with large vision language models at foundation model scale, and

507
01:16:25.400 --> 01:16:42.050
Sukrut Rao: Then, to coming back to the question you had in the beginning, so far, we only build models that are close in performance, but with this kind of regularization, at least on specific problems, specific issues that we want to fix, we would hope that this extra guidance could then be used to improve the performance as well.

508
01:16:42.050 --> 01:16:44.019
Sukrut Rao: So maybe that is a conclusion.

509
01:16:45.260 --> 01:16:45.949
Sukrut Rao: Pardon me.

510
01:16:54.480 --> 01:16:56.319
Sukrut Rao: Yeah, I was gonna ask about the M.

511
01:16:56.520 --> 01:17:00.750
Sukrut Rao: Were you able to benchmark and find any, like, large teams from?

512
01:17:01.280 --> 01:17:02.380
Sukrut Rao: localization, right?

513
01:17:02.740 --> 01:17:07.560
Sukrut Rao: For which one? The fine-tuning you do to make sure that they'll follow the right folks.

514
01:17:08.100 --> 01:17:16.930
Sukrut Rao: So, in this context, yes. So, for example, on waterbirds, we get significant gains. Doing with foundation models is something we are wanting to do now, so we don't have reasons for that.

515
01:17:21.740 --> 01:17:27.250
Sukrut Rao: Yeah. So, what's your general take on interim technology itself? Where do you…

516
01:17:27.540 --> 01:17:30.250
Sukrut Rao: Where do you think it should go towards? It feels like…

517
01:17:32.140 --> 01:17:34.999
Sukrut Rao: Are pushing more towards, let's…

518
01:17:35.420 --> 01:17:38.119
Sukrut Rao: Change the architecture a little bit, kind of.

519
01:17:38.480 --> 01:17:55.529
Sukrut Rao: So, this is one of the things we… I worked on during my PhD, but in a broader sense, I would say what I'm interested is in actionability. So, not interpretability just for its own sake, but then using it as a tool to fix things in models like this, for instance.

520
01:17:55.530 --> 01:18:07.000
Sukrut Rao: I believe that the changing things is maybe one way of doing it, but I don't… not the only way of doing it. Just the general notion that you can use explanations as a tool to then

521
01:18:07.080 --> 01:18:16.320
Sukrut Rao: find what's going wrong, and use that understanding then to fix the model itself. And the downstream gains, I believe, are probably more interesting to the broader community as well.

522
01:18:31.040 --> 01:18:32.349
Sukrut Rao: Any other questions?

523
01:18:34.940 --> 01:18:36.789
Sukrut Rao: Thank you, Susan. Thank you.

524
01:18:36.890 --> 01:18:38.169
Sukrut Rao: Thanks very much.

525
01:18:45.460 --> 01:18:46.739
Sarah Wang: Great, thank you.