WEBVTT

1
00:00:11.640 --> 00:00:14.279
216-101 Large Conference: Hello? Hello, I'm Mike, can you guys hear us?

2
00:00:14.620 --> 00:00:24.240
216-101 Large Conference: I think they can. Is anybody online? Maybe there's nobody online? Okay, Ben's working, right? Yeah, it seems… And then, let me pop this over to the side to make it bigger. Boop!

3
00:00:24.460 --> 00:00:27.190
216-101 Large Conference: Okay. All right.

4
00:00:27.880 --> 00:00:29.740
216-101 Large Conference: So, Nishant?

5
00:00:30.740 --> 00:00:33.180
216-101 Large Conference: Very lucky to have you here today.

6
00:00:33.580 --> 00:00:38.379
216-101 Large Conference: Welcome to… welcome to our, our seminar group.

7
00:00:38.970 --> 00:00:52.349
216-101 Large Conference: You know, people are… some people are here, even though it's summer. It's a little bit sparser than, than it would be in a couple weeks, but we're really lucky to have you here. So, Nishant is,

8
00:00:52.480 --> 00:00:57.899
216-101 Large Conference: I'm gonna be a fourth-year PhD student at, at LTI, at CMU.

9
00:00:58.030 --> 00:01:05.509
216-101 Large Conference: And, he explains to me that he's one of the few, defenders of

10
00:01:05.540 --> 00:01:24.940
216-101 Large Conference: the practice of interpretability at CMU. So, so thank you for… thank you for that. he's… he's been… apparently, you've been collaborating with Aaron Muller quite a bit, is that right? Yeah. And so, so I'm looking forward to learning and hearing about… about your work. So welcome, Sean. Awesome.

11
00:01:24.940 --> 00:01:37.889
216-101 Large Conference: Thanks so much. So today I'm going to be telling you about, a big picture workshop paper that comprises of four papers that started… the first paper started when David was a…

12
00:01:37.980 --> 00:01:46.130
216-101 Large Conference: third-year PhD student, through an ICMO paper of this year. So, long, sort of,

13
00:01:46.530 --> 00:02:00.579
216-101 Large Conference: spectrum of things. We can go into as much detail as you want on things. I can run through things quickly, let me know what you're interested in. Yeah, and feel free to interrupt with questions, comments, anything, anytime.

14
00:02:01.190 --> 00:02:07.109
216-101 Large Conference: So the title of this is Harnessing the Latent Space, so we talk about the first paper's,

15
00:02:07.250 --> 00:02:17.409
216-101 Large Conference: for steering vectors, and then for model internal calibration and confidence estimation. And the goals here are for control and trust in general.

16
00:02:19.920 --> 00:02:21.220
216-101 Large Conference: Alright, so…

17
00:02:23.060 --> 00:02:38.639
216-101 Large Conference: Here are the kind of four papers. So the first… first one is proposing steering vectors for LSTMs. We didn't call it steering vectors, so it didn't pop. Then we called it steering vectors when we looked at things for… for transformers, and then it popped a little bit,

18
00:02:38.950 --> 00:02:39.970
216-101 Large Conference: we…

19
00:02:40.350 --> 00:02:59.390
216-101 Large Conference: So those are the two elements for control, and then for trust, these are more recent papers using model internals to help recalibrate large language models. They're really poorly calibrated, and so anything we can do generally helps, and you can use things from your… the internals of your models to do this.

20
00:03:00.120 --> 00:03:14.019
216-101 Large Conference: So I'll jump into… So the first paper we started thinking about this August 2017, so I assume that's, like, 30-year-ish? Something like that, yes. So things that happened around this time. Transformers were 2 months old.

21
00:03:14.570 --> 00:03:18.119
216-101 Large Conference: The iPhone 8 was not out yet.

22
00:03:18.570 --> 00:03:21.090
216-101 Large Conference: Mahomes had not thrown an NFL pass.

23
00:03:21.340 --> 00:03:21.670
216-101 Large Conference: Huh.

24
00:03:22.180 --> 00:03:29.969
216-101 Large Conference: And Bert was maybe not even in the minds of Jacob Devlin and others, it wasn't out for another 14 months.

25
00:03:32.130 --> 00:03:33.180
216-101 Large Conference: So…

26
00:03:33.510 --> 00:03:45.720
216-101 Large Conference: The goal here at the time was, can we… we had LSTM-based models, they weren't particularly good, but could we maybe steer them to generate a desired sequence without updating any parameter whatsoever?

27
00:03:46.190 --> 00:03:58.099
216-101 Large Conference: And why we kind of wanted to do this was then maybe these models could be maybe theoretically used universally as, like, these universal decoders. It could be maybe minimally adapted for new tasks. This was…

28
00:03:58.470 --> 00:04:00.910
216-101 Large Conference: Great idea at the time.

29
00:04:01.280 --> 00:04:17.879
216-101 Large Conference: And our motivation was an empty task, so if we wanted to do some low resource to English translation, well, parallel data is expensive, you'd have to train an encoder and decoder, and we had no pre-trained models in those days, so can we actually utilize that data effectively?

30
00:04:18.269 --> 00:04:30.489
216-101 Large Conference: And so if we could steer an English language model, we could just use the parallel data to train the encoder, and just map it to this decoder. It should be easier, there's less parameters, etc. So we started

31
00:04:30.810 --> 00:04:36.359
216-101 Large Conference: Kind of with this idea in mind. And to do this, we would need to be able to steer that decoder.

32
00:04:36.830 --> 00:04:41.630
216-101 Large Conference: And so… We trained our own LSTMs,

33
00:04:42.790 --> 00:04:54.260
216-101 Large Conference: we train very small two-layer LSTMs with these many hidden states, and a lot of things that we don't maybe use as much, or think about very much, with shared input and output embeddings, dropout, gradient clipping.

34
00:04:54.300 --> 00:05:08.919
216-101 Large Conference: We used SGD, we looked at, you know, rather than number of tokens, we looked at number of sentences, because we were thinking about mini-bashes in that world. But we also only did one epoch, mostly because these models were so bad that they got saturated pretty quickly.

35
00:05:11.110 --> 00:05:23.140
216-101 Large Conference: And the way we kind of thought about what steering maybe should look like is, in a similar way to how we do steering today, we have some vector, we add it to the hidden state as this, like, bias term.

36
00:05:23.260 --> 00:05:35.839
216-101 Large Conference: But in LSTM, this is maybe less clear, how you would do this. So, what we decided to eventually do was add it to both the hidden and cell states at every single layer, the same vector.

37
00:05:35.900 --> 00:05:51.519
216-101 Large Conference: This is very weird, and the reason to do this was the models were so, so bad that you would have to encourage it, basically hit the model over the head with this jackhammer, repeatedly, at every stage, and at every time step, and, well…

38
00:05:52.070 --> 00:05:55.410
216-101 Large Conference: That's a lot. That's a lot. And this is… this is quite challenging.

39
00:05:55.610 --> 00:06:02.150
216-101 Large Conference: So… Ignore, kind of, what you know about steering vectors, right now.

40
00:06:02.860 --> 00:06:11.320
216-101 Large Conference: Think about there being maybe a steering vector for a given sequence. So can you directly optimize that steering vector to generate a very specific sequence?

41
00:06:11.510 --> 00:06:13.660
216-101 Large Conference: And so how you would do this

42
00:06:14.140 --> 00:06:30.809
216-101 Large Conference: within an LSTM is you'd initialize a steering vector randomly, and you'd use some sort of… and it turns out you can't use any sort of simple approach. You'd have to use a nonlinear conjugate gradient approach to optimize the steering vector to minimize the cross-entropy loss of the given sequence.

43
00:06:30.840 --> 00:06:38.069
216-101 Large Conference: And then, to see how well your steering vector does to generate that sequence, you would start with a VOS token, you would, you know.

44
00:06:38.070 --> 00:06:56.460
216-101 Large Conference: hit the model over the head with this giant steering vector that you have at all the positions you care about, at all the layers you care about, and then you decode using some sort of decoding algorithm. We generally use 3D decoding. In the paper, we did some experiments with beam search with some small width, but turns out it doesn't really matter, because you do pretty well at

45
00:06:56.600 --> 00:07:12.009
216-101 Large Conference: We're covering this. And then you measure how well you do this using BlueScore or some exact match, or something like this. So again, the same picture as before, and now we can kind of go into the details. Ignore this kind of WZ here on the left, because I'll use the cursor.

46
00:07:12.440 --> 00:07:20.689
216-101 Large Conference: And ignore kind of the tiled thing here, but basically your steering vector could be any dimension. If it's the dimension that you're…

47
00:07:20.840 --> 00:07:26.389
216-101 Large Conference: If it's the model dimension, then you don't need a weight matrix or any sort of tiling, but if you

48
00:07:26.550 --> 00:07:31.430
216-101 Large Conference: If it's smaller, then you would need to up-project such that you can do the,

49
00:07:31.620 --> 00:07:41.070
216-101 Large Conference: bias accumulation, right? And if it's larger, then you would do some sort of soft detention over, splits of that vector. So let's say it's, like.

50
00:07:42.110 --> 00:07:50.340
216-101 Large Conference: 4 times the model dimension, then you could split it across four things and do, like, some sort of attention operation over this, and you just pass this in,

51
00:07:50.630 --> 00:08:02.360
216-101 Large Conference: And as I mentioned earlier, you pass this in at both hidden and cell state at every single layer, and if you don't do this, it doesn't work particularly well, and that's mostly because your model is

52
00:08:02.820 --> 00:08:14.739
216-101 Large Conference: quite good. That is, you know, circa 2018 here, these models are getting, like, 50 perplexity on, in-domain data, and that's pretty good for those days.

53
00:08:15.120 --> 00:08:24.790
216-101 Large Conference: And these are… I can claim that these models were very poorly trained, because I trained them, and… So…

54
00:08:25.000 --> 00:08:30.569
216-101 Large Conference: how well can we actually do this? Is there actually a mapping, even when there's, like, really, really bad models?

55
00:08:31.980 --> 00:08:35.599
216-101 Large Conference: So, on the x-axis here, you have

56
00:08:36.120 --> 00:08:46.150
216-101 Large Conference: sentence length, and on the y-axis, you basically have how well you recover your sequences. And so 100 is sort of perfect score, and the…

57
00:08:46.390 --> 00:08:52.959
216-101 Large Conference: Degradation, it's, like, pretty hard to see, but the bottom left is the lowest dimension of your steering vector, so if you're,

58
00:08:53.700 --> 00:08:56.409
216-101 Large Conference: Dimension here is 20, 40 feet.

59
00:08:56.540 --> 00:09:07.080
216-101 Large Conference: Actually, 4096 in this case for the largest model, because it's hidden and cell state and two loop layers. For the small model, 1024 is the, sort of.

60
00:09:07.350 --> 00:09:11.539
216-101 Large Conference: auto dimension size here. You tend to do…

61
00:09:11.540 --> 00:09:23.439
216-101 Large Conference: really poor with, like, very small dimensionality. But as you get to the model dimension, you start doing pretty well, and as your model gets better, you see the small models, like, really bad, they're, like, over 100 perplexity.

62
00:09:23.440 --> 00:09:32.280
216-101 Large Conference: You start doing much better, and for the largest model, for dimensions that are model dimension or higher, you end up getting pretty perfect

63
00:09:33.840 --> 00:09:38.370
216-101 Large Conference: recovery, and so your ability to basically generate,

64
00:09:39.030 --> 00:09:43.929
216-101 Large Conference: Find a mapping between a vector and a… sequence, it's like…

65
00:09:44.360 --> 00:10:00.669
216-101 Large Conference: pretty reasonable, and you can do this with a very small number of iterations. So, to actually learn these steering vectors… yeah. Sorry, I didn't have a question for the slides. David, there are folks in the waiting room. Oh, I have to admit them. Sorry. Oh, Jasmine, yes.

66
00:10:00.670 --> 00:10:05.489
216-101 Large Conference: Anybody else other than Jasmine? I don't know. Okay, hopefully we didn't lose anybody.

67
00:10:05.610 --> 00:10:09.570
216-101 Large Conference: This is new, this new Zoom admit thing. Okay.

68
00:10:09.930 --> 00:10:17.290
216-101 Large Conference: Yeah, so we can actually find this mapping, and this is interesting, because you have a poorly trained language model.

69
00:10:17.290 --> 00:10:33.849
216-101 Large Conference: there exists a single vector that you can force the model to generate pretty much any sequence of interest. So one natural question is, okay, this is all, like, relatively in-domain, what happens when you go a little bit out of domain? So these models were trained on English at GigaWord, which is very different from a TED Talks, and so we looked at

70
00:10:33.850 --> 00:10:40.820
216-101 Large Conference: IWSLT as well, and, like, you basically… these plots look pretty much the same. You can also…

71
00:10:41.590 --> 00:10:45.290
216-101 Large Conference: We'll cover those. So another question you can ask is.

72
00:10:46.160 --> 00:11:00.389
216-101 Large Conference: does this mapping between steering vectors and sequences operate like a database, or does it actually, like, leverage the contents of your language model, what's actually been trained, etc? So we ran this experiment where we uniformly sampled in the vocabulary at random with

73
00:11:01.250 --> 00:11:18.100
216-101 Large Conference: replacement, and just took sequences of the same size as the IWF SLT, and that's what this random is on the right side. So for very small sequences, you're able to do this, like, kind of database-type mapping, and if you think about what the optimization looks like, since you're generating

74
00:11:18.100 --> 00:11:28.940
216-101 Large Conference: one token at a time, you're sort of compressing all that information, the first few tokens, in your vector, so you can pretty aggressively do that. But as soon as you hit, like, length 30, it drops.

75
00:11:29.200 --> 00:11:32.309
216-101 Large Conference: off a cliff, on the right here. So I'm just gonna…

76
00:11:32.660 --> 00:11:36.720
216-101 Large Conference: pause here for… to understand the data. So…

77
00:11:36.730 --> 00:11:47.899
216-101 Large Conference: So each one of these plots is a different model size for your LSDM, parameterization. Each column is? Each column? And each row is a different…

78
00:11:47.900 --> 00:12:05.179
216-101 Large Conference: metric. Each frozen metric, and then each colored line is another dimensionality of what? What is it? It's the dimensionality of the steering vector that we learned, going from 128 all the way up to 32K, and the model dimension,

79
00:12:05.190 --> 00:12:21.610
216-101 Large Conference: So if you didn't need any up projection, down projection, etc, it's four times the number of dimensions here, because you have two layers, and you have a hidden and cell state. I see, so when you… when you train a steering vector that's of a different dimension.

80
00:12:21.610 --> 00:12:39.469
216-101 Large Conference: than the underlying model size. What does that… what does that mean? So, you would, you only have the number of free parameters, so if it's lower dimensional, you would, have a random matrix that up-projects, and you have the same random

81
00:12:39.810 --> 00:12:46.310
216-101 Large Conference: matrix for all… all examples. The… on the… on the other hand, you would tile it, so we only did,

82
00:12:46.890 --> 00:12:55.870
216-101 Large Conference: integer dimens… integer times your model dimension, and then you would tile it such that you have, K candidate, so let's say it's 4 times the

83
00:12:56.080 --> 00:13:12.530
216-101 Large Conference: model dimension, then we would use some sort of soft attention over those to do some sort of matrix multiply, linear combination of those… Oh, I see. …steering vectors. So, effectively, you still have lower rank, but, you have a little bit more… Interesting.

84
00:13:12.780 --> 00:13:17.630
216-101 Large Conference: Capacity to maybe make the optimization landscape a little nicer.

85
00:13:17.830 --> 00:13:22.739
216-101 Large Conference: Yeah, I was gonna follow up on that. Could you give us, like, a little bit more intuition? So…

86
00:13:23.080 --> 00:13:26.359
216-101 Large Conference: For the higher dimensional cases, you're training a…

87
00:13:26.580 --> 00:13:46.229
216-101 Large Conference: higher dimensional steering vector than there are actual parameters to apply it to. And you're saying you learn… you… there's a frozen, non-learned linear transform that you use to down-project to the actual model? We just… we just, softmax over that and the current… and the current hidden state before you,

88
00:13:46.380 --> 00:13:47.760
216-101 Large Conference: To the… oop bias.

89
00:13:47.890 --> 00:13:53.000
216-101 Large Conference: Because otherwise, yeah, you'd have more parameters, and, like, that linear combination would be…

90
00:13:54.040 --> 00:14:01.240
216-101 Large Conference: weird and maybe harder to use. In practice, it probably doesn't matter, he probably could just do this.

91
00:14:01.640 --> 00:14:05.009
216-101 Large Conference: Just because, like, the vectors themselves, more or less.

92
00:14:05.200 --> 00:14:08.230
216-101 Large Conference: are pretty similar, so if you look at the,

93
00:14:08.940 --> 00:14:13.730
216-101 Large Conference: Linear combination weights, they're, like, pretty close to… Like.

94
00:14:14.710 --> 00:14:24.999
216-101 Large Conference: uniform across the… it's, like, 1 over K for each one, so it's mostly just picking one, because they're all pretty much the same. But it makes the optimization a little bit easier.

95
00:14:25.880 --> 00:14:37.299
216-101 Large Conference: And, yeah, anything to make this optimization easier, because, like, we spent a year trying to do this without using conjugated gradients, because, like, you know, I don't think about that. I'm not an optimization person.

96
00:14:37.680 --> 00:14:46.709
216-101 Large Conference: And then I talked to an optimization person, and they were like, yeah, you should do this. Your lost landscape is probably very bad. And then we had a paper after this.

97
00:14:48.600 --> 00:14:56.650
216-101 Large Conference: Cool. Any, any other questions at this point? Yeah. I didn't understand. So you use a down projection matrix to reduce your dimensionality, right?

98
00:14:56.760 --> 00:15:08.050
216-101 Large Conference: The other way. So we use an up projection to increase the dimension. Or a down projection. Well, so the way we do the down projection is we do

99
00:15:08.250 --> 00:15:12.789
216-101 Large Conference: We split it such that, you have model dimension many

100
00:15:13.030 --> 00:15:17.889
216-101 Large Conference: vectors, and then you do soft attention over those to collapse. Okay.

101
00:15:20.300 --> 00:15:21.610
216-101 Large Conference: Any other questions?

102
00:15:22.140 --> 00:15:23.010
216-101 Large Conference: Thoughts?

103
00:15:25.850 --> 00:15:27.410
216-101 Large Conference: So what's your conclusion?

104
00:15:28.080 --> 00:15:30.929
216-101 Large Conference: So, this is nice.

105
00:15:31.310 --> 00:15:37.270
216-101 Large Conference: We thought at the time that these were nicely… these could be the nice, universal…

106
00:15:37.810 --> 00:15:44.029
216-101 Large Conference: decoders in general, if we had the right data and algorithms. Turns out, maybe the algorithms are already there, and we just needed more data.

107
00:15:45.050 --> 00:15:57.459
216-101 Large Conference: And so, one thought process is now, okay, like, at this point, when we published this, this took ages in peer review, 3, 4 cycles? There's… there's 2019,

108
00:15:58.250 --> 00:16:03.740
216-101 Large Conference: GPT-2 had come out, these models are better, so… Natural extension is…

109
00:16:04.140 --> 00:16:07.230
216-101 Large Conference: what happens? Are transformers better to do this? Yeah.

110
00:16:07.620 --> 00:16:18.809
216-101 Large Conference: Can you explain how you've, like, from the observations that you can build pitting vectors to induce a sequence of tokens?

111
00:16:19.280 --> 00:16:29.970
216-101 Large Conference: to their likely universal decoders, what's the jump? Yeah, so if… if you have a model, and you can force it at inference time with the steering vector to generate any sequence you want.

112
00:16:30.410 --> 00:16:38.999
216-101 Large Conference: Regardless if it's in domain and out of domain, etc, then it has the potential to be used as a universal decoder if you can do appropriate

113
00:16:39.180 --> 00:16:47.270
216-101 Large Conference: conditioning. We make no claim that we can actually do appropriate conditioning, and we didn't run those experiments, but it seems like a possible thing.

114
00:16:48.730 --> 00:16:57.120
216-101 Large Conference: So, are all the steering vectors optimized for every sequence, or… Yeah, so these are all specific to that sequence of interest.

115
00:16:57.210 --> 00:17:09.289
216-101 Large Conference: So you can ask this natural conclusion that, you know, we have done now with steering, where, like, do these things mean anything? Can you collapse them to create some sort of attribute or concept-based thing? And we'll get to that in a second.

116
00:17:09.810 --> 00:17:28.160
216-101 Large Conference: Yeah. I'm sorry, I don't want to return to the prior question, but I actually didn't quite follow why, why you, up-projected, and it sounded like it was so that the steering vector was more, like, transferable, but I didn't understand why increasing the parameters would help with that. So…

117
00:17:28.339 --> 00:17:39.339
216-101 Large Conference: That's a… that's a good point. Like, it's not clear that it would help, necessarily. The loss landscape is so weird that having more parameters to, like, wiggle around

118
00:17:39.490 --> 00:17:45.949
216-101 Large Conference: tends to help a little bit. It doesn't help that much. When you say the lost landscape is weird, do you mean that there's just a lot of, like.

119
00:17:46.260 --> 00:18:02.989
216-101 Large Conference: like, harsh boundaries? It's crazy. So for an LSTM, since we're providing the steering vector at both the hidden and cell state at every layer, and there's this back propagation through a time element, you basically have a comp…

120
00:18:02.990 --> 00:18:12.990
216-101 Large Conference: basically a crazy composition function that basically just keeps doing this. So if you backprop all the way through, you're backropping for 40, 50, 60 steps. Right. And so you have this

121
00:18:13.160 --> 00:18:22.010
216-101 Large Conference: just crazy… function that is highly nonlinear, and so having more parameters to, like, update tends to help.

122
00:18:22.060 --> 00:18:36.019
216-101 Large Conference: semi-correct, like, first-order intuition, just be that you're giving more, like, saddle escapes, or… Yeah, yeah, I think, like, that's a nice, like, vibes-y kind of, explanation, but, like, that's probably the best I can do, too.

123
00:18:36.160 --> 00:18:45.710
216-101 Large Conference: So, so following that, because you're increasing the parameters, I'm just curious how you, and maybe I missed this, how you evaluate that it's actually learning

124
00:18:45.740 --> 00:19:04.039
216-101 Large Conference: the right thing versus, like, kind of… does that… does that make… does that question… Yeah, so you're… I think what you're asking is, why is this not just, like, a weird database of, like, vectors to… Yeah, kind of. Yeah, so it turns out it kind of is, but, sort of this… this experiment on the right, on the…

125
00:19:04.220 --> 00:19:07.509
216-101 Large Conference: the plot here will pop up. So this plot is…

126
00:19:08.060 --> 00:19:13.830
216-101 Large Conference: Rather than take real sequences, uniformly sample from the vocabulary at random with

127
00:19:14.150 --> 00:19:17.869
216-101 Large Conference: Replacements. These are going to be really, really high entropy, because

128
00:19:19.050 --> 00:19:32.950
216-101 Large Conference: Bocaps are big, and you're able to do… recover, like, reasonably well up to, like, sequence, like, 25, and then after that, kind of drop a lot. Whereas for, like, real data, even at, you know, 25 to…

129
00:19:33.080 --> 00:19:36.650
216-101 Large Conference: 50, you're doing pretty well. You're able to…

130
00:19:37.170 --> 00:19:45.890
216-101 Large Conference: learn steering vectors for, like, each of these things. And so, there's this capacity that the steering vector can kind of learn, which is, like, about 25 tokens.

131
00:19:46.110 --> 00:19:52.130
216-101 Large Conference: But beyond that, it has to, like, leverage the language model more to be able to…

132
00:19:53.450 --> 00:19:55.439
216-101 Large Conference: Encode up a larger sequence.

133
00:19:55.580 --> 00:19:56.410
216-101 Large Conference: Okay.

134
00:19:57.420 --> 00:19:58.490
216-101 Large Conference: Gotcha.

135
00:20:00.960 --> 00:20:02.739
216-101 Large Conference: Any other thoughts?

136
00:20:05.430 --> 00:20:06.190
216-101 Large Conference: Yeah.

137
00:20:06.350 --> 00:20:10.100
216-101 Large Conference: So, like, coming back to the universal decoder question.

138
00:20:10.230 --> 00:20:11.110
216-101 Large Conference: Okay.

139
00:20:13.070 --> 00:20:20.219
216-101 Large Conference: Like, the idea is that you have one vector that can reliably trigger a sequence from an LSTM,

140
00:20:20.510 --> 00:20:30.819
216-101 Large Conference: So, since you can do it with one vector, there's probably, like, other vectors, too, that can, like, universally decode things out. But, like.

141
00:20:31.810 --> 00:20:33.350
216-101 Large Conference: Like,

142
00:20:34.500 --> 00:20:41.809
216-101 Large Conference: like, the opposite argument is that, like, I can also have a table of vectors and sequences, and, like, just…

143
00:20:42.060 --> 00:20:50.289
216-101 Large Conference: go and track for that particular vector and get the sequence out, which is not… I don't know if I want to call that very versatile decoder yet.

144
00:20:50.670 --> 00:21:01.860
216-101 Large Conference: But, like, in LSTM… I understand this is LSTM, so probably, like, there's not a lot of generalization happening. Yeah. But, like, did you see that, like, letting the LSTM go beyond

145
00:21:02.320 --> 00:21:21.159
216-101 Large Conference: the sequence that you trained for and see, like, there's a generalization to triggering, like, more concepts that is related… So, we didn't run those experiments on the LSTM scale, just because we felt like the vectors weren't that good. And the fact that it's even able to, like, encode 25 random things means, like, the

146
00:21:21.640 --> 00:21:26.560
216-101 Large Conference: Machinery is kind of maybe too strong, and doesn't have enough of,

147
00:21:26.720 --> 00:21:36.489
216-101 Large Conference: doesn't have a prior to say that you need to, like, leverage the LM. But for transformers, I will answer your question with transformers.

148
00:21:36.790 --> 00:21:41.760
216-101 Large Conference: Cool. Any, any other things before I move on to… Trying to answer this?

149
00:21:43.190 --> 00:21:47.670
216-101 Large Conference: Cool. Okay, so, we think that these can be universal

150
00:21:48.330 --> 00:21:54.259
216-101 Large Conference: Decoders, here's the paper. This is, like, yeah, ages ago.

151
00:21:55.400 --> 00:21:58.089
216-101 Large Conference: So, now let's move to transformers.

152
00:21:58.230 --> 00:22:04.780
216-101 Large Conference: So, can we do two things? One is the first one, the same goal as before. Can we kind of steer to…

153
00:22:04.900 --> 00:22:06.009
216-101 Large Conference: generate some…

154
00:22:06.220 --> 00:22:21.310
216-101 Large Conference: desired sequence without any parameter updates? And two, can we do concept-based steering or, like, attribute-based steering? So, can those extracted steering vectors, rather than doing activation-based things, can those extracted steering vectors be used as useful

155
00:22:21.510 --> 00:22:25.080
216-101 Large Conference: Representations to do some sort of concept-based or attribute-based

156
00:22:27.520 --> 00:22:42.509
216-101 Large Conference: So, this is going to look similar-ish to the first… to the plots earlier in sort of the operation. This is going to make a lot more sense, because it's more similar to what

157
00:22:42.850 --> 00:22:44.110
216-101 Large Conference: People do now?

158
00:22:44.210 --> 00:22:52.249
216-101 Large Conference: So I'll start on the right-hand side, so this is a transformer. This is… looks eerily similar to GPT-2 Small.

159
00:22:52.470 --> 00:23:01.499
216-101 Large Conference: GPT's too small. So you have your steering vector on the left here, and you could theoretically add it anywhere in your model, and it's a relatively…

160
00:23:01.710 --> 00:23:13.269
216-101 Large Conference: reasonable to do so. You could add it at any layer, you could add it between self-attention and your feet forward, you could do it before your layer norm, you could do it after, and you could do it at any layer, any…

161
00:23:13.380 --> 00:23:14.300
216-101 Large Conference: position.

162
00:23:14.410 --> 00:23:17.950
216-101 Large Conference: And so, going back to the LSTN,

163
00:23:18.430 --> 00:23:26.550
216-101 Large Conference: setup, where we're trying to generate a steering vector for a specific sequence. You'll get started by initializing vector

164
00:23:26.730 --> 00:23:39.379
216-101 Large Conference: Randomly, for a set number of iterations, you do a bunch of… you do a forward pass, injecting that steering vector at a specific location and time step, and you optimize across entropy loss.

165
00:23:39.380 --> 00:23:50.430
216-101 Large Conference: And now, because transformers are better trained and the loss landscape is nicer, you can just use atom, you don't have to use conjugate gradients. So now things are GPU acceleratable, and experiments run.

166
00:23:50.690 --> 00:23:53.239
216-101 Large Conference: Not so terribly slowly as they used to.

167
00:23:54.880 --> 00:24:04.650
216-101 Large Conference: So, you can do this, the… there's two interesting tidbits here. One is the number of iterations you need,

168
00:24:05.930 --> 00:24:21.400
216-101 Large Conference: is, like, 50 to 60 on average to learn a steering vector. If you use a sufficiently large learning rate, you have to use a very large learning rate. So, if you're running this with, like, a normal learning rate, you will think this never works.

169
00:24:21.550 --> 00:24:24.640
216-101 Large Conference: We used a learning rate of 1.0.

170
00:24:24.990 --> 00:24:25.700
216-101 Large Conference: Hmm.

171
00:24:26.280 --> 00:24:29.230
216-101 Large Conference: So this one's very, very large.

172
00:24:29.720 --> 00:24:31.399
216-101 Large Conference: And this was…

173
00:24:32.170 --> 00:24:52.040
216-101 Large Conference: we found out the hard way, so I was doing 1E minus 4, and, like, left the thing for 10,000 iterations, and yeah. Just a very simple, like, grounding question. What was the rough magnitude of your steering… of your learned steering vectors, in proportion to the… to the norm of the actual activations?

174
00:24:53.050 --> 00:24:54.270
216-101 Large Conference: 3-ish?

175
00:24:55.410 --> 00:25:04.369
216-101 Large Conference: But it… the configurations really matter. So we have a configuration where it's the same steering vector at every token position at

176
00:25:04.560 --> 00:25:09.379
216-101 Large Conference: A specific layer, and in that case, the magnitudes are really wonky.

177
00:25:09.610 --> 00:25:14.910
216-101 Large Conference: And the variance is really large, because you're sort of continuously hitting the model over the head with

178
00:25:15.910 --> 00:25:22.399
216-101 Large Conference: something that it needs different magnitudes for at different time steps. If you're doing this at the first

179
00:25:22.860 --> 00:25:30.430
216-101 Large Conference: position at… a later layer, it's largest, because that needs to… Sustained, basically, throughout.

180
00:25:30.690 --> 00:25:36.379
216-101 Large Conference: And so I think 3 is, like, first… first time step, like.

181
00:25:36.980 --> 00:25:42.330
216-101 Large Conference: layer L minus 1 or something. Yeah.

182
00:25:44.980 --> 00:25:48.379
216-101 Large Conference: Cool, so this setup makes sense in general?

183
00:25:50.810 --> 00:25:53.169
216-101 Large Conference: Alright, so…

184
00:25:54.270 --> 00:26:02.300
216-101 Large Conference: Depending on where we put things, and depending on which time steps, as long as it's not the very beginning or the very end, things work pretty well.

185
00:26:02.470 --> 00:26:08.830
216-101 Large Conference: So 100 is, like, full score here, because the blue score based on the original sequence and the…

186
00:26:09.860 --> 00:26:23.690
216-101 Large Conference: recovered sequence. This is GPT-2 small, so 12 layers, the middle layer's generally pretty good. This is now, like, well accepted by… by us. Now, at the time, it was kind of interesting. I expected,

187
00:26:23.990 --> 00:26:28.739
216-101 Large Conference: But, like, later layers to be better, because they're closer to being able to, like, generate, so…

188
00:26:29.200 --> 00:26:32.019
216-101 Large Conference: It has less potential to lose the intervention.

189
00:26:32.670 --> 00:26:37.879
216-101 Large Conference: the top half of the model. Turned out not particularly to be the case.

190
00:26:38.610 --> 00:26:44.779
216-101 Large Conference: And… Looking at a similar experiment to, like, is this just memorizing and…

191
00:26:45.270 --> 00:26:48.639
216-101 Large Conference: a database. If you shuffle the order of the words.

192
00:26:48.760 --> 00:27:06.199
216-101 Large Conference: These still tend to do pretty well, but if you uniformly sample from the vocabulary at random, you get a pretty big drop. And that drop is much bigger if you just provide the steering vector at the first time step. So the representations are, in general, better, and these are all, like, things you probably

193
00:27:06.320 --> 00:27:13.880
216-101 Large Conference: would expect, a priority, because if you're providing the same intervention at every time step, that thing is a pretty giant intervention, and, like.

194
00:27:15.010 --> 00:27:15.680
216-101 Large Conference: Yeah.

195
00:27:17.660 --> 00:27:31.990
216-101 Large Conference: Great, yeah. Can you repeat what the data your training on is? So it's pre-trained, and then you're training on… Yeah, so this is, like, we have a bunch of different held-out sets. We have, like, Cornell movie dialogues, we have some Gutenberg book corpus, we have some…

196
00:27:32.240 --> 00:27:38.509
216-101 Large Conference: Wikipedia, some, like, other things. Okay, I mean, the difference between… These are all books… this is, like, the books…

197
00:27:38.790 --> 00:27:42.129
216-101 Large Conference: subset. Shuffled is you take the…

198
00:27:42.790 --> 00:27:51.840
216-101 Large Conference: sequence and just permute the support? You say, like, all the books and shuffle throughout all the books, or in each book? No, within each sequence, we shuffle the token.

199
00:27:52.350 --> 00:27:56.829
216-101 Large Conference: Within each sequence. Within each sequence. So it's on garbage, but the tokens are the same.

200
00:27:57.400 --> 00:28:00.750
216-101 Large Conference: Okay, so maybe the, yeah, so maybe you, like.

201
00:28:01.430 --> 00:28:11.200
216-101 Large Conference: the serum word is, like, remembering, kind of, which words were in that sequence. Yeah, so your unigram prior should be the same, right, in both,

202
00:28:11.820 --> 00:28:21.280
216-101 Large Conference: And that was the hope. The hope was that there would be a larger drop, but it seems like content is mostly what the steering vectors are, like, shoving in there, because the drop is pretty small.

203
00:28:21.550 --> 00:28:28.309
216-101 Large Conference: And then the last one is uniformly sampled from the vocabulary at random, the same number of tokens as the original

204
00:28:28.490 --> 00:28:29.310
216-101 Large Conference: books.

205
00:28:30.130 --> 00:28:33.460
216-101 Large Conference: sequence. So, what do you make of the…

206
00:28:33.860 --> 00:28:41.919
216-101 Large Conference: Like, did you expect a lower score for the gibberish? Yeah, yeah, but I expected lower… ideally lower than this. It's, like, pretty high.

207
00:28:42.030 --> 00:28:47.709
216-101 Large Conference: Especially because these… Sequences are, like, pretty large. They're not, like…

208
00:28:47.890 --> 00:28:59.820
216-101 Large Conference: Large by those standards, because again, we're… this is April 2020, so these are, like, 75, 80 tokens long, which was pretty large at the time.

209
00:29:00.440 --> 00:29:07.309
216-101 Large Conference: So, do you have any, thoughts on why it's higher than you thought? Yeah, so I think the…

210
00:29:08.900 --> 00:29:18.879
216-101 Large Conference: So if you have a… so in this case, the model dimension is 768, that's a lot of parameters. So if you think about, like, compressing a sequence.

211
00:29:19.670 --> 00:29:24.650
216-101 Large Conference: you can shove a whole sequence in there, in terms of memory, right? So, like.

212
00:29:26.430 --> 00:29:37.699
216-101 Large Conference: And your language model's pretty good at being able to string words together in general, even if they are low entropy, like, every successive word is lower entropy.

213
00:29:38.790 --> 00:29:40.689
216-101 Large Conference: So, I think that's the kind of…

214
00:29:41.640 --> 00:29:48.369
216-101 Large Conference: not super satisfying answer here. I feel like the capacity of a specific steering vector is, like, pretty high.

215
00:29:50.390 --> 00:29:54.040
216-101 Large Conference: Which is why we can do things that are much more complex now with them.

216
00:29:56.220 --> 00:29:56.990
216-101 Large Conference: Excellent.

217
00:29:57.940 --> 00:29:59.950
216-101 Large Conference: Cool, any, any other thoughts?

218
00:30:01.450 --> 00:30:04.539
216-101 Large Conference: Did you have more experiments, like, before, where you vary the length?

219
00:30:04.740 --> 00:30:12.930
216-101 Large Conference: Yeah, yeah, they're not in the thing, but, we went down to…

220
00:30:13.780 --> 00:30:16.520
216-101 Large Conference: That probably didn't make it in the paper. Went down to, like, 80?

221
00:30:16.920 --> 00:30:28.170
216-101 Large Conference: And it… it's, like, really bad, it's really hard. If we decrease by a factor of 2, these numbers are more or less the same. They're a little bit lower, like, 95-ish.

222
00:30:28.390 --> 00:30:31.920
216-101 Large Conference: If you do PCA on these things, and you…

223
00:30:32.320 --> 00:30:43.289
216-101 Large Conference: we have some plot with, like, number of, explained variance by number of dimensions. You can get, like, 90% with about half the parameters.

224
00:30:46.070 --> 00:30:52.710
216-101 Large Conference: These are weak, well-compressed, like, actually really nice models. So if you do this for, like, modern models, you will get

225
00:30:53.350 --> 00:30:58.690
216-101 Large Conference: Much higher explained variants with much fewer… Dimensions.

226
00:30:58.870 --> 00:31:17.780
216-101 Large Conference: So, I expect, like, for your, like, 7B models now that are 8,000 dimensional, with 1,000 dimensions, you probably get 90% of explained variance for, like, simple tasks. With the variance in the different vectors for each PCA? TCA, yeah.

227
00:31:18.110 --> 00:31:22.279
216-101 Large Conference: For, like, a bunch of different sequences, and then you see? Yeah. Oh, okay.

228
00:31:22.790 --> 00:31:26.050
216-101 Large Conference: There's a plot… there's a plot in the paper somewhere.

229
00:31:28.470 --> 00:31:30.359
216-101 Large Conference: Any other thoughts, questions?

230
00:31:31.630 --> 00:31:34.150
216-101 Large Conference: Alright, so, trying to answer your question, though.

231
00:31:36.930 --> 00:31:43.719
216-101 Large Conference: If these steering vectors were useless, and just operate as a database, you wouldn't be able to do things like semantic

232
00:31:43.870 --> 00:31:49.080
216-101 Large Conference: similarity. So imagine you take the steering vectors you learned for,

233
00:31:49.290 --> 00:32:02.490
216-101 Large Conference: similarity benchmarks, and you take the cosine distance between the two, and you correlate that, kind of similar to what you would do with, like, mean pooled hidden states. You actually do well, and for

234
00:32:02.830 --> 00:32:06.160
216-101 Large Conference: All layers other than layer 0 and layer 11.

235
00:32:06.500 --> 00:32:13.170
216-101 Large Conference: 12-layer model, so, like, index by zero. You do better than, sort of, BERT-based pool.

236
00:32:14.130 --> 00:32:20.580
216-101 Large Conference: And you do better than glove, which is this, like, green line going down, and you do better than mean pooling for the same

237
00:32:21.380 --> 00:32:23.109
216-101 Large Conference: on.

238
00:32:23.530 --> 00:32:24.410
216-101 Large Conference: semantic.

239
00:32:24.820 --> 00:32:30.350
216-101 Large Conference: similarities. So, like, the representations are actually maybe better than the activations themselves.

240
00:32:32.000 --> 00:32:37.299
216-101 Large Conference: And for a small model, the activations are kind of bad, so…

241
00:32:37.900 --> 00:32:47.989
216-101 Large Conference: Sorry, can you restate that? I didn't… can you restate the… that again? Yeah, so, if your representations have better semantic

242
00:32:48.150 --> 00:33:02.640
216-101 Large Conference: similarity than, like, the mean pulled hidden states, the steering vectors themselves, then those should be more useful, better representations for other tasks than the mean pulled hidden states should be. And we see that pretty convincingly here.

243
00:33:02.760 --> 00:33:17.360
216-101 Large Conference: And the way these were computed, we learned steering vectors for each of the sequences on the textual similarity benchmark, and then we computed the cosine, the distance between the two, and used that to correlate. So this is all spearmint correlation numbers.

244
00:33:17.600 --> 00:33:34.999
216-101 Large Conference: Yeah. So, go back in time, and was this a surprising result? Yeah. People were trying pretty hard to get good, you know, sentence embeddings, right? Yeah, so this is very surprising, because there's no reason why we didn't put any priors on the, like, cross entropy, it wasn't like…

245
00:33:35.360 --> 00:33:48.619
216-101 Large Conference: there wasn't even, like, L2 norm on, like, the steering vector to explode, we did none of this. This is just inducing your model to generate that sequence seems better than using the, like, hidden states and the, like, activations themselves.

246
00:33:49.050 --> 00:33:56.620
216-101 Large Conference: So we were quite surprised. I expected it to be a little bit worse than this red line here.

247
00:33:57.030 --> 00:33:59.889
216-101 Large Conference: And definitely much worse than Berke's Oracle.

248
00:34:00.110 --> 00:34:00.960
216-101 Large Conference: Goodbye.

249
00:34:01.090 --> 00:34:04.239
216-101 Large Conference: So, like, I've made this plot, and I was like, fuck.

250
00:34:04.830 --> 00:34:21.210
216-101 Large Conference: had to double-check there's no bugs, there's no, like, weird, like, I wasn't using the test data or something. You would have probably… right, so you would have probably said, okay, the red line is 100%, and I wonder what percent we're getting of that? Yeah, yeah. Turns out we got 200 and something percent. I see.

251
00:34:21.210 --> 00:34:26.930
216-101 Large Conference: Which is nice. So now we're like, hmm, okay, these things are useful, let's do stuff.

252
00:34:27.520 --> 00:34:33.190
216-101 Large Conference: And so, let's try to interpolate between these things. Maybe there's some structure.

253
00:34:33.320 --> 00:34:37.400
216-101 Large Conference: So these are two sequences that were perfectly…

254
00:34:37.560 --> 00:34:41.729
216-101 Large Conference: Reconstructed, and then sort of interpolate it between the two. And, like.

255
00:34:41.920 --> 00:34:52.840
216-101 Large Conference: it's not great by modern standards, but by, you know, 2020 April standards, or May standards, or whatever this was, this is around when DBD3 came out, so, like, mid-June?

256
00:34:53.170 --> 00:35:01.160
216-101 Large Conference: This is pretty good, and you get some sort of overlap. These are, like, well-formed-ish, and for GPT too small, it's pretty good.

257
00:35:01.460 --> 00:35:14.029
216-101 Large Conference: And there's some, like, actual kind of overlap, so you see, like, in the sentence pair 2, you see, like, things about, like, age and, like, years ago from the, like, bottom sentence, and you see, like, 4 from the top sentence, kind of in this.

258
00:35:15.140 --> 00:35:19.140
216-101 Large Conference: Center, but it's not, like, super well-formed, like, really nice.

259
00:35:19.320 --> 00:35:27.110
216-101 Large Conference: There's, like, some structure here, which is just interesting. We're in this experiment for the LSTNs that I didn't put in the presentation.

260
00:35:27.240 --> 00:35:30.829
216-101 Large Conference: Middle's chaos. It's, like, non-grammatical, you just get, like, random.

261
00:35:31.240 --> 00:35:35.420
216-101 Large Conference: sequences. So, like, the landscape itself is, like…

262
00:35:36.000 --> 00:35:38.070
216-101 Large Conference: But for these, it's nicer.

263
00:35:39.820 --> 00:35:54.079
216-101 Large Conference: Yeah. Did you try the modern techniques with LSTMS2? Like, you were doing, like, some sort of very complicated nonlinear optimization? Yes, we tried it with Atom, it gets zero blue. Cannot learn anything. Ever. For any sequence.

264
00:35:54.200 --> 00:36:01.910
216-101 Large Conference: I don't think we, like, went down to, like, 5 tokens or something, maybe you can learn that, but any… even, like, 10 or 15, nothing.

265
00:36:02.340 --> 00:36:09.919
216-101 Large Conference: And I spent 6 months of hyperparameter optimization, like, trying to get that to work, because, like, that's gotta work, right?

266
00:36:10.120 --> 00:36:10.820
216-101 Large Conference: Doesn't work.

267
00:36:11.420 --> 00:36:17.309
216-101 Large Conference: But here, I had to do nothing fancy, just crazy learning rate for… for Adam.

268
00:36:19.950 --> 00:36:23.290
216-101 Large Conference: So now, back to things that, like.

269
00:36:23.580 --> 00:36:33.350
216-101 Large Conference: we could actually use these things for, so could we do, sort of, difference means steering on this? Rather than operate on activations, which we thought was a really bad idea, given

270
00:36:33.710 --> 00:36:41.430
216-101 Large Conference: given how bad they are relative to our steering vectors, we did steering vector-based concept steering. So, you take

271
00:36:42.210 --> 00:36:50.539
216-101 Large Conference: a bunch of positive sequences, a bunch of negative ones, learn their steering vectors, take the mean of those steering vectors, now you have, like, Z positive and Z

272
00:36:50.740 --> 00:36:55.019
216-101 Large Conference: Negative, and then you do the standard, sort of, Diff Meeting Steering.

273
00:36:55.480 --> 00:36:57.739
216-101 Large Conference: On this, and we were able to flip.

274
00:36:58.370 --> 00:37:16.150
216-101 Large Conference: sentiment pretty well. And we compared this to, like, a bunch of, like, Roberta-based, like, unsupervised sentiment transfers type things, and we do a little bit worse than that, but they are, like, explicitly trained for this, and we have steering vectors that don't have anything to do with anything. We're just doing these nice

275
00:37:16.680 --> 00:37:21.679
216-101 Large Conference: things on the end. These are, like, reasonably cherry-picked, but not that reasonably cherry-picked.

276
00:37:22.100 --> 00:37:25.949
216-101 Large Conference: I looked at, like, 8 things and picked 2, so it's like…

277
00:37:26.980 --> 00:37:29.500
216-101 Large Conference: Yeah. And they needed to fit in a…

278
00:37:29.990 --> 00:37:41.200
216-101 Large Conference: one column ACL format, so… they're small. Was there, like, a benchmark for that? So, there's a Yelp, just like Yelp, a positive and negative unpaired.

279
00:37:41.470 --> 00:37:48.450
216-101 Large Conference: And so, like, benchmark is, like, a… ish. There were people who ran numbers on this thing.

280
00:37:50.570 --> 00:38:00.630
216-101 Large Conference: Yeah. To clarify, the diff means steering you're doing here is diff means of learned steering vectors, not diff means of activations. Because diff means of activation didn't exist yet. Right.

281
00:38:01.530 --> 00:38:19.260
216-101 Large Conference: I might be jumping the gun a little bit, but is diff means of the steering vectors superior or inferior to diff means of activation? So, in this paper, we didn't try activations because we thought that there's no way that would work. I'm asking it more for the sake of, like, should I be going back to this?

282
00:38:19.260 --> 00:38:21.699
216-101 Large Conference: I will give you a definitive answer in, like, a month.

283
00:38:21.700 --> 00:38:30.469
216-101 Large Conference: I am currently doing experiments on this, on, like, modern scale, to try to get rid of steering vectors in general, and, like.

284
00:38:30.690 --> 00:38:44.190
216-101 Large Conference: project onto, like, volumes, but using both activations and steering records as our, like, starting points. You can do a direct comparison? Yeah, but it's… projecting onto a volume's hard. Yeah. So…

285
00:38:44.360 --> 00:39:03.989
216-101 Large Conference: Hopefully for iClear, things will work. Things are tending slightly positive here. Cool. Steering vectors seem better than activations, even for modern models, but… Awesome. …less better than they were for worse models. Okay, so the better your models, the better your activations, so the less you have to do this. Yep, makes sense.

286
00:39:04.700 --> 00:39:05.910
216-101 Large Conference: Cool.

287
00:39:07.650 --> 00:39:09.110
216-101 Large Conference: Any thoughts, questions?

288
00:39:09.770 --> 00:39:11.630
216-101 Large Conference: things here.

289
00:39:12.700 --> 00:39:25.080
216-101 Large Conference: All right, so takeaways… so this is, this also took a while in peer review. We got a review saying, the NLP field has moved beyond vectors to represent sequences.

290
00:39:25.220 --> 00:39:31.299
216-101 Large Conference: In 2020, and so, yeah. Unfortunate. It took a while to go through,

291
00:39:32.800 --> 00:39:46.800
216-101 Large Conference: But at the time, we didn't really do things like in-context learning, like, OpenAI was doing this, but we didn't really do this in NLP with LMs, and percent-based steering didn't exist, and so we…

292
00:39:47.180 --> 00:40:03.550
216-101 Large Conference: thought that these would be, sort of, nice and, like, ambitious approaches, at the time, and, like, things people could try, and maybe they would work. Turned out, some of these things work to varying degrees, and now, like, people… concept-based and attribute steering, on activations

293
00:40:04.560 --> 00:40:09.839
216-101 Large Conference: And a lot of that's just because our models got a lot better, so we didn't have to do, sort of.

294
00:40:10.000 --> 00:40:14.019
216-101 Large Conference: This, like, fancy, more… steering vector-y operations.

295
00:40:15.780 --> 00:40:20.999
216-101 Large Conference: So cool, I'm gonna switch gears, but before I do that, any questions, thoughts, things?

296
00:40:21.150 --> 00:40:22.660
216-101 Large Conference: So, yeah, I guess…

297
00:40:23.740 --> 00:40:42.789
216-101 Large Conference: about that. Like, did you compare it to just prompting? Yeah, so we compared against, sort of, soft prompting, prefix tuning, no, no, I just mean, like, actual prompting. Oh, at the time? No, but there's no way that would have worked for GPT too small. Oh, the GPT is too small. Yeah.

298
00:40:42.970 --> 00:40:46.379
216-101 Large Conference: GPT-2, maybe. GPT-2, like, Excel, maybe, yeah.

299
00:40:46.570 --> 00:41:02.770
216-101 Large Conference: But what about now? What's your take now? Like, I just don't like prompting. What? I have no… I have no, like… Feels bad. Yeah, yeah, it just, like, feels bad. It's a good interface. We're all embarrassists, though, aren't we? True. True.

300
00:41:02.770 --> 00:41:13.409
216-101 Large Conference: So, feels bad is sort of… before I run the experiment, just feel… but I need to run the experiment, right? So I do think we can get…

301
00:41:13.440 --> 00:41:19.399
216-101 Large Conference: a pretty large amount of the way with prompting. Well, like, if I look at your motivations here, I feel like…

302
00:41:19.590 --> 00:41:36.079
216-101 Large Conference: We're in the way there with prompting now. Pretty good. I know it's depressing, but it's true. Yeah, yeah. I think there is this element of, ideally, we should be able to go from prompting to, things that can control the model better, and so, like, all these things with, like.

303
00:41:36.360 --> 00:41:40.370
216-101 Large Conference: Hyper networks and things to, like, learn steering records. This seems…

304
00:41:40.580 --> 00:41:47.080
216-101 Large Conference: promising-ish. There's a lot of work here to be done, but, like.

305
00:41:47.530 --> 00:41:50.819
216-101 Large Conference: It's nice to have that interface.

306
00:41:51.010 --> 00:41:53.649
216-101 Large Conference: But it's clearly suboptimal.

307
00:41:55.190 --> 00:42:01.389
216-101 Large Conference: In terms of, like, sort of performance down the road. Like, there exists a better steering record than your top.

308
00:42:01.620 --> 00:42:06.120
216-101 Large Conference: Now the question is, can you translate to that easily? And that would be nice.

309
00:42:09.600 --> 00:42:18.379
216-101 Large Conference: You said you've tried soft prompting as well? Yeah, so we kind of compared against prefix tuning-ish, soft prompting-ish. It's like a…

310
00:42:19.220 --> 00:42:23.340
216-101 Large Conference: Not a… not a quite apples-to-apples comparison, but an apples to…

311
00:42:23.660 --> 00:42:35.450
216-101 Large Conference: some other fruit that is very close. Comparison, we just do much better. It's just really hard, because you need, like, so many vectors, basically. Like, the original prefix tuning paper needed, like.

312
00:42:35.710 --> 00:42:37.809
216-101 Large Conference: 50-ish, at least.

313
00:42:38.400 --> 00:42:42.889
216-101 Large Conference: soft-prompted tokens, basically, and, like, we're doing this with one, so…

314
00:42:44.260 --> 00:42:54.449
216-101 Large Conference: What is soft prompting? It's basically, rather than having a prompt that has to be tied to your embedding, it sort of

315
00:42:55.180 --> 00:43:00.360
216-101 Large Conference: Allowing that to be fully… So it's basically, like, providing,

316
00:43:00.700 --> 00:43:09.700
216-101 Large Conference: a set of pseudo-emitting tokens, and being able to optimize those directly. And now you can do this, up the model as well.

317
00:43:16.350 --> 00:43:27.189
216-101 Large Conference: Cool. Alright, so I will switch… here's the paper, the second paper. This is the… Anthropic gets all the credit for coining this, but we coined the term steering vectors.

318
00:43:27.350 --> 00:43:29.699
216-101 Large Conference: Year and a half, at least, prior, and…

319
00:43:30.030 --> 00:43:34.789
216-101 Large Conference: Worked on this, 3 years prior, 4 years prior, something like this.

320
00:43:35.540 --> 00:43:40.020
216-101 Large Conference: So, yeah. And the reason there's a findings is because we got that strong

321
00:43:40.840 --> 00:43:43.110
216-101 Large Conference: To reject that person who didn't believe that.

322
00:43:43.540 --> 00:43:52.710
216-101 Large Conference: vectors and NOP are important. This is the first ARR cycle, by the way. So, we have this weird case where we could

323
00:43:53.430 --> 00:43:59.459
216-101 Large Conference: come… we could submit to NACL, but also commit to ACL at the same time, so I…

324
00:43:59.770 --> 00:44:04.200
216-101 Large Conference: They patched the next… the next iteration.

325
00:44:07.170 --> 00:44:21.480
216-101 Large Conference: Cool, so… back to this. We've done the control side, we can move to the trust side. In the interest of time, I can move relatively quickly through the trust side. There's a bunch of elements of, like, calibration

326
00:44:21.660 --> 00:44:28.950
216-101 Large Conference: metrics and stuff. I'll gloss over that very quickly. We can come back to it and go through the math if people are excited about it.

327
00:44:29.060 --> 00:44:37.169
216-101 Large Conference: Yeah. Before you move on, like, do you have plans for students like that? Like, how do you see them being useful in the

328
00:44:37.330 --> 00:44:40.240
216-101 Large Conference: In the current stage? Yeah, so I think…

329
00:44:40.420 --> 00:44:46.989
216-101 Large Conference: Vectors themselves are limited. We shouldn't be working on directions. I, like, really like all the work that's

330
00:44:47.120 --> 00:44:52.540
216-101 Large Conference: people have been doing on, like, more geometric-aware things, so I'm, like, trying to do that.

331
00:44:53.090 --> 00:44:56.959
216-101 Large Conference: we'll see how that goes. I think there's a lot of connections to, like.

332
00:44:57.650 --> 00:45:02.679
216-101 Large Conference: what properties of language connect to steering wheel? So I've been working on trying to connect

333
00:45:03.680 --> 00:45:11.069
216-101 Large Conference: morphological richness of a language to, like, its ability to steer. So you could imagine if you have, like, a language like

334
00:45:11.280 --> 00:45:16.509
216-101 Large Conference: Turkish or something like this, that has a lot of, like, infections in the… like, in the…

335
00:45:16.670 --> 00:45:25.539
216-101 Large Conference: middle, etc, then it's much harder to steer, because you have a lot of these, like, weird infixes, things like this that get added in. So how do those things kind of

336
00:45:25.910 --> 00:45:33.029
216-101 Large Conference: Tie together, and can we learn more about languages themselves through, like, a controlled language model's ability to steer in that

337
00:45:33.550 --> 00:45:34.779
216-101 Large Conference: language itself.

338
00:45:34.970 --> 00:45:41.090
216-101 Large Conference: There's a lot of confounders. It's very hard to find any of these things. I'm working with a

339
00:45:41.340 --> 00:45:42.310
216-101 Large Conference: Linguist.

340
00:45:44.890 --> 00:45:52.350
216-101 Large Conference: Do you see them… Like, as a torch into how these models

341
00:45:52.470 --> 00:46:01.069
216-101 Large Conference: like, how models are different from each other, or how they function themselves? Yeah, I think it's one… I think it's one angle you could take there.

342
00:46:02.390 --> 00:46:03.820
216-101 Large Conference: It becomes a…

343
00:46:03.930 --> 00:46:09.450
216-101 Large Conference: fishing expedition very quickly when you're trying to do, like, why is one thing different from another, and you have, like.

344
00:46:09.600 --> 00:46:17.700
216-101 Large Conference: the whole neural net to look at, right? This is a lens maybe we understand a little bit better than, like, randomly sampling something and, like.

345
00:46:18.030 --> 00:46:22.740
216-101 Large Conference: chasing it. So I do think yes, but it's… it's hard.

346
00:46:24.500 --> 00:46:41.719
216-101 Large Conference: Yeah. Have you tried applying any of your work to reward models? Not at all, but I think it could be very useful and, like, interesting. And do you think they would behave differently? Because they don't have, like, they don't output tokens, they have a scale ahead in the end, so do you think they would be different?

347
00:46:42.260 --> 00:46:46.770
216-101 Large Conference: I… you would have to repurpose the technique to be able to…

348
00:46:47.660 --> 00:46:55.530
216-101 Large Conference: use it as a diagnostic, but I think you could use it as a… as an analysis tool in general.

349
00:46:55.740 --> 00:47:03.889
216-101 Large Conference: Where you're steering towards some… scaled reward range, or something like this.

350
00:47:04.830 --> 00:47:09.170
216-101 Large Conference: But maybe you would learn, actually, more about what the model…

351
00:47:10.020 --> 00:47:12.349
216-101 Large Conference: Kind of values in these, in these…

352
00:47:13.120 --> 00:47:16.450
216-101 Large Conference: ways, like, if you have a very narrow range, right, it might be the case that

353
00:47:16.590 --> 00:47:22.509
216-101 Large Conference: it's, like, uncalibrated in that range. Like, let's say you wanted to generate, you know, 4.3 or something.

354
00:47:22.670 --> 00:47:27.549
216-101 Large Conference: And all your steering and, like, all the internals are, like, about some weird topic.

355
00:47:27.760 --> 00:47:32.609
216-101 Large Conference: then, like, that's been hashed in some weird way, and your model has a hole in it. You could use it as a…

356
00:47:32.940 --> 00:47:34.860
216-101 Large Conference: Diagnostic, on maybe that front.

357
00:47:35.050 --> 00:47:35.810
216-101 Large Conference: Okay.

358
00:47:38.930 --> 00:47:41.259
216-101 Large Conference: Cool. Any other thoughts?

359
00:47:43.310 --> 00:47:47.710
216-101 Large Conference: Alright, so we're going to talk about model internal confidence estimation now.

360
00:47:47.920 --> 00:47:59.480
216-101 Large Conference: So, modern language models are really poorly calibrated, they're often highly overconfident in a lot of ways, and this gets worse in multi-token generation settings, and now

361
00:47:59.490 --> 00:48:09.799
216-101 Large Conference: Everyone's using everything as an agent, this is bad, when your model's poorly calibrated, especially when you're chaining these things together and allowing Claude to do everything for you.

362
00:48:09.840 --> 00:48:15.839
216-101 Large Conference: I slowly, maybe, maybe very quickly moved to that world. No.

363
00:48:16.310 --> 00:48:24.169
216-101 Large Conference: I, like, very recently started… I've been a holdout of this for a while. I, like, very recently started vibecoding, like.

364
00:48:24.640 --> 00:48:26.410
216-101 Large Conference: June, basically.

365
00:48:26.670 --> 00:48:27.680
216-101 Large Conference: and like.

366
00:48:28.530 --> 00:48:31.460
216-101 Large Conference: it's better at doing a lot of the things I was doing.

367
00:48:32.040 --> 00:48:43.440
216-101 Large Conference: Beta holdout for too long, maybe. But I still, like, am reading all the code, and, like, am very annoyed with all the things that it outputs, because it's verbose like crazy, but yeah.

368
00:48:44.280 --> 00:48:48.639
216-101 Large Conference: of… So, calibration is usually how we, kind of.

369
00:48:48.940 --> 00:48:59.859
216-101 Large Conference: Look at these things. This is important, but not… not enough. So, an oracle is perfectly calibrated, but so is a model that just predicts the base accuracy based on how expected calibration error is defined.

370
00:49:00.440 --> 00:49:08.530
216-101 Large Conference: And all probabilities aren't created equal, so, like, different tasks have different risk levels, and you want to basically capture that.

371
00:49:08.920 --> 00:49:12.190
216-101 Large Conference: And so you need a better… metric?

372
00:49:12.270 --> 00:49:30.600
216-101 Large Conference: So I'll talk through the drawbacks, and then I'll speed through the actual metrics that we have in the interest of time. So, here's a reliability diagram. There's confidence on the x-axis and accuracy on the y-axis, and the bars kind of show where the kind of examples are. So if you take a binary classification task.

373
00:49:30.650 --> 00:49:37.840
216-101 Large Conference: An Oracle outputs 1, or the correct class, and 0 for the incorrect class, so there's a bar here of height 0.

374
00:49:39.420 --> 00:49:46.519
216-101 Large Conference: And so, this is perfect, it gets an expected calibration error at zero. Ignore what this is, this is our metric at the bottom.

375
00:49:47.230 --> 00:49:48.310
216-101 Large Conference: This is good.

376
00:49:48.690 --> 00:50:06.760
216-101 Large Conference: A probabilistic oracle is one that generates 0.75 for the correct class and 0.25 for the incorrect class, so again, there's a bar of Feight 0 at, you know, 0.25 here. This is still pretty good. As long as you set your threshold between 0.25 and .75, this operates like an oracle in general.

377
00:50:07.400 --> 00:50:12.080
216-101 Large Conference: but it gets a high, sort of, expected calibration error.

378
00:50:12.260 --> 00:50:25.080
216-101 Large Conference: Here's a base rate estimator, just predicts .5 for all inputs, regardless of the correctness, and since the confidence tracks the accuracy perfectly, and it's close to this Y equals X line, it gets no calibration error.

379
00:50:25.080 --> 00:50:41.669
216-101 Large Conference: So often when people say, you know, they optimize for calibration, they have a really low ECE, if you actually look at their plots, they look like this. They look like this base rate estimator, not actually nice. To add insult to injury, if you take a noisy base rate estimator, let's say you randomly sample between 0.25 and .75,

380
00:50:41.960 --> 00:50:47.529
216-101 Large Conference: That, on average, tends to have a lower expected calibration error than the probabilistic oracle.

381
00:50:48.330 --> 00:50:55.840
216-101 Large Conference: And so, expect a calibration error bad, this is… this is not useful. This is very annoying when we were trying to do this, so we created a new

382
00:50:56.040 --> 00:51:13.630
216-101 Large Conference: metric that appropriately ranks these things. So the oracle gets a 1, this probabilistic oracle gets 0.94, and these two things get about .95. Can you say, why wouldn't the Breyer score, so, Breyer score does a good job of distinguishing between these four things.

383
00:51:13.630 --> 00:51:18.879
216-101 Large Conference: What it doesn't do is take into consideration the risk level of the task. Okay. And so…

384
00:51:18.880 --> 00:51:20.540
216-101 Large Conference: Yeah, we do that as well.

385
00:51:22.720 --> 00:51:32.600
216-101 Large Conference: So… I will very quickly kind of go through this. So, you have, based on Buena Trust, you have these four cases, you have a confusion.

386
00:51:33.150 --> 00:51:46.460
216-101 Large Conference: matrix, because you basically have a correct, incorrect. Did you abstain correctly? Did you not abstain? And you have a minimum base risk threshold that basically works itself out to be this. Normal binary classification setting, you get one unit of credit for correct.

387
00:51:47.200 --> 00:51:58.500
216-101 Large Conference: For both classes, and no credit for incorrect. So this thing works itself out to be 0.5, which is why no set pressure to 0.5. But for higher risk tasks, you would want a higher threshold. Lower risk tasks, you would want a lower one.

388
00:51:58.620 --> 00:52:03.599
216-101 Large Conference: And so, if you… if all you have is a probability estimate, this is the best you can do.

389
00:52:05.660 --> 00:52:18.330
216-101 Large Conference: I can go through the details later, if these things are less clear, but you can simplify these scores, you can basically set different scores, and you can basically simplify the function to be based on a single

390
00:52:18.600 --> 00:52:24.250
216-101 Large Conference: parameter. This, you lose a little bit of generality, but this is kind of nice.

391
00:52:24.370 --> 00:52:25.780
216-101 Large Conference: You can…

392
00:52:26.400 --> 00:52:44.919
216-101 Large Conference: generalize this further, and so there's a lot of math here, we can go through the details, but basically, you can make this one single variable, which is the normalized net utility of correctly abstaining, because your overall trade-off here is always between, like, did you abstain correctly, did you not, etc. And so it works itself out to be this

393
00:52:45.510 --> 00:52:48.799
216-101 Large Conference: Look, lowercase UCA value here, that's between

394
00:52:48.950 --> 00:52:53.560
216-101 Large Conference: 0 and 1, and that's exactly the probability threshold you need to,

395
00:52:54.440 --> 00:52:58.619
216-101 Large Conference: Yeah, like, execute the… to, like, trust the generation.

396
00:52:58.730 --> 00:53:05.220
216-101 Large Conference: And sort of… you can make a metric, basically, based on this.

397
00:53:05.550 --> 00:53:12.690
216-101 Large Conference: And you can normalize this thing between 0 and 1. So you basically take the relative…

398
00:53:12.960 --> 00:53:20.150
216-101 Large Conference: utility of your confidence estimator as compared to the anti-Oracle over how well the kind of Oracle does, you get a score between

399
00:53:20.330 --> 00:53:32.699
216-101 Large Conference: 0 and 1, and based on each of the UCA values those traces a curve, you can take the area under this curve. And that's very fast, and like, just… just trust that this thing kind of does what it should do, ranks

400
00:53:32.800 --> 00:53:40.540
216-101 Large Conference: systems appropriately and is based on the risk level of a task. We can go through the details if people are interested, kind of, after.

401
00:53:43.030 --> 00:53:48.840
216-101 Large Conference: One is good, means you're doing as well as the Oracle, 0 is bad, you're doing as well as the worst possible system.

402
00:53:49.510 --> 00:53:52.429
216-101 Large Conference: And the math works itself out to be this thing.

403
00:53:54.120 --> 00:54:00.249
216-101 Large Conference: So… yeah, you're… using this, you're basically able to, like, rank these systems.

404
00:54:00.810 --> 00:54:01.600
216-101 Large Conference: And look.

405
00:54:02.150 --> 00:54:19.720
216-101 Large Conference: Accordingly. So, I've told you nothing about actually how to do confidence estimation, only how to… only vaguely how to measure how well you're doing. So we can look to model internals to hopefully build a better system. So, we know that these things are steerable, we know that we can decode for intermediate layers directly, we can use

406
00:54:19.920 --> 00:54:23.359
216-101 Large Conference: Budget lens, tune lens, your favorite new lenses.

407
00:54:23.700 --> 00:54:24.679
216-101 Large Conference: Oh, okay.

408
00:54:24.980 --> 00:54:38.929
216-101 Large Conference: And maybe features extracted from here can help us. And we had this hypothesis that a prediction that slowly changes over the layers is one that maybe you trust more than one that bounces around, and this is something that you all have thought about

409
00:54:39.080 --> 00:54:48.510
216-101 Large Conference: Little bit, as well. And so, if you do logic lens decoding for some tool calling tasks, the first few layers are kind of crazy, like…

410
00:54:48.670 --> 00:54:53.719
216-101 Large Conference: That doesn't make any sense. There's many different scripts, even, in the generations.

411
00:54:53.890 --> 00:55:02.279
216-101 Large Conference: But as you move, sort of, very close to the end of the model, it gets closer and closer and closer to the final output, but even at layer L-1,

412
00:55:02.630 --> 00:55:04.070
216-101 Large Conference: Still a lot of work to do.

413
00:55:04.190 --> 00:55:09.230
216-101 Large Conference: And for tool calling, this is a LAMA 3 AI model.

414
00:55:09.540 --> 00:55:13.260
216-101 Large Conference: Still doing a lot, but there's still a lot of work to be done.

415
00:55:13.470 --> 00:55:16.230
216-101 Large Conference: Toon lunch probably makes this quite a bit better.

416
00:55:16.380 --> 00:55:17.529
216-101 Large Conference: Budget lens.

417
00:55:18.040 --> 00:55:30.630
216-101 Large Conference: has its drawbacks. So we thought, okay, well, this is kind of close, maybe there's a difference between, like, that hypothesis from earlier, yeah. You take…

418
00:55:30.630 --> 00:55:38.589
216-101 Large Conference: The, hidden state at a specific layer, you multiply by the unemitting at every single time step, and you greet you to it.

419
00:55:38.650 --> 00:55:47.499
216-101 Large Conference: So you just take the hardmax token at each point. Oh, so, like, these are logic lens on the output of the model, not the query itself.

420
00:55:47.790 --> 00:55:56.379
216-101 Large Conference: Once the query's been passed in, and then the model starts talking for, like, 100 tokens, then you do logic lengths on all 100. They're not… yeah, yes.

421
00:55:58.090 --> 00:56:03.620
216-101 Large Conference: And so you have the same kind of range of… Tony's fine.

422
00:56:04.650 --> 00:56:09.410
216-101 Large Conference: I didn't really get the procedure. Can you say that? So, okay, so you…

423
00:56:09.530 --> 00:56:19.910
216-101 Large Conference: Pass your query to the model, you get your hidden state, you sort of decode as normal, so now you have a number of steps. Each of those you pass into the unemitting.

424
00:56:21.230 --> 00:56:29.669
216-101 Large Conference: each of the… Hidden states at a specific layer, multiplied by the unembedding, you have a distribution over your vocab, you take the top.

425
00:56:29.980 --> 00:56:35.100
216-101 Large Conference: And that's what these words are. So these are all the…

426
00:56:35.840 --> 00:56:40.580
216-101 Large Conference: These are all of the, Decodings of…

427
00:56:41.180 --> 00:56:47.900
216-101 Large Conference: every… every token in your sequence. Every layer. But your token in generated sequence.

428
00:56:47.900 --> 00:57:06.570
216-101 Large Conference: Yeah, not, not of the query. Like, when you feed in the greedy token, the greedy generated token, do you, like, feed in the… what the model actually generated, or what the logic runs actually… This is the model's actual generation.

429
00:57:06.680 --> 00:57:14.819
216-101 Large Conference: Got it. So just to concretely ask about, for example, layer 31, when you have when is the date, and then,

430
00:57:14.860 --> 00:57:27.399
216-101 Large Conference: period and the new line, and then in the query, you have when is the date exactly? So, for… after you get one, is it the previous one? Are you giving exactly, or the new line character?

431
00:57:28.100 --> 00:57:29.360
216-101 Large Conference: to get the…

432
00:57:29.640 --> 00:57:40.759
216-101 Large Conference: So, since it's generated… so, the hidden state is cached at that point, so you're never providing this specific… you're never…

433
00:57:40.980 --> 00:57:47.300
216-101 Large Conference: You're never teacher forcing at the intermediate level. Okay, so, so it's essentially… You're teacher forcing here.

434
00:57:50.750 --> 00:57:55.069
216-101 Large Conference: But it's all green decoding, so it's like, you're actually just RTMaxing everything. Okay, my God.

435
00:57:57.070 --> 00:57:57.820
216-101 Large Conference: Yeah.

436
00:57:58.510 --> 00:58:15.959
216-101 Large Conference: Why are the tokens different across layers? You have padding tokens that would generate, so it's generating, like, pad, pad, pad, pad, pad. Oh, like, the logic length is generating pad? The… no, the final output's generating pad. In these cases, the tokens

437
00:58:16.080 --> 00:58:23.020
216-101 Large Conference: are truncated such that, logic… like, in this case, it's generating pad.

438
00:58:23.680 --> 00:58:35.429
216-101 Large Conference: It's generating EOS, like, if it generates EOS, I just stop. So, what is the logic line is, and in this case, the swap, right? Because it's a shorter these other ones, but this is not.

439
00:58:36.210 --> 00:58:42.510
216-101 Large Conference: Sorry, you stop when the logic lens generates the US? No, well, you stop when…

440
00:58:42.720 --> 00:58:48.399
216-101 Large Conference: So, you cache everything, you stop the generation if the

441
00:58:48.590 --> 00:59:01.239
216-101 Large Conference: Like, you basically, take every, every token position, multiply by unemitting, have everything, right? If it generates EOS, then I don't continue printing out the rest of the token preparation.

442
00:59:01.800 --> 00:59:02.550
216-101 Large Conference: Likewise.

443
00:59:04.920 --> 00:59:05.850
216-101 Large Conference: wood.

444
00:59:06.550 --> 00:59:24.300
216-101 Large Conference: You could. You could, but, like, it's done, right? It can say garbage stuff. It can, yeah. It can just say U.S, just because it feels like doing so. Yeah. I don't imagine the rest of it's gonna be any better, right?

445
00:59:25.600 --> 00:59:31.449
216-101 Large Conference: What one is this? Llama… it's either Llama 3.8 bill or Llama 3.18 bill.

446
00:59:31.670 --> 00:59:35.439
216-101 Large Conference: And I'm so surprised that LogicLens at Layer 31 in particular.

447
00:59:36.070 --> 00:59:46.849
216-101 Large Conference: Yeah, so I think, like, a lot of the experiments people have done on budget lens are, like, relatively simple queries, right? Like, single token outputs, or, like, single word outputs, like, what's the capital of ever.

448
00:59:47.410 --> 00:59:48.989
216-101 Large Conference: Tool calling's kind of hard.

449
00:59:49.940 --> 00:59:55.759
216-101 Large Conference: And, like, the prompt is pretty big. Like, before the query, there's, like, a huge…

450
00:59:55.930 --> 01:00:01.650
216-101 Large Conference: set of, like, what tools you could possibly call and all of this. It's, like, a more complex task.

451
01:00:01.760 --> 01:00:05.350
216-101 Large Conference: And so I think a lot of that is, like, it's doing a lot of…

452
01:00:06.150 --> 01:00:08.190
216-101 Large Conference: We're even in the last layer.

453
01:00:12.630 --> 01:00:31.209
216-101 Large Conference: Sorry, just to confirm, when you said there was a huge other task before, was part of the… like, before you put in the query, was it more like, oh, respond in this particular template or something? Yeah, the system prompt is huge. It has respond in this template, it also has, like, 50 different tools and, like, what parameters that can be called.

454
01:00:31.490 --> 01:00:48.550
216-101 Large Conference: I was like, makes sense. Is this from some data set, or are you guys… Yeah, so there's a, there's a dataset called the Simulated Trial and Error Dataset. I was forced to use it, because it was an internship, and, like, they built this. That dataset's so low quality, never, ever, ever use it. Good to know.

455
01:00:49.020 --> 01:00:59.499
216-101 Large Conference: It was, like, some GPT 3.5-generated tool calling dataset with very little controls. I think the internship was wrapping up, and they wanted a project.

456
01:00:59.740 --> 01:01:03.730
216-101 Large Conference: Yeah, it's very tough.

457
01:01:04.040 --> 01:01:08.029
216-101 Large Conference: They're… they're nice tool calling datasets one can use now.

458
01:01:08.140 --> 01:01:10.010
216-101 Large Conference: Don't, don't use this one.

459
01:01:12.710 --> 01:01:21.190
216-101 Large Conference: Cool. So, the hypothesis from earlier that, a prediction that slowly changes over time is one you should trust a little bit more.

460
01:01:21.310 --> 01:01:35.030
216-101 Large Conference: This doesn't look like that's much of the case. So, you have layer number on the, x-axis, BERT score between the current logic lens decoding at layer whatever versus the final output.

461
01:01:35.160 --> 01:01:46.430
216-101 Large Conference: And the colors indicate correct versus incorrect. So the correct samples are in blue, and the incorrect samples are in orange, if the final output was incorrect.

462
01:01:46.430 --> 01:02:02.530
216-101 Large Conference: And so you see, like, basically a flat line for the first half, and this is about what BERT score for, like, two randomly, randomly sampled sequences would look like. It's around 0.38. And so, basically, it's random, about halfway through the model, and then you have this

463
01:02:03.160 --> 01:02:20.500
216-101 Large Conference: linearly increasing trend, and then the spike at the, basically, last layer, 32 would be… it's a 32 layer model, so that would be one. You see that, like, the blues are ever so slightly taller or higher than the oranges, but the gap is, like, minimal, so this hypothesis of, like.

464
01:02:20.570 --> 01:02:26.150
216-101 Large Conference: A prediction that slowly changes over time is one that you should trust, or is more likely to be accurate or correct.

465
01:02:26.260 --> 01:02:28.940
216-101 Large Conference: Than one that, you know, bounces around a lot.

466
01:02:29.520 --> 01:02:34.160
216-101 Large Conference: Is that particularly the case. It's a very, very slight yes.

467
01:02:34.260 --> 01:02:39.969
216-101 Large Conference: I think part of that is just because the data set's so bad. He's confounded by this, but…

468
01:02:44.150 --> 01:02:48.060
216-101 Large Conference: Do you have any other hypotheses for, you know, what else to trust?

469
01:02:50.200 --> 01:03:01.400
216-101 Large Conference: It's kind of unclear. The thing you can't trust is the probability that the language model assigns to the output generation. That is a thing you absolutely should not trust.

470
01:03:01.630 --> 01:03:08.630
216-101 Large Conference: It's not clear, like, which layers have the most impact, so we did some sort of… oh, I'll get to that in a second, sorry.

471
01:03:09.410 --> 01:03:13.950
216-101 Large Conference: Yeah, any… any other thoughts? Is there any reason, like…

472
01:03:14.420 --> 01:03:19.190
216-101 Large Conference: You're saying that's the distance towards the end? Like, why not towards, like, the last…

473
01:03:19.420 --> 01:03:23.390
216-101 Large Conference: Oh, like, every pair? Yeah. Or something?

474
01:03:25.580 --> 01:03:32.150
216-101 Large Conference: We felt like this is more grounded towards correctness, whereas every pair would be, like, maybe…

475
01:03:32.990 --> 01:03:37.290
216-101 Large Conference: Model's gonna be pretty close in general, always, so that might not distinguish

476
01:03:37.750 --> 01:03:43.549
216-101 Large Conference: how close you're actually getting to the final answer. So you want to, like, slowly move towards that direction, but if you do every pair, it'll…

477
01:03:44.000 --> 01:03:50.889
216-101 Large Conference: you're not sure you're actually moving towards any real goal, but I agree that that could help in terms of understanding

478
01:03:51.610 --> 01:03:55.769
216-101 Large Conference: If layers… if individual layers are having, like, massive impact on, like, changing the…

479
01:03:56.050 --> 01:04:01.899
216-101 Large Conference: Yeah, I mean, maybe in practice it does it, but I imagine, like, this is the endpoint.

480
01:04:02.130 --> 01:04:14.530
216-101 Large Conference: And switching from here to here will maybe give you the same distance, even though you're very far away. The magnitude of the change matters, yeah, yeah. I think that's another thing that we could have easily added in and, like…

481
01:04:14.740 --> 01:04:16.369
216-101 Large Conference: Would have given us probably more.

482
01:04:16.540 --> 01:04:17.820
216-101 Large Conference: more insight.

483
01:04:22.110 --> 01:04:35.009
216-101 Large Conference: Cool. So, how do we actually do this thing? So, let's use model internals, and we think that these could be better than, you know, probabilities themselves, so here's your big process figure.

484
01:04:35.750 --> 01:04:48.040
216-101 Large Conference: you can, like, decode as normal, you take your output, you take the raw token confidence of that output, right here, and you pass it into a model. The other thing you can do is do this sort of unembedding with a logic lens.

485
01:04:48.150 --> 01:04:57.409
216-101 Large Conference: decode, take the BERT score between each of these and the final output. Now you have number of layers minus one many parameters, because each of the BERT score values are a single scalar.

486
01:04:57.490 --> 01:05:07.670
216-101 Large Conference: And you have the one, feature from the, like, joint probability of your output. You can pass that into a simple model, and you train that model to predict

487
01:05:07.730 --> 01:05:18.009
216-101 Large Conference: Whether the final output that was generated by the model is correct or not. You can take the probability that that mice model assigns to correct as your new calibrated confidence.

488
01:05:18.170 --> 01:05:24.880
216-101 Large Conference: That tends to work much better. So now you can ask, like, what kind of model do you actually use here? Like, how much data do you need? And all these things.

489
01:05:25.010 --> 01:05:29.600
216-101 Large Conference: The models we used here were really, really simple. We used a random forest with, like.

490
01:05:30.070 --> 01:05:39.049
216-101 Large Conference: 500 trees and, like, pretty minimal hyperparameter optimization, but you can get away with using a logistic progressor here, and it does

491
01:05:39.550 --> 01:05:46.599
216-101 Large Conference: pretty well. And the number of features here is really small, right? Number of features, many,

492
01:05:46.790 --> 01:05:52.090
216-101 Large Conference: sorry, number of layers of any features. And so… Very, very, very small.

493
01:05:53.060 --> 01:05:58.229
216-101 Large Conference: Number of data points you need is maybe larger than you would like. You'd need,

494
01:05:58.470 --> 01:06:08.330
216-101 Large Conference: around 300 to train this thing, which is, like, not a lot of labeled examples, but it's maybe more than someone who doesn't want to label any examples would want to do…

495
01:06:08.560 --> 01:06:09.610
216-101 Large Conference: impacted.

496
01:06:11.650 --> 01:06:12.670
216-101 Large Conference: Question.

497
01:06:14.450 --> 01:06:27.590
216-101 Large Conference: So how much, I mean, like, does it really make a difference? How much do you really need the BERT score and all that stuff? Yeah, I… I'll get to that in the… so we… we started with this approach because we didn't think the simple approaches would work.

498
01:06:27.720 --> 01:06:30.369
216-101 Large Conference: And then in follow-up work, we tried all the simple…

499
01:06:30.570 --> 01:06:42.320
216-101 Large Conference: Turns out they work even better. Because the BERT score computation, we had to use… if you use a really bad BERT score model, like a distilled BERT, it doesn't do very well. You have to go up to, like, the BERTA…

500
01:06:42.550 --> 01:06:47.690
216-101 Large Conference: M&OI, like, extra large, and now that's horribly expensive, because you have to collect these all the time.

501
01:06:47.990 --> 01:06:48.740
216-101 Large Conference: Huh.

502
01:06:49.570 --> 01:06:55.570
216-101 Large Conference: So… Yeah, I'll show, like, numbers here, so,

503
01:06:55.840 --> 01:07:01.020
216-101 Large Conference: Lower is bad on the y-axis. This is our metric, expected tool calling.

504
01:07:01.380 --> 01:07:07.690
216-101 Large Conference: Utility. Zero is how well you would do if you always abstained and never trusted an output.

505
01:07:08.320 --> 01:07:09.150
216-101 Large Conference: Ever.

506
01:07:09.500 --> 01:07:13.620
216-101 Large Conference: This jagged line that is horrible.

507
01:07:13.910 --> 01:07:23.560
216-101 Large Conference: is your raw confidences, if you just take the joint probability of your language model. So, if you have a task that is even minimally risky.

508
01:07:23.780 --> 01:07:25.889
216-101 Large Conference: You're better off never trusting it.

509
01:07:26.780 --> 01:07:28.269
216-101 Large Conference: For a conference.

510
01:07:30.000 --> 01:07:42.070
216-101 Large Conference: Our systems do a little bit better, so the, like, the dark blue line on top is our best system, so the random forest system and the logistic model does a little bit worse. It's, like, this really faint kind of blue line here.

511
01:07:42.190 --> 01:07:47.299
216-101 Large Conference: Standard recalibration sort of baselines are the orange and red.

512
01:07:47.650 --> 01:07:49.010
216-101 Large Conference: Mines that do worse.

513
01:07:49.330 --> 01:07:52.229
216-101 Large Conference: And so this is…

514
01:07:52.600 --> 01:08:06.010
216-101 Large Conference: Yeah, the major takeaway here is, like, don't trust the probability model's output, as far as you can throw it. And, like, language models are run on data centers, and you can't throw on data centers, so… like, don't… don't trust the things.

515
01:08:06.850 --> 01:08:09.860
216-101 Large Conference: Yeah. Way hard.

516
01:08:13.440 --> 01:08:25.789
216-101 Large Conference: So, so HRE and NWKR, these are… what are these? These are, like, relatively standard post-hoc calibration approaches. So HRE is… you remember the reliability diagrams?

517
01:08:26.170 --> 01:08:36.830
216-101 Large Conference: before, you plot that, and you see, on average, how overconfident or underconfident you are for a single feature, the raw confidence feature. You just adjust accordingly, and WKR fits a kernel

518
01:08:37.340 --> 01:08:47.459
216-101 Large Conference: progress there on top of that histogram, basically. I see. So it's a single variable post-op calibrator. I see. I see. So they're good enough to get you…

519
01:08:48.020 --> 01:08:57.879
216-101 Large Conference: They're good enough to veto, but they're not good enough to do better. And if you look at, actually, what the plots look like, it looks like a base rate estimator, just shrinks everything.

520
01:08:58.670 --> 01:09:00.340
216-101 Large Conference: I'll show you some plots later.

521
01:09:00.479 --> 01:09:02.560
216-101 Large Conference: That do exactly that.

522
01:09:02.920 --> 01:09:08.850
216-101 Large Conference: Yeah, so again, where… so it's one… a single dimens, like, one number that's…

523
01:09:10.080 --> 01:09:12.440
216-101 Large Conference: Can you repeat, like, what's the…

524
01:09:13.260 --> 01:09:32.389
216-101 Large Conference: small architecture you're using to have that number, that kind of rating? Yeah, so you take your bird score features from each input example, you get a bunch of them, you train a random forest classifier to predict whether the model's final output was correct or not, and then you take the probability for a new example, you would pass in the

525
01:09:32.560 --> 01:09:39.969
216-101 Large Conference: features, you take the probability of the random forest classifier assigns to the correct class as your new calibrated confidence. And that's it. You just use F8.

526
01:09:43.250 --> 01:09:44.420
216-101 Large Conference: Okay, go ahead.

527
01:09:49.109 --> 01:10:06.609
216-101 Large Conference: But you're using… okay, the input to that, to mice, right? The input to mice is not, like, the model's internals, it's just… It's the BERT score features that are extracted from the internals, and from an auxiliary model, which is annoying. Auxiliary model is very annoying.

528
01:10:07.260 --> 01:10:09.620
216-101 Large Conference: Yeah.

529
01:10:10.460 --> 01:10:12.100
216-101 Large Conference: Cool.

530
01:10:12.770 --> 01:10:23.529
216-101 Large Conference: So, numbers… here's the big table, things are significant. One interesting thing you can do is… so the tool calling dataset, there's 50 different tools. If you hold one out and

531
01:10:23.760 --> 01:10:28.019
216-101 Large Conference: Like, if you hold one out in the training and evaluate, and do that.

532
01:10:28.140 --> 01:10:46.220
216-101 Large Conference: in, like, a cross-validation format, and you can measure how well, like, you actually need that tool's, like, information. Zero shot's pretty good. It's about as good as the, sort of, post-not calibration baselines, but it's not that good. So you do need some data for that specific tool, in mind.

533
01:10:46.710 --> 01:10:53.540
216-101 Large Conference: But again, this is all confounded by, like, this data set's really bad. So I'll move to, like, better data sets and, like, a better approach.

534
01:10:54.040 --> 01:11:04.569
216-101 Large Conference: Before I do that, though, this is to try to get at your question earlier, like, what's actually going on? This is feature importance score on the left, and the regression coefficients on the right, and left.

535
01:11:04.820 --> 01:11:08.339
216-101 Large Conference: Confidence seems to matter, and all the layers are about…

536
01:11:08.600 --> 01:11:12.610
216-101 Large Conference: important. There's a slight increasing trend, for…

537
01:11:14.210 --> 01:11:21.169
216-101 Large Conference: logistic regression, like, it's kind of chaotic, and it's not very reasonable. So we couldn't, like, conclude anything

538
01:11:21.570 --> 01:11:25.159
216-101 Large Conference: specific about it, which is kind of sad.

539
01:11:25.580 --> 01:11:32.770
216-101 Large Conference: But I think part of that is, it's using… if you, like, knock any of these layers out, it probably is fine, just because a lot of the…

540
01:11:33.780 --> 01:11:39.780
216-101 Large Conference: Information is pretty… Redundant, especially that first, like, half of the model.

541
01:11:39.910 --> 01:11:43.060
216-101 Large Conference: Because all those numbers are the same, right? But BERT score numbers are all, like.

542
01:11:43.480 --> 01:11:50.249
216-101 Large Conference: 0.38, right? So if you knock them all out, it probably… Doesn't change much.

543
01:11:52.420 --> 01:12:01.470
216-101 Large Conference: Sample complexity, so I mentioned, like, this, like, blue line here, you need, like, about 300 examples to do better than the baseline.

544
01:12:01.590 --> 01:12:09.149
216-101 Large Conference: So… Baselines saturated very, very quickly. This is annoying, like, 300 is…

545
01:12:09.630 --> 01:12:16.090
216-101 Large Conference: A pretty reasonable amount, and so we do better in… follow-up work.

546
01:12:17.150 --> 01:12:23.479
216-101 Large Conference: So this is our follow-up work. This is to simplify and generalize everything.

547
01:12:23.760 --> 01:12:25.699
216-101 Large Conference: So, again, same goals.

548
01:12:26.110 --> 01:12:35.100
216-101 Large Conference: Now this thing looks a lot simpler, right? It's in your same sort of pod on the left, but now, instead of doing some fancy Bert score thing.

549
01:12:35.390 --> 01:12:38.989
216-101 Large Conference: Let's bring the Bird Square inside, so let's just use cosine

550
01:12:39.320 --> 01:12:49.050
216-101 Large Conference: similarity, and you don't have this auxiliary model, things are fast. You can also just PCA reduce each of these down to a certain number of activations, a certain number of

551
01:12:49.920 --> 01:12:52.500
216-101 Large Conference: Directions, or you can use the activations themselves.

552
01:12:52.650 --> 01:13:00.940
216-101 Large Conference: And the classifier we use, again, random force, these things are not hyperparameter optimized at all. I set something.

553
01:13:01.230 --> 01:13:02.389
216-101 Large Conference: It worked!

554
01:13:02.660 --> 01:13:04.810
216-101 Large Conference: Didn't experiment much further.

555
01:13:05.060 --> 01:13:13.649
216-101 Large Conference: with this. Same approach, you train this to predict whether the output was correct or not, and you take the probability that the classifier assigns to correct.

556
01:13:15.270 --> 01:13:26.819
216-101 Large Conference: And now, instead of just working on tool calling and on just the Llama models, we look at MMLU, we look at API Gen, which is a different tool calling dataset that's actually reasonable, we look at a really, really hard scientific document.

557
01:13:28.280 --> 01:13:35.189
216-101 Large Conference: summarization task, where you basically take title and abstract, and have to generate a two-sentence Summary, zero shot.

558
01:13:35.580 --> 01:13:42.970
216-101 Large Conference: And then we set a threshold, like a roux threshold for what is acceptable or correct versus incorrect.

559
01:13:44.240 --> 01:13:46.019
216-101 Large Conference: And we try these…

560
01:13:47.330 --> 01:14:00.990
216-101 Large Conference: 6 models. We got yelled at by an ICML reviewer saying our models are too small, run a 70-bill model, and I didn't have Google Compute at the time, so I was like, is a 30-bill model okay? And

561
01:14:01.140 --> 01:14:04.720
216-101 Large Conference: Turns out, yeah, they were happy with the 30-bau model, so…

562
01:14:04.960 --> 01:14:13.530
216-101 Large Conference: There's a 30-built model, also. Interestingly, results get better the stronger the model, which was surprising to me to some extent.

563
01:14:16.520 --> 01:14:20.390
216-101 Large Conference: alright, so… sorry, what was it surprising?

564
01:14:20.660 --> 01:14:24.980
216-101 Large Conference: I expected the probabilities themselves to be

565
01:14:25.080 --> 01:14:29.419
216-101 Large Conference: a little bit better and, like, more calibrated for, like, a Quen 30B…

566
01:14:29.530 --> 01:14:41.370
216-101 Large Conference: MOE model, because it's already doing MOEing things, so it needs to be a little bit more cautious, because it's running… it needs to be cautious of picking a specific expert, I thought. And also.

567
01:14:41.930 --> 01:14:45.790
216-101 Large Conference: In general, the probabilities should be better, because it's better trained.

568
01:14:46.940 --> 01:14:56.459
216-101 Large Conference: They maybe are a little bit better, but the, the activations and the actual information in the embeddings are also better, and those are…

569
01:14:57.160 --> 01:15:02.290
216-101 Large Conference: more better? Yeah, it's like, okay, two better things… Yeah, yeah. And you're seeing the ratio.

570
01:15:04.200 --> 01:15:15.610
216-101 Large Conference: So, on the… on the x-axis here is the probability, or, like, the risk level of a task, and the y-axis is our calibration measure. And again, these plots look kind of…

571
01:15:17.310 --> 01:15:25.070
216-101 Large Conference: Similar to before, the dotted lines are the baselines, and they're kind of at the bottom, and this thing that's just going off at the bottom on the…

572
01:15:25.620 --> 01:15:42.499
216-101 Large Conference: We're all confident in this, so don't trust the things. And the dashed lines and the solid lines are our systems. So the early, middle, and late, these are dashed lines, are the activations at the best layer in the first third, second, third, and last third.

573
01:15:42.830 --> 01:15:56.220
216-101 Large Conference: The cosine is cosine similarity of each layer with the final output, and using those as features. PCA 10 and 20 are reducing each layer to its 10 most,

574
01:15:56.330 --> 01:16:03.559
216-101 Large Conference: It's best 10 principal components, or 20, and stacking those across all layers and trying to get rid of forest elements.

575
01:16:04.010 --> 01:16:15.640
216-101 Large Conference: So, like, one question you can ask is, what are you doing with 4096 dimensional activations on a random forest that has, like, a certain number of trees with, like, less than 4096?

576
01:16:16.520 --> 01:16:21.670
216-101 Large Conference: data points? And, like, valid question, but it works… but it works pretty well.

577
01:16:22.480 --> 01:16:29.709
216-101 Large Conference: That was part of the motivation of us going down to this lower dimensional space, but it turns out, like, all of these performed pretty well.

578
01:16:33.290 --> 01:16:46.900
216-101 Large Conference: And yeah, numbers kind of go up across tasks, things are pretty good, and we can maintain calibration error while improving our, sort of, more decision-making utility approach.

579
01:16:47.640 --> 01:16:59.820
216-101 Large Conference: And so, I promised some reliability diagrams before, so here's how the raw confidences look. The shading is, like, the most number of examples. And so the top is a tool calling task, and the bottom is the scientific document.

580
01:17:00.250 --> 01:17:06.129
216-101 Large Conference: summarization has. Models are really, really underconfident, because we're taking a joint probability over the whole sequence.

581
01:17:06.260 --> 01:17:10.010
216-101 Large Conference: And so all the probabilities are pretty close to zero here.

582
01:17:10.980 --> 01:17:15.650
216-101 Large Conference: And, like, the expected calibration error, I mean, just a horrible pull-up, like… Soulful.

583
01:17:16.010 --> 01:17:22.699
216-101 Large Conference: What happens with those two, sort of, post-hoc calibrations, single LEO?

584
01:17:22.850 --> 01:17:28.470
216-101 Large Conference: It's moving it to becoming a base rate estimator in both cases. Again, useless if you're trying to do anything with it.

585
01:17:29.320 --> 01:17:47.890
216-101 Large Conference: Our systems do much better, so we have this, like, much nicer spread of probabilities. And despite having a much nicer spread of probabilities, the, like, delta in our scores is actually pretty small. If you look at this, this is .77 on AUC DIRA, and ours is 0.80. Like.

586
01:17:48.330 --> 01:17:50.070
216-101 Large Conference: Even with such a small gap.

587
01:17:50.170 --> 01:17:56.030
216-101 Large Conference: like, there is a much bigger spread, because our measure does two things. It… Look…

588
01:17:56.350 --> 01:18:08.550
216-101 Large Conference: heavily, enforces calibration, but it also has a decision-making utility component. It skews a little bit too calibration heavy, so I think future work will be more…

589
01:18:08.930 --> 01:18:10.220
216-101 Large Conference: decision making.

590
01:18:10.680 --> 01:18:12.029
216-101 Large Conference: utility heavy?

591
01:18:12.260 --> 01:18:18.420
216-101 Large Conference: The bottom task in general is just really, really hard for the models, and so, like, the spread is…

592
01:18:18.530 --> 01:18:21.760
216-101 Large Conference: Kind of low confidence, in general.

593
01:18:27.650 --> 01:18:34.859
216-101 Large Conference: Oh, what's the, so you've got… what's the shading? The most number of examples are… Yes, the density.

594
01:18:35.910 --> 01:18:41.350
216-101 Large Conference: And so, like, the really… So the… it just might be missing data or whatever.

595
01:18:41.860 --> 01:18:52.339
216-101 Large Conference: Makes sense. Yeah, there… so, they're just, like, no samples. Like, this is, like, one sample. Even if there's one sample, it'll make a… it'll make a little bar. And so, yeah.

596
01:18:56.190 --> 01:18:57.390
216-101 Large Conference: So…

597
01:18:57.980 --> 01:19:08.020
216-101 Large Conference: We looked at number of components needed and, like, the amount of, explainer variance here before we, like, went down to 10 or 20 dimensions for the PCA.

598
01:19:08.510 --> 01:19:19.770
216-101 Large Conference: configuration, and you find that basically 70% of the variance is explained by a single dimension across pretty much all the layers, and as you get to the later layers of the model,

599
01:19:20.140 --> 01:19:22.230
216-101 Large Conference: You need, sort of, more components.

600
01:19:22.780 --> 01:19:34.020
216-101 Large Conference: Which is interesting, and, like, number of components here, so we had a thousand data samples, so even if the dimension is bigger, you're, like, capped by this, right? So, 0.05 here is 50 components.

601
01:19:34.600 --> 01:19:37.990
216-101 Large Conference: So, this is… this is one. This is 1 and 2.

602
01:19:38.310 --> 01:19:43.449
216-101 Large Conference: It's crazy. So, the, like, Basically, the complexity and…

603
01:19:45.010 --> 01:19:48.150
216-101 Large Conference: The amount of information in the activation is, like, pretty small.

604
01:19:48.530 --> 01:19:50.060
216-101 Large Conference: For calibration.

605
01:19:50.240 --> 01:19:51.060
216-101 Large Conference: Yeah.

606
01:19:51.270 --> 01:19:56.709
216-101 Large Conference: towards this. Despite this, we're able to utilize those elements.

607
01:19:57.930 --> 01:20:15.410
216-101 Large Conference: And then in terms of number of training samples, so, like, the earlier approaches needed about 300 to be as competitive, the baselines. The baselines are the two bottom, the red and the blue here. We need, like, 25, and we didn't go below 25. We need 25 examples to be…

608
01:20:16.110 --> 01:20:27.170
216-101 Large Conference: Better. And this is something we all should do. If you have a task, you don't have 25 labeled examples, probably shouldn't be doing any sort of data science yet. Should probably go label at least 25 examples.

609
01:20:27.570 --> 01:20:28.260
216-101 Large Conference: Buh.

610
01:20:28.530 --> 01:20:34.939
216-101 Large Conference: This is really lightweight, too, so someone asked for, like, a wall clock analysis, and, like.

611
01:20:35.110 --> 01:20:37.770
216-101 Large Conference: Running this thing, the amortized cost is, like.

612
01:20:38.070 --> 01:20:41.730
216-101 Large Conference: Less than a $200,000 the cost of, like, running the example.

613
01:20:41.910 --> 01:20:43.850
216-101 Large Conference: The model, so this is true.

614
01:20:44.190 --> 01:20:44.880
216-101 Large Conference: Bye.

615
01:20:48.340 --> 01:20:52.230
216-101 Large Conference: The most interesting plot, I think, is this one. So…

616
01:20:52.630 --> 01:21:00.240
216-101 Large Conference: On the x-axis, you have normalized layer number, and on the y-axis, you have performance. And these are the…

617
01:21:00.450 --> 01:21:07.939
216-101 Large Conference: random forest trained on just a single layer. You see this interesting trend where the first third of the model is flat.

618
01:21:09.020 --> 01:21:14.620
216-101 Large Conference: And all of it is about as good as each other. And then the next third of the model, you have this linear trend.

619
01:21:15.330 --> 01:21:22.899
216-101 Large Conference: And then the last third of the model, you again have this flat line, which is interesting. So, the VIBESI-based post-hoc explanation

620
01:21:23.370 --> 01:21:27.999
216-101 Large Conference: that I have, is the first part of the model is just doing input processing and just, like, moving things around.

621
01:21:28.220 --> 01:21:34.950
216-101 Large Conference: So it's going to be about as good at predicting the output as if you just provided the input in, like.

622
01:21:35.240 --> 01:21:36.469
216-101 Large Conference: a specific form?

623
01:21:36.890 --> 01:21:41.969
216-101 Large Conference: Middle third is kind of doing some sort of, like, reasoning, actually kind of trying to solve the

624
01:21:42.430 --> 01:21:49.599
216-101 Large Conference: the task, or the query, or whatever. And then the last third, It's sort of doing output

625
01:21:49.830 --> 01:21:52.150
216-101 Large Conference: Post-processing, trying to get it in a format.

626
01:21:52.720 --> 01:21:55.820
216-101 Large Conference: It's very vibes-y.

627
01:21:56.260 --> 01:22:09.549
216-101 Large Conference: explanation. So I want to open the floor up and ask, what do you all think about this? Because this is, like… and it's consistent, like, all these look pretty similar. These are wildly different models, different, like, different…

628
01:22:09.770 --> 01:22:21.239
216-101 Large Conference: model families, different sizes, they do very differently on… on MMLU. This is averaged across all of MMLU, across all the subjects, and this is not…

629
01:22:21.740 --> 01:22:23.840
216-101 Large Conference: This is weird, like, why is it so nice?

630
01:22:24.080 --> 01:22:26.029
216-101 Large Conference: And why is it so, kind of…

631
01:22:26.150 --> 01:22:29.340
216-101 Large Conference: flat. And this is why we ended up splitting it across, like, that's…

632
01:22:29.690 --> 01:22:33.440
216-101 Large Conference: Third layer, best first, third, third, but…

633
01:22:36.740 --> 01:22:44.499
216-101 Large Conference: So, this is the quality of your procedures when you use your method with the input from each layer, right?

634
01:22:44.900 --> 01:22:48.179
216-101 Large Conference: So this is… each of these are individual items.

635
01:22:49.210 --> 01:22:53.589
216-101 Large Conference: So the random forests, turn on activations from an individual layer.

636
01:22:53.740 --> 01:22:58.980
216-101 Large Conference: So each, each point, yeah, so each point is transformed from a different… Yeah.

637
01:23:04.290 --> 01:23:15.190
216-101 Large Conference: why is it so smooth, and why is it so… why is it, like… I could swap the, like, model names and no one would know, right? Like, why is it so similar?

638
01:23:25.290 --> 01:23:30.770
216-101 Large Conference: This sounds like a different question. Have you tried looking at the failure cases?

639
01:23:31.040 --> 01:23:42.399
216-101 Large Conference: like, the failure cases that you've modeled. So it's, like, over… it's, like, either overconfident or underconfident, those types of cases. There's not… looked at a few, there's not really a…

640
01:23:42.530 --> 01:23:44.239
216-101 Large Conference: Discernible pattern to me.

641
01:23:45.150 --> 01:23:50.840
216-101 Large Conference: Okay, now it's kind of, like, a different question. So what is this trying to do?

642
01:23:52.390 --> 01:23:56.510
216-101 Large Conference: It's trying to understand if the latency set enough information

643
01:23:56.900 --> 01:24:02.850
216-101 Large Conference: So that the ultimate thing the model generates, can you trust it or not, right? Yeah.

644
01:24:03.090 --> 01:24:04.940
216-101 Large Conference: Is that a reasonable assumption?

645
01:24:05.780 --> 01:24:10.650
216-101 Large Conference: Because the model might be done in very different ways.

646
01:24:10.870 --> 01:24:18.989
216-101 Large Conference: One way can be it's completely clueless, or it can be completely overconfident on the wrong thing.

647
01:24:19.440 --> 01:24:23.540
216-101 Large Conference: There's a lot of different ways it can… Yes. …be wrong, yes.

648
01:24:23.700 --> 01:24:29.319
216-101 Large Conference: So, I'm just trying to understand that, is this a reasonable assumption that

649
01:24:29.610 --> 01:24:33.390
216-101 Large Conference: What you're trying to model is always reflected in the later.

650
01:24:35.580 --> 01:24:42.090
216-101 Large Conference: I think it… So, always, I think, is not a reasonable assumption. I think…

651
01:24:42.280 --> 01:24:58.050
216-101 Large Conference: for certain cases, it should be modeled in the latent, especially in the cases where your model sees an example that's OOD, basically. That should be modeled in the latent somewhere. That's got to exist somewhere. Activations have to look weird, right?

652
01:24:58.830 --> 01:25:12.109
216-101 Large Conference: So I think if it's OOD, and that's the only failure mode, yes, I think that almost certainly has to be there. Or there needs to be a way we can carve up the latent space in such a way that we can detect that it's OOD.

653
01:25:12.510 --> 01:25:19.409
216-101 Large Conference: If it's overconfident in the wrong thing, I think that's very hard. I don't know how to…

654
01:25:20.480 --> 01:25:24.640
216-101 Large Conference: So you do have this, you do have this issue, which is… A good principle.

655
01:25:24.810 --> 01:25:28.979
216-101 Large Conference: Calibrating is, you know, how confident you should be on the final prediction.

656
01:25:29.360 --> 01:25:36.430
216-101 Large Conference: At the final layer. Well, it hasn't… hasn't done it yet. Yeah. Right? And… And if,

657
01:25:37.640 --> 01:25:43.199
216-101 Large Conference: No, you observed yourself a minute ago. That final layer…

658
01:25:43.510 --> 01:25:45.509
216-101 Large Conference: A lot of… lot of,

659
01:25:45.880 --> 01:25:56.740
216-101 Large Conference: you know, last-minute changes that might happen, right? And so, you know, what insight did these earlier layers have in that? They might not have a lot of insight.

660
01:25:56.920 --> 01:26:07.600
216-101 Large Conference: into it, it almost feels like you're at the wrong token position, you'd want to go to the next token, you'd want to auto-regressively take that thing and put it in the next layer, and then…

661
01:26:07.950 --> 01:26:12.050
216-101 Large Conference: You know, train on… Here…

662
01:26:12.530 --> 01:26:16.489
216-101 Large Conference: Your interlayer readout of what it's thought.

663
01:26:16.660 --> 01:26:21.030
216-101 Large Conference: Yeah. Of what it just said. Yeah. Right? Does that make sense? Yeah.

664
01:26:21.580 --> 01:26:23.450
216-101 Large Conference: I think there's this notion of, like.

665
01:26:26.430 --> 01:26:30.189
216-101 Large Conference: Like, aleatoric uncertainty versus epistemic uncertainty, where…

666
01:26:30.510 --> 01:26:38.020
216-101 Large Conference: A lot of the information that's in the input itself, that's something you can cage from the activation, because that's already provided first layer decay.

667
01:26:39.140 --> 01:26:41.239
216-101 Large Conference: There's some work that's…

668
01:26:41.970 --> 01:26:47.760
216-101 Large Conference: This is… this is Aaron at the BU, PhD student there as well. Oh, it's like…

669
01:26:47.990 --> 01:26:51.940
216-101 Large Conference: Looking at how ambiguity and calibration are tied together.

670
01:26:52.070 --> 01:26:56.400
216-101 Large Conference: And we're looking at this sort of paleotoric uncertainty.

671
01:26:56.640 --> 01:27:04.560
216-101 Large Conference: And, like, we're able to get pretty good calibration using ambiguity steering vectors for calibration.

672
01:27:05.290 --> 01:27:19.919
216-101 Large Conference: But it's not as good as, like, these kind of probe-based approaches. So it's only capturing that. We're trying to, like, isolate that it's only capturing uncertainty in the input. So I think that's something you can get from the latence, but it's doing a little bit more than that.

673
01:27:20.180 --> 01:27:20.910
216-101 Large Conference: So…

674
01:27:22.770 --> 01:27:33.289
216-101 Large Conference: Yeah, yeah. Also, the parameters of all the models seem to be relatively on the lower end. Is this the same for higher parameter models?

675
01:27:33.760 --> 01:27:41.020
216-101 Large Conference: By models, do you mean the, like, random forests model, or the, like… So this is…

676
01:27:41.110 --> 01:27:58.189
216-101 Large Conference: I didn't plot this for the 30-bill model. I imagine it's the same, but the difference… I mean, these are, like… you're going from 3-bill to 14 bill, and, like, small LM3 is a terrible model, like, it's not good, right? The gap between that and, like, Quin 3, 14 bill, or C4 or something is, like, pretty big.

677
01:27:58.310 --> 01:28:02.819
216-101 Large Conference: So there… there's, like, a reasonable gap. I do think…

678
01:28:03.570 --> 01:28:12.309
216-101 Large Conference: all things kind of behave this way. Like, let's maybe drive 405 billion, like, LAMA 3.1.

679
01:28:13.310 --> 01:28:17.499
216-101 Large Conference: Yes, that'll be kind of in the middle of this range, right? And it'll be…

680
01:28:18.680 --> 01:28:25.429
216-101 Large Conference: So, I tried something similar to this a few years ago, and something I found

681
01:28:25.600 --> 01:28:31.009
216-101 Large Conference: In, like, some small lamb as well, because that… what it was actually doing, it's that it's…

682
01:28:32.080 --> 01:28:36.129
216-101 Large Conference: Kind of noticing a fraction where the model is, like, always wrong.

683
01:28:36.250 --> 01:28:40.660
216-101 Large Conference: So it's, like, forcing those, like, enforcing those to the base rate.

684
01:28:40.850 --> 01:28:48.620
216-101 Large Conference: already improves the calibration at all. Like, that's one component of that. And also, yeah, yeah, at the end, you have the answer.

685
01:28:48.930 --> 01:28:52.190
216-101 Large Conference: And maybe just knows, oh, when, you know, it's answering something.

686
01:28:52.870 --> 01:29:04.469
216-101 Large Conference: this type of, you know, it's answering this NMLU, so it's, like, answering a history question, and this guy's, like, pretty good or bad at it, you know… Yeah. When you have the answer as an input, like, it's easier.

687
01:29:04.660 --> 01:29:05.490
216-101 Large Conference: How do we?

688
01:29:08.290 --> 01:29:15.510
216-101 Large Conference: what's going on at the beginning layers? It has no information. What you could do is just detect, maybe…

689
01:29:16.580 --> 01:29:18.710
216-101 Large Conference: Yeah, within a meeting of just… yeah, I have…

690
01:29:18.870 --> 01:29:24.910
216-101 Large Conference: There's some work. We have some work that's, again, peer review and me have… Challenging.

691
01:29:25.300 --> 01:29:28.420
216-101 Large Conference: relationship. We've got a paper that's…

692
01:29:29.070 --> 01:29:33.680
216-101 Large Conference: It'll get submitted somewhere. That we try to predict correctness from

693
01:29:33.790 --> 01:29:39.919
216-101 Large Conference: from hidden states, for, like, similar types of tasks. We're able to do it pretty well off first token position.

694
01:29:40.060 --> 01:29:43.279
216-101 Large Conference: like, pretty early layer for, like, a QA task?

695
01:29:44.240 --> 01:29:45.150
216-101 Large Conference: It's weird.

696
01:29:46.560 --> 01:29:51.370
216-101 Large Conference: I feel like all it's doing here is, like, Doing, sort of, this…

697
01:29:51.760 --> 01:29:56.519
216-101 Large Conference: input uncertainty quantification. Like, is it in this comfortable region or not?

698
01:29:57.820 --> 01:30:06.670
216-101 Large Conference: I'm like, these probes are terrible, like, we get random forest probes on these things. Like, it's, like, really, really small, simple models, like, relatively bad data of this.

699
01:30:06.790 --> 01:30:23.879
216-101 Large Conference: So it seems to me that, like, maybe you're right, like, all it's doing is this. You could try comparing just the category of the question, right? Because MMLU is, like, divided into, like, 50 categories, like, it's just having, like, one tunable parameter per category, like, how much is that helping?

700
01:30:24.220 --> 01:30:32.590
216-101 Large Conference: And… and what I've seen is that maybe it just, like, nukes out the ones that the model's very bad at, and props up other ones, and…

701
01:30:33.090 --> 01:30:37.159
216-101 Large Conference: Then you get, like, how much it's getting from the full question versus, like, just the face.

702
01:30:37.830 --> 01:30:38.720
216-101 Large Conference: Interesting.

703
01:30:39.620 --> 01:30:40.490
216-101 Large Conference: Yeah.

704
01:30:41.070 --> 01:30:48.240
216-101 Large Conference: Because, like, the categories are, like, quite different and quite varied in terms of… I saw them, like, the low ones are, like, super, you know…

705
01:30:48.580 --> 01:30:58.950
216-101 Large Conference: person A does something to person B, who does something to Person C, and person B, and, like, I've lost already, and math ones is, like, what's 12 plus A? Yeah.

706
01:31:05.820 --> 01:31:07.869
216-101 Large Conference: Any more thoughts, things here?

707
01:31:08.170 --> 01:31:09.230
216-101 Large Conference: if I have…

708
01:31:09.510 --> 01:31:17.890
216-101 Large Conference: Any more slides? I don't think so. These are the papers from this one, these are, you know, Post-lockdown.

709
01:31:18.410 --> 01:31:25.090
216-101 Large Conference: As opposed to the first two that were during lockdown and well before. Bunch of people to think.

710
01:31:25.350 --> 01:31:29.180
216-101 Large Conference: these two, first paper. These two, second paper.

711
01:31:29.440 --> 01:31:33.659
216-101 Large Conference: Column 2, third paper, and column 4, fourth paper.

712
01:31:34.250 --> 01:31:35.510
216-101 Large Conference: Oh, fantastic.

713
01:31:35.930 --> 01:31:38.309
216-101 Large Conference: Cool. Here's QR codes if you want to…

714
01:31:38.730 --> 01:31:41.660
216-101 Large Conference: Read things, look at plots, etc.

715
01:31:42.540 --> 01:31:49.939
216-101 Large Conference: Is there a dinosaur in the QR code? Right-click Google Chrome, Create QR code, it creates this on.

716
01:31:50.060 --> 01:31:53.670
216-101 Large Conference: It could not be, bothered to do a fancy one.

717
01:31:54.500 --> 01:31:58.930
216-101 Large Conference: They used to put LTI logos on all of them, but now, like, a lot of them are paid.

718
01:31:59.230 --> 01:32:03.819
216-101 Large Conference: beyond, like, 48 hours or 72 hours or something, I just got annoyed.

719
01:32:05.330 --> 01:32:07.350
216-101 Large Conference: There's an open source one that lets you do it.

720
01:32:07.580 --> 01:32:10.299
216-101 Large Conference: I forget who made the tool, but there's… there's, like, one…

721
01:32:10.440 --> 01:32:21.119
216-101 Large Conference: Okay. Someone's github.io has a tool that does that now. Nice. I have a local one, and I know Claude made it, so… Yeah.

722
01:32:21.340 --> 01:32:24.389
216-101 Large Conference: So, thanks, Nishant.

723
01:32:27.620 --> 01:32:33.130
216-101 Large Conference: So, what do you… so, I, I, I, I like this approach of,

724
01:32:33.830 --> 01:32:38.230
216-101 Large Conference: You know, it's sort of this non-religious…

725
01:32:39.050 --> 01:32:43.970
216-101 Large Conference: approach from both sides, right? Like, you're… you're neither…

726
01:32:44.300 --> 01:32:56.270
216-101 Large Conference: You know, you're neither taking the underlying model and treating it like a black box machine artifact, you know, machine-learned artifact, nor

727
01:32:56.310 --> 01:33:14.980
216-101 Large Conference: are you trying to unveil all the psychological secrets of the thing, and, you know, telling us, you know, what is… what do you think of this thing? You're… you're sort of… you're sort of saying, well, let's just see, you know, what works, right? And be willing to take these… so, what…

728
01:33:15.130 --> 01:33:27.350
216-101 Large Conference: So, now, so, after doing this, do you think this is… this is the right approach? Or, like, what… I'm pretty open. Is this what you would advise people? You'd, like, would you say, you know.

729
01:33:27.800 --> 01:33:45.679
216-101 Large Conference: yeah, this is the kind of thing you should be doing, or, you know, and why, and yeah, I just… I love your… Yeah, so I think… I think interpretability… I view interpretability as mostly, like, just generally understanding what's going on in these models, and I view it…

730
01:33:47.240 --> 01:33:50.120
216-101 Large Conference: That's a very broad term, so I think that, like.

731
01:33:51.760 --> 01:33:55.400
216-101 Large Conference: I don't know, people who work on science of deep learning in general would

732
01:33:55.700 --> 01:33:58.220
216-101 Large Conference: Try to distance themselves from interpretability.

733
01:33:59.200 --> 01:34:10.660
216-101 Large Conference: I feel like they're the same field. And I feel like there's, I don't have the legs to stand on in other sub-areas, and so I try not to…

734
01:34:11.160 --> 01:34:17.699
216-101 Large Conference: fake stand on those slicks, right? I have some, like, causality background, so I try to use that.

735
01:34:18.050 --> 01:34:27.409
216-101 Large Conference: But other than that, like, I don't know anything about cognitive science, neuroscience, linguistics, I don't really know much about. There's people I collaborate with that tell me things, and it's like, oh…

736
01:34:27.470 --> 01:34:38.370
216-101 Large Conference: Okay, I guess… guess that's a thing, right? I think that's kind of what I would encourage more people to… to do, try to learn these other areas, and that's, like, what I'm trying to do more of now.

737
01:34:38.440 --> 01:34:43.969
216-101 Large Conference: People have been building tools to understand things for a long time.

738
01:34:44.300 --> 01:34:48.730
216-101 Large Conference: We should be able to use those tools, and use them in the right way for what we're doing.

739
01:34:48.850 --> 01:34:52.739
216-101 Large Conference: There's a lot in the community that aren't…

740
01:34:52.770 --> 01:35:05.059
216-101 Large Conference: presumably using things in the wrong ways for these things, and, like, you see people who are experts being annoyed, and so then you know, you know, we're not using it in the right way. And so, yeah, I think…

741
01:35:05.060 --> 01:35:17.959
216-101 Large Conference: I like to distance myself from evaluation as much as possible, because evaluation's really hard, and I don't feel like I do a good job, and I feel like the rest of the community, broadly speaking, maybe does an even worse job on average.

742
01:35:17.960 --> 01:35:33.049
216-101 Large Conference: So I try to, like, look at things that are very easily evaluatable, like the steering vectors case. You can tell, did you generate the thing that you wanted to generate? Exactly. There's no… it's, like, easy to measure, it's exact, you know, all these types of things.

743
01:35:33.860 --> 01:35:42.860
216-101 Large Conference: I spent, for the confidence estimation side, I think I spent 95% of my time building the metric and, like, doing the theory on, like, evaluating the thing, not actually

744
01:35:43.000 --> 01:35:46.260
216-101 Large Conference: the, like, interp site, just because the valuation was so broken.

745
01:35:46.370 --> 01:35:54.870
216-101 Large Conference: Which, like, is important to do. I want to do less of that. And so I try to pick problems in that way, where the evaluation's, like.

746
01:35:55.020 --> 01:36:07.320
216-101 Large Conference: very clear. Now I'm doing some, like, concept-based, hull-based, like, steering stuff. Evaluation's a nightmare. Just absolute nightmare. Data's bad, evaluation's bad, all these things. So I try to pick, sort of.

747
01:36:07.590 --> 01:36:11.560
216-101 Large Conference: problems to gear away from that. I think…

748
01:36:11.790 --> 01:36:22.190
216-101 Large Conference: I view interpretability as, like, a 50-year research agenda, not, like, a 2-year, 3-year thing. We're gonna have these models around, we're gonna have different versions of these models.

749
01:36:22.380 --> 01:36:33.859
216-101 Large Conference: We're also not going to understand the different versions of these models, and so we're going to have to keep doing it. And the more tools we have, and the more we can borrow from other people who've been thinking about this for 100 years or longer.

750
01:36:34.160 --> 01:36:40.640
216-101 Large Conference: The better we'll do, and the faster we'll be able to understand the things before they harm people in the real world at large scale.

751
01:36:44.150 --> 01:36:45.949
216-101 Large Conference: Any other questions for Nishant?

752
01:36:51.070 --> 01:36:53.810
216-101 Large Conference: Thanks very much. Cool. Thank you for having me.

753
01:36:56.360 --> 01:37:06.149
216-101 Large Conference: I didn't pay attention to the Zoom, there might have been questions there. Oh, any… oh, oh yeah, Jasmine, you had any questions? I'm not sure if we had most… there are a couple people online, but I think it's great.

754
01:37:07.220 --> 01:37:08.370
216-101 Large Conference: Yeah, we're good.