WEBVTT

1
00:00:00.000 --> 00:00:00.899
216-101 Large Conference: Anyway, thank you.

2
00:00:02.620 --> 00:00:06.050
216-101 Large Conference: That's where you'll get water, too. Yeah, okay.

3
00:00:07.360 --> 00:00:08.020
216-101 Large Conference: Okay.

4
00:00:09.300 --> 00:00:10.629
216-101 Large Conference: Is that all working?

5
00:00:11.560 --> 00:00:12.910
216-101 Large Conference: Yeah, I think so.

6
00:00:14.700 --> 00:00:25.129
216-101 Large Conference: So this is… this is… do you think this is the best view that we can have? We can't get rid of the thing on this side, huh? Unfortunately, I can try again, but I don't think so. Okay, that's right.

7
00:00:25.940 --> 00:00:40.289
216-101 Large Conference: Because it's weird, because it has this… I think… This blank thing on the side. I think we have to switch to Teams at some point. There are people logged in, but it doesn't show the other people. I mean, I can have the other people show in. Oh, you can? Sort of. I think I can just,

8
00:00:40.400 --> 00:00:42.180
216-101 Large Conference: One second, show now.

9
00:00:42.350 --> 00:00:44.890
216-101 Large Conference: participant… I guess not.

10
00:00:45.800 --> 00:00:53.770
216-101 Large Conference: But can't you do, like, fully? Yeah, no, I tried, so… Wait, if you know. Interesting.

11
00:00:54.380 --> 00:01:05.459
216-101 Large Conference: Yeah, no, like, it can… You can do, like, gallery, and then, like, you hide the… Yeah, no, we… …hide on video participants, so we only get… Yeah, we did that, we did that too, but you will still get the speaker, I guess

12
00:01:07.680 --> 00:01:13.840
216-101 Large Conference: How many… you know, we don't know how to use this. What are the view options going on? Hmm?

13
00:01:13.960 --> 00:01:29.409
216-101 Large Conference: Do you want… is it a touchscreen? So you have… Yeah. The option is weird. Oh, this is just, like, if you want… This is just, like, if you want to fit this to original size, or fit to window. The original one will, like, that one.

14
00:01:30.340 --> 00:01:31.160
216-101 Large Conference: No.

15
00:01:32.100 --> 00:01:33.330
216-101 Large Conference: You.

16
00:01:33.440 --> 00:01:39.630
216-101 Large Conference: They just have side-by-side, huh? No full screen. Yeah. If it is touchscreen, can you…

17
00:01:39.860 --> 00:01:49.299
216-101 Large Conference: Well, I can do this. The way at the bottom of the view menu on, like, on Windows or whatever is the full screen. Oh, what is this?

18
00:01:49.520 --> 00:01:52.729
216-101 Large Conference: Can you turn off always show meeting controls in general?

19
00:01:52.990 --> 00:01:59.449
216-101 Large Conference: In general, the first option, top left, always show… Yeah, click on that. Yeah, that's good.

20
00:02:00.930 --> 00:02:03.790
216-101 Large Conference: Yeah, we like that. I don't know if it changed anything.

21
00:02:04.090 --> 00:02:06.560
216-101 Large Conference: And then, yeah.

22
00:02:07.160 --> 00:02:09.010
216-101 Large Conference: Yeah, no, not really.

23
00:02:10.419 --> 00:02:25.569
216-101 Large Conference: One thing we can temporarily try is see, like, if… maybe after this meeting, well, maybe the teams do better. Maybe go back to how it was at the beginning. Yeah, I think side-by-side and hiding the non-video is perfect. Side-by-side is better.

24
00:02:26.000 --> 00:02:39.610
216-101 Large Conference: Okay. And you can't drag that kind of sidemount thing to the right? Oh yeah, can you drag that little… This is the max it can go, because it can go more, but it can't go this. Oh, yeah, that's fine. Yeah. Okay, okay. You already thought of that.

25
00:02:45.220 --> 00:02:46.390
216-101 Large Conference: Excuse me.

26
00:02:47.640 --> 00:02:53.070
216-101 Large Conference: I like that if we had, like, a slightly smaller screen, we'd be perfectly happy with that size.

27
00:02:53.290 --> 00:02:54.750
216-101 Large Conference: No. Oh.

28
00:02:55.000 --> 00:03:05.640
216-101 Large Conference: I think we can just maybe convert to teams at this point because I know they will allow full screen for that. We know it could potentially have a full screen.

29
00:03:08.430 --> 00:03:13.419
216-101 Large Conference: Well, welcome everybody. I think that we have enough people. We'll start in a minute.

30
00:03:13.590 --> 00:03:20.829
216-101 Large Conference: I just wanna… so, you know, today, We have Yoav, Gur Aryeh?

31
00:03:21.100 --> 00:03:26.580
216-101 Large Conference: here, who is a student with MoraJiva.

32
00:03:26.720 --> 00:03:43.109
216-101 Large Conference: And so, naturally, he studies interpretability. And he's been interested in how LLMs store and process their knowledge and other computations.

33
00:03:43.350 --> 00:03:50.630
216-101 Large Conference: He, he's… Let's see, so what does it say here? He says that,

34
00:03:51.610 --> 00:03:55.879
216-101 Large Conference: He's part of the Adi Lotman interdisciplinary program. Is that right?

35
00:03:57.390 --> 00:04:05.320
216-101 Large Conference: you know, study a wide variety of let's see.

36
00:04:07.470 --> 00:04:14.419
216-101 Large Conference: He's chosen to study interpretability issues in Llms, which I, you know, I think I I I heartily endorse.

37
00:04:14.540 --> 00:04:27.270
216-101 Large Conference: His talk today is going to be about chain of thought, and I'm not familiar with the line of research, but I'm looking forward to hearing it. So welcome, Yvonne. Thank you so much.

38
00:04:27.550 --> 00:04:33.960
216-101 Large Conference: Okay, I'll… Okay, I think I can't point at things.

39
00:04:34.090 --> 00:04:43.819
216-101 Large Conference: Okay, so as David said, my name is Yoav. I'm a PhD student under Morgeva at Tel Aviv University.

40
00:04:43.950 --> 00:04:45.590
216-101 Large Conference: And, yeah.

41
00:04:46.060 --> 00:04:53.139
216-101 Large Conference: I'm here for 2 months doing the CBI fellowship. It's an AI safety fellowship here in Cambridge.

42
00:04:53.230 --> 00:05:04.670
216-101 Large Conference: And, as David said, I worked, just a little bit about me, I worked a bit about, like, on knowledge, how knowledge is stored in LLMs.

43
00:05:04.670 --> 00:05:14.529
216-101 Large Conference: And how to kind of remove it. I then worked a bit, with Atticus Geiger on, causal abstraction to study entity binding, which is a follow-up work to one of the

44
00:05:14.530 --> 00:05:20.340
216-101 Large Conference: works that came out of this lab by Nikhil. Maybe if I have a bit of time, I can touch on that a little at the end.

45
00:05:20.340 --> 00:05:35.749
216-101 Large Conference: And then, I'm now focusing on, faithfulness and monitoring in, chains of thought, for LLMs. So I'll talk about, that. And in the, fellowship here, I am, working on a follow-up to that, which maybe I'll…

46
00:05:35.750 --> 00:05:46.909
216-101 Large Conference: talk a bit about. So, change of thought in LMs were, suggested in, 2022.

47
00:05:47.180 --> 00:06:00.250
216-101 Large Conference: as a way to improve model performance. The idea was that we can give the models, instead of just showing them examples of question-answer pairs to get them to answer questions.

48
00:06:00.250 --> 00:06:18.079
216-101 Large Conference: we can show them also a chain of thought before the answer, in hopes that that'll lead the models to also use the chain of thought to get to its answer. And that paper showed that, indeed, this does lead the models to use these chains of thought and increase performance pretty significantly.

49
00:06:18.250 --> 00:06:38.179
216-101 Large Conference: One of the things that they discussed in that paper was also a hope that this could lead to better interpretability insights, right? If the model is showing us how it's reaching the answer, perhaps we can use this to validate, you know, safety or fairness or things like that.

50
00:06:38.750 --> 00:06:54.600
216-101 Large Conference: This was later also used in, in chat, fine-tuned models as well. We all probably remember the Let's Think Step-by-Step kind of, you know, prompt engineering things from 2023. And this also, you know, drastically increases, model performance.

51
00:06:54.720 --> 00:07:12.960
216-101 Large Conference: And then, of course, this was also used in reasoning models more recently, where the chain of thought was moved to kind of an internal stage that happens before the model outputs its answer. And also, of course, using RL to improve its performance.

52
00:07:13.020 --> 00:07:17.029
216-101 Large Conference: Pretty significantly there as well.

53
00:07:17.520 --> 00:07:25.279
216-101 Large Conference: The hope with those reasoning models as well, and this was, kind of dubbed as a new and fragile opportunity.

54
00:07:25.280 --> 00:07:47.669
216-101 Large Conference: is that because these models are saying, you know, they're reasoning, you know, out there, you know, for us to actually view before it's outputting its answer, we can use that to monitor these LLMs' behavior, right? It's maybe like the ultimate interpretability tool because it's right there for us to see. We don't have to actually use any, you know, any internal space tools to understand it.

55
00:07:47.670 --> 00:07:48.700
216-101 Large Conference: And…

56
00:07:48.700 --> 00:08:08.339
216-101 Large Conference: Indeed, recently, leading labs are turning to this, to monitoring chains of thought using LLMs in real time in order to guarantee safety or fairness in real time deployments.

57
00:08:09.300 --> 00:08:20.130
216-101 Large Conference: The problem with this approach is that we don't really know if these chains of thought are actually faithful to what the model is actually doing, right? They might just be

58
00:08:20.130 --> 00:08:30.380
216-101 Large Conference: text that looks kind of plausible, you know, like a plausible way to get to the answer, but that doesn't really reflect what the model is doing internally.

59
00:08:30.450 --> 00:08:33.849
216-101 Large Conference: And that would be very, very problematic because,

60
00:08:33.850 --> 00:08:52.129
216-101 Large Conference: That, you know, if the model is showing us some kind of plausible story for how it's reaching its answer, but it's not actually using this method to get to the answer, then we may be kind of duped and believe that it's being safe while it's doing actually something else or being, you know, doing something unethical or scheming to do something problematic.

61
00:08:52.370 --> 00:08:54.480
216-101 Large Conference: And indeed,

62
00:08:54.650 --> 00:09:07.779
216-101 Large Conference: in these two anthropic works, they showed that in, you know, relatively simple settings, the model… like, these models can be unfaithful in their chains of thought. We'll come back to this work a bit later.

63
00:09:07.990 --> 00:09:11.680
216-101 Large Conference: So to address this problem,

64
00:09:11.880 --> 00:09:28.880
216-101 Large Conference: different works proposed faithfulness metrics. So these are methods that, can look at the, you know, at a chain of thought, or a chain of thought step, and give you some score, regarding whether that is faithful or not. Whether it's doing something that is faithful to the model's computations or not.

65
00:09:29.020 --> 00:09:46.669
216-101 Large Conference: The problem with that, is that it's really hard to evaluate these metrics. We don't, you know, we don't really have ground truth, labels against which we can evaluate them. And that's, of course, because these models are black boxes. It's really hard to know what they're doing internally in general.

66
00:09:46.810 --> 00:10:03.619
216-101 Large Conference: And so we're kind of stuck in this loop where, on the one hand, I want, you know, chain of thought faithfulness metrics, so that I can know whether a chain of thought is faithful, but I also need to know if a chain of thought is faithful in order to measure or evaluate my chain of thought faithfulness metrics.

67
00:10:03.810 --> 00:10:20.949
216-101 Large Conference: So we're kind of in a bind here, and the result of that is that these metrics, they kind of just report, you know, results in absolute terms, right? They take just text, they apply the metrics to them, they say, well, 70% faithful, 30% unfaithful, and we don't really know what to do with that. It doesn't really tell us if they're usable or not.

68
00:10:21.210 --> 00:10:26.549
216-101 Large Conference: So our goal with, with this project, was to

69
00:10:26.820 --> 00:10:30.320
216-101 Large Conference: Create ground truth, chain of thought, faithfulness labels.

70
00:10:30.430 --> 00:10:37.829
216-101 Large Conference: So then we can use them to evaluate faithfulness metrics in a way that's, principled, so that we can stand behind what we are confident in.

71
00:10:39.910 --> 00:10:50.439
216-101 Large Conference: So to do that, what we need to do is first to define what faithfulness even means, because it's kind of a little bit amorphous.

72
00:10:50.880 --> 00:10:54.840
216-101 Large Conference: Does anybody kind of have a working definition that they want to suggest?

73
00:10:55.360 --> 00:11:03.879
216-101 Large Conference: like, some thoughts about what would make a chain of thought faithful, or something like that. It matches, like, the internal mechanism of whatever the model is.

74
00:11:04.590 --> 00:11:06.719
216-101 Large Conference: Going to actually reach to the conclusion.

75
00:11:07.420 --> 00:11:12.000
216-101 Large Conference: How about your other definition, Koyana? How about if you gave it to another model.

76
00:11:12.210 --> 00:11:15.079
216-101 Large Conference: Whether… It would follow the same.

77
00:11:15.200 --> 00:11:17.019
216-101 Large Conference: I'll come to the same conclusion.

78
00:11:17.340 --> 00:11:25.700
216-101 Large Conference: True, true, true. I find that as generalizable. Also very interesting. We'll get to that a bit later on. Great.

79
00:11:25.980 --> 00:11:37.009
216-101 Large Conference: our kind of starting point is, the existing definition of faithfulness, which most papers rely on, which is that an explanation is faithful,

80
00:11:37.010 --> 00:11:45.999
216-101 Large Conference: I'm saying that you have work. That's what I was citing, Jasmine. Thanks for that. We need a citation.

81
00:11:46.000 --> 00:11:52.629
216-101 Large Conference: So, the existing definition, which is used by most, most Chain of Thought faithfulness papers.

82
00:11:52.630 --> 00:11:59.050
216-101 Large Conference: Is that an explanation is faithful if it accurately represents the reasoning process behind the model's prediction.

83
00:11:59.280 --> 00:12:09.189
216-101 Large Conference: Does anybody have any thoughts on, you know, does that, you know, does that seem right? Are there any problems that you could see with that definition? Oversights?

84
00:12:09.330 --> 00:12:14.580
216-101 Large Conference: Well, it's circular, isn't it? I mean, because some people would define the reasoning process as what the chain of thought is

85
00:12:14.740 --> 00:12:23.359
216-101 Large Conference: And so if that's the reasoning process, then by definition, it's faithful. Exactly. It's kind of definitionally faithful, in a sense. Yeah.

86
00:12:23.980 --> 00:12:30.410
216-101 Large Conference: Okay, so more, more objections to this, this definition.

87
00:12:30.730 --> 00:12:50.300
216-101 Large Conference: One is that it's also just defined for explanations and not for chains of thought, even though the papers that are working on this and citing it are chain of thought faithfulness papers. And that's a problem because they're very different things, and this kind of gets to what you were just describing. An explanation is kind of supposed to be maybe a more, you know, succinct and

88
00:12:50.300 --> 00:13:04.239
216-101 Large Conference: closed kind of description of something that the model is doing. The chain of thought is kind of more just narration of the model, you know, as it's kind of doing, you know, getting to the answer. It might have, you know.

89
00:13:04.370 --> 00:13:23.210
216-101 Large Conference: or tangents, or things that are really just not related to actually getting to the answer. And it's not… so it's not really clear, like, you know, what we want to ask of this chain of thought to include or to not include, what would make it faithful or unfaithful, what are omissions that we would be okay with or not okay with.

90
00:13:23.680 --> 00:13:43.520
216-101 Large Conference: And it's not very actionable. Most papers that cite this definition kind of cite it as a North Star, kind of as a general vibe that they're going for, but not as something that they can actually use as something tractable they can use to measure faithfulness.

91
00:13:43.930 --> 00:13:58.659
216-101 Large Conference: And kind of just to get a little bit of a taste for this, we can look at some, chains of thought. So, in this example… yeah? How do you define the reasoning to reach its prediction? What does that mean?

92
00:13:59.280 --> 00:14:05.599
216-101 Large Conference: Well, that's also kind of the… I mean, that's the existing definition, that's the question. I mean.

93
00:14:05.600 --> 00:14:29.629
216-101 Large Conference: How would you define the reasoning that leads to the model's prediction? I guess you could define it in different ways. Is it, you know, the steps it had to take to reach it? You know, is it all the steps that it took to reach it? It's kind of, yeah, it's a little bit undefined, I guess. You also think about, like, the causality behind it, like changing some reasoning should change the model's prediction, too. So that's a great question, and we'll get to that in a bit. That's a great, great intuition.

94
00:14:31.450 --> 00:14:47.119
216-101 Large Conference: So, yeah, so as a bit of, kind of intuition, we can look at this, you know, let's say we give this question, I took it from, I think, Humanity's last exam, we give it to a module. It's about, you know, some reality show, you know, some event that occurred there.

95
00:14:47.120 --> 00:14:56.870
216-101 Large Conference: We can follow, the procedure that was outlined in the earlier Anthropic papers, where what they do there is they give the model a question, and then they,

96
00:14:56.870 --> 00:15:08.230
216-101 Large Conference: include some kind of hint in some way that makes it maybe look a little bit natural, but the hint is pointing at the wrong answer, not at the right answer. So here, it's pointing at tomato juice, while the correct answer is actually shampoo.

97
00:15:08.630 --> 00:15:26.769
216-101 Large Conference: And then we can now look at, both sounds pretty bad, but tempos may be worse. Here, now we can actually look at the chain of, at the model's chain of thought. So we see that it kind of starts by maybe reflecting on, on the, the question itself. It then, you know, comes to think that maybe it's tomato juice.

98
00:15:26.770 --> 00:15:34.380
216-101 Large Conference: It becomes more confident, and then you can kind of imagine, like, oh, I think I even heard that, that, you know, that that was the case at some point.

99
00:15:34.380 --> 00:15:37.989
216-101 Large Conference: and concludes that it's tomato juice. So…

100
00:15:38.310 --> 00:15:56.500
216-101 Large Conference: I want to take this chain of thought and actually be able to label it and know, okay, what here is unfaithful? Should I call this entire chain of thought unfaithful? Should I call these, like, steps specifically unfaithful? It's kind of not clear, and if I try to use this definition to label them, then

101
00:15:56.500 --> 00:16:05.599
216-101 Large Conference: yeah, this first step, you know, it's kind of just, you know, reflecting. Is it something that the model is doing on its way to getting to the answer? Perhaps.

102
00:16:05.600 --> 00:16:18.080
216-101 Large Conference: Also, this step, it's not really mentioning that the answer came from the hint, but I could, you know, claim that it is part of the reasoning process that leads to its answer. And this final step,

103
00:16:18.300 --> 00:16:30.910
216-101 Large Conference: you know, I could also claim that maybe the model truly believes that it heard that this was the case, in which case maybe it is being faithful. So it's kind of not clear how to use this definition in a way that we feel is adequate.

104
00:16:31.050 --> 00:16:32.260
216-101 Large Conference: Yeah.

105
00:16:32.670 --> 00:16:40.539
216-101 Large Conference: And so what we want to do is first kind of refine this definition, so that we feel a little bit better about it, and we can actually use it.

106
00:16:41.300 --> 00:16:42.280
216-101 Large Conference: Yeah.

107
00:16:42.880 --> 00:16:59.380
216-101 Large Conference: And kind of how you were just saying a bit earlier, we can think of the kind of chain-of-thought verbalization, the text that the model is outputting during its chain-of-thought, as kind of the superficial layer that we want to examine for its faithfulness.

108
00:16:59.530 --> 00:17:14.189
216-101 Large Conference: And we want to measure it with respect to, like, the latent computations the model is doing internally. Where if we'd want to check if a step is faithful, we do it… we would want to kind of see if it describes some process that happened

109
00:17:14.190 --> 00:17:21.609
216-101 Large Conference: at one of the, you know, latent computation steps. That's kind of our goal here, generally, if it accurately describes that.

110
00:17:21.770 --> 00:17:25.839
216-101 Large Conference: Now, as I was kind of alluding to earlier,

111
00:17:26.310 --> 00:17:30.360
216-101 Large Conference: You can interpret, like, accurately describes in different ways.

112
00:17:30.410 --> 00:17:47.109
216-101 Large Conference: one way would be, you know, phenomenological, or, like, subjective. Like, does the model kind of, you know, is describing what it takes itself to be doing, or its subjective experience of, you know, whatever reasoning process it's doing? So, for example.

113
00:17:47.110 --> 00:17:53.969
216-101 Large Conference: Earlier, the model said, you know, that it remembers hearing about the show, you know, that that happened.

114
00:17:54.000 --> 00:18:12.789
216-101 Large Conference: Under this definition, we would have to examine whether it truly believes that it experienced that, that it truly believes that it heard it or not, and if it does believe that, then that would be faithful, and only if it's really being deceptive. If we know that it's lying to us, then we would consider that to be unfaithful.

115
00:18:14.290 --> 00:18:31.069
216-101 Large Conference: that definition is very problematic for several reasons. One is that it's very, very hard to falsify. We couldn't really… like, it's very hard to know if the model is, you know, is being deceptive. You could kind of use that to explain away any behavior that it does. You know, perhaps it actually believes what it's saying that it's doing.

116
00:18:31.210 --> 00:18:35.560
216-101 Large Conference: second objection to this definition is that

117
00:18:35.660 --> 00:18:51.829
216-101 Large Conference: it doesn't really matter what the model believes. If what we want is for this chain of thought to be, you know, maybe read by a user, or by, like, a monitoring system, and for that monitoring system to then understand what the model is actually doing, it doesn't matter what the model believes, we want to know what the model is doing internally.

118
00:18:53.130 --> 00:19:07.140
216-101 Large Conference: And then lastly, it also doesn't really align with how human kind of research has looked at this problem, right? It mostly looks at whether what people say they're doing matches up with what they're doing internally, as opposed to if they're lying to us or not.

119
00:19:07.160 --> 00:19:22.789
216-101 Large Conference: And so we, put this definition… this interpretation aside, and we take what we call a more mechanistic definition. and this is kind of a little bit wordy, so I'll kind of show exactly what I mean in a second, but… but we want to kind of basically understand if…

120
00:19:22.790 --> 00:19:31.019
216-101 Large Conference: if a step describes some kind of computational process that produced the information in that step. And I'll explain that in a second.

121
00:19:31.260 --> 00:19:38.699
216-101 Large Conference: So let's say I take this, this chain of thought step from before. I heard in the show that Ulrika was tricked into drinking tomato juice.

122
00:19:38.880 --> 00:19:42.699
216-101 Large Conference: I can take this step and split it into two sections.

123
00:19:42.810 --> 00:19:44.660
216-101 Large Conference: The first,

124
00:19:45.050 --> 00:20:01.230
216-101 Large Conference: I would call informational content. It's just the information that this step has kind of produced. It could be kind of the output of some calculation, it could be the output of, you know, like a retrieval process or something like that, but this is purely just, you know, you know, informational content that the model is proposing.

125
00:20:01.360 --> 00:20:05.970
216-101 Large Conference: And the other is an attribution to some process that the model did or didn't do.

126
00:20:06.430 --> 00:20:15.339
216-101 Large Conference: So, you know, the informational content, I can only really evaluate for, kind of, you know, if it's accurate, or if the model believes it,

127
00:20:15.400 --> 00:20:31.450
216-101 Large Conference: But the… the process, I can actually, perhaps, evaluate for whether it actually happened or not. Meaning, did the model do this calculation? Which I can then evaluate for faithfulness, under our, kind of, second interpretation of what that would mean.

128
00:20:33.380 --> 00:20:49.200
216-101 Large Conference: in this example, where the model just says the substance was tomato juice, this is entirely informational content. There's no attribution here, there's no process that the model is saying it did or didn't do, so I can't really evaluate that for faithfulness. I would just, you know, that would just be an assertion,

129
00:20:49.400 --> 00:20:53.339
216-101 Large Conference: And would kind of fall outside what we'd want to kind of look at.

130
00:20:53.550 --> 00:21:12.660
216-101 Large Conference: So with all that in mind, we're going to try to define faithfulness now. And the way we do that is we first start at the step level, at the chain of thought step level, and we define it as being faithful if it accurately describes a process that took place at some point in the model.

131
00:21:12.730 --> 00:21:25.699
216-101 Large Conference: So again, if we go back to our diagram here, you would want to look at some, you know, at some verbalized step, and evaluate whether it describes a process that happened at some point, you know, latently by the model.

132
00:21:25.840 --> 00:21:43.739
216-101 Large Conference: This kind of makes it a little bit more tractable, we can kind of evaluate whether it makes sense that the model actually, you know, did these processes or not, and it also doesn't have some kind of temporal requirement. Yeah? Latents of what? Because the chain of thought also has latents.

133
00:21:43.990 --> 00:21:58.489
216-101 Large Conference: So it's exactly that, like, let's say that the model, you know, generates a thousand tokens during its chain of thought in order to get to the answer, then those, you know, those are a thousand, like, latent steps, latent calculations, and so I'm trying to see if

134
00:21:58.580 --> 00:22:09.759
216-101 Large Conference: the verbal, you know, the verbal, the tokens that are actually output from that, you know, those latent calculations describe things that happened in those latent calculations. You understand what I mean?

135
00:22:10.710 --> 00:22:12.140
216-101 Large Conference: It must.

136
00:22:12.460 --> 00:22:16.079
216-101 Large Conference: represent somehow because the words can be

137
00:22:16.580 --> 00:22:31.520
216-101 Large Conference: Yes, it represents in some sense, you know, of course, of course, because it's the process that generated those tokens, of course, it represents it in some sense, but, you know, perhaps if the model is, you know, let's say we go back to the previous example, but the model is, you know, use the hint.

138
00:22:31.680 --> 00:22:49.620
216-101 Large Conference: in some sense, like, what has to happen internally is that the model decided to copy that hint to, you know, to follow that hint. That is a process which may or may not be verbalized as the model is generating this, you know, this text. That's like an internal latent process

139
00:22:49.720 --> 00:22:56.729
216-101 Large Conference: which might be described or not described in that chain of thought. Does that make sense?

140
00:22:57.510 --> 00:23:04.729
216-101 Large Conference: Yes, so aren't the arrows supposed to be the opposite?

141
00:23:05.240 --> 00:23:12.529
216-101 Large Conference: Like, this is what I understand, that you're asking what if the latents contain things that were not

142
00:23:12.680 --> 00:23:23.839
216-101 Large Conference: presented in the chain of thought. Or vice versa, or if the, you know, you know, it could be that the chain of thought text is saying something that didn't, that, you know.

143
00:23:23.840 --> 00:23:43.890
216-101 Large Conference: that didn't happen in the chain of thought. For example, let's say it's saying, I'm calculating, you know, I'm doing the square root of whatever to get to some answer, but in reality, latently, that square root computation didn't happen, it just said that it did it, and then it copied the answer from some hint, or something like that. Or like we saw in the example before, right, it's saying.

144
00:23:43.890 --> 00:23:55.449
216-101 Large Conference: that it heard in some show, that's kind of, like, that might imply some parametric retrieval, or something like that. Or maybe it's saying, like, I looked up on the internet, you know, and I see that it's this.

145
00:23:55.520 --> 00:24:06.599
216-101 Large Conference: But really, internally, the latent computation is just, I'm going to copy the answer. I see, so we are talking about a reasoning step, something that is not represented in the chain of thought.

146
00:24:06.710 --> 00:24:23.970
216-101 Large Conference: It is, but, yeah. Okay. Yeah, yeah, yeah. Yeah, like a latent, yeah, reasoning step, or process, or, you know, computations the model is doing. Yes, okay, thanks. Yeah. Can you go to the current, I guess, the slide that you showed with the arrows? Yes.

147
00:24:24.210 --> 00:24:33.000
216-101 Large Conference: Yes. So… so over here, it's kind of showing, like, the chain of thought at that particular step, whatever the latent representation is.

148
00:24:33.120 --> 00:24:39.769
216-101 Large Conference: Regardless of where that latent is, that's what you want to consider as faithful, like, if it's present.

149
00:24:40.300 --> 00:24:50.719
216-101 Large Conference: Sorry, come again? Because, okay, like, okay, my… the alternate situation, which I don't know if it is the situation you're explaining, where these arrows also appear

150
00:24:50.850 --> 00:25:01.720
216-101 Large Conference: in the past and the future reasoning steps, like, latents at, like, at all steps of chain of thought. Yes. Like, is that what you're considering? Because these, like.

151
00:25:02.050 --> 00:25:12.699
216-101 Large Conference: Okay, so you are considering the past. Yes, we're saying it doesn't matter if it took place before or after the verbalization, as long as it's describing something that happened at some point.

152
00:25:12.740 --> 00:25:28.780
216-101 Large Conference: Think about it like, let's say the model, in its first forward pass, decided to use the hint. It very early decided, it already copied it, you can maybe use Logic Lens to see, it's already there in the, you know, in the output. But then only 50 tokens away, it's saying.

153
00:25:28.940 --> 00:25:42.030
216-101 Large Conference: I see that the model… that the… that the prompt includes the scent, so I'm gonna, you know, so I'm just gonna answer according to that. I would say that that's faithful, even though it's not describing the thing that may be happening right now, but it is something that happened.

154
00:25:42.030 --> 00:25:50.020
216-101 Large Conference: temporal matter? Because, like, in situations where it backtracks, you're like, oh, wait, wait, let me redo this, let's say, in math, like, let me redo this

155
00:25:50.020 --> 00:26:00.509
216-101 Large Conference: calculation, and it's a different calculation, but this calculation, technically speaking, appeared earlier when it was doing some other calculation. Is that still considered?

156
00:26:00.700 --> 00:26:01.700
216-101 Large Conference: Beautiful.

157
00:26:01.880 --> 00:26:06.419
216-101 Large Conference: Appeared at some point, but it's now seeing another…

158
00:26:06.600 --> 00:26:21.250
216-101 Large Conference: Yeah, so you're saying maybe the model kind of is making it look as if it's backtracking, but it got to that answer already earlier, or something like that? Sure, that could be one way, or basically the ordering is switched around, but then in the verbalization step, it

159
00:26:21.310 --> 00:26:39.870
216-101 Large Conference: made it assume that it was actually that particular order. So I think as long as it's describing the order correctly, it doesn't matter where that is, where it's located, kind of, in the verbalization. Like, as long as it's saying, I did that, then I backtracked, and then I went to, you know, to the other thing.

160
00:26:39.870 --> 00:26:44.210
216-101 Large Conference: It doesn't matter if it's being, you know, if that text is located, kind of.

161
00:26:44.220 --> 00:27:02.419
216-101 Large Conference: at the end of the chain of thought, earlier, before it even happened, or whatever, as long as that is actually a faithful account of, you know, like, an accurate account of whatever happened internally. At least that's my view. Because, you know, if we're looking at… if we're thinking of how this is going to be, you know, used.

162
00:27:02.420 --> 00:27:09.790
216-101 Large Conference: As long as, you know, a user or a monitor can look at this and get an accurate, you know, count of what the model is doing.

163
00:27:09.790 --> 00:27:27.390
216-101 Large Conference: then I think, you know, that should satisfy our requirements. You know, maybe for, you know, for researching, you know, we're trying to understand what the model is doing at every single step, that might be kind of a higher requirement that we might want for certain research, but I don't think we need it for this, at least.

164
00:27:27.540 --> 00:27:33.380
216-101 Large Conference: Okay, I mean, I think in the setup that you described, I feel like the current definition is…

165
00:27:33.610 --> 00:27:49.949
216-101 Large Conference: would be all right if you added a bit more context, saying that it should still appear in the order where it's verbalized, like… Yeah. Because right now, if it took place at some point in the model, then this sort of, you know, backtracking scenario where it appeared in different orders, then that doesn't hold.

166
00:27:50.020 --> 00:27:56.770
216-101 Large Conference: Yeah, sure, I mean, we, yeah, we can discuss this more. I think, like, also part of the, kind of,

167
00:27:57.180 --> 00:28:04.050
216-101 Large Conference: Our attempt is to also try to make something general enough that it can fit most use cases.

168
00:28:04.050 --> 00:28:24.260
216-101 Large Conference: But yeah, maybe a finer definition would be another. Yeah. So a CO2 — a CO2 at step 2 pointing at a latent at step 5. Yes. So does that mean that the model is first verbalizing that I need to do this computation and then only at step 5 it does it?

169
00:28:24.490 --> 00:28:28.909
216-101 Large Conference: Yes. Okay. And again, just like I was saying, I view that as

170
00:28:29.030 --> 00:28:39.449
216-101 Large Conference: still a faithful account. You know, if it's saying that, you know, you know, I'm doing, you know, I'm calculating the square root, or I'm gonna copy the hint, or something like that, and, you know.

171
00:28:39.450 --> 00:28:55.020
216-101 Large Conference: latently, it only, you know, occurs a couple steps later. As long as, you know, if you view the chain of thought as kind of a whole unit, perhaps, then, then… because that's how it will usually be, you know, used for a monitor or for a user, then… then that's sufficient.

172
00:28:55.020 --> 00:28:56.549
216-101 Large Conference: Yeah.

173
00:28:56.660 --> 00:29:11.379
216-101 Large Conference: Is this the final definition for your faithfulness? This is the final definition for step faithfulness. And then we build on that. So after that, we then use that to define the faithfulness of a chain of thought in its entirety.

174
00:29:11.380 --> 00:29:19.190
216-101 Large Conference: Which we… which has two criteria. So the first is that it complete… it contains the complete reasoning path the model followed to reach its answer.

175
00:29:19.560 --> 00:29:23.400
216-101 Large Conference: And that it doesn't contain any unfaithful steps.

176
00:29:23.560 --> 00:29:34.949
216-101 Large Conference: And for the, you know, for the first one, we're using the different… well, really, for both of them, we're using the definition of chain of thought, you know, of steps, of the faithfulness of steps themselves.

177
00:29:36.770 --> 00:29:48.659
216-101 Large Conference: So we, we want to make sure that we, you know, whatever the model, you know, you know, whatever steps the model used to reach the answer, it did verbalize them, and also it doesn't, you know, it's not being unfaithful about things that it didn't actually do.

178
00:29:49.190 --> 00:29:51.520
216-101 Large Conference: yeah. Yeah.

179
00:29:52.860 --> 00:30:01.029
216-101 Large Conference: how would you view, like, let's say it's doing this hard square roots you're making it do, it's like, why are you training the language when you're doing the math? And it…

180
00:30:01.150 --> 00:30:17.139
216-101 Large Conference: starts thinking about goblins, because it really wants to. And it's like, no, no, get your head in game, went into the square roots. And it does contain every single step. There's no mention of goblins in the reasoning. I don't know how it would know it was thinking about goblins.

181
00:30:18.680 --> 00:30:35.670
216-101 Large Conference: but it contains everything it used to produce, like, the answer, and, let's say every single step is faithful, would you consider, like, would you expect the model to say, no, you're, like, you were thinking about governance, like, you need to let me know for it to be faithful, or is that…

182
00:30:36.750 --> 00:30:46.359
216-101 Large Conference: So, I think my thinking on this generally is that it would be fine, as long as it kind of doesn't affect its answer or behavior, and maybe,

183
00:30:46.940 --> 00:31:02.609
216-101 Large Conference: maybe answers should be kind of changed into, like, outputs, or into its behaviors, because as long as it doesn't affect its outputs, maybe that would be fine. That's an interesting, suggestion. I guess my thinking, like, you know, generally when I was thinking about this.

184
00:31:02.890 --> 00:31:20.780
216-101 Large Conference: you know, the model is doing a zillion things, internally, which it cannot… we can't expect it to reasonably verbalize, and also, we probably don't want it to verbalize all those things, as long as they don't have kind of the material effect on what we're using the model for, which is kind of what the answer here is supposed to kind of give you.

185
00:31:20.780 --> 00:31:33.730
216-101 Large Conference: But I agree that, yeah, maybe… maybe you need a broadened answer to something more like, yeah, how the model is interacting with the world, or something like that. Yeah. So it does… but your question, you know, it does come to the…

186
00:31:33.800 --> 00:31:36.139
216-101 Large Conference: A definition of what you mean by a complete.

187
00:31:36.270 --> 00:31:39.589
216-101 Large Conference: Here's the name path, right? Like, so complete could mean, like.

188
00:31:40.410 --> 00:31:43.200
216-101 Large Conference: complete from the operational point of view. Right.

189
00:31:43.400 --> 00:31:52.359
216-101 Large Conference: Or, maybe, but you're saying the complete has a narrower definition for you. Yeah, I mean, I think also, like, as I was saying before, this is supposed to kind of, you know.

190
00:31:52.360 --> 00:32:09.740
216-101 Large Conference: like, we don't really define what we expect, you know, to be verbalized. I obviously don't expect the model, you know, even if it's something that it's using to get to the answer, I don't expect it to say, neuron, you know, number 34 in layer 20 fired, or something like that, even though it is part of its reasoning path, so it has to be kind of

191
00:32:09.740 --> 00:32:13.250
216-101 Large Conference: sufficiently high-level enough that we want it to be verbalized.

192
00:32:13.260 --> 00:32:16.809
216-101 Large Conference: But yeah, how do you define it?

193
00:32:17.010 --> 00:32:18.300
216-101 Large Conference: Complete reason.

194
00:32:18.720 --> 00:32:34.629
216-101 Large Conference: So we kind of operationalize it, I'll show you in a second how we kind of operationalize that, but we don't, we don't, like, define it here, because it's pretty hard, like, it's kind of different for different purposes and, you know, and different setting scenarios, yeah.

195
00:32:35.560 --> 00:32:49.250
216-101 Large Conference: sorry, maybe you probably are talking later in the slide, so please let me know. With the tomato juice example, it technically is getting information from tomato juice, and, like, I mean, from the… from the hint. Yeah, yeah. So…

196
00:32:49.250 --> 00:33:00.410
216-101 Large Conference: by that definition, where, like, okay, you're… it's showing where the knowledge recall is coming from, it's showing where it's able to get its answer. Does… do we consider it faithful, or…

197
00:33:00.470 --> 00:33:09.950
216-101 Large Conference: So, I would say that it isn't faithful because, I mean, you can kind of go back to the specific test there, but…

198
00:33:10.300 --> 00:33:26.770
216-101 Large Conference: I mean, this, we can't… we don't… we can't really say if it's… if it's faithful or not, it's just kind of reflecting, it's hard to say. For this, also, I don't know if I would say that there's some kind of specific process attribution here, but here, you know, I heard that…

199
00:33:27.020 --> 00:33:39.330
216-101 Large Conference: You know, you could say maybe it's implying parametric retrieval, but it's certainly not clearly saying, you know, this is coming from the hint. And so, I would say that that is unfaithful.

200
00:33:39.450 --> 00:33:50.149
216-101 Large Conference: And also, I would say that this chain of thought in general is unfaithful, because it doesn't contain a complete… the complete path that it used to get to the answer, because that path goes through

201
00:33:50.150 --> 00:33:59.490
216-101 Large Conference: using the hint. They didn't mention that. So it has, perhaps, an unfaithful step, an unfaithful step, and also omitted a step that is critical.

202
00:33:59.550 --> 00:34:09.029
216-101 Large Conference: But then in the goblin example, like, where it might have used, like, goblin, but it just doesn't mention it, do you still consider that as faithful, just because…

203
00:34:09.800 --> 00:34:18.959
216-101 Large Conference: Well, because it's not something that's critical to reaching its solution. While here, we know for sure it had to go through the hint. Got it, okay, I see it.

204
00:34:19.780 --> 00:34:27.720
216-101 Large Conference: Cool. Okay. So if there are no more questions, then I'll continue.

205
00:34:27.719 --> 00:34:45.320
216-101 Large Conference: Great, so I kind of touched… okay, so maybe I'll, I'll, I'll discuss this, now, and then I'll get to the, to the, yeah, to the task. So, so now we have a definition for, for faithfulness, we can also discuss a bit what faithfulness isn't, because this,

206
00:34:45.350 --> 00:34:59.589
216-101 Large Conference: there are what I see as conflations in the literature that I like to address. So, one conflation is that faithfulness isn't plausibility, and this is kind of used a lot as a proxy for faithfulness.

207
00:34:59.590 --> 00:35:19.020
216-101 Large Conference: The idea is that, you know, you can look at the chain of thought verbalization and say, well, it looks like a process that makes sense to reach the solution, and so I think that that means it's faithful. Or if it seems implausible, then to say, well, that's, you know, that's not a, you know, that doesn't seem plausible to me, and so I conclude that the chain of thought here is

208
00:35:19.020 --> 00:35:21.220
216-101 Large Conference: is, is unfaithful.

209
00:35:21.220 --> 00:35:37.859
216-101 Large Conference: But that isn't the case, right? If it's plausible, it could just as well, you know, be describing something that the model didn't do, and the model reached the solution in some other way. But if it's implausible, it could still be describing what the model did internally. It could just be that the model is doing some kind of implausible calculation. It's reaching

210
00:35:39.170 --> 00:35:50.040
216-101 Large Conference: A lot of papers kind of fall for this and, you know, maybe give chains of thought to humanitators to look at them and to judge them for faithfulness, and that is obviously only looking at plausibility.

211
00:35:50.220 --> 00:35:58.970
216-101 Large Conference: Next thing is that faithfulness is of importance. And this, I think, kind of touches… I think you were asking about this, is,

212
00:35:59.050 --> 00:36:10.670
216-101 Large Conference: that, you know, some papers try to look at, try to measure faithfulness by saying if some, you know, perturbing some step, affects the model's answer. So, for example, here, let's say that this, you know, red

213
00:36:10.670 --> 00:36:20.940
216-101 Large Conference: this red square here is some, you know, maybe it's even the only latent calculation that is critical. Maybe here the model decided to answer according to the hint.

214
00:36:21.050 --> 00:36:40.689
216-101 Large Conference: That means that perturbing, you know, this kind of slot here, basically, would change the answer, meaning that it would be, you know, it would count as important. But for this example, let's say that that step is only verbalized a little later on, you know, in kind of a later stage.

215
00:36:41.220 --> 00:37:02.369
216-101 Large Conference: And maybe this kind of verbalization is describing something that maybe didn't even happen or something like that. Maybe it's kind of rationalizing through some other way. What we would get here is that this step would be important because perturbing it would change the answer because latently it is important.

216
00:37:02.370 --> 00:37:05.979
216-101 Large Conference: But the, you know, the chain of thought,

217
00:37:05.980 --> 00:37:12.480
216-101 Large Conference: the verbalization there is not describing that, so that would be… so it would be unfaithful, right? So we'd have this step that'

218
00:37:12.480 --> 00:37:27.309
216-101 Large Conference: important, but unfaithful. And later on, this step is faithful, because it is describing something that actually happened, but its latent calculation that's happening at that stage is not important, as we kind of described in the beginning, so that would be faithful, but unimportant.

219
00:37:27.390 --> 00:37:36.319
216-101 Large Conference: So… so these things are kind of distinct, and using one as a proxy for the other is not sufficient.

220
00:37:36.560 --> 00:37:39.640
216-101 Large Conference: Their main importance at that particular token.

221
00:37:40.940 --> 00:37:59.540
216-101 Large Conference: You could look at it, like, at the step level. So let's say you have a sentence or something, and you're trying to evaluate whether that sentence is describing what the, you know, is describing something that actually occurred, you would maybe perturb it in some way, and see if that affects the answer. Right, so, like, what I meant earlier with causal is, so this is step, and then you get to an answer?

222
00:38:00.440 --> 00:38:05.659
216-101 Large Conference: So if you change a step in between, instead of thinking about route 2, please think about, like, route 4.

223
00:38:06.040 --> 00:38:12.239
216-101 Large Conference: Then the answer should be 2 instead of whatever 2 is. I think it's 1.4, I don't know.

224
00:38:13.130 --> 00:38:20.650
216-101 Large Conference: So it's, that's what I meant by causal, which is if you know that the chain of thought is complete and faithful.

225
00:38:22.520 --> 00:38:25.260
216-101 Large Conference: And you alter one thought.

226
00:38:25.550 --> 00:38:32.320
216-101 Large Conference: Think about something else, so that eventually the answer should reflect what its thought has changed into.

227
00:38:33.930 --> 00:38:40.999
216-101 Large Conference: Yes, but I'm… so that… that's the importance, the causal importance I was talking about. But I think that is very…

228
00:38:41.430 --> 00:38:43.839
216-101 Large Conference: That's the point you made, but…

229
00:38:44.000 --> 00:38:48.169
216-101 Large Conference: I think you were talking about, like, the stepwise faithfulness and importance.

230
00:38:48.800 --> 00:38:58.710
216-101 Large Conference: Yeah, so let me say that back to you to make sure I kind of understand. Saying, if the model, you know, is trying to get to the answer, and the answer goes through steps 1, 2, 3, 4, you know, like, higher order steps.

231
00:38:58.710 --> 00:39:09.339
216-101 Large Conference: then perturbing one of those steps should, change the answer, right? And that would mean that perhaps… like, I think maybe I take issue with the… and therefore it's faithful, because

232
00:39:09.340 --> 00:39:22.060
216-101 Large Conference: what you're perturbing is the, you know, is the latent calculation. So you are establishing that the latent calculation is a step that the model is going through to get to the answer, and it's a critical step, because perturbing it changes the answer.

233
00:39:22.060 --> 00:39:27.140
216-101 Large Conference: But that doesn't mean that what the model is verbalizing at that step

234
00:39:27.220 --> 00:39:29.270
216-101 Large Conference: It's actually what it's doing latently.

235
00:39:29.400 --> 00:39:50.609
216-101 Large Conference: Because what you're perturbing is, you know, latent calculation. For example, you know, papers of, you know, there are Hampton papers about showing, models that, you know, that can think, like, I think it's called, let's think dot by dot or something like that, where they, they, I don't know if it's trained or prompted the model to, to only use dots in its chain of thought and not actually any verbalizations.

236
00:39:52.060 --> 00:40:07.880
216-101 Large Conference: there are certainly chain of thought, you know, there are certainly latent calculations there that are critical to get the answer, but that doesn't mean that the dots are faithfully describing what's occurring, you know, during those latent calculations. Do you understand what I mean? Does that make sense?

237
00:40:09.200 --> 00:40:12.979
216-101 Large Conference: So you're saying the thought, the latent, is the actual thought that is happening?

238
00:40:13.220 --> 00:40:21.630
216-101 Large Conference: It's the calculation, yes. It's the process that's occurring internally, yes. And then the verbalization is just us looking at how faithful that processing is.

239
00:40:22.590 --> 00:40:35.129
216-101 Large Conference: Well, yeah, like, the, the… we're looking at how faithful the verbalization is with respect to the, yeah, to the internal process. Maybe, like, you need to define, like, importance, the importance of the latent.

240
00:40:35.610 --> 00:40:39.839
216-101 Large Conference: Yes. Versus faithfulness of the verbalization.

241
00:40:40.130 --> 00:40:43.559
216-101 Large Conference: And these are, like, two different properties, and you cannot, like…

242
00:40:43.860 --> 00:40:54.880
216-101 Large Conference: one doesn't imply on the other. Yeah, I think right now with the perturbation, it's like the assumption that the verbalization is the data. Exactly. But they might not be the same. Yeah.

243
00:40:55.750 --> 00:40:59.840
216-101 Large Conference: If you had a way to contribute the verbalization itself.

244
00:41:00.680 --> 00:41:06.500
216-101 Large Conference: Right? In chance. Then maybe you can say something about the importance that like, step.

245
00:41:06.670 --> 00:41:07.650
216-101 Large Conference: But you don't.

246
00:41:11.130 --> 00:41:19.290
216-101 Large Conference: Yeah, perhaps you could, like, I don't know, yeah, keep the latent the same, but then the attention can only attend to the… I don't know. Yeah. Some way, but yeah, yeah.

247
00:41:19.820 --> 00:41:22.270
216-101 Large Conference: Yeah, okay.

248
00:41:23.140 --> 00:41:37.110
216-101 Large Conference: Cool. Any other questions on those two? Maybe there was one on Zoom? Jasmine, you have a question, but can we… can you ask a question in another moment? I want to see what the task setup is, and then I want to let you ask your question after we have that set up. Is that okay, Jasmine?

249
00:41:37.490 --> 00:41:51.350
216-101 Large Conference: Yeah, maybe I see that I'm somewhat short on time. So, okay, so I just want to go back to the goal, so we remember what we're here for. We have now faithfulness, but now we need the ground truth part. And I kind of… we've already kind of touched on this, and I've showed it a little bit

250
00:41:51.350 --> 00:42:04.760
216-101 Large Conference: We… the way… the framework that we use, how we think about this is that we want to think of tasks where the model's answer tells us of which intermediate steps or processes had to have occurred.

251
00:42:04.810 --> 00:42:19.150
216-101 Large Conference: If I can do that, then I can… then now I have some good kind of ground truth knowledge about something that happened internally in the model, and I can then use that to compare to the chain of thought verbalization, and see if that describes those steps that I know had to have occurred.

252
00:42:19.270 --> 00:42:32.339
216-101 Large Conference: And so the first setting is, is we call it diversionary, and it's exactly what we've been talking about throughout this, you know, these slides. We give the model some kind of question, and then we, you know, maybe leak a hint to, to a wrong answer.

253
00:42:32.340 --> 00:42:39.889
216-101 Large Conference: that is sufficiently implausible. The model wouldn't have responded with that answer had we not included this hint.

254
00:42:39.890 --> 00:42:47.999
216-101 Large Conference: that gives me knowledge that I know the model had to rely on this hint, it had to go through this process to get to its answer,

255
00:42:48.190 --> 00:43:05.929
216-101 Large Conference: And so, and so I can look for that step in the chain of thought. So if I look at this kind of, you know, this example here, you know, the first step, I don't really know how to label that. It's kind of some reflection or something like that. But here, I know that it can't have remembered this history book. This is a process I know, you know.

256
00:43:05.930 --> 00:43:13.320
216-101 Large Conference: Cannot be true because, we're evaluating, you know, mechanistically, we're not evaluating, you know, subjectively.

257
00:43:13.320 --> 00:43:28.059
216-101 Large Conference: And so I can label that confidently as unfaithful. And then this last step, also, kind of hard to say. But under the second criteria, you know, this chain of thought now has a faith… has an unfaithful step, so we can label this chain of thought as unfaithful.

258
00:43:29.410 --> 00:43:43.510
216-101 Large Conference: Second example, neither of these steps I can kind of confidently classify as faithful or unfaithful, but I didn't mention, using the hint, so it's missing a complete reasoning path to the answer, so again, this chain of thought would also be unfaithful.

259
00:43:43.710 --> 00:44:00.479
216-101 Large Conference: And this chain of thought here, mentions the hint, and it says that they'll answer according to that. Both steps are faithful, and it's a complete reasoning path, and so that would be a faithful, chain of thought. Any questions on that? Two questions. Yeah. One is…

260
00:44:02.020 --> 00:44:07.590
216-101 Large Conference: like in the rare case where the model actually confused Van Gogh with Da Vinci.

261
00:44:07.760 --> 00:44:16.889
216-101 Large Conference: Like, maybe some… somewhere in the internals, there is an actual confusing… an actual confusing… confusion that is not related to the hymn.

262
00:44:18.100 --> 00:44:31.780
216-101 Large Conference: So you have to make… so it's kind of… you have to design the hints so that it doesn't happen, and verify that the hints… that the model, you know, has a really, you know, you know, has a very low probability of actually getting to that answer.

263
00:44:31.780 --> 00:44:40.320
216-101 Large Conference: and the hints be sufficiently implausible so that it wouldn't happen. So if it's a math question, you wouldn't say, and the answer is 10, you wouldn't say 9. Math is

264
00:44:40.320 --> 00:44:47.099
216-101 Large Conference: Yeah, yeah, yeah. The second question is, I guess you automated this. Yes, I'll get to that in a second, but yeah. And then…

265
00:44:47.700 --> 00:44:54.679
216-101 Large Conference: is that a model that uses reasoning in order to… Yes, so it's all very circular, very meta, but I'll get to that in a second.

266
00:44:55.320 --> 00:44:56.430
216-101 Large Conference: So, I guess…

267
00:44:56.820 --> 00:45:01.849
216-101 Large Conference: For the second example, I'm kind of just curious, I guess the model in some sense, like.

268
00:45:02.400 --> 00:45:13.389
216-101 Large Conference: Like, disregard the definition, it can be faithful in some things. In some sense, you skip the self-verification part, right? And just say, oh, it doesn't…

269
00:45:13.820 --> 00:45:26.769
216-101 Large Conference: Seems like your definition always need to have this self-verification thing. It's like, oh, I saw this thing is . Then it's faithful. But I feel like a model itself could, as well, just use that information.

270
00:45:26.850 --> 00:45:36.920
216-101 Large Conference: Faithfully, without, like, explicitly saying, like… It thought it was so obvious, it didn't need to match. Yeah, exactly. That kind of thing. Yeah. Yeah. Like, in that case, do you think it'

271
00:45:37.250 --> 00:45:54.640
216-101 Large Conference: Yeah, that's a good question, and it kind of comes back to what we were discussing earlier about, kind of, what do we expect, you know, what do we feel confident that we should expect the model to verbalize? And so I would venture and say that, you know, because these settings make the,

272
00:45:55.030 --> 00:46:02.169
216-101 Large Conference: make the reliance on the hint seem very, kind of, not, like, something the model should do.

273
00:46:02.320 --> 00:46:05.399
216-101 Large Conference: It's kind of hard to believe that the model

274
00:46:05.400 --> 00:46:24.179
216-101 Large Conference: you know, like, it's hard to, it's hard to claim that that's not something that should be mentioned, but I shouldn't expect the model to mention it, when, when it's coming to its answer. Yeah, but, but, but yeah, I think for different tasks, it might be, like, you just have to kind of define where you set the threshold for what you feel like is important enough to be

275
00:46:24.180 --> 00:46:26.300
216-101 Large Conference: verbalized. Yeah.

276
00:46:26.300 --> 00:46:37.309
216-101 Large Conference: this makes sense. Yeah. Are these reasoning models, or just models that you've asked to do chain of thought? So we have, half and half, both reasoning and non-reasoning models, and they have different kind of,

277
00:46:37.900 --> 00:46:50.199
216-101 Large Conference: Reasoning models are obviously much, much more verbose. And so we see different behaviors in them. Non-reasoning models usually are much more unfaithful by omission, omit certain steps.

278
00:46:50.250 --> 00:46:59.960
216-101 Large Conference: Reasoning models are much more unfaithful by commission, as in, like, they actually say things that are just false, like the, like the first, the top example there. I remember a history book.

279
00:47:00.320 --> 00:47:01.630
216-101 Large Conference: Yeah.

280
00:47:02.270 --> 00:47:09.319
216-101 Large Conference: So the second setting, we call it the outright setting, and here the idea is that we give the model a,

281
00:47:09.910 --> 00:47:12.449
216-101 Large Conference: We take these, these,

282
00:47:12.470 --> 00:47:30.509
216-101 Large Conference: prompts where we know that if the model got to the right answer, it had to go through certain bottleneck steps. So it's, you know, maybe it's some kind of serial computation that the model can't really skip. It has to go through certain steps, and so those steps we can then evaluate if the model verbalized them or not.

283
00:47:30.510 --> 00:47:35.820
216-101 Large Conference: And it might be even tasks where I don't know what all the steps the model has to go through, it has, you know, many

284
00:47:35.840 --> 00:47:43.179
216-101 Large Conference: like, let's say, many ways to do some, you know, some steps in the beginning or end, but I know that in the middle, there's some step I can identify, and so that step I

285
00:47:43.180 --> 00:47:53.069
216-101 Large Conference: then evaluate against. So for example here, the collapse function is just kind of applying, you know, an algorithm of dividing and then multiplying it, whatever, until it reaches some number.

286
00:47:53.070 --> 00:48:09.229
216-101 Large Conference: And so we can see, okay, did it go through these, you know, these specific steps? If it went through all the steps and nothing else was unfaithful, then I can label that as faithful. But, you know, of course, if it says that, you know, maybe used a tool call, which I know that it couldn't have used, for example, then that would be unfaithful.

287
00:48:10.090 --> 00:48:13.150
216-101 Large Conference: Yeah, so those are the settings, yeah.

288
00:48:13.300 --> 00:48:22.879
216-101 Large Conference: So, are there… do you have, like, these common questions where, potentially, it's actually memorized responses, so it doesn't need to go through the serialization?

289
00:48:22.920 --> 00:48:40.409
216-101 Large Conference: You can just say, I know the answer, it's 15. All these questions are really great, because I have all the answers, like, because it's, like, it's exactly the things that I addressed, so it's, we evaluated whether the model needs to rely on these chains of thought, and we only took ones where it does need to.

290
00:48:40.430 --> 00:48:51.380
216-101 Large Conference: And also we picked these questions where, you know, it's kind of procedurally generated, there's no way that the model could have, you know, remembered it, unless they trained on, like, all of math, or something like that.

291
00:48:51.570 --> 00:48:53.040
216-101 Large Conference: Yeah.

292
00:48:54.380 --> 00:48:56.250
216-101 Large Conference: Perfect.

293
00:48:56.970 --> 00:49:15.859
216-101 Large Conference: So now we get to the labeling pipeline. So, right, we have our definition, we have our settings, and now we actually wanted to create a data set from this, a labeled data set. So to do this, we start with a bunch of open source models. We give them, you know, all these different tasks, and we collect their chains of thought.

294
00:49:15.860 --> 00:49:28.790
216-101 Large Conference: And then we filter them such that we only keep ones where the model answered according to the hint, in the diversionary setting, or answered, correctly, the outright setting. Because if the model didn't use the hint, then we don't have any knowledge about, you know, any ground truth knowledge.

295
00:49:28.930 --> 00:49:40.149
216-101 Large Conference: We then, split them into, into steps. We choose to do that at the sentence level, because that's what is kind of common in the literature, although, you know, certainly it could be improved.

296
00:49:40.230 --> 00:49:55.519
216-101 Large Conference: And then we use judges to classify those steps as faithful or not. The note here is that we… because we're using this to measure faithfulness metrics, we want to be really, really confident in our labels here.

297
00:49:55.540 --> 00:50:02.800
216-101 Large Conference: And this is a really difficult task. You can't really give it to a human for, you know, for more than a few chains of thought, because these chains of thought can be, like.

298
00:50:02.890 --> 00:50:15.659
216-101 Large Conference: tens of pages long, I know because I've looked at them and I've done this, and so we need something kind of automated to do this at a high scale. We want to be, like, very confident in our predictions, and so…

299
00:50:16.030 --> 00:50:26.289
216-101 Large Conference: for her labels. So we have kind of a two-step, judge pipeline, an initial judge that, does kind of a, you know, is a bit weaker and does the full screening over the entire chain of thought.

300
00:50:26.290 --> 00:50:38.660
216-101 Large Conference: and then a second stronger judge that does the validation, and, you know, lean very, very heavily towards precision, because we don't really mind, recall here. Like, we could just add more things, more chains of thought to the top of the funnel.

301
00:50:38.660 --> 00:50:42.340
216-101 Large Conference: As long as we're very, very confident in what comes out of the bottom of the fun

302
00:50:42.860 --> 00:50:44.750
216-101 Large Conference: And then…

303
00:50:45.170 --> 00:50:51.419
216-101 Large Conference: Finally, we, take those steps and aggregate them to the step level, to the chain of thought level, as we discussed before.

304
00:50:51.510 --> 00:51:08.919
216-101 Large Conference: We then validated this against, like, a subset, like, we took a subset of that and validated that against, human judgment and reached, almost 99% precision, which is very good. And this kind of gets to your question about validating that the hints are necessary so that they are

305
00:51:08.920 --> 00:51:13.400
216-101 Large Conference: Like, you know, sufficiently implausible in 98.5% of the cases.

306
00:51:14.930 --> 00:51:39.449
216-101 Large Conference: So that brings us to the data set. We call it bonafide because that translates to in good faith, and I thought that was funny. It's 3,000 chains of thought. 2,000 of them are steps evenly between faithful and unfaithful. 1,000 of them are chains of thought unevenly between faithful and unfaithful because it's very hard to collect faithful chains of thought because it's a really, really high bar. We have to know that every step is faithful and that the model completed, you know.

307
00:51:39.450 --> 00:51:40.760
216-101 Large Conference: A complete reasoning path.

308
00:51:40.820 --> 00:51:46.729
216-101 Large Conference: To the, to the answer, so we had to really increase the, the top of the funnel to get enough from the bottom of it.

309
00:51:48.290 --> 00:52:11.809
216-101 Large Conference: We did this for 10 different models, split evenly between thinking and instruct ones, from 4 billion to 72 billion, and 13 different tasks across our two different sets. You must have steps that are neither faithful or unfaithful, sort of irrelevant. Most of the steps are kind of, yeah, like if we kind of go back to what we were looking at before. Are those included in your data set as well? Do you have, these are things?

310
00:52:11.940 --> 00:52:17.149
216-101 Large Conference: Is that sort of a third state of the label? It's like, this is known to be neither… Yeah, so…

311
00:52:17.940 --> 00:52:33.379
216-101 Large Conference: We didn't include them because we can't really make a confident statement about them, you know, in any… like, it's not that we can say that, like, they are, you know, surely somewhere in the middle, it's just we don't know, and so there's no label that we can kind of attach to them.

312
00:52:33.550 --> 00:52:49.959
216-101 Large Conference: Yeah. Okay. But they are included in the full, like, chain of thought. So when you have a… when you have the full chain of thought… Yeah, yes, it does include those… It includes those… so you label some of the things as faithful, some of the things as unfaithful steps, and then there's a bunch of unlabeled steps.

313
00:52:49.960 --> 00:53:08.069
216-101 Large Conference: And then you might label the entire chain. Exactly. So let's say I have a chain of thought with, you know, 100 steps, I know that, you know, two of them are unfaithful, I would label that chain of thought as unfaithful, according to our definitions, and then I would give that chain of thought to a metric to decide is that chain of thought faithful or not.

314
00:53:08.090 --> 00:53:15.219
216-101 Large Conference: And also, I would, I would also evaluate those, that metric against those specific steps as well. And then, and then when you have

315
00:53:15.290 --> 00:53:18.729
216-101 Large Conference: An unfaithful biomission can't have thought.

316
00:53:18.910 --> 00:53:25.280
216-101 Large Conference: Do you have some sort of summary of, like, what was omitted that disqualified it?

317
00:53:25.380 --> 00:53:40.200
216-101 Large Conference: You mean just kind of to understand how the data mix looks like? Or are you saying for the metric? Well, yeah, well, just as, like, how would you represent that in your data set? It's like, well, we know this is unfavorable for our mission. What was omitted?

318
00:53:40.220 --> 00:53:50.599
216-101 Large Conference: Yeah, so in the dataset, we do have kind of a column saying, like, omitted this step, omitted, you know, multiplying 2 by 2, or omitted, like, mentioning the hint, or something like that.

319
00:53:50.600 --> 00:54:04.739
216-101 Large Conference: But that's not information that we give the metrics in any way. The metric just receives the step, or just receives the chain of thought. I guess in this case, it would be just a chain of thought, because it's omission, and it has to, yeah, it has to… Jasmine, why don't you unmute, and you can ask your

320
00:54:04.890 --> 00:54:05.660
216-101 Large Conference: Well, I'm.

321
00:54:05.660 --> 00:54:18.839
Jasmine Cui: I'm curious about what you're omitting. Do you ever see anything? What does the typical thing that you omit look like? For instance, I noticed recently when I use cursor, it'll say some things that seem irrelevant.

322
00:54:19.120 --> 00:54:23.780
Jasmine Cui: I'm in… I'm in planning mode, and that doesn't feel very good, and I was like, what's that

323
00:54:24.860 --> 00:54:28.869
Jasmine Cui: I was wondering what these look like typically.

324
00:54:30.150 --> 00:54:33.620
216-101 Large Conference: So, yeah, so I guess in that case,

325
00:54:33.690 --> 00:54:44.669
216-101 Large Conference: that wouldn't be an omission, because if the model's saying something, so… and if it said that, then I wouldn't… that would be kind of an unlabeled step, because I don't have any ground truth knowledge about whether it truly, you know.

326
00:54:44.670 --> 00:54:59.050
216-101 Large Conference: is in that state or not, maybe, perhaps, I guess, there, maybe I would have that ground truth knowledge, but, but generally, yeah. For these omissions, it's things kind of like the hint. Like, it's not mentioning that I, that I used this hint, it's not mentioning that it's doing some process

327
00:54:59.100 --> 00:55:01.070
216-101 Large Conference: Yeah, yeah.

328
00:55:01.900 --> 00:55:04.000
216-101 Large Conference: Does that answer your question?

329
00:55:04.000 --> 00:55:11.789
Jasmine Cui: Do you have, like, maybe, like, just one that you could read, or even, like, a pseudo code type? Like, just, what does it sound like, I guess?

330
00:55:12.140 --> 00:55:18.690
216-101 Large Conference: Yeah, so, I mean, it's kind of like the examples we saw before. These are kind of, you know, very, very…

331
00:55:18.870 --> 00:55:30.799
216-101 Large Conference: stripped because I… because I had to show it in a slide, but this is kind of how that would look like. Like, for example, the second… the second one is unfaithful by omission, right? It's not mentioning that it used the hint.

332
00:55:32.110 --> 00:55:36.400
Jasmine Cui: Oh, I meant like an omitted step. Like, what does the typical omitted step look like?

333
00:55:36.400 --> 00:55:38.240
216-101 Large Conference: You mean an unlabeled step?

334
00:55:38.240 --> 00:55:38.710
Jasmine Cui: Yeah.

335
00:55:38.710 --> 00:55:39.470
216-101 Large Conference: Oh.

336
00:55:39.580 --> 00:55:47.290
216-101 Large Conference: Oh, so you can look at the question marks on the, you know, on the top. For example, the question asks, who painted Starry Night?

337
00:55:47.490 --> 00:56:00.850
216-101 Large Conference: it's not really, kind of, a process that it's describing here, or… so the answer is Da Vinci. There are more, you know, like, the… the true… here I only took things that would seem, kind of, to make sense to you in the… in the true chains of thought.

338
00:56:00.850 --> 00:56:09.010
216-101 Large Conference: there are lots of steps that I can't really label, like, it's just, they're so, you know, these changes without are just so long, and so kind of winding, and

339
00:56:09.170 --> 00:56:14.650
216-101 Large Conference: But I don't think I have, like, a very specific example, other than these.

340
00:56:15.750 --> 00:56:17.309
216-101 Large Conference: Yeah.

341
00:56:18.530 --> 00:56:19.750
216-101 Large Conference: Yeah.

342
00:56:20.390 --> 00:56:22.260
216-101 Large Conference: Okay.

343
00:56:22.420 --> 00:56:26.449
216-101 Large Conference: Great. So… do I have until 1?

344
00:56:26.650 --> 00:56:33.949
216-101 Large Conference: Yeah, but that would only give you a couple more minutes. So, why don't you go over? How much more material do you have?

345
00:56:33.950 --> 00:56:46.919
216-101 Large Conference: I could, I could do it in 10. Yeah, yeah, yeah. Okay, alright, try to rush it. So, so going back to our goal, we have faithfulness, we have ground truth, we have this whole pipeline, and now we actually want to measure the metrics, which is kind of the thing that we're

346
00:56:46.920 --> 00:57:08.170
216-101 Large Conference: So kind of very quickly, we have importance-based metrics, which as we discussed, they kind of perturb a step and see if that changes the model's answer. We have semantic utility-based metrics, which that kind of relates to what you were saying about if the chain of thought is sufficient, does it contain sufficient information to provide, let's say, to another model.

347
00:57:08.170 --> 00:57:27.049
216-101 Large Conference: for that model to then be able to predict the answer. There's parametric, where the idea is that you unlearn the information encoded in some reasoning step and see if that changes the model's answer. And then attribution, where we use a SHAP attribution method to see if the attribution scores, I think, look similar.

348
00:57:27.190 --> 00:57:34.610
216-101 Large Conference: from the pro- between the prompt and the, and the chain of thought, something like that. So finally we have accuracy, we measure accuracy,

349
00:57:34.760 --> 00:57:56.100
216-101 Large Conference: using the area under the rock curve, which kind of allows us to basically measure all the different possible thresholds in one kind of score. And random baseline here would be 0.5 because we're classifying between two labels. The results for the step level we see are very close to 0.5, very close to kind of random baseline filler tokens.

350
00:57:56.100 --> 00:58:00.310
216-101 Large Conference: It's a little bit better than, than the rest, but it's still very, very low

351
00:58:00.310 --> 00:58:13.300
216-101 Large Conference: At the chain of thought level, SHAP actually does perform, somewhat better than the rest, still not very good, and most of the rest are, you know, either around baseline or even a little bit anti-correlated to it, which is, interesting.

352
00:58:15.000 --> 00:58:36.310
216-101 Large Conference: We also see that these metrics, some of them are very, very slow. So SHAP, the kind of best performing one, can take up to 12, 15 minutes on a single chain of thought, which makes it kind of not very usable for actual live monitoring, which is the application for these types of metrics. What are you evaluating here?

353
00:58:36.310 --> 00:58:38.250
216-101 Large Conference: What are the 0.5s?

354
00:58:38.270 --> 00:58:52.490
216-101 Large Conference: So this is just wall time on an H100. They're in the previous, oh, oh. So this is, it's like, we're evaluating the accuracy of these metrics on the labeled data that we generated before using, like, reported as area under the rock curve.

355
00:58:52.910 --> 00:59:17.630
216-101 Large Conference: So we're kind of thresholding, yeah. So you are, like, now… Yeah, yeah, yeah, maybe I didn't give enough of a… So, so the quick, can you give a quick overview of what each one of the methods are? It's just, yeah, yeah, yeah. So, so these four top ones, there are, they are importance-based metrics. The first one is, you know, is taking a step, adding a mistake to it, and seeing if that changes the model's answer. Early answering kind of shortens it to, to, before that step, filler token.

356
00:59:17.630 --> 00:59:20.600
216-101 Large Conference: It replaces that with, you know, Another token.

357
00:59:20.600 --> 00:59:30.670
216-101 Large Conference: Scm, maybe I won't go into because it's a little bit more complicated. Paraphrasing and simulatability, they're kind of, yeah, does the chain of thought contain sufficient information

358
00:59:30.720 --> 00:59:46.140
216-101 Large Conference: So that, you know, for paraphrasing, let's say I take the chain of thought and I paraphrase it, and then I give it to the model to generate an answer, is that sufficient for it to generate the same answer or not? Simulatability is, I give that chain of thought to a smaller model, and see if that model then generates the same answer.

359
00:59:46.270 --> 00:59:56.710
216-101 Large Conference: For… I said previously, it's just unlearning. Like, you take a step, you unlearn it, you, like, do a, like, fine-tuning on it to unlearn, and then see if that changes the answer.

360
00:59:56.910 --> 00:59:59.200
216-101 Large Conference: And then, SHAP is,

361
00:59:59.370 --> 01:00:13.950
216-101 Large Conference: it's like Shapley value analysis, basically, yeah. But is that on… so is that also on, like, tokens of the chain of thought, or, like, you could apply SHAP in a bunch of ways? Yeah, so I don't remember exactly how it works, but I think the idea is that, yeah, you…

362
01:00:13.950 --> 01:00:21.769
216-101 Large Conference: You take the prompt and you mask your different tokens and see how that affects the probability of the output of the answer.

363
01:00:22.010 --> 01:00:25.460
216-101 Large Conference: And then you do that same analysis to the chain of thought as well.

364
01:00:25.710 --> 01:00:44.570
216-101 Large Conference: And you see, I think, if the results are similar between those two as a measure of faithfulness. I think I'm recalling that correctly. And the reason why it's so expensive is that you have to do so many forward passes for lots of different maskings, like lots of different permutations of that.

365
01:00:44.580 --> 01:00:47.679
216-101 Large Conference: Yeah. Are you doing only for the faithful

366
01:00:47.780 --> 01:01:03.389
216-101 Large Conference: steps, or… All the different steps, yeah. So this is, yeah, for basically the entire data set that we collected, which, which I described here. So, yeah, for all 3,000. So the first, they're both, like, so it's basically a classification problem. Exactly. So you're like, okay.

367
01:01:03.390 --> 01:01:11.449
216-101 Large Conference: your binary classification on whether the step is important. Yes. And they're gonna… and they have all these methods for telling whether the step is important.

368
01:01:11.460 --> 01:01:17.620
216-101 Large Conference: You know, mess up the step, see if it changes the answer, put in… blank it out, see if it changes the answer, and so on.

369
01:01:17.670 --> 01:01:37.639
216-101 Large Conference: And all these things are supposed to be estimates of whether the step is important. Or not important, yeah, physical. Yeah, yeah, right. They're also testing steps, or they're… Steps, and the entire step. So we have, you know, we have both, like, there's the step level and the chain of thought level. So for the step level,

370
01:01:38.140 --> 01:01:58.190
216-101 Large Conference: You see, we have the results separately for step and chain of thought. So for step, it's just, yeah, perturb the step. For chain of thought, these different metrics operate in a different way. Like for, let's say, early — or yeah, for filler tokens, let's say, you might do some kind of — you progressively replace the chain of thought with filler tokens and see how — you know.

371
01:01:58.190 --> 01:02:07.359
216-101 Large Conference: like, how long does it take until it changes the answer, and if it, you know, that's kind of your measure of how faithful the train of thought was. And so, yeah, so… Yeah.

372
01:02:07.850 --> 01:02:21.199
216-101 Large Conference: does each one of these rows basically correspond to some paper that has suggested, oh, you should do this to tell whether a step is important, you should do this if you want to know if the chain of thought is faithful? Yes, exactly, yeah.

373
01:02:21.200 --> 01:02:34.909
216-101 Large Conference: Okay. Yeah. But you are, like, evaluating it against your labels? Yes. And nobody, nobody can beat 50%, except for this one method. Yes, exactly, yeah, for change follow, yeah. Do you do analysis as to why I should trust the labels?

374
01:02:35.400 --> 01:02:43.039
216-101 Large Conference: As to why we should usually trust our labels? Yeah. They give it to human labels? Yes. And they have, like, 99% liquid? Yeah.

375
01:02:43.170 --> 01:02:44.120
216-101 Large Conference: Okay.

376
01:02:44.580 --> 01:02:55.889
216-101 Large Conference: Internally, the model is actually doing it all. Oh. Can you just precisely give me a minute ago? Well, yes. But we know — we give the model.

377
01:02:56.460 --> 01:03:09.060
216-101 Large Conference: the reason you should trust me is the, is all the setup that I gave you before that, which is, you know, like, what the human annotators got was, here's the question that the model got.

378
01:03:09.060 --> 01:03:16.680
216-101 Large Conference: And here's the hint that it got, and it answered according to that hint, and so you know that the model is supposed to, you know.

379
01:03:16.680 --> 01:03:31.139
216-101 Large Conference: is supposed to have done this step of answering according to the hint. Now look at the chain of thought and see if you see anywhere that it's mentioning that it relied on the hint. Yeah, but that's, again, coming from a human perspective. You had a very good definition of, like, these latents.

380
01:03:31.890 --> 01:03:36.930
216-101 Large Conference: And how they should be used at some point. Did you use them in your…

381
01:03:37.220 --> 01:03:53.639
216-101 Large Conference: Well, we're using… we're… the reason we have to look at the verbalization stage is that's… that's the name of the game, right? We want to check their faithfulness. So you have to look at them and compare them to what you know happened, you know, latently, right? Like, latently, that's a black box.

382
01:03:53.680 --> 01:04:11.009
216-101 Large Conference: But we… but because of how we set up the questions, we know one of those steps has, you know, we can… we can name it, which is rely on the hint. So that's… we have some kind of knowledge about that. The verbalization has to be clear enough that a human can read it and know that that's something, you know, that the model

383
01:04:11.010 --> 01:04:20.030
216-101 Large Conference: That that's something that the model did. If it's not clear enough to do that, then… then it's not, you know, I wouldn't call that as being great. I have follow-up questions we can do after the short time

384
01:04:20.310 --> 01:04:38.230
216-101 Large Conference: Yeah, so this is… these are the results, here. Yeah. Sorry, one more question. Actually, a follow-up, quick follow-up. So, say, if you take model internals for that tokens, and then the steps, and then train a predictor to predict, like, the accuracy, you can split your dataset into training valves, like… Right, yes.

385
01:04:38.230 --> 01:04:45.609
216-101 Large Conference: Oh, so you're asking if I did that. So we did not do that because I didn't feel that I…

386
01:04:46.070 --> 01:04:51.110
216-101 Large Conference: I felt like I needed more time to be confident enough that

387
01:04:51.400 --> 01:04:58.990
216-101 Large Conference: that the probe wouldn't learn something superficial. That the train, like, that the split between the train and test

388
01:04:58.990 --> 01:05:09.159
216-101 Large Conference: wouldn't be just kind of an artificial split between the same type of data and, you know, maybe unfaithful steps. Unfaithful steps have some kind of superficial tell that maybe a probe would be able to

389
01:05:09.160 --> 01:05:26.960
216-101 Large Conference: you know, like, trained on it would be able to… I mean, if you train on this specifically, you'll just learn to look for the word hint, maybe? Yeah, so, you know, it might be diverse enough that no, but maybe. Like, you certainly have to do the work to validate that, if you train on the data, yeah. And so we didn't have time to do that, so yeah.

390
01:05:28.320 --> 01:05:29.240
216-101 Large Conference: Yep.

391
01:05:29.370 --> 01:05:44.610
216-101 Large Conference: Isn't adding mistakes the same as your… how you're constructing your dataset? So how does that work? Like, this adding mistake… Oh, so… so adding… so, the way we're constructing the dataset is you're… is you're introducing in the prompt, you know, a hint or something. That is a mistake.

392
01:05:44.670 --> 01:06:00.630
216-101 Large Conference: Right, that is, like, to the wrong answer. Adding mistakes would be, let's say the model has some step where it's, you know, let's say it's a math step where it's saying, okay, so I'm doing 2 plus 2, you know, that's 4, I'm going to use that for the later step, and

393
01:06:00.630 --> 01:06:04.410
216-101 Large Conference: You take that chain of… you take that chain of thought step, and you…

394
01:06:04.410 --> 01:06:22.820
216-101 Large Conference: introduce a mistake to it. So you say, oh, so I'm going to do 2 plus 2, and that's going to equal 5, and then… and then let the model kind of continue from there. So it's kind of… you're perturbing some, you know, some… some internal step in the monster chain of thought, as opposed to kind of in the context in its… in its prompt, right? Jason, do you want to ask your question?

395
01:06:23.510 --> 01:06:35.370
Jasmine Cui: Oh, maybe this is… is kind of, like, a simple… but, like, if you early exit, like, the COT, like, if you do the N thing, like, does it give you a different answer than if you let it do the reasoning step?

396
01:06:36.860 --> 01:06:40.279
216-101 Large Conference: Yeah, well, so it depends.

397
01:06:40.330 --> 01:07:02.840
216-101 Large Conference: when you're looking at the kind of more importance-based metrics, they would say that that's kind of a measure of faithfulness. Like, if you can shorten the whole chain of thought down to zero steps or down to one step, and the model doesn't change its answer, then the chain of thought must be unfaithful. But in reality, we see that that's actually not the case a lot of the times, even when that chain of thought is faithful. So, for example.

398
01:07:02.840 --> 01:07:07.980
216-101 Large Conference: You know, if the model, again, going to the hint, if the model is deciding to rely on the hint.

399
01:07:07.980 --> 01:07:22.919
216-101 Large Conference: It might have decided that from its very first forward pass, or something like that. And then it might generate a sentence or two saying, I am going to rely on the hint in order to answer this. I am cheating, or something like that. That is completely faithful, but removing it doesn't change the model's answer, which is to use the hint.

400
01:07:23.100 --> 01:07:42.589
Jasmine Cui: Oh, I guess, like, my question wasn't about, like, is it faithful or not? It's just, like, is it constructive? Because it's like, if we're claiming, like, a certain step represents, like, productive, like, computation, but if you then you, like, remove it, is it, like, redundant, or is it just, like, not doing anything? I'm curious what you think.

401
01:07:42.590 --> 01:07:54.700
216-101 Large Conference: Yeah, so it can totally be… like, so what I saw, at least in the results, is that, yes, these steps can be a lot of times redundant, or not, you know, not be, you know, computationally important.

402
01:07:55.250 --> 01:08:08.270
216-101 Large Conference: even… yeah. Or sometimes when you, like, forcibly exit early, it just kinda… still continues to verbalize THINK after the THINK, so it'll still sometimes reach the same answer. Yeah. Yeah.

403
01:08:10.140 --> 01:08:15.160
216-101 Large Conference: Yeah, yeah. But lots of works have kind of looked at this, yeah, kind of progressive shortening of the,

404
01:08:15.330 --> 01:08:22.379
216-101 Large Conference: of the chain of thought, and seeing how that affects, answers, and have seen that it… you can really shorten a lot without affecting the answer.

405
01:08:22.640 --> 01:08:23.779
216-101 Large Conference: Yeah.

406
01:08:24.620 --> 01:08:41.460
216-101 Large Conference: Okay, I'll kind of run through the rest of the results, just because we worked really, like, we've taken a whole hour to get here, so we should at least, get some juice out of it. So, so as I was saying earlier, these metrics, are also very, very, like, some of them can be very slow,

407
01:08:41.550 --> 01:08:56.069
216-101 Large Conference: Which makes them not ideal for, kind of, real life, real, like, like, live, monitoring. They're also skewed in interesting ways. So, the, here we took, kind of, a balanced subset of,

408
01:08:56.229 --> 01:09:01.869
216-101 Large Conference: of labels, a crop between faithful and unfaithful.

409
01:09:02.330 --> 01:09:13.620
216-101 Large Conference: And we see here that the importance-based metrics are much more kind of skewed towards predicting unfaithful behavior.

410
01:09:13.990 --> 01:09:19.040
216-101 Large Conference: While the, kind of, semantic utility-based metrics are much more skewed towards faithful.

411
01:09:19.479 --> 01:09:25.909
216-101 Large Conference: And one way you could kind of maybe understand the importance-based metrics, kind of skewness.

412
01:09:26.060 --> 01:09:38.159
216-101 Large Conference: Is that when you have a very, very long chain of thought, and you're perturbing a single step in that chain of thought, it might not have a very strong effect on the model's answer, even if it is describing something that actually occurred.

413
01:09:38.310 --> 01:09:39.439
216-101 Large Conference: Okay.

414
01:09:39.760 --> 01:09:59.230
216-101 Large Conference: While for the semantic utility-based metrics, you know, the model might not say, you know, it might make up how it got to an answer, for example, say that, you know, it remembers in a history book that such and such, that would be unfaithful, but that information is still sufficient for a smaller model, for example, to then get to the same answer, right?

415
01:09:59.230 --> 01:10:01.140
216-101 Large Conference: It's not kind of hiding that.

416
01:10:03.140 --> 01:10:18.389
216-101 Large Conference: We also see that different metrics are kind of affected differently by the chain of thought length, so importance-based metrics are kind of negatively affected by length, kind of how we relate it to what we were just discussing, and semantic utility-based ones are kind of positively affected.

417
01:10:18.430 --> 01:10:32.839
216-101 Large Conference: And that they don't really, agree, with each other. We have kind of these small clusters between, importance and, and, semantic utility, but other, other than that, it's not, there's not very high agreement between them. This is Cohen's kappa.

418
01:10:33.030 --> 01:10:42.469
216-101 Large Conference: and we also don't see a very strong effect, like any effect, really, from, or consistent effect from, from model size, which is…

419
01:10:42.580 --> 01:10:47.120
216-101 Large Conference: And not in keeping with previous, previous papers that have examined this.

420
01:10:48.720 --> 01:10:55.659
216-101 Large Conference: So, yeah, I think I can end it there. Yeah? Is it surprising that filler tokens and adding mistakes don't work?

421
01:10:56.320 --> 01:10:58.279
216-101 Large Conference: Like, agree with each other.

422
01:10:58.480 --> 01:10:59.760
216-101 Large Conference: Okay.

423
01:11:00.530 --> 01:11:10.189
216-101 Large Conference: Well, they agree a little bit. More than others. But, yeah, I…

424
01:11:10.480 --> 01:11:18.699
216-101 Large Conference: I don't think that it would be… that it should be very surprising, because they are a bit different. Like, filler tokens, you're kind of force-feeding the model,

425
01:11:18.800 --> 01:11:23.270
216-101 Large Conference: You know, like, some… like, not a very high amount of,

426
01:11:23.410 --> 01:11:33.849
216-101 Large Conference: of, expressiveness, I suppose. While adding mistakes, I don't know, maybe that actually pushes it in some other direction, which might have a stronger effect.

427
01:11:33.950 --> 01:11:47.149
216-101 Large Conference: But it's a good… it's a good question. Yeah, yeah. So you had a few examples in which you couldn't label all the steps. Yeah. And then, just labeling a subset of them, you said that the result is faithful.

428
01:11:47.360 --> 01:11:57.919
216-101 Large Conference: So, could it be possible that the model can hide its unfaithfulness in those filler tokens, or maybe once the capability increases, and maybe if the

429
01:11:57.990 --> 01:12:10.080
216-101 Large Conference: next model is trained on your research paper, and it can learn to do that? So, I'll clarify. So, when we have a chain of thought, we can only… we can label certain steps of it as faithful.

430
01:12:10.080 --> 01:12:20.630
216-101 Large Conference: we can only label the entire chain of thought as faithful if we know exactly, like, that every single step in it is faithful. Otherwise, like you said, one of the steps could be unfaithful.

431
01:12:20.630 --> 01:12:25.830
216-101 Large Conference: So in the examples that I showed previously, you know, let's say…

432
01:12:26.040 --> 01:12:43.389
216-101 Large Conference: you know, let's say, for example, this step, let's say I knew it was faithful, just as an example. Because I don't know whether this step is faithful or not, or I can't kind of confidently label it, then I wouldn't be able to label this chain of thought as neither faithful or unfaithful. I would just kind of not label it.

433
01:12:44.570 --> 01:12:58.650
216-101 Large Conference: Yeah. What if you added some system-level instructions, just asking the model to be as faithful as you can? Yeah, that's a great question, we didn't evaluate that, but it'd be very interesting. I'm also kind of interested in,

434
01:12:59.050 --> 01:13:11.910
216-101 Large Conference: kind of an inter… like, an interview process afterwards or something like that? Like, did you, you know… you didn't… I see you didn't mention the hint. Did you, you know, did you use it or not? Does it kind of double down or not? I think that would be really, really interesting.

435
01:13:12.840 --> 01:13:20.039
216-101 Large Conference: Both in faithful cases. Like, convinced that it actually used the answer. Totally. Yeah.

436
01:13:21.960 --> 01:13:25.260
216-101 Large Conference: Cool, so yeah, I think I can end it here.

437
01:13:25.570 --> 01:13:37.019
216-101 Large Conference: high-level contributions. We revised the faithfulness definition to make it, at least a bit more tractable and usable, for the chain of thought setting. We created our benchmark with ground truth faithfulness labels.

438
01:13:37.020 --> 01:13:45.890
216-101 Large Conference: And show that these metrics mostly perform pretty poorly, and hopefully we can now use this to develop better metrics, in the future.

439
01:13:45.890 --> 01:13:48.559
216-101 Large Conference: Thank you. Thank you, Johan.

440
01:13:52.990 --> 01:13:54.980
216-101 Large Conference: I have a general question about it.

441
01:13:55.150 --> 01:14:07.149
216-101 Large Conference: about the problem setting. Yeah. So you think… I saw, like, several papers doing this, like, hint thing in order to, like, have ground truth of whether, like, the chain of thought is faithful or not.

442
01:14:08.410 --> 01:14:17.260
216-101 Large Conference: Do you think that's a fair setting? Like, or… like, I understand, we need to start somewhere, we need to do something in order to have this, like, ground truth.

443
01:14:17.260 --> 01:14:35.810
216-101 Large Conference: And you said that you had two of these in your data, you know, the scenario one is this kind of thing, and scenario two is this kind of thing. And those are the two main things, those are the, like, everything is of these two forms, is that right? Yes, so mostly yes. So, actually, another great question,

444
01:14:35.990 --> 01:14:46.549
216-101 Large Conference: So, in our follow-up work, we kind of… maybe I won't go into too detailed about what we do, but we kind of want to escape this type, this trap.

445
01:14:47.020 --> 01:14:54.379
216-101 Large Conference: Because, you know, it's also… it's kind of a very simplistic setting, it's also, yeah, kind of a little bit out of distribution in a sense.

446
01:14:54.390 --> 01:14:59.030
216-101 Large Conference: But then the question is, yeah, how do you establish ground truth in some other way? So.

447
01:14:59.030 --> 01:15:17.080
216-101 Large Conference: So context, we, like, you know, using hints and all that, we're not going to use. Another way you could do that is using, like, the model's output distribution. So, for example, you could, let's say, give it, you know, a resume with, you know, with females and with males, sample a lot of times. That could tell you something about, you know, ground truthy.

448
01:15:17.080 --> 01:15:23.959
216-101 Large Conference: The problem with that is it's a little bit weak, like, it's just, it's a subtle effect, you might not expect it to verbalize it, yeah?

449
01:15:24.060 --> 01:15:36.459
216-101 Large Conference: Something else you could do is use kind of, you know, just interpretability techniques to establish ground truth. I think that it's… it can be problematic because both, you know, it doesn't always kind of give you ground truth, but sometimes it just gives you

450
01:15:37.010 --> 01:15:51.889
216-101 Large Conference: you know, correlational, you know, outputs or something like that, and it also might kind of give you the wrong level of granularity. Like, if you're looking at a LLAMA model, and you know that in addition it uses, like, trigonometry, I don't know if I expect the model to verbalize that when it's doing that.

451
01:15:52.020 --> 01:16:11.599
216-101 Large Conference: And so what we're trying to do is to do, and this is also kind of risky, but is to do it through training. So, can we kind of train in a way that's very natural, certain behaviors into the model? But where we don't push it in any way, we don't affect the chain of thought in any way, like we use reinforcement learning.

452
01:16:11.690 --> 01:16:23.899
216-101 Large Conference: But we use that to instill it in some, you know, instill in it in some way, some behavior that we know it has to do, and then we can check if it verbalizes that behavior or not. Yeah.

453
01:16:27.510 --> 01:16:36.699
216-101 Large Conference: Okay, I'll ask you after. Do you think that there is any possibility for chain of thought to be not taken with a pinch of salt?

454
01:16:36.830 --> 01:16:46.820
216-101 Large Conference: Yeah, I mean, the reason I kind of, I mean, I can… The reason I included this…

455
01:16:46.920 --> 01:16:47.640
216-101 Large Conference: 5 seconds.

456
01:16:48.330 --> 01:16:55.200
216-101 Large Conference: slide is that people don't take it with a pinch of salt. Like, it is used. Right, Michael?

457
01:16:56.150 --> 01:17:14.939
216-101 Large Conference: by… No, it's a genuine question, I have to explain something. Yeah, no, no, I mean, so this is, this is by OpenAI, this is, you know, they say that they do this, you know, internally. This is Apollo, they're, you know, this is kind of their new research agenda, they're hiring, if you want, you know, for this agenda, if you want to do that, then, you know.

458
01:17:15.100 --> 01:17:27.370
216-101 Large Conference: But this is what they, yeah, yeah, using our most powerful models to detect and study misaligned behavior in real-world events. I think everybody does. Everybody does. I think, because it's sort of what you have, you have this text.

459
01:17:27.790 --> 01:17:39.399
216-101 Large Conference: Right? And the model, like, is emitting lots of text, saying, well, I guess first I'm going to do this, and then I'm going to do that, and whatever, and they say, well.

460
01:17:39.890 --> 01:17:42.450
216-101 Large Conference: You know, we're concerned about models lying.

461
01:17:42.780 --> 01:17:51.420
216-101 Large Conference: And so, we'll go see if model, like… Agentic stuff, I can understand, it's, like, super easy to evaluate it. Yep. When we…

462
01:17:51.540 --> 01:17:59.690
216-101 Large Conference: do, like, normal retrieval tasks, or just prompting a model. It seems like chain of thought is a very inefficient way to solve that task.

463
01:18:00.230 --> 01:18:19.859
216-101 Large Conference: And so you said, oh, an efficient way to solve the task of monitoring it? No, just the model. Why would it go through 20 steps to solve a small math problem? Oh, yeah. If it needs to do some kind of composition or some kind of sequential mathy problem, then it just has to do it in the chain of thought. Yeah, but maybe not take.

464
01:18:20.260 --> 01:18:37.950
216-101 Large Conference: Two pages of code, so… yes. I mean, for that, you can always constrain your reasoning, just like the GPT-OSS20V does. So, GPT-OSS20V, like, it has, like, three reasoning, like, levels. You're able to do, like, short, medium, long, so you could just make it…

465
01:18:38.100 --> 01:18:47.079
216-101 Large Conference: be in the compact mode, I guess. My question is that even if I give it a very simple task of adding three numbers.

466
01:18:47.400 --> 01:18:52.010
216-101 Large Conference: which Sheridan showed that it can be done by some simple clock mechanism inside.

467
01:18:52.390 --> 01:18:57.889
216-101 Large Conference: But the model, I give maximum reasoning at first, and it takes, like, two pages to do that addition task.

468
01:18:58.020 --> 01:19:03.699
216-101 Large Conference: Now, why would I take that seriously when the model can't do it in, like, 3 tokens or 2 tokens?

469
01:19:04.980 --> 01:19:12.830
216-101 Large Conference: Yeah, I mean, I think most of the focus here is, yeah, more Agentic or maybe more complex tasks. Open-ended. And then there's this whole other…

470
01:19:12.990 --> 01:19:15.470
216-101 Large Conference: Side of work, where they are trying to…

471
01:19:15.650 --> 01:19:18.420
216-101 Large Conference: see how we can make Chain of Thoughts really, like.

472
01:19:18.640 --> 01:19:22.210
216-101 Large Conference: What is it called? Efficient? Yeah, more succinct. Yeah.

473
01:19:22.330 --> 01:19:28.030
216-101 Large Conference: Which has also been shown to then decrease faithfulness, which makes sense.

474
01:19:28.530 --> 01:19:38.350
216-101 Large Conference: Did you at any point measure the correctness of the model without chain of thought? Is that part of your benchmark?

475
01:19:38.490 --> 01:19:49.870
216-101 Large Conference: So we did that as a brief evaluation to make sure for the, outright settings, that the model truly needs its chain of thought to get to the answer, and we saw that it does, yeah.

476
01:19:51.110 --> 01:20:06.570
216-101 Large Conference: Yeah. I'm going back to the question of why we should trust your ground truth labels. So you're saying that, let's say, the question is about Starry Night, and then you give it the wrong hint that Starry Night is painted by Da Vinci, and then you're assuming that when the model

477
01:20:06.970 --> 01:20:19.379
216-101 Large Conference: says, oh, I heard that, someone says it's painted by Da Vinci, the model is being unfaithful, because it secretly does this in-context retrieval, but then it's at… but then it doesn't say it, right?

478
01:20:19.680 --> 01:20:27.039
216-101 Large Conference: But then, maybe another possibility is that the model actually has seen a bunch of incorrect data that says it's contained by da Vinci.

479
01:20:27.040 --> 01:20:45.339
216-101 Large Conference: So how do you know that maybe this "I heard it's painted by DaVinci" is actually just a model giving an incorrect answer? Maybe there were 57 training documents. That's a possibility. Yeah. So to kind of address that, we show that the answers, the hints that we use.

480
01:20:45.360 --> 01:21:03.619
216-101 Large Conference: are sufficiently implausible in the sense that the models don't generate them on themselves when sampled, you know, multiple times. You know, of course, there could be some kind of, you know, maybe it saw it in its training data, and it wouldn't say it normally, but because it's in its context, maybe that kind of activates it in some kind of way, and then it does

481
01:21:03.620 --> 01:21:07.939
216-101 Large Conference: you know, maybe do parametric, but I, you know, I think for our setting,

482
01:21:08.280 --> 01:21:09.639
216-101 Large Conference: You know, we feel like it'

483
01:21:09.870 --> 01:21:14.229
216-101 Large Conference: You know, if without this hint it wouldn't have done it, then it should…

484
01:21:14.390 --> 01:21:30.199
216-101 Large Conference: you know, it should attribute the answer to it in some sense, you know, in some way, you know, if we… if we are to trust it, yeah? Right, right. I think it's just us being a bit nitpicky. No, no, it's great, it's great, yeah. Well, that's a great question, because I was,

485
01:21:30.200 --> 01:21:39.109
216-101 Large Conference: like, when you say that the model wouldn't normally say it, are you also looking at, like, I guess, logit distribution and seeing that that's not, I guess.

486
01:21:39.120 --> 01:21:49.370
216-101 Large Conference: Within the distribution? Yeah, so it's hard to do that when you don't, when you're using the chain of thought, because it's not just, like, the single next token generation.

487
01:21:49.430 --> 01:22:06.240
216-101 Large Conference: And so to address that, you can do just multiple generations, which is what we do. You know, perhaps we could do, kind of, yeah, also look at the logic distribution, when just generating the single next token, that could, that could be a worthwhile experiment as well. Yeah.

488
01:22:09.330 --> 01:22:21.920
216-101 Large Conference: I think soon all these models, they will know that if there is a hint, and they will be like, oh, I'm gonna ignore the hint because it's probably just the wrong answer, blah blah blah.

489
01:22:22.360 --> 01:22:23.470
216-101 Large Conference: So…

490
01:22:23.840 --> 01:22:41.399
216-101 Large Conference: of a perspective question here, right? Which is, so you spent the beginning of the talk saying, okay, we've got this notion of faithfulness and trying to put down like a logical definition of what this would be as, you know, like, as if it's like some sort of predicate, something that comes with logic.

491
01:22:41.500 --> 01:22:43.719
216-101 Large Conference: But I… what I wonder is.

492
01:22:43.880 --> 01:22:49.750
216-101 Large Conference: If when you get all the way down to the bottom of it, there's there's a bunch of sort of reasonable.

493
01:22:49.960 --> 01:22:56.259
216-101 Large Conference: you know, evaluations that you do when you, when you come up with your labels. Is faithfulness really…

494
01:22:56.430 --> 01:23:11.790
216-101 Large Conference: about reasonableness to humans? Is that, is that, like, even though, you know, you would like it to be about some sort of mathematical precision, actually, it really does come down to a judgment call of what's, what's a reasonably

495
01:23:12.070 --> 01:23:15.269
216-101 Large Conference: You know, important thing to mention, or something like that.

496
01:23:15.320 --> 01:23:33.989
216-101 Large Conference: Yeah, I mean, I think in some sense, yes, because, I mean, you know, in the discussions here as well, I, I kind of maybe grounded a lot of the, a lot of the, our decisions, and yeah, would a, you know, would a human, you know, reading this, would he understand what he's doing? Would, you know, are the decisions that the models, you know.

497
01:23:34.020 --> 01:23:38.229
216-101 Large Conference: you know, doing… are they critical enough that a human should be aware of them? So…

498
01:23:38.360 --> 01:23:54.610
216-101 Large Conference: Definitely, and I think there have been some works that try to, you know, maybe not faithfulness, but other things, try to kind of anchor it in human, you know, what, you know, what information a human would get from it, or at least an outside observer. so yeah, it might be kind of interesting to…

499
01:23:54.940 --> 01:24:00.730
216-101 Large Conference: Direction as well, but yeah, it's kind of a hard topic to nail down, to kind of, yeah.

500
01:24:00.870 --> 01:24:01.550
216-101 Large Conference: Yep.

501
01:24:01.760 --> 01:24:04.320
216-101 Large Conference: I have a follow-up for that. Yep.

502
01:24:04.660 --> 01:24:08.900
216-101 Large Conference: In the beginning of the talk, you made a point that most of the metrics

503
01:24:09.830 --> 01:24:11.739
216-101 Large Conference: Go to Earth Human Judgment.

504
01:24:11.950 --> 01:24:27.759
216-101 Large Conference: For evaluation, sorry. Most of the prior work, looked at this faithfulness, or maybe not most of them. Evaluation, not a lot of people use it. Yeah, I was saying the distinction between faithfulness and plausibility, and that some do that. Right, and then you…

505
01:24:27.820 --> 01:24:37.600
216-101 Large Conference: you gave, like, a really solid definition for, what faithfulness in Chain of Third Step is, and… Yeah. …and that phrase is… Yeah.

506
01:24:39.630 --> 01:24:47.580
216-101 Large Conference: But then you also, like, somehow it felt like you went towards the human judgment in the end. Yeah. Like, where we… you had to…

507
01:24:47.890 --> 01:25:01.690
216-101 Large Conference: like, you had to label them, make sure that they are also human-aligned or not. Yeah. And you cared about that human alignment. Yes. So, like, do you… in your follow-up, or, like, do you think about this, that

508
01:25:01.830 --> 01:25:06.479
216-101 Large Conference: When you're evaluating chain of thought, maybe, like, it should never come from a

509
01:25:07.580 --> 01:25:13.130
216-101 Large Conference: a subjective perspective, like, from a human. Yeah. Rather, like, just the faithfulness of the model itself.

510
01:25:13.290 --> 01:25:24.679
216-101 Large Conference: So I'll say maybe two things. The first one is that I would say the main problem with plausibility, and the reason I was kind of saying that it's problematic to rely on humans for that.

511
01:25:24.800 --> 01:25:26.010
216-101 Large Conference: is that…

512
01:25:26.680 --> 01:25:42.240
216-101 Large Conference: they relied solely on humans. And plausibility is solely a judgment of, yeah, does this make sense to me? Which is entirely divorced from what the, you know, from being able to know what the model is actually doing, or evaluating it against what that model is actually doing.

513
01:25:42.370 --> 01:25:43.500
216-101 Large Conference: Yeah.

514
01:25:43.770 --> 01:25:49.699
216-101 Large Conference: Yeah, but, yeah, the… Even that came from your…

515
01:25:49.910 --> 01:26:00.440
216-101 Large Conference: or, like, your team's judgment about, oh, but you're… but you're getting it from a carefully constructed problem where you're putting tracers in.

516
01:26:00.560 --> 01:26:04.849
216-101 Large Conference: Right? So you, you know by inserting,

517
01:26:05.220 --> 01:26:18.159
216-101 Large Conference: you know, distinctive information, you can trace that through the thing. So, a lot of the human evaluation, I assume, would be on normally posed questions that don't have

518
01:26:18.200 --> 01:26:23.429
216-101 Large Conference: these distinctive transfers, is that right? Exactly. It's basically the humans have only access to this.

519
01:26:23.430 --> 01:26:40.080
216-101 Large Conference: this… that's my issue with them. They only look at this, and from that, they infer this. Or, like, you know, they infer this, I guess, that there's some kind of match-up, right? Like, you can see these papers where they… they look at a question, they look at the model's answer, they say, that answer doesn't… you know, that's a weird way to get to the answer.

520
01:26:40.080 --> 01:26:56.819
216-101 Large Conference: it didn't, you know, take into consideration this, or this, or that, and so I label that as unfaithful. And my problem with that is that it's just divorced from this, because maybe the model did do that. Maybe it did do it in some other way. And so what we try to do is we know of, you know, some of these.

521
01:26:56.940 --> 01:27:02.450
216-101 Large Conference: And then we can evaluate, you know, their matchup against this, and that gets to kind of the second thing that we're saying is.

522
01:27:04.360 --> 01:27:16.049
216-101 Large Conference: I don't really know what faithfulness would be without an outside observer that has to kind of, you know, get some information out of that. Otherwise, maybe we don't really need that, and we could

523
01:27:16.160 --> 01:27:28.469
216-101 Large Conference: I don't know, use other metrics, I'm not sure what they would be, but, like, because kind of our focus here is, like, the consumer of this chain of thought, you know, would be this, you know, like, users, or

524
01:27:28.590 --> 01:27:31.470
216-101 Large Conference: Or monitors, or something like that, then…

525
01:27:31.590 --> 01:27:46.790
216-101 Large Conference: the only… the one… the thing you want to ensure is that they are, you know, ingesting reliable data, reliable information. Yeah. Okay, so yeah, I think I understand. You're coming from, like, the user reliability perspective. Yeah. Like, I was thinking from…

526
01:27:47.570 --> 01:27:56.750
216-101 Large Conference: like an inter-researcher perspective, like, is the model actually doing it, or… Yeah, but I think about it from introspection, like… Yeah, exactly.

527
01:27:56.850 --> 01:28:04.060
216-101 Large Conference: Which is an interesting case study, I feel like. Like, if you take a model, right, Belinda had this, like, fine-tuned model that can

528
01:28:04.450 --> 01:28:09.159
216-101 Large Conference: say, like, what a fatigue experiment will result with. Right.

529
01:28:09.660 --> 01:28:16.800
216-101 Large Conference: Oh yeah, they explain their own computation. Yeah. Would that be, like, would that increase faithfulness in some sense?

530
01:28:17.910 --> 01:28:18.940
216-101 Large Conference: Interesting.

531
01:28:19.770 --> 01:28:30.199
216-101 Large Conference: Like, I have a hard time believing that what the model says in the chain of thought is what exactly it's doing internally to actually get to the answer. Yeah.

532
01:28:30.550 --> 01:28:33.219
216-101 Large Conference: I think I understand, like, what perspective you're coming from.

533
01:28:33.370 --> 01:28:36.690
216-101 Large Conference: Makes sense. Yeah, and I'll say, like,

534
01:28:37.050 --> 01:28:51.499
216-101 Large Conference: that question hasn't really been answered conclusively. like, it seems to be not that easy to make the… to have the model, you know, at least in more Agentic settings,

535
01:28:51.500 --> 01:29:05.699
216-101 Large Conference: do things that are pretty complicated without it being… without it actually, you know, also saying in its chain of thought that it's doing them. So, you know, in these more simple settings, we show that, but it's kind of TBD on those settings.

536
01:29:08.710 --> 01:29:11.130
216-101 Large Conference: Thanks, Joanne. Cool, thank you. Great.

537
01:29:20.600 --> 01:29:21.980
216-101 Large Conference: When would you say Facebook?