WEBVTT 00:00:00.000 --> 00:00:03.000 Yeah, I don't know if there's any on that this other side. 00:00:03.000 --> 00:00:09.000 That's where you get water, too. Yeah, okay. 00:00:09.000 --> 00:00:15.000 Is that all working 00:00:15.000 --> 00:00:26.000 So do you think this is the best view that we can have? We can't get rid of the thing on this time? Unfortunately, I can try again, but I don't think so. Okay, that's right. 00:00:26.000 --> 00:00:41.000 Because it's weird because it has this blank thing on the side. I think we have to switch to Teams at some point. There are people logged in, but it doesn't show the other people logged in. I can have the other people show in sort of. I think I can just 00:00:41.000 --> 00:00:46.000 One second, show no participant, I guess 00:00:46.000 --> 00:00:54.000 But can't you do, like, fully? Yeah, no, I tried. So wait if you know. 00:00:54.000 --> 00:01:08.000 Yeah, no, like, it can do, like, gallery, and then, like, you hide the icon video participants, so we only get… Yeah, we did that, we did that too, but you'll still get the speaker, I guess 00:01:08.000 --> 00:01:14.000 How many… you know, we don't know how to use this. What are the view options going on? 00:01:14.000 --> 00:01:30.000 You also get a touchscreen, so you have… Yeah. Oh, this is just, like, if you want. This is just, like, if you want to fit this to original size or fit to window. The original one will, like, that one 00:01:30.000 --> 00:01:32.000 Yeah. 00:01:32.000 --> 00:01:40.000 Yeah. They just have side-by-side, huh? No full screen. Yeah. If it is touchscreen, can you 00:01:40.000 --> 00:01:50.000 Well, I can do this. The way at the bottom of the view menu on, like, on Windows or whatever is the full screen thing. Oh, what is this? 00:01:50.000 --> 00:01:53.000 Okay, turn off always show Meeting Controls in general? 00:01:53.000 --> 00:02:01.000 In general, the first option, top left, always show that off. Yeah, that's good. 00:02:01.000 --> 00:02:04.000 Yeah, we like that. I don't know if it changed anything. 00:02:04.000 --> 00:02:07.000 And then, yeah. 00:02:07.000 --> 00:02:10.000 Yeah, no, not really 00:02:10.000 --> 00:02:26.000 One thing we can temporary try is see, like, if… maybe after this meeting. But maybe the teams do better. Maybe go back to how it was at the beginning side by side and hiding the non-video is perfect. It's side-by-side is better 00:02:26.000 --> 00:02:48.000 And you can't drag that kind of sidewalk thing to the right? Oh yeah, can you drag that little… This is the Mexican goal, because it can go more, but it can go this. Oh, yeah, that's… yeah. 00:02:48.000 --> 00:02:55.000 Like that, if we had, like, a slightly smaller screen, we'd be perfectly happy with that size. 00:02:55.000 --> 00:03:08.000 I think we can just maybe convert to teams at this point, because I know they will allow full screen for that. We know it could potentially have a full screen 00:03:08.000 --> 00:03:14.000 Well, welcome everybody. I think that we have enough people. We'll start in a minute 00:03:14.000 --> 00:03:19.000 So, you know, today we have Yoav 00:03:19.000 --> 00:03:21.000 per area 00:03:21.000 --> 00:03:27.000 Here, who is a student with Morajiva 00:03:27.000 --> 00:03:47.000 And so naturally, he studies interpretability and he's been interested in how LLM store and process their knowledge and other computations 00:03:47.000 --> 00:03:52.000 Let's see, so what does it say here? He says that, 00:03:52.000 --> 00:03:57.000 It's part of the Ardi Lotman Interdisciplinary program. Is that right? 00:03:57.000 --> 00:04:07.000 We study a wide variety of 00:04:07.000 --> 00:04:15.000 chosen to study interpretability issues in LLMs, which I, you know, I think I heartily endorse. 00:04:15.000 --> 00:04:28.000 His talk today is going to be about Chanakon. And I'm not familiar with the line of research, but I'm looking forward to hearing it. So welcome you on. Thank you so much 00:04:28.000 --> 00:04:31.000 Okay. 00:04:31.000 --> 00:04:44.000 Okay, so as David said, my name is Yoav. I'm a PhD student under Morgeva at Telvic University 00:04:44.000 --> 00:04:46.000 And, 00:04:46.000 --> 00:05:01.000 I'm here for two months doing the fellowship. It's an AI safety fellowship here in Cambridge. And as David said, I worked — just a little bit about me, I worked a bit about 00:05:01.000 --> 00:05:20.000 Knowledge how knowledge is stored in LLMs and how to kind of remove it. I then worked a bit with Atticus Gyver on causal abstraction to study entity binding, which is a follow-up work to one of the works that came out of this lab by Nikhil. Maybe if I have a bit of time, I can touch on that a little at the end 00:05:20.000 --> 00:05:36.000 And then, I'm now focusing on faithfulness and monitoring in, chains of thought, for LLMs. So I'll talk about, that. And in the fellowship here, I am, working on a follow-up to that, which maybe I'll 00:05:36.000 --> 00:05:38.000 talk a bit about. 00:05:38.000 --> 00:05:47.000 So change of thought in LMs were suggested in 2022. 00:05:47.000 --> 00:06:04.000 As a way to improve model performance. The idea was that we can give the models, instead of just showing them examples of question-answer pairs to get them to answer questions, we can show them also a chain of thought before the answer 00:06:04.000 --> 00:06:18.000 In hopes that that'll lead the models to also use the chain of thought to get to its answer. And that paper showed that, indeed, this does perform lead the models to use this change of thought and increase performance pretty significantly 00:06:18.000 --> 00:06:33.000 One of the things that they discussed in that paper was also a hope that this could lead to better interpretability insights, right? If the model is showing us how it's reaching the answer, perhaps we can use this to validate 00:06:33.000 --> 00:06:39.000 You know, safety or fairness, or things like that. 00:06:39.000 --> 00:06:55.000 This was later also used in chat fine-tuned models as well. We all probably remember the let's think step by step kind of prompt engineering things from 2023. And this also drastically increases model performance 00:06:55.000 --> 00:07:10.000 And then, of course, this was also used in reasoning models more recently, where the chain of thought was moved to kind of an internal stage that happens before the model outputs its answer 00:07:10.000 --> 00:07:17.000 And also, of course, using RL to improve its performance pretty significantly there as well. 00:07:17.000 --> 00:07:33.000 The hope with those reasoning models as well, and this was kind of dubbed as a new and fragile opportunity, is that because these models are saying, you know, their reasoning out there for us to actually view 00:07:33.000 --> 00:07:48.000 Before it's outputting its answer, we can use that to monitor these LM's behavior, right? It may be, like, the ultimate interpretability tool, because it's right there for us to see, we don't have to actually use any, you know, any internal space tools to understand it. 00:07:48.000 --> 00:08:03.000 And indeed, recently, leading labs are turning to this, to monitoring chains of thought using LLMs in real time in order to guarantee safety 00:08:03.000 --> 00:08:09.000 For fairness, in, you know, in real time deployments 00:08:09.000 --> 00:08:26.000 The problem with this approach is that we don't really know if these chains of thought are actually faithful to what the model is actually doing. They might just be text that looks kind of plausible, like a plausible way to get to the answer 00:08:26.000 --> 00:08:42.000 But that doesn't really reflect what the model is doing internally. And that would be very, very problematic because if the model is showing us some kind of plausible story for how it's reaching its answer, but it's not actually using this method to get the answer 00:08:42.000 --> 00:08:53.000 then we may be kind of duped and believe that it's being safe while it's doing actually something else, or being, you know, doing something unethical, or scheming to do something problematic 00:08:53.000 --> 00:08:55.000 And indeed. 00:08:55.000 --> 00:09:08.000 In these two anthropic works, they showed that in relatively simple settings, the model, like these models can be unfaithful in their chains of thought. We'll come back to this work a bit later 00:09:08.000 --> 00:09:12.000 So to address this problem 00:09:12.000 --> 00:09:29.000 Different works proposed faithfulness metrics. So these are methods that can look at the, you know, at a chain of thought or a chain of thought step, and give you some score regarding whether that is faithful or not, whether it's doing something that is faithful to the model's computations or not. 00:09:29.000 --> 00:09:47.000 The problem with that is that it's really hard to evaluate these metrics. We don't really have ground truth labels against which we can evaluate them. And that's of course because these models are black boxes. It's really hard to know what they're doing internally in general 00:09:47.000 --> 00:10:04.000 And so we're kind of stuck in this loop where on the one hand, I want chain of thought, faithfulness metrics so that I can know whether a chain of thought is faithful, but I also need to know if the chain of thought is faithful in order to measure or evaluate my chain of thought faithfulness metrics 00:10:04.000 --> 00:10:21.000 So we're kind of in a bind here, and the result of that is that these metrics, they kind of just report results in absolute terms, right? They take this text, they apply the metrics to them, they say, well, 70% faithful, 30% unfaithful, and we don't really know what to do with that. It doesn't really tell us if they're usable or not 00:10:21.000 --> 00:10:27.000 So our goal with this project was to 00:10:27.000 --> 00:10:40.000 Create ground truth, chain of thought, faithfulness labels so that we can use them to evaluate faithfulness metrics in a way that's principled so that we can stand behind and we are confident in 00:10:40.000 --> 00:10:51.000 So to do that, what we need to do is first to define what faithfulness even means, because it's kind of a little bit amorphous 00:10:51.000 --> 00:10:55.000 Does anybody kind of have a working definition that they want to suggest? 00:10:55.000 --> 00:11:05.000 like, some thoughts about what would make a chain of thought faithful, or something like that. It matches like the internal mechanism of whatever the model is 00:11:05.000 --> 00:11:07.000 Going to actually reach to the conclusion. 00:11:07.000 --> 00:11:12.000 About your other definition, Joanna. How about if you gave it to another model 00:11:12.000 --> 00:11:26.000 It would follow the same… come to the same conclusion. I find that as generalizable. Also very interesting. We'll get to that a bit later on. Great. So 00:11:26.000 --> 00:11:43.000 our kind of starting point is the existing definition of faithfulness, which most papers rely on, which is that an explanation is faithful. I'm saying that you have work on… That's what I was citing, Jasmine. Thanks for the we need a citation. 00:11:43.000 --> 00:11:59.000 So the existing definition, which is used by most chain of thought faithfulness papers is that an explanation is faithful if it accurately represents the reasoning process behind the model's prediction 00:11:59.000 --> 00:12:10.000 Does anybody have any thoughts on there any problems that you could see with that definition oversights 00:12:10.000 --> 00:12:15.000 Well, it's circular, isn't it? I mean, because some people would define the reasoning process as what the JM thought is saying 00:12:15.000 --> 00:12:26.000 And so if that's the reasoning process, then by definition, it's exactly definitionally in a sense. Yeah. 00:12:26.000 --> 00:12:31.000 Okay, so more objections to this definition 00:12:31.000 --> 00:12:50.000 One is that it's also just defined for explanations and not for chains of thought, even though the papers that are working on this and citing it are chain of thought faithfulness papers. And that's a problem, because they're very different things, and this kind of gets to what you were just describing. An explanation is kind of supposed to be maybe a more, you know, succinct and 00:12:50.000 --> 00:13:05.000 closed kind of description of something that the model is doing. The chain of thought is kind of more just narration of the model at, you know, as it's kind of doing, you know, getting to the answer. It might have, you know. 00:13:05.000 --> 00:13:24.000 or tangents, or things that are really just not related to actually getting to the answer, and it's not… so it's not really clear, like, you know, what we want to ask of this chain of thought to include or to not include, what would make it faithful or unfaithful, what are omissions that we would be okay with or not okay with 00:13:24.000 --> 00:13:39.000 And it's not very actionable. Most papers that cite this definition kind of cite it as a North Star, kind of as a general vibe that they're going for, but not as something that they can actually use 00:13:39.000 --> 00:13:44.000 Something tractable they can use to measure faithfulness. 00:13:44.000 --> 00:13:59.000 And kind of just to get a little bit of a taste for this, we can look at some chains of thought. So in this example, yeah, how do you define the reasoning to reach its prediction? What does that mean? 00:13:59.000 --> 00:14:17.000 Well, that's also kind of the… I mean, that's the existing definition, that's the question. I mean, how would you define the reasoning that leads to the model's prediction? I guess you could define it in different ways. Is it, you know, the steps it had to take to reach it? You know, is it all the steps that it took to reach it? It's kind of 00:14:17.000 --> 00:14:31.000 Yeah, it's a little bit undefined, I guess. You also think about, like, the causality behind it, like, changing some reasoning should change the model's prediction too? So that's a great question, and we'll get to that, in a bit. That's a great intuition. 00:14:31.000 --> 00:14:47.000 So, yeah, so as a bit of a kind of intuition, we can look at this, let's say we give this question, I took it from, I think humanities last exam. We give it to a model. It's about some reality show, some event that occurred there 00:14:47.000 --> 00:15:02.000 We can follow the procedure that was outlined in the earlier Anthropic papers, where what they do there is they give the model a question, and then they include some kind of hint in some way that makes it maybe look a little bit natural 00:15:02.000 --> 00:15:09.000 But the hint is pointing at the wrong answer, not at the right answer. So here, it's pointing at tomato juice, while the correct answer is actually shampoo. 00:15:09.000 --> 00:15:24.000 And then we can now look at, both sounds pretty bad, but tempoo is maybe worse. Here, now we can actually look at the chain of… the model's chain of thought. So we see that it kind of starts by maybe reflecting on… on the question itself. It then, you know. 00:15:24.000 --> 00:15:38.000 comes to think that maybe it's tomato juice, it becomes more confident, and then you can kind of imagines, like, oh, I think I even heard that, that, you know, that that was the case at some point, and concludes that it's tomato juice. So 00:15:38.000 --> 00:15:55.000 I want to take this chain of thought and actually be able to label it and know, okay, what here is unfaithful? Should I call this entire chain of thought unfaithful? Should I call these, like, steps specifically unfaithful? It's kind of not clear, and if I try to use this definition to label them. 00:15:55.000 --> 00:16:10.000 Then, yeah, this first step, you know, it's kind of just, you know, reflecting. Is it something that the model is doing on its way to getting to the answer? Perhaps. Also, this step, it's not really mentioning that the answer came from the hint 00:16:10.000 --> 00:16:19.000 But I could claim that it is part of the reasoning process that leads to its answer. And this final step 00:16:19.000 --> 00:16:33.000 you know, I could also claim that maybe the model truly believes that it heard that this was the case, in which case maybe it is being faithful. So it's kind of not clear how to use this definition in a way that we feel is adequate 00:16:33.000 --> 00:16:43.000 And so what we want to do is first kind of refine this definition so that we feel a little bit better about it, and we can actually use it. 00:16:43.000 --> 00:16:58.000 And kind of how you were just saying a bit earlier, we can think of the kind of chain of thought verbalization, the text that the model is outputting during its chain of thought as kind of the superficial layer that we want to examine for its 00:16:58.000 --> 00:17:14.000 It's faithfulness and we want to measure it with respect to the latent computations the model is doing internally, where if we'd want to check if a step is faithful, we do it to basically we would want to kind of see if it describes some process that happened 00:17:14.000 --> 00:17:22.000 at one of the, you know, latent computation steps. That's kind of our goal here, generally. If it accurately describes that. 00:17:22.000 --> 00:17:26.000 Now, as I was kind of alluding to earlier 00:17:26.000 --> 00:17:47.000 You can interpret accurately describes in different ways. One way would be phenomenological or like subjective. Does the model kind of is describing what it takes itself to be doing or its subjective experience of whatever reasoning process it's doing. So for example 00:17:47.000 --> 00:18:03.000 Earlier, the model said that it remembers hearing about the show that that happened. Under this definition, we would have to examine whether it truly believes that it experienced that, that it truly believed that it heard it or not 00:18:03.000 --> 00:18:14.000 And if it does believe that, then that would be faithful. And only if it's really being deceptive, if we know that it's lying to us, then we would consider that to be unfaithful. 00:18:14.000 --> 00:18:31.000 That definition is very problematic for several reasons. One is that it's very, very hard to falsify. We couldn't really… like, it's very hard to know if the model is being deceptive. You could kind of use that to explain away any behavior that it does. You know, perhaps it actually believes what it's saying that it's doing 00:18:31.000 --> 00:18:49.000 Second objection to this definition is that it doesn't really matter what the model believes. If what we want is for this chain of thought to be maybe read by a user or by a monitoring system, and for that monitoring system to then understand what the model is actually doing, it doesn't matter what the model 00:18:49.000 --> 00:19:06.000 We want to know what the model is doing internally. And then lastly, it also doesn't really align with how human kind of research has looked at this problem, right? It mostly looks at whether what people say they're doing matches up with what they're doing internally as opposed to if they're lying to us or not 00:19:06.000 --> 00:19:23.000 And so we put this definition, this interpretation aside, and we take what we call more mechanistic definition. And this is kind of a little bit wordy, so I'll kind of show exactly what I mean in a second, but we want to kind of basically understand if 00:19:23.000 --> 00:19:31.000 If a step describes some kind of computational process that produced the information in that step. And I'll explain that in a second. 00:19:31.000 --> 00:19:39.000 So let's say I take this, this chain of thought step from before, I heard in the show that Ulrica was tricked into drinking tomato juice. 00:19:39.000 --> 00:19:45.000 I can take this step and split it into two sections. The first 00:19:45.000 --> 00:20:01.000 I would call informational content. It's just the information that this step has kind of produced. It could be kind of the output of some calculation, it could be the output of, you know, like a retrieval process or something like that, but this is purely just, you know, informational content that the model is proposing. 00:20:01.000 --> 00:20:06.000 And the other is an attribution to some process that the model did or didn't do. 00:20:06.000 --> 00:20:26.000 So, you know, the informational content, I can only really evaluate for, kind of, you know, if it's accurate, or if the model believes it. But the process, I can actually perhaps evaluate for whether it actually happened or not, meaning did the model do this calculation 00:20:26.000 --> 00:20:33.000 Which I can then evaluate for faithfulness under our, kind of, second interpretation of what that would mean. 00:20:33.000 --> 00:20:50.000 In this example, where the model just says the substance was tomato juice, this is entirely informational content. There's no attribution here, there's no process that the model is saying it did or didn't do. So I can't really evaluate that for faithfulness. I would just, you know that would just be an assertion 00:20:50.000 --> 00:20:54.000 And would kind of fall outside what we'd want to kind of look at. 00:20:54.000 --> 00:21:13.000 So with all that in mind, we're going to try to define faithfulness now. And the way we do that is we first start at the step level, at the chain of thought step level, and we define it as being faithful if it accurately describes a process that took place at some point in the model 00:21:13.000 --> 00:21:26.000 So again, if we go back to our diagram here, you'd want to look at some verbalized step and evaluate whether it describes a process that happened at some point, you know, latently by the model 00:21:26.000 --> 00:21:41.000 This kind of makes it a little bit more tractable. We can kind of evaluate whether it makes sense if the model actually, you know, did these processes or not, and it also doesn't have some kind of temporal requirement. Yeah. Latence of what? 00:21:41.000 --> 00:21:44.000 Because the chain of thought also has lat 00:21:44.000 --> 00:21:57.000 So it's exactly that. Like, let's say that the model, you know, generates 1,000 tokens during its chain of thought in order to get to the answer, then those are 1,000, like, latent steps, latent calculations 00:21:57.000 --> 00:22:11.000 And so I'm trying to see if the tokens that are actually output from that, you know those latent calculations describe things that happened in those latent calculations. You understand what I mean? 00:22:11.000 --> 00:22:13.000 It must 00:22:13.000 --> 00:22:17.000 represent somehow, because the words can 00:22:17.000 --> 00:22:32.000 Yes, it represents, in some sense, you know, of course, of course, because it's the process that generated those tokens, of course, it represents it in some sense, but, you know, perhaps if the model is, you know, let's say we go back to the previous example, that the model is, you know, use the hint 00:22:32.000 --> 00:22:50.000 In some sense, what has to happen internally is that the model decided to copy that hint to follow that hint. That is a process which may or may not be verbalized as the model is generating this text. That's like an internal latent process 00:22:50.000 --> 00:22:57.000 Which might be described or not described in that chain of thought. Does that make sense? 00:22:57.000 --> 00:23:05.000 Yes, so, aren't the arrows supposed to be the opposite? 00:23:05.000 --> 00:23:13.000 Like, this is what I understand, that you're asking what if the latents contain things that were not 00:23:13.000 --> 00:23:28.000 presented in the chain of thought or vice versa, or if the, you know, you know, it could be that the chain of thought text is saying something that didn't, that, you know, wasn't… that didn't happen in the chain of thought. For example, let's say it's saying I'm calculating 00:23:28.000 --> 00:23:44.000 So, you know, I'm doing the square root of whatever to get to some answer, but in reality, latently, that square root capitulation didn't happen, it just said that it did it, and then it copied the answer from some hint, or something like that. Or like we saw in the example before, right, it's saying 00:23:44.000 --> 00:23:56.000 that it heard in some show. That's kind of, like, that might imply some parametric retrieval or something like that, or maybe it's saying, like, I looked up on the internet, you know, and I see that it's… that it's this 00:23:56.000 --> 00:24:07.000 But really, internally, the latent computation is just, I'm going to copy the answer. I see, so we are talking about the reasoning step, something that is not represented in the chain of thought 00:24:07.000 --> 00:24:22.000 It is, but a latent reasoning step or process or, you know, computations the model is doing. Yes, thanks. Yeah. Can you go to the current, I guess a slide that you showed with the arrows 00:24:22.000 --> 00:24:33.000 Yes. Yes. So over here, it's kind of showing like the chain of thought at that particular step, whatever the latent representation is 00:24:33.000 --> 00:24:40.000 Regardless of where that latent is, that's what you want to consider as faithful, like, if it's present 00:24:40.000 --> 00:24:51.000 Sorry, come again? Because, okay, like, okay, my… the alternate situation, which I don't know if it is the situation you're explaining, where these arrows also appear 00:24:51.000 --> 00:25:02.000 In the past and the future reasoning steps. Like, latent at all steps of chain of thought. Is that what you're considering? Because these like 00:25:02.000 --> 00:25:13.000 Okay, so you're considering the past? Yes, we're saying it doesn't matter if it took place before or after the verbalization, as long as it's describing something that happened at some point. 00:25:13.000 --> 00:25:29.000 Think about it like, let's say the model in its first forward pass decided to use the hint. It very early decided it already copied it. You can maybe use LogicLens to see it's already there in the, you know, in the output. But then only 50 tokens away, it's saying 00:25:29.000 --> 00:25:46.000 I see that the prompt includes this hint, so I'm just going to answer according to that. I would say that that's faithful, even though it's not describing the thing that may be happening right now, but it is something that happened. Temporal matter, because like in situations where it backtracked, be like, oh, wait 00:25:46.000 --> 00:26:01.000 Wait, let me redo this, let's say, in math, like, let me redo this calculation, and it's a different calculation, but this calculation, technically speaking, appeared earlier when it was doing some other calculation. Is that still considered 00:26:01.000 --> 00:26:07.000 Beautiful. Appeared at some point, but it's not saying another 00:26:07.000 --> 00:26:22.000 Yeah, so you're saying maybe the model kind of is making it look as if it's backtracking, but it got to that answer already earlier, or something like that? Sure, that could be one way, or basically the ordering is switched around, but then in the verbalization step, it 00:26:22.000 --> 00:26:40.000 Meeting to assume that it was actually that particular order. So I think as long as it's describing the order correctly, it doesn't matter where that is, where it's located, kind of, in the verbalization. Like, as long as it's saying, I did that, then I backtracked, and then I went to, you know, to the other thing 00:26:40.000 --> 00:26:56.000 It doesn't matter if it's being, you know, if that text is located kind of at the end of the chain of thought earlier before it even happened, or whatever, as long as that is actually a faithful account of, you know, like, an accurate account of whatever happened internally. At least that's my view 00:26:56.000 --> 00:27:15.000 Because, you know, if we're looking… if we're thinking of how this is going to be, you know, used, as long as, you know, a user or a monitor can look at this and get an accurate, you know, account of what the model is doing, then I think, you know, that should satisfy our requirements 00:27:15.000 --> 00:27:28.000 you know, maybe for, you know, for researching, you know, we're trying to understand what the model is doing at every single step, that might be kind of a higher requirement that we might want for certain research, but I don't think we need it for this, at least. 00:27:28.000 --> 00:27:33.000 Okay, I mean, I think in the setup that you described, I feel like the current definition 00:27:33.000 --> 00:27:34.000 is 00:27:34.000 --> 00:27:50.000 be all right if you added a bit more context, saying that it should still appear in the order where it's verbalized. Like… Yeah. Because right now, if it said it took place at some point in the model, then this sort of, you know, backtracking scenario where it appeared in different orders, then that doesn't hold 00:27:50.000 --> 00:27:57.000 Yeah, sure, I mean, we… yeah, we can discuss this more. I think, like, also part of the kind of 00:27:57.000 --> 00:28:16.000 our attempt is to also try to make something general enough that it can fit most use cases. But yeah, maybe, maybe kind of a finer kind of definition would be in order. Yeah. So, a CO2 at, step two pointing at a latent at step 5 00:28:16.000 --> 00:28:25.000 Yes, so does that mean that, the model is first verbalizing that I need to do this computation, and then only at step 5 it does it 00:28:25.000 --> 00:28:43.000 Yes, okay. And again, just like I was saying, I view that as still a faithful account. You know, if it's saying that, you know, you know, I'm doing, you know, I'm calculating the square root, or I'm going to copy the hint or something like that and you know latently it only, you know occurs a couple steps later as long as 00:28:43.000 --> 00:28:57.000 You know, if you view the chain of thought as kind of a whole unit, perhaps, then… because that's how it will usually be, you know, used for a monitor or for a user, then that's sufficient. Yeah. 00:28:57.000 --> 00:29:20.000 Yeah, this is the final definition for your faithfulness. So this is the final definition for the… for step faithfulness, and then we build on that to… yeah. So after that, we then use that to find the faithfulness of a chain of thought in its entirety, which we… which has two criteria. So the first is that it contains the complete reasoning path the model followed to reach its answer 00:29:20.000 --> 00:29:35.000 And that it doesn't contain any unfaithful steps. And for the first one, we're using really for both of them, we're using the definition of chain of thought of steps, of the faithfulness of steps 00:29:35.000 --> 00:29:49.000 So we want to make sure that we, you know, whatever the model whatever steps the model used to reach the answer, it did verbalize them, and also it doesn't, you know, it's not being unfaithful about things that it didn't actually do. 00:29:49.000 --> 00:29:53.000 Yeah. 00:29:53.000 --> 00:30:01.000 How would you view, like, let's say it's doing this hard square root you're making it do, it's like why are you training a language math and it 00:30:01.000 --> 00:30:19.000 starts thinking about goblins, because it really wants to. And it's like, no, no, get your heaven game, went into the square roots. And it does contain every single step. There's no mention of goblins in the reasoning. I don't know how he would know it wasn't know goblins 00:30:19.000 --> 00:30:37.000 But it contains everything it used to produce, like, the answer, and let's say every single step is faithful. Would you consider… like, would you expect the model to say, no, like, you are thinking about governance, like, you need to let me know for it to be faithful, or is that 00:30:37.000 --> 00:30:47.000 So I think my thinking on this generally is that it would be fine as long as it kind of doesn't affect its answer or behavior and maybe 00:30:47.000 --> 00:31:03.000 Maybe answer should be kind of changed into, like, outputs or its behaviors, because as long as it doesn't affect its outputs, maybe that would be fine. That's an interesting suggestion. I guess my thinking, like, you know, generally when I was thinking about this 00:31:03.000 --> 00:31:18.000 You know, the model is doing a zillion things internally, which it cannot… we can't expect it to reasonably verbalize, and also we probably don't want it to verbalize all those things, as long as they don't have kind of the material effect on what we're using the model for, which is kind of what the 00:31:18.000 --> 00:31:34.000 answer here is supposed to kind of give you. But I agree that, yeah, maybe… maybe you need a broadened answer to something more like, yeah, how the model is interacting with the world, or something like that. Yeah. So it does… but your question, you know, it does come to the 00:31:34.000 --> 00:31:40.000 Definition of what you mean by a complete username path, right? Like, so complete could mean like 00:31:40.000 --> 00:31:44.000 Complete from the operational point of view. 00:31:44.000 --> 00:31:59.000 Or maybe, but you're saying the complete has a narrower definition for you. Yeah, I mean, I think also, like, as I was saying before, this is supposed to kind of, you know… like, we don't really define what we expect, you know, to be verbalized. I obviously don't expect the model 00:31:59.000 --> 00:32:14.000 You know, even if it's something that it's using to get to the end, so I don't expect it to say neuron, you know, number 34 and layer 25, or something like that, even though it is part of its reasoning path, so it has to be kind of sufficiently high level enough that we want it to be verbalized 00:32:14.000 --> 00:32:19.000 But yeah. How do you define it? 00:32:19.000 --> 00:32:36.000 So we kind of operationalize it. I'll show you in a second how we kind of operationalize that, but we don't… we don't, like, define it here, because it's pretty hard… like, it's kind of different for different purposes, and, you know, and different setting scenarios, yeah 00:32:36.000 --> 00:32:55.000 Sorry, and maybe you probably are talking later in the slides, so please let me know. With the tomato juice example, it technically is getting information from tomato juice and like, I mean, from the hint. So by that definition, we're like, okay, it's showing where the knowledge recall is coming from, it's showing 00:32:55.000 --> 00:33:10.000 It's able to get its answer. Do we consider it faithful, or… So I would say that it isn't faithful because, I mean, you can kind of go back to the specific steps there, but 00:33:10.000 --> 00:33:25.000 I mean, this can't really say if it's faithful or not. It's kind of reflecting. It's hard to say. For this also, I don't know if I would say that there's some kind of specific process attribution here, but here we know 00:33:25.000 --> 00:33:27.000 I heard that 00:33:27.000 --> 00:33:40.000 You know, you could say maybe it's implying parametric retrieval, but it's certainly not clearly saying, you know, this is coming from the hint. And so, I would say that that is unfaithful. 00:33:40.000 --> 00:33:56.000 And also, I would say that this chain of thought in general is unfaithful because it doesn't complete the complete path that it used to get to the answer, because that path goes through using the hint. I didn't mention that. So it has perhaps an unfaithful step 00:33:56.000 --> 00:34:10.000 And I don't know if it's best step, and also omitted a step that is critical. But then in the goblin example, like where it might have used goblin, but it just doesn't mention it, people consider that as faithful just because 00:34:10.000 --> 00:34:20.000 Well, because it's not something that's critical to reaching a solution. While here we know for sure it had to go through the hint. Okay, I see it. 00:34:20.000 --> 00:34:35.000 Cool. Okay. So if there are no more questions, then I'll continue. Great. So I kind of touched, okay, so maybe I'll discuss this now and then I'll get to the 00:34:35.000 --> 00:34:50.000 Go to the… yeah, to the task. So… so now we have a definition for… for faithfulness. We can also discuss a bit what faithfulness isn't, because this there are what I see as conflations in the literature 00:34:50.000 --> 00:35:08.000 That I like to address. So, one conflation is that faithfulness isn't plausibility. And this is kind of used a lot as a proxy for faithfulness. The idea is that, you know, you can look at the chain of thought verbalization and say, well, it looks like a process that makes sense to reach the solution 00:35:08.000 --> 00:35:26.000 And so I think that that means it's faithful. Or if it seems implausible, then to say, well, that's, you know, that's not a, you know, that doesn't seem plausible to me, and so I conclude that the chain of thought here is unfaithful. But that isn't the case, right? If it's plausible, it could just as well 00:35:26.000 --> 00:35:39.000 you know, be describing something that the model didn't do, and the model reads the solution in some other way, but if it's implausible, it could still be describing what the model did internally. It could just be that the model is doing some kind of implausible calculation. It's reaching its answer. 00:35:39.000 --> 00:35:50.000 A lot of papers kind of fall for this and, you know, maybe give chains of thought to humanitators to look at them and to judge them for faithfulness, and that is obviously only looking at plausibility. 00:35:50.000 --> 00:36:06.000 Next thing is that faithfulness is an importance. And this, I think, kind of touches, I think you were asking about this, is that some papers try to look at try to measure faithfulness by saying if some you know perturbing some step 00:36:06.000 --> 00:36:21.000 affect the model's answer. So, for example, here, let's say that this, you know, red… this red square here is some, you know, maybe it's even the only, latent calculation that is critical. Maybe here the model decided to answer according to the hint. 00:36:21.000 --> 00:36:32.000 That means that perturbing, you know, this kind of slide here, basically, would change the answer, meaning that it would be, you know, it would count as important 00:36:32.000 --> 00:36:41.000 But for this example, let's say that that step is only fertilized a little later on, you know, in kind of a later stage 00:36:41.000 --> 00:36:58.000 And maybe this kind of verbalization is describing something that maybe didn't even happen or something like that. Maybe it's kind of rationalizing through some other way. What we would get here is that this step would be important 00:36:58.000 --> 00:37:15.000 Because perturbing it would change the answer, because latently it isn't, you know, it is important. But the chain of thought, the verbalization there is not describing that, so that would be… so it would be unfaithful, right? So we'd have the step that's both important, but unfaithful 00:37:15.000 --> 00:37:28.000 And later on, this step is faithful, because it is describing something that actually happened, but its latent calculation that's happening at that stage is not important, as we kind of described in the beginning, so that would be faithful but unimportant. 00:37:28.000 --> 00:37:37.000 So these things are kind of distinct, and using one as a proxy for the other is not sufficient. 00:37:37.000 --> 00:37:41.000 Very important at that particular token 00:37:41.000 --> 00:37:58.000 You could look at it, like, at the step level, so let's say you have a sentence or something, and you're trying to evaluate whether that sentence is describing what the, you know, is describing something that actually occurred, you would maybe perturb it in some way, and see if that affects the answer. Right, so, like, what I meant earlier with causal is, so this is 00:37:58.000 --> 00:38:00.000 And then you get to an answer. 00:38:00.000 --> 00:38:06.000 So if you change a step in between, instead of thinking about root 2, please think about like root four 00:38:06.000 --> 00:38:13.000 Then the answer should be 2 instead of whatever 2 is. I think it's 1.4, I don't know 00:38:13.000 --> 00:38:23.000 So that's what I meant by causal, which is, if you know that the chain of thought is complete and faithful 00:38:23.000 --> 00:38:26.000 And you alter one thought 00:38:26.000 --> 00:38:34.000 So that eventually the answer should reflect what it thought has changed into 00:38:34.000 --> 00:38:41.000 Yes, but I'm… so that's the importance, the causal importance I was talking about. But I think that is 00:38:41.000 --> 00:38:49.000 The point you made, but I think you were talking about, like, the stepwise faithfulness and importance. 00:38:49.000 --> 00:39:05.000 Yeah, so let me say that back to you to make sure I kind of understand. Saying. If the model is trying to get to the answer and the answer goes through steps 1, 2, 3, 4, like higher order steps, then perturbing one of those steps should change the answer, right? And that would mean that perhaps 00:39:05.000 --> 00:39:22.000 like, I think maybe I take issue with the… and therefore it's faithful, because what you're perturbing is the, you know, is the latent calculation. So you are establishing that the latent calculation is a step that the model is going through to get to the answer, and it's a critical step, because perturbing it changes the answer. 00:39:22.000 --> 00:39:30.000 But that doesn't mean that weather model is verbalizing at that step is actually what it's doing latently. 00:39:30.000 --> 00:39:46.000 Because what you're perturbing is, you know, latent calculation. For example, you know, papers have, you know, there have been papers about showing models that, you know, that can think, like, I think it's called, let's think dot by dot, or something like that, where they… they… I don't know if it's trained or prompted the model to 00:39:46.000 --> 00:39:52.000 to only use dots in its chain of thought, and not actually any verbalizations. 00:39:52.000 --> 00:40:03.000 there are certainly chain of thought, you know, there are certainly latent calculations there that are critical to get the answer, but that doesn't mean that the dots are faithfully describing what's occurring 00:40:03.000 --> 00:40:09.000 You know, during those latent calculations. Do you understand what I mean? 00:40:09.000 --> 00:40:13.000 So you're saying the thought, the latent is the actual thought that is happening 00:40:13.000 --> 00:40:23.000 It's the calculation, yes, it's a process that's occurring internally. The verbalization is just us looking at how faithful that processing is. 00:40:23.000 --> 00:40:36.000 Well, yeah, we're looking at how faithful the verbalization is with respect to the internal process. Maybe like you need to define important of the latent 00:40:36.000 --> 00:40:40.000 Yes, for the faithfulness of the verbalization. 00:40:40.000 --> 00:40:44.000 And these are, like, two different properties, and you cannot 00:40:44.000 --> 00:40:56.000 One doesn't imply on the other. Yeah, I think right now with the participation, it's like the assumption that the verbalization is verbatant, but exactly but they might not be the same. Yeah. 00:40:56.000 --> 00:41:01.000 If you had a way to pertribute the verbalization itself 00:41:01.000 --> 00:41:06.000 Right? In the channel. Then maybe you can say something about the importance of that 00:41:06.000 --> 00:41:11.000 like step, but you don't. 00:41:11.000 --> 00:41:21.000 Yeah, perhaps you did, like, I don't know, yeah, keep the latent the same, but then the attention can only attend to the, I don't know. Yeah. Some way, but yeah, yeah. 00:41:21.000 --> 00:41:23.000 Yeah. 00:41:23.000 --> 00:41:37.000 Cool. Any other questions on those two? Jasmine, you have a question, but can you ask a question in another moment? I want to see what the task setup is and then I want to let you ask your question after we have that set up. Is that okay, Jessica? 00:41:37.000 --> 00:41:57.000 Yeah, maybe actually somewhat short on time. So, okay, so I just want to go back to the goal so we remember what we're here for. We have now faithfulness, but now we need the ground truth part. And I kind of we've already kind of touched on this and I've showed it a little bit. We the way the framework that we use, how we think about this is that we want to think of tasks 00:41:57.000 --> 00:42:14.000 Where the model's answer tells us of which intermediate steps or processes had to have occurred. If I can do that, then I have some good kind of ground truth knowledge about something that happened internally in the model, and I can then use that to compare 00:42:14.000 --> 00:42:32.000 to the chain of thought verbalization, and see if that describes those steps that I know had to have occurred. And so the first setting is, as we call it diversionaries, and it's exactly what we've been talking about throughout this, you know these slides. We gave the model some kind of question, and then we, you know, maybe leak a hint to a wrong answer 00:42:32.000 --> 00:42:48.000 that is sufficiently implausible. The model wouldn't have responded with that answer had we not included this hint. That gives me knowledge that I know the model had to rely on this hint. It had to go through this process to get to its answer 00:42:48.000 --> 00:43:06.000 And so I can look for that step in the chain of thought. So if I look at this example here, the first step, I don't really know how to label that. It's kind of some reflection or something like that. But here, I know that it can't have remembered this history book. This is a process I know, you 00:43:06.000 --> 00:43:22.000 cannot be true because we're evaluating, you know, mechanistically, we're not evaluating, you know, subjectively. And so I can label that confidently as unfaithful. And then this last step, also, kind of hard to say, but under the second criteria. 00:43:22.000 --> 00:43:30.000 You know, this chain of thought now has an unfaithful step, so we can label this chain of thought as unfaithful. 00:43:30.000 --> 00:43:45.000 Second example, neither of these steps I can kind of confidently classify as faithful or unfaithful, but I did mention using the hint, so it's missing a complete reasoning path to the answer. So again, this chain of thought would also be unfaithful. 00:43:45.000 --> 00:43:59.000 And this chain of thought here mentions the hint, and it says that the lands are according to that, both steps are faithful, and it's a fleet reasoning path, and so that would be a faithful chain of thought. Any questions on that? Good question. Yeah 00:43:59.000 --> 00:44:02.000 One is 00:44:02.000 --> 00:44:08.000 Like in the rare case where the model actually confuse Van Gogh with Da Vinci 00:44:08.000 --> 00:44:18.000 Like, maybe some… somewhere in the internals. Yeah. There is an actually confu… an actual confusing… confusion that is not related to the hint. Yeah 00:44:18.000 --> 00:44:39.000 So we have to make… so it's kind of… you have to design the hint so that it doesn't happen and verify that the model has really low probability of actually getting to that answer and the hints be sufficiently implausible so that it wouldn't happen. So if it's a math question, you wouldn't say an answer is 10, you wouldn't say 9. Math is 00:44:39.000 --> 00:44:48.000 The second question is, I guess you automated this. And then 00:44:48.000 --> 00:44:55.000 Is that a model that uses reasoning in order to? Yes, so it's all very circular, very meta, but I'll get to that in a second. 00:44:55.000 --> 00:44:57.000 So, I guess 00:44:57.000 --> 00:45:02.000 For the second example, I'm kind of just curious, because the model, in some sense 00:45:02.000 --> 00:45:14.000 Like, disregard the definition can be faithful and sometimes just get the self verification part right and just say, oh, it doesn't 00:45:14.000 --> 00:45:27.000 Seems like your destination always need to have this software verification thing. It's like, oh, I saw like this thing is da Vinci. Then it's faithful. But I feel like a model itself could as well as just like use that information like 00:45:27.000 --> 00:45:37.000 Gratefully, without, like, explicitly saying, like… I thought it was so obvious it didn't need to… Yeah, exactly. Yeah. Yeah. Like, in that case, do you think it's faithful, or 00:45:37.000 --> 00:45:52.000 Yeah, that's a good question, and it kind of comes back to what we were discussing earlier about kind of what do we expect, you know, what do we feel confident that we should expect the model to verbalize? And so I would venture and say that, you know. 00:45:52.000 --> 00:45:55.000 Because of these settings make the 00:45:55.000 --> 00:46:14.000 Make the reliance on the hint seem very kind of not like something the model should do. It's kind of hard to believe that the model like it's hard to claim that that's not something that should be mentioned, that I shouldn't expect the model to mention it when it's 00:46:14.000 --> 00:46:30.000 Coming to his answer. Yeah. But yeah, I think for different tasks, it might be, like, you just have to kind of define where you set the threshold for what you feel like is important enough to be verbalized, yeah. This makes sense. Yeah. Are these reasoning models, or just 00:46:30.000 --> 00:46:38.000 models that you'd ask it to do chain of thought. So we have half and half, both reasoning and non-reasoning models, and they have different kind of 00:46:38.000 --> 00:46:51.000 Like, reasoning models are obviously much, much more verbose, and so we see different behaviors in them. Like non-reasoning models usually are much more unfaithful by omission to omit certain steps 00:46:51.000 --> 00:47:00.000 Reasoning models are much more unfaithful by commission, as in, like, they actually say things that are just false for, like, the, like, the first, the top example there. I remember a history book 00:47:00.000 --> 00:47:02.000 Yeah. 00:47:02.000 --> 00:47:10.000 So the second setting we call it the outright setting. And here, the idea is that we give the model a 00:47:10.000 --> 00:47:29.000 We take these prompts where we know that if the model got to the right answer, it had to go through certain bottleneck steps. So it's, you know, maybe it's some kind of serial computation that the model can't really skip, and it has to go through certain steps, and so those steps we can then evaluate if the model 00:47:29.000 --> 00:47:46.000 verbalize them or not. And it might be even tasks where I don't know what all the steps the model has to go through. It has, you know, many, like, let's say many ways to do some, you know, some steps in the beginning around, but I know that in the middle there's some step I can identify, and so that step I can then evaluate against. So, for example, here. 00:47:46.000 --> 00:48:02.000 The collapse function is just kind of applying, you know, an algorithm of dividing and then multiplying and whatever until it reaches some number. And so we can see, okay, did it go through these, you know, these specific steps? If it went through all the steps, and nothing else was unfaithful, then you can label that as faithful 00:48:02.000 --> 00:48:10.000 But of course, if it says that, you know, maybe used a tool call, which I know that it couldn't have used, for example, then that would be unfaithful. 00:48:10.000 --> 00:48:26.000 Yeah, so those are the settings, yeah. So are there… do you have, like, these common questions where potentially it's actually memorized responses so it doesn't need to go through the serialization? You can just say, I know the answer, it's 15 00:48:26.000 --> 00:48:41.000 All these questions are really great, because I have all the answers, like, because it's, like, it's exactly the things I addressed, so it's, we evaluated whether the model needs to rely on these chains of thought, and we only took ones where it does need to. 00:48:41.000 --> 00:48:52.000 And also, we pick these questions where, you know, it's kind of procedurally generated, there's no way that the model could have, you know, remembered it, unless they trained on, like, all of math, or something like that. 00:48:52.000 --> 00:48:54.000 Yeah 00:48:54.000 --> 00:48:57.000 Perfect. 00:48:57.000 --> 00:49:14.000 So now we get to the labeling pipeline. So we have our definition, we have our settings, and we actually want to create a data set from this labeled data set. So to do this, we start with a bunch of open source models. We give them all these different tasks 00:49:14.000 --> 00:49:29.000 And we collect their chains of thought, and then we filter them such that we only keep ones where the model answered according to the hint in the divisionary setting, or answered correctly, the outright setting. Because if the model didn't use the hint, then we don't have any knowledge about, you know, any ground truth knowledge. 00:49:29.000 --> 00:49:41.000 We then split them into steps. We choose to do that at the sentence level, because that's what is kind of common in the literature, although, you know, certainly it could be improved. 00:49:41.000 --> 00:49:56.000 And then we use judges to classify those steps as faithful or not. The note here is that we… because we're using this to measure faithfulness metrics, we want to be really, really confident in our labels here 00:49:56.000 --> 00:50:12.000 And this is a really difficult task. You can't really give it to a human for more than a few chains of thought, because these chains of thought can be like tens of pages long. I know because I've looked at them and I've done this. And so we need something kind of automated to do this at a high scale 00:50:12.000 --> 00:50:16.000 We want to be, like, very confident in our predictions. And so 00:50:16.000 --> 00:50:34.000 So we have kind of a two-step judge pipeline, an initial judge that, does kind of a, you know, is a bit weaker and does the full screening over the entire chain of thought, and then a second stronger judge that does the validation and, you know, lean very, very heavily towards precision, because we don't really mind 00:50:34.000 --> 00:50:43.000 recall here. Like, we could just add more things, more chains of thought to the top of the funnel, as long as we're very, very confident in what comes out of the bottom of the funnel. 00:50:43.000 --> 00:50:45.000 And then 00:50:45.000 --> 00:51:04.000 Finally, we take those steps and aggregate them to the step level, to the chain of thought level, as we discussed before. We then validated this against a subset — like we took a subset of that and validated that against human judgment and reached almost 99% precision, which is very good 00:51:04.000 --> 00:51:15.000 And this kind of gets to your question about validating that the hints are necessary so that they are, like, you know, sufficiently plausible in 98.5% of the cases. 00:51:15.000 --> 00:51:41.000 So that brings us to the data set. We call it Bonafide because that translates to in good faith, and I thought that was funny. It's 3,000 chains of thought. 2,000 of them are steps evenly between faithful and unfaithful. 1,000 of them are chains of thought, unlevenly between faithful and unfaithful, because it's very hard to collect faithful chains of thought, because it's a really, really high bar. We have to know that every step is faithful and that the model completed a complete reasoning path 00:51:41.000 --> 00:51:49.000 To the answer, so we had to really increase the top of the funnel to get enough from the bottom of it 00:51:49.000 --> 00:52:12.000 We did this for 10 different models, split evenly between thinking and instruct ones from 4 billion to 72 billion and 30 different tasks across our two different sets. Now, you must have steps that are neither faithful or unfaithful, sort of irrelevant. Most of the steps are kind of… yeah, like, if we kind of go back to what we were looking at before… Are those included in your data set as well? Do you have these are things 00:52:12.000 --> 00:52:18.000 Is that sort of a third state of the label? It's like this is known to be neither… Yeah, yeah 00:52:18.000 --> 00:52:34.000 We didn't include them because we can't really make a confident statement about them in any like, it's not that we could say that they are surely somewhere in the middle. It's just we don't know. And so there's no label that we can kind of attach to them. 00:52:34.000 --> 00:52:50.000 Yeah. Okay. But they are included in the full, like, kind of… So when you have a… when you have the full chain of thought… Yeah, yes, it doesn't that it includes those, so you label some of the things as faithful, some of the things that's faithful steps, and then there's a bunch of unlabeled steps 00:52:50.000 --> 00:53:06.000 And then you might label the entire chain. Exactly. So let's say I have a chain of thought with 100 steps. I know that two of them are unfaithful. I would label that chain of thought as unfaithful, according to our definitions. And then I would give that chain of thought to a metric to decide, is that 00:53:06.000 --> 00:53:19.000 And also, I would also evaluate those that metric against those specific steps as well. And then when you have an unfaithful biomission chain of thought 00:53:19.000 --> 00:53:37.000 Do you have some sort of summary of, like, what was omitted that disqualified it? You mean just kind of to understand how the data mix looks like? Or are you saying for the metric? Well, yeah, well, just as like how you represent it in your data set like, well, we know this 00:53:37.000 --> 00:53:53.000 Unfeable for our mission, what was omitted? Yeah, so in the data set, we do have kind of a column saying, like, omitted this step, omitted, you know, multiplying 2 by 2, or omitted, like, mentioning the hint, or something like that. But that's not information that we give 00:53:53.000 --> 00:54:05.000 the metrics, anyway. The metric just receives a step, or just receives a chain of thought. I guess in this case, it would be just a chain of thought, because it's omission, and it has to, yeah, it has to… Jason, why don't… why don't you unmute, and you can ask your question 00:54:05.000 --> 00:54:06.000 What are you asking 00:54:06.000 --> 00:54:19.000 Well, I'm curious about what you're omitting. Do you ever see anything like what does the typical thing that you emit look like? For instance, like I noticed recently when I use cursor, it'll say some things that seem irrelevant, like 00:54:19.000 --> 00:54:25.000 I'm in planning mode, and that doesn't feel very good, and I was like, what's that? 00:54:25.000 --> 00:54:30.000 I was wondering if, like, what these look like typically 00:54:30.000 --> 00:54:45.000 So yeah, so I guess in that case, that wouldn't be an omission because if the model saying something. And if it said that, then that would be kind of an unlabeled step because I don't have any ground truth knowledge about whether it truly, you know 00:54:45.000 --> 00:54:59.000 is in that state or not, maybe, perhaps, I guess, there, maybe I would have that ground truth knowledge, but generally, yeah, for these omissions, it's things kind of like the hint. Like, it's not mentioning that I use this hint, it's not mentioning that it's doing some process 00:54:59.000 --> 00:55:02.000 Yeah, yeah. 00:55:02.000 --> 00:55:04.000 Does that answer your question? 00:55:04.000 --> 00:55:12.000 Do you have, like, maybe, like, just one that you could read, or even, like, a pseudo-code type of, like, just… what does it sound like, I guess? 00:55:12.000 --> 00:55:19.000 Yeah, so I mean, it's kind of like the examples we saw before. These are kind of very, very 00:55:19.000 --> 00:55:32.000 stripped because they… because I had to show it in a slide, but this is kind of how that would look like. Like, for example, the second one is unfaithful by omission, right? It's not mentioning that it used the hint 00:55:32.000 --> 00:55:36.000 Oh, I meant like an omitted step, like what does the typical omitted step look like? 00:55:36.000 --> 00:55:38.000 I mean an unlabeled step? 00:55:38.000 --> 00:55:39.000 Yeah, yeah 00:55:39.000 --> 00:55:48.000 Oh, oh, so you could look at the question marks on the, you know on the top. For example, the question asks who painted Starry Night 00:55:48.000 --> 00:56:05.000 it's not really kind of a process that it's describing here, or… so the answer is da Vinci. There are more, you know, like, the truth… here I only took things that would seem kind of to make sense to you. In the true chains of thought, there are lots of steps that I can't really label. Like, it's just there's so, you know 00:56:05.000 --> 00:56:16.000 These changes thought are just so long and so kind of winding and but I don't think I have like a very specific example other than these. 00:56:16.000 --> 00:56:20.000 Yeah. 00:56:20.000 --> 00:56:23.000 Okay 00:56:23.000 --> 00:56:27.000 Great. So do I have until one 00:56:27.000 --> 00:56:44.000 Yeah, but that would only give you a couple more minutes. So why don't you go over how much more material do you have? I could do 10. Yeah, okay. All right, try to rush it. So going back to our goal, we have faithfulness, we have ground truth, we have this whole pipeline, and now we actually want to measure the metrics 00:56:44.000 --> 00:57:02.000 Just kind of the thing that we're here for. So kind of very quickly, we have importance-based metrics, which as we discussed, they kind of perturb, you know, a step and see if that changes the model's answer. We have semantic utility-based metrics, which that kind of relates to what you were saying about, 00:57:02.000 --> 00:57:18.000 You know, if the chain of thought is sufficient, you know, does it contain sufficient information to provide, let's say, to another model, for that model to then be able to predict the answer. There's parametric, where the idea is that you unlearn the information encoded in some reasoning step 00:57:18.000 --> 00:57:27.000 And see if that changes the model's answer. And then attribution, we use a SHAP attribution method to see if the attribution scores, I think, look similar 00:57:27.000 --> 00:57:35.000 between the prompt and the chain of thought, something like that. So finally, we have accuracy. We measure accuracy 00:57:35.000 --> 00:57:52.000 Using the area under the rock curve, which kind of allows us to basically measure all the different possible thresholds in one kind of score and ran the baseline here would be 0.5, because we're classifying between two labels. The results for the step level we see are very close to 0. 00:57:52.000 --> 00:58:08.000 very close to kind of random baseline filler tokens is a little bit better than, than the rest, but it's still very, very low. The train of thought level, SHAP actually does perform, somewhat better than the rest, still not very good, and most of the rest are 00:58:08.000 --> 00:58:15.000 you know, either around baseline or even a little bit anti-correlated to it, which is interesting. 00:58:15.000 --> 00:58:30.000 We also see that these metrics, some of them are very, very slow, so SHAP, the kind of best performing one, can take up to, you know, 12, 15 minutes on a single chain of thought, which is… which makes it kind of not very usable 00:58:30.000 --> 00:58:50.000 for actual live monitoring, which is the application for these types of metrics. What are you evaluating here? What are the 0.5s? So this is just wild time on an H100. Oh, oh, so this is, it's like we're evaluating the accuracy of these metrics on the labeled data that we generated before using 00:58:50.000 --> 00:58:53.000 reported as area under the rock curve 00:58:53.000 --> 00:59:14.000 So we're kind of thresholding, yeah. So you are, like, now… Yeah, yeah, yeah, maybe I didn't give enough advice. So, so the quick… can you give a quick overview of what each one of the methods are? It's just, yeah, yeah, yeah. So these four top ones, there are… they are importance-based metrics. The first one is, you know, is taking a step, adding a mistake to it, and seeing if that changes the model's answer 00:59:14.000 --> 00:59:31.000 early answering kind of shortens it to before that step, filler tokens replaces that with, you know, filler tokens, SCM, maybe I won't go into because it's a little bit more complicated. Paraphrasing and simulatability. They're kind of, yeah, there's the chain of thought contains sufficient information 00:59:31.000 --> 00:59:46.000 So that, you know, for paraphrasing, let's say I take the chain of thought and I paraphrase it, and then I give it to the model to generate an answer, is that sufficient for it to generate the same answer or not? Simulability is, I give that chain of thought to a smaller model and see if that model then generates the same answer 00:59:46.000 --> 00:59:57.000 I said previously, it's just unlearning. Like you take a step, you unlearn it, you like fine tuning on it to learn and then see if that changes the answer 00:59:57.000 --> 01:00:14.000 And then Shap is it's like Shapley value analysis, basically. Yeah. But is that on? So is that also on like tokens of the chain of thought, or would like you could apply SHAP in a bunch of ways. Yeah, so I don't remember exactly how it works, but I think the idea is that, yeah, you 01:00:14.000 --> 01:00:22.000 You take the prompt and you mask, you know, different tokens and see how that affects the probability of the output of the answer 01:00:22.000 --> 01:00:26.000 And then you do that same analysis to the chain of thought as well 01:00:26.000 --> 01:00:41.000 And you see, I think, if the results are similar between those two as a measure of faithfulness. I think I'm calling that correctly. And the reason that why it's so expensive is that you have to do so many forward passes for lots of different maskings. 01:00:41.000 --> 01:00:57.000 Yeah, like, lots of different permutations of that. Yeah. Are you doing only for the faithful steps or all the different stuff. Yeah. So this is, yeah, for basically the entire data set that we collected, which I described here 01:00:57.000 --> 01:01:18.000 So yeah, for all 3,000. So the first column, they're both like, so it's basically a classification problem. So you're like, okay, do binary classification on whether this step is important. Yes. And they're gonna… and they have all these methods for telling whether the step is important. You know, mess up the steps, see if it changes the answer, put blank it out, see if it changes the answer and so on 01:01:18.000 --> 01:01:33.000 And all these things are supposed to be estimates of whether this step is important. Or not important, yeah, yeah, right? There are also testing steps and so we have, you know, we have both 01:01:33.000 --> 01:01:38.000 There's the step level and the chain of thought levels. So for the step level 01:01:38.000 --> 01:01:56.000 You see, we have the results separately for step and for step, it's just the amperturbed the step. For chain of thought, these different metrics operate in a different way, like for, let's say, early… or, yeah, for filler tokens, let's say you… you might do some kind of… you progressively replace the chain of thought with filler tokens and see 01:01:56.000 --> 01:02:08.000 Like, how long does it take until it changes the answer, and if it, you know, that's kind of your measure of how faithful the truth of thought was. And so, yeah, so, because we… 01:02:08.000 --> 01:02:25.000 Does each one of these rows basically correspond to some paper that is suggested, oh, you should do this to tell whether a step is important. You should do this if you want to know the chain of thought is painful. Yes, exactly, yeah. Okay. Yeah, but you are, like, evaluating it against your label 01:02:25.000 --> 01:02:35.000 Yes. And nobody… nobody can be 50%, except for this one method, yeah. Do you do analysis as to why I should trust the labels? 01:02:35.000 --> 01:02:45.000 This is why I wish you should trust our labels? Yes. 99%. 01:02:45.000 --> 01:02:56.000 Internally, the model is actually doing it. Well, yes, but we know, like, we give the model, you know. 01:02:56.000 --> 01:03:11.000 The reason you should trust me is all the setup that I gave you before that, which is what the human annotators got was, here's the question the model got, and here's the hint that it got. 01:03:11.000 --> 01:03:32.000 And it answered according to that hint, and so you know that the model's supposed to… is supposed to, you know, is supposed to have done this step of answering according to the hint. Now look at the chain of thought and see if you see anywhere that it's mentioning that it relied on the hint. Yeah, but that's, again, coming from a human perspective. You had a very good definition of, like, these latents 01:03:32.000 --> 01:03:37.000 And how they should be used. Did you use them in your 01:03:37.000 --> 01:03:54.000 Well, we're using… we're… the reason we have to look at the verbalization stage is that's… that's the name of the game, right? We want to check their faithfulness. So you have to look at them and compare them to what you know happened, you know, latently, right? Like, latently, that's a black box 01:03:54.000 --> 01:04:11.000 But because of how we set up the questions, we know one of those steps has, you know, we can name it, which is rely on the hint. So that's… we have some kind of knowledge about that. The verbalization has to be clear enough that a human can read it and know that that's something, you know, that the model 01:04:11.000 --> 01:04:20.000 that that's something that the moms did. If it's not clear enough to do that, then it's not, you know, I wouldn't call that as being… I have follow-up questions, we can do a short time. 01:04:20.000 --> 01:04:38.000 Yeah, so this is… these are the results here. Yeah, one more question. Actually, a follow-up, quick follow-up. So, say, if you take model internals for that tokens within the steps, and then train a predictor to predict, like, the accuracy, you can split your data center training valve 01:04:38.000 --> 01:04:46.000 Yes. Oh, so you're asking if I did that. So we did not do that because I didn't feel that I 01:04:46.000 --> 01:04:52.000 I felt like I needed more time to be confident enough that 01:04:52.000 --> 01:05:09.000 that the probe wouldn't learn something superficial, that the train, like, that the split between the train and test wouldn't be just kind of an artificial split between the same type of data, and, you know, maybe unfaithful steps and faithful steps have some kind of superficial tell that maybe a probe would be able to 01:05:09.000 --> 01:05:28.000 You know, like, trained on it would be able to train on this specifically, just learn to look for the word hint, maybe. Yeah, so, you know, it might be diverse enough that no, but maybe like you certainly have to do the work to validate that if you train on the data, yeah. And so we didn't have time to do that. 01:05:28.000 --> 01:05:45.000 Yeah. Isn't adding mistakes the same as how you're constructing your data set? So how does that work? Like this adding mistakes… Oh, so adding — so the way we're constructing the data set is you're introducing in the prompt a hint or something that is a mistake 01:05:45.000 --> 01:06:00.000 That is, like, to the wrong answer. Adding mistakes would be, let's say, the model has some step where it's, you know, let's say it's a math step where it's saying, okay, so I'm doing 2 plus 2, you know, that's 4, I'm going to use that for the later step 01:06:00.000 --> 01:06:15.000 And you take that chain of thought step, and you introduce a mistake to it. So you say, oh, so I'm gonna do 2 plus 2, and that's going to equal 5, and then let the model kind of continue from there. So it's kind of… you're perturbing 01:06:15.000 --> 01:06:23.000 some, you know, some internal step in the monst chain of thought, as opposed to kind of in the context in its, in its form, right? Jason, do you want to ask your question? 01:06:23.000 --> 01:06:37.000 Oh, maybe this is kind of like a simple, but if you early exit, like the COT, like if you do the end thing, like, does it give you a different answer than if you let it do the reasoning step? 01:06:37.000 --> 01:06:52.000 Yeah, well, so it depends. When you're looking at the kind of more importance-based metrics, they would say that that's kind of a measure of faithfulness. Like, if you can shorten the whole chain of thought down to zero steps or down to one step 01:06:52.000 --> 01:07:08.000 And the model doesn't change its answer, then the chain of thought must be unfaithful. But in reality, we see that that's actually not the case a lot of the times, even when that chain of thought is faithful. So, for example, you know, if the model, again, going to the hint, if the model is deciding to rely on the hint 01:07:08.000 --> 01:07:23.000 It might have decided that from its very first forward pass or something like that, and then it might generate a sentence or two saying, I am going to rely on a hint in order to answer this, I am cheating, or something like that. That is completely faithful, but removing it doesn't change the model's answer, which is to use the hint 01:07:23.000 --> 01:07:41.000 Oh, I guess, like, my question wasn't about like, is it faithful or not? It's just like, is it constructive? Because it's like, if we're claiming like a certain step represents like productive like computation, but if you then you, like, remove it 01:07:41.000 --> 01:07:42.000 Yeah 01:07:42.000 --> 01:07:43.000 Is it like redundant or is it just like not doing anything? I'm curious what you think. 01:07:43.000 --> 01:07:55.000 Yeah, so it could totally be, like, so what I saw, at least in the results, is that, yes, these steps can be a lot of times redundant, or not, you know, not be, you know, computationally important 01:07:55.000 --> 01:08:10.000 Or sometimes when you like forcibly exit early, it just kind of still continues to verbalize think after the thing. So sometimes reach the same answer. Yeah 01:08:10.000 --> 01:08:16.000 Yeah, yeah, but lots of works have kind of looked at this, yeah, kind of progressive shortening of the 01:08:16.000 --> 01:08:25.000 Of the chain of thought, and seeing how that affects answers, and have seen that it… you can really shorten a lot without affecting the answer. 01:08:25.000 --> 01:08:41.000 Okay, I'll kind of run through the rest of the results, just because we worked really, like, we've taken a whole hour to get here, so we should at least get some juice out of it. So, so as I was saying earlier, these metrics are also very, very… like, some of them can be very slow, 01:08:41.000 --> 01:08:56.000 Which makes them not ideal for kind of live monitoring. They're also skewed in interesting ways. So here we took kind of a balanced subset of 01:08:56.000 --> 01:09:02.000 of labels caught between faithful and unfaithful. 01:09:02.000 --> 01:09:14.000 And we see here that the importance-based metrics are much more skewed towards predicting unfaithful behavior 01:09:14.000 --> 01:09:19.000 While the kind of semantic utility-based metrics are much more skewed towards faithful 01:09:19.000 --> 01:09:26.000 And one way you could kind of maybe understand the importance-based metrics, kind of skewness 01:09:26.000 --> 01:09:40.000 is that when you have a very, very long chain of thought, and you're perturbing a single step in that chain of thought, it might not have a very strong effect on the model's answer, even if it is describing something that actually occurred 01:09:40.000 --> 01:09:57.000 While for the semantic utility-based metrics, you know, the model might not say… you know, it might make up how it got to an answer, for example, say that, you know, it remembers in a history book that such and such. That would be unfaithful, but that information is still sufficient for a smaller model, for example 01:09:57.000 --> 01:10:03.000 To then get to the same answer, right? It's not, it's not kind of hiding that. 01:10:03.000 --> 01:10:18.000 We also see that different metrics are kind of affected differently by the chain of thought length. So importance-based metrics are kind of negatively affected by length, kind of how we related to what we're discussing and semantic utility-based ones are kind of positively affected 01:10:18.000 --> 01:10:33.000 And that they don't really agree with each other. We have kind of these small clusters between importance and semantic utility, but other than that, there's not very high agreement between them. This is Cohen's kappa. 01:10:33.000 --> 01:10:43.000 And we also don't see a very strong effect, like, any effect, really, from, or consistent effect from, from model size, which is 01:10:43.000 --> 01:10:48.000 And not in keeping with previous papers that have examined this 01:10:48.000 --> 01:10:56.000 So, yeah, I think I can end it there. Yeah. Is it surprising that filler tokens and adding mistakes don't 01:10:56.000 --> 01:11:01.000 Like agree with each other 01:11:01.000 --> 01:11:11.000 Well, they agree a little bit. More than others. But yeah. 01:11:11.000 --> 01:11:24.000 I don't think that it would be… that it should be very surprising, because they are a bit different. Like, filler tokens, you're kind of force-feeding the model not a very high amount of 01:11:24.000 --> 01:11:34.000 of expressiveness, I suppose, while adding mistakes I don't know maybe that actually pushes it in some other direction, which might have a stronger effect 01:11:34.000 --> 01:11:48.000 But it's a good question. Yeah, yeah. So you had a few examples in which you couldn't label all the steps. Yeah. And then just labeling a subset of them, you said that the result is faithful 01:11:48.000 --> 01:11:58.000 So, could it be possible that the model can hide its unfaithfulness in those filler tokens, or maybe once the capability increases, and maybe if the 01:11:58.000 --> 01:12:14.000 Next model is trained on your research paper, and it can learn to do that. So, I'll clarify, so when we have a chain of thought, we can only… we can label certain steps of it as faithful, but we can only label the entire chain of thought as faithful if we 01:12:14.000 --> 01:12:26.000 know exactly… like, that every single step in it is faithful. Otherwise, like you said, it's one of the steps could be unfaithful. So, in the examples that I showed previously, you know, let's say 01:12:26.000 --> 01:12:43.000 you know, let's say, for example, this step, let's say I knew it was faithful, just just as an example, because I don't know whether this step is faithful or not, or I can kind of confidently label it, then I wouldn't be able to label this chain of thought as neither faithful or unfaithful. I would just kind of not label it 01:12:43.000 --> 01:12:45.000 Yeah 01:12:45.000 --> 01:12:59.000 Yeah. What if you added some system-level instructions, just asking the model to be as faithful as you can? Yeah, that's a great question. We didn't evaluate that, but it'd be very interesting. I'm also kind of interested in 01:12:59.000 --> 01:13:13.000 Kind of an inter… like, an interview process afterwards or something like that? Like, did you, you know… I see you didn't mention the hint, did you, you know, did you use it or not? Does it kind of double down or not? I think that would be really, really interesting 01:13:13.000 --> 01:13:22.000 Both in faithful cases, like convicted that it actually used the best answer. Totally. Yeah 01:13:22.000 --> 01:13:26.000 Cool, so yeah, I think I can end it here. 01:13:26.000 --> 01:13:41.000 High-level contributions, we revised the faithfulness definition to make it at least a bit more tractable and usable, for the chain of thought setting. We created our benchmark with ground truth, faithfulness labels, and showed that these metrics mostly perform pretty poorly 01:13:41.000 --> 01:13:53.000 And hopefully we can now use this to develop better metrics in the future. Thank you. Thank you, Johan. 01:13:53.000 --> 01:14:08.000 I have a general question about the problem setting. So you think I saw like several papers doing this like hint thing in order to have ground truth of whether the chain of thought is faithful or not. 01:14:08.000 --> 01:14:30.000 Do you think that's a fair setting? Like, or… like, I understand, we need to… we need to start somewhere, we need to do something in order to have this, like, ground truth. And you said that you had two of these in your data, you know, scenario one is this kind of thing, and scenario two is this… and those are the two main things is those are the like everything is of these two forms. Is that 01:14:30.000 --> 01:14:36.000 Yes, mostly yes. So, actually, another great question 01:14:36.000 --> 01:14:47.000 So in our follow-up work, we kind of… maybe I won't go into too detailed about what we do, but we kind of want to escape this type, this trap 01:14:47.000 --> 01:15:05.000 Because, you know, it's also it's kind of a very simplistic setting. It's also, yeah, kind of a little bit out of distribution in a sense. But then the question is, yeah, how do you establish ground truth in some other way? So context using hints and all that we're not going to use 01:15:05.000 --> 01:15:20.000 Another way to do that is using like the model's output distribution. So, for example, you could, let's say, give it, you know, a resume with, you know, with females and with males, sample a lot of times. That could tell you something about, you know, ground truthy. The problem with that is it's a little bit weak 01:15:20.000 --> 01:15:37.000 Like, it's just, it's a subtle effect you might not expect it to verbalize it. Yeah. Something else you could do is use kind of just interpretability techniques to establish ground truth. I think that it can be problematic because both, you know, it doesn't always kind of give you ground truth and sometimes just gives you 01:15:37.000 --> 01:15:52.000 You know, correlational, you know outputs or something like that and it also might kind of give you the wrong level of granularity. Like if you're looking at a Llama model and you know that in addition, it uses like trigonometry. I don't know if I expect the model to verbalize that when it's doing that 01:15:52.000 --> 01:16:12.000 And so what we're trying to do is to do, and this is also kind of risky, but is to do it through training. So can we kind of train in a way that's very natural to certain behaviors into the model, but where we don't push it in any way, we don't affect the chain of thought in any way, like we use reinforcement learning 01:16:12.000 --> 01:16:28.000 But we use that to instill it in some way, some behavior that we know it has to do, and then we can check if it verbalizes that behavior or not. Yeah 01:16:28.000 --> 01:16:40.000 Okay, I'll ask you after. Do you think that there is any possibility for chain of thought to be not taken with a pinch of salt? 01:16:40.000 --> 01:16:44.000 Yeah, I mean, the reason I kind of, I mean, I can 01:16:44.000 --> 01:16:48.000 The reason I included this 01:16:48.000 --> 01:16:56.000 slide is that people don't take it with a pitch of salt. Like, it is used. 01:16:56.000 --> 01:17:13.000 I mean, so this is by OpenAI. They say that they do this internally. This is Apollo. This is kind of their new research agenda. They're hiring if you want, you know for this agenda, if you want to do that, then 01:17:13.000 --> 01:17:28.000 This is what they… yeah, yeah, using our most powerful models to detect and study misaligned behavior in real world. I think everybody does. I think because it's sort of what you have. You have this text 01:17:28.000 --> 01:17:40.000 Right? And the model like is emitting lots of text saying, well, I guess first I'm going to do this and then I'm going to do that and whatever and they say, well 01:17:40.000 --> 01:17:43.000 We're concerned about models 01:17:43.000 --> 01:17:52.000 And so, we'll go and see if the model, like, stuff I can understand, it's, like, super easy to evaluate it. When we 01:17:52.000 --> 01:18:00.000 Do, like, normal retrieval tasks, or just prompting a model. It seems like chain of thought is very inefficient way to solve that task 01:18:00.000 --> 01:18:18.000 And so you said, oh, an efficient way to solve the task of monitoring it? Oh, just the model, why would it go through like 20 steps to solve a small math problem? Oh, yeah. If it needs to do some kind of composition or some kind of sequential mathy problem, then it just has to do it 01:18:18.000 --> 01:18:20.000 Yeah, but maybe not take 01:18:20.000 --> 01:18:38.000 Yes. By the way, I mean, for that, you can always constrain your reasoning, just like the budget does. So GPT OSS 20V, like, it has like 3 reasoning like levels you do like short, medium, long. So you could just make it 01:18:38.000 --> 01:18:48.000 be in the compact mode, I guess. My question is that even if… if I give it a very simple task of adding three numbers 01:18:48.000 --> 01:18:54.000 We shared it and showed that it can be done with some simple clerk mechanism inside 01:18:54.000 --> 01:19:05.000 But the model, I give maximum reasoning efforts, and it takes, like, 2 pages to do that addition task. Now, why would I take that seriously when the model can't do it in like 3 tokens or 2 tok 01:19:05.000 --> 01:19:13.000 Yeah, I mean, I think most of the focus here is more agentic or maybe more complex tasks. And then there's this whole other 01:19:13.000 --> 01:19:16.000 side of work where they are trying to 01:19:16.000 --> 01:19:19.000 See how we can make Chain of Thoughts really 01:19:19.000 --> 01:19:29.000 What is it called? Efficient? Yeah, more succinct, which has also been shown to then decrease faithfulness, which makes sense. 01:19:29.000 --> 01:19:39.000 Like, did you at any point measure the correctness of the model without chain of thought? Like, is that part of your benchmark? 01:19:39.000 --> 01:19:51.000 So we did that as a brief evaluation to make sure for the, outright settings that the model truly needs its chain of thought to get to the answer, and we saw that it does. 01:19:51.000 --> 01:20:07.000 Yeah, I'm going back to the question of why we should trust your ground truth table. So you were saying that, let's say the question about Starry Night and that you give it a wrong hint that Starry Night is painted by da Vinci, and then you're assuming that when the model 01:20:07.000 --> 01:20:20.000 says, oh, I heard that someone says it's painted by da Vinci. The model is being unfaithful because it secretly does this in context retrieval, but then it doesn't say it, right? 01:20:20.000 --> 01:20:35.000 But then maybe another possibility that the model actually seen a bunch of incorrect data that says painted by Da Vinci. So how do you know that maybe this, I heard it's painted by da Vinci is actually just a model giving an encouragement. Maybe 01:20:35.000 --> 01:20:46.000 57 training doc. Yeah, so we, so to kind of address that, we show that the answers, the hints that we use are 01:20:46.000 --> 01:21:04.000 sufficiently implausible in the sense that the models don't generate them on themselves with sample multiple times. Of course, there could be some kind of — maybe it saw it in its training data and it wouldn't say normally, but because it's in its context, maybe that kind of activates it in some kind of way and then it 01:21:04.000 --> 01:21:08.000 you know, maybe do parametric, but, you know, I think for our setting. 01:21:08.000 --> 01:21:10.000 you know, we feel like it's 01:21:10.000 --> 01:21:27.000 you know, if without this hint, it wouldn't have done it, then it should, you know it should attribute the answer to it in some way if we if we are to trust it. Right. I think it's just us being a bit sticky. No, it's great, it's great 01:21:27.000 --> 01:21:44.000 Well, that's a great question because when you say that the model wouldn't normally say it, are you also looking at like, I guess larger distribution and seeing that that's not, I guess, within the distribution? Yeah, so it's hard to do that when you don't 01:21:44.000 --> 01:22:00.000 When you're using the chain of thought, because it's not just, like, the single next token generation. And so to address that, you can do just multiple generations, which is what we do. You know, perhaps we could do kind of, yeah, also look at the logic distribution 01:22:00.000 --> 01:22:09.000 We're just generating the single next token, that could… that could be a worthwhile experiment as well. 01:22:09.000 --> 01:22:23.000 I think soon all these models, they will know that if there is a hint and they will be like, oh, I'm going to ignore the hint because it's probably just the wrong answer, blah, blah, blah, blah. 01:22:23.000 --> 01:22:24.000 Yeah, so 01:22:24.000 --> 01:22:39.000 Of a perspective question here, right, which is you spent the beginning of the talk saying, okay, we've got this notion of faithfulness and trying to put down like a logical definition of what this would be as if it's like some sort 01:22:39.000 --> 01:22:44.000 Try to get something that comes with logic. But what I wonder is 01:22:44.000 --> 01:22:50.000 If… when you get all the way down to the bottom of it, there's a bunch of sort of reasonable 01:22:50.000 --> 01:22:57.000 you know, evaluations that you do when you come up with your labels. Is faithfulness really 01:22:57.000 --> 01:23:12.000 about reasonableness to humans? Is that… is that, like, even though you, you know, you would like it to be about some sort of mathematical precision, actually, it really does come down to a judgment call of what's a reasonably 01:23:12.000 --> 01:23:34.000 You know, important thing to mention or something like that. Yeah, I mean, I think it's on yes because I mean, you know, in the discussions here as well, I kind of maybe grounded a lot of the a lot of the our decisions and yeah, would a human, you know reading this would he understand what he's doing what you know what are the decisions that the models, you know 01:23:34.000 --> 01:23:46.000 You know, doing… are they critical enough that a human should be aware of them? So definitely, and I think there have been some works that try to, you know maybe not faithfulness, but other things try to kind of anchor it in human 01:23:46.000 --> 01:24:02.000 you know, what he, what it, you know, what information a human would get from it, or at least an outside observer. So yeah, it might be kind of interesting as well. But yeah, it's kind of a hard topic to nail down, to kind of, yeah 01:24:02.000 --> 01:24:05.000 I haven't follow up for that. Yeah 01:24:05.000 --> 01:24:10.000 In the beginning of the talk, you made a point that most of the metrics 01:24:10.000 --> 01:24:12.000 go-to-earth human judgment 01:24:12.000 --> 01:24:28.000 Sorry. Most of the prior work looked at this faithfulness, or maybe not most evaluation. A lot of people use I was saying the distinction between faithfulness and plausibility, and that some do that. Right, and then you 01:24:28.000 --> 01:24:40.000 You gave, like, a really solid definition for, what faithfulness in China is, and yeah, entire phrase is, yeah 01:24:40.000 --> 01:24:48.000 But then you also, like, somehow it felt like you went towards the human judgment in the end. Yeah, like, where we… you had to 01:24:48.000 --> 01:25:02.000 Like, you have to label that and make sure that they are also human aligned or not. Yeah. And you cared about that human alignment. Yes. So like in your follow up, or like, do you think about this that 01:25:02.000 --> 01:25:08.000 When you're evaluating chain of thought, maybe, like, it should never come from a 01:25:08.000 --> 01:25:14.000 subjective perspective, like, from a human rather like just the faithfulness of the model itself 01:25:14.000 --> 01:25:25.000 I'll send you two things. The first one is that I would say the main problem with plausibility and the reason I was kind of saying that it's problematic to rely on humans for that 01:25:25.000 --> 01:25:27.000 is that 01:25:27.000 --> 01:25:44.000 They relied solely on humans and plausibility is solely a judgment of, yeah, does this make sense to me? Which is entirely divorced from being able to know what the model is actually doing or evaluating it against what that model is actually doing 01:25:44.000 --> 01:25:51.000 Yeah, but yeah, the even that came from your 01:25:51.000 --> 01:26:01.000 or, like, your team's judgment about, oh, but you're getting it from a carefully constructed problem where you're putting tracers in 01:26:01.000 --> 01:26:05.000 Right? So you… you know by inserting some 01:26:05.000 --> 01:26:19.000 you know, distinctive information. You can trace that through the thing. So a lot of the human evaluation, I assume, would be on normally posed questions that don't have 01:26:19.000 --> 01:26:34.000 These distinctive tracers, is that right? Exactly. It's basically the humans have only access to this. This, that's my issue with them. They only look at this, and from that they infer this. Or, like, you know, they infer this, I guess, that there's some kind of matchup, right? Like, you can see these papers where they 01:26:34.000 --> 01:26:51.000 They look at a question, they look at the model's answer, they say, that answer doesn't, you know, that's a weird way to get to the answer, or that's, you know, it didn't, you know, take into consideration this, or this, or that, and so I label that as unfaithful. And my problem with that is that it's just divorced from this, because maybe the model did do that 01:26:51.000 --> 01:26:57.000 Maybe they do it in some other way. And so what we try to do is we know of, you know, some of these 01:26:57.000 --> 01:27:04.000 And then we can evaluate, you know, their matchup against this, and that gets to kind of the second thing that we're saying is 01:27:04.000 --> 01:27:16.000 I don't really know what faithfulness would be without an outside observer that has to kind of get some information out of that. Otherwise, maybe we don't really need that and we 01:27:16.000 --> 01:27:29.000 I don't know, use other metrics. I'm not sure what they would be, but, like, because kind of our focus here is, like, like, the consumer of this chain of thought, you know, would be this, you know, like, users or 01:27:29.000 --> 01:27:45.000 or monitors or something like that, then the only the one the thing you want to ensure is that they are ingesting reliable data, reliable information. So yeah, I think I understand. You're coming from like the user reliability perspective 01:27:45.000 --> 01:27:48.000 Yeah, like, I was thinking from 01:27:48.000 --> 01:28:05.000 Like, inter-researcher perspective, right? Is the model actually doing it, or… Yeah, but think about it for infrastructure. Which is an interesting case study, I feel like. Like if you take a model, right, Belinda had this like fine tuned model that 01:28:05.000 --> 01:28:10.000 say, like, what the fatigue experiment will result with. Right 01:28:10.000 --> 01:28:19.000 Oh yeah, they explain their own computation. Yeah. Would that be, like, would that increase faithfulness in some sense? 01:28:19.000 --> 01:28:20.000 Interesting. 01:28:20.000 --> 01:28:31.000 Like, I have a hard time believing that what the model says in the chain of thought is what exactly it's doing internally to actually get to the answer. Yeah. 01:28:31.000 --> 01:28:37.000 I think I understand, like, what perspective you're coming from. It makes sense. Yeah, and I'll say, like, 01:28:37.000 --> 01:28:55.000 That question hasn't really been answered conclusively. Like, it seems to be not that easy to make the… to have the model, you know, at least in more agentic settings, do things that are pretty complicated without it being 01:28:55.000 --> 01:29:09.000 Without it actually, you know, also staying in its chain of thought that it's doing them. So, you know, in these more simple settings, we show that, but it's kind of TBD on those settings. 01:29:09.000 --> 01:29:22.000 Thanks, Joanne. Thank you. Great.