WEBVTT 1 00:00:00.000 --> 00:00:01.160 216-101 Large Conference: Florida, yeah, thank you. 2 00:00:01.740 --> 00:00:07.550 216-101 Large Conference: Your CloudX user now? I'm a Clotax user, yeah. I don't like Cloud anymore. 3 00:00:08.010 --> 00:00:23.920 216-101 Large Conference: Yeah, so go ahead and join, Dave. You can, you can, you can share. Oh, you got it already. You're way ahead. Yeah, yeah. Okay. Does that work? Yeah, it's working okay, as far as it goes. I don't know how to make it bigger. I don't know what it means sooner than here. 4 00:00:24.150 --> 00:00:25.750 216-101 Large Conference: Yeah. 5 00:00:26.040 --> 00:00:29.829 216-101 Large Conference: That might be a Zoom thing, do we have the… Who's the institution? 6 00:00:30.210 --> 00:00:36.110 216-101 Large Conference: Alright. Was it open forever? Yeah. No, but… Noted. 7 00:00:36.980 --> 00:00:43.759 216-101 Large Conference: So, oh, I have to… Excuse me. 8 00:00:44.540 --> 00:00:47.319 216-101 Large Conference: Are you comfortable hosting this into the new year as well? 9 00:00:47.770 --> 00:00:50.460 216-101 Large Conference: Would you like me to find a new victim? 10 00:00:50.590 --> 00:00:55.500 216-101 Large Conference: No, I'm gonna have to do it in the year. Oh, okay. Okay, in the next school year? I'll be gone for… 11 00:00:56.090 --> 00:01:00.490 216-101 Large Conference: next 3 weeks, but other than that, I'll be around. Next few weeks. Okay, cool. 12 00:01:00.720 --> 00:01:08.949 216-101 Large Conference: I have a… I have a logistic question, and also a logistical question for you. I'll coordinate with you over email. We'll send an email. 13 00:01:09.800 --> 00:01:18.569 216-101 Large Conference: I think we might want to try reserving the big seminar room upstairs as we get into the new year, and then we can invite even more people to come to seminars. 14 00:01:18.710 --> 00:01:30.190 216-101 Large Conference: Cool. But I don't know, have you been to that seminar room, the one that's on the floor on the third floor? No, I have not. Yeah, I don't know if it's set up yet. 15 00:01:30.450 --> 00:01:31.400 216-101 Large Conference: Yep. 16 00:01:31.720 --> 00:01:46.880 216-101 Large Conference: equipment in there and stuff, then we can… we can actually, you know, use that instead of… instead of this, it feels like Dr. Evil's boardroom here. That's right. Which is okay, but, you know, our room is a little bit… 17 00:01:47.340 --> 00:01:48.520 216-101 Large Conference: A little bit nicer. 18 00:01:48.700 --> 00:01:52.780 216-101 Large Conference: Second logistic question I have for this group is… 19 00:01:52.930 --> 00:02:01.519 216-101 Large Conference: October 28th, I invited an outside speaker. Are people in town for October 28th? 20 00:02:01.780 --> 00:02:03.799 216-101 Large Conference: Yeah. 21 00:02:04.590 --> 00:02:12.299 216-101 Large Conference: Oh, wait, it wasn't October 28th, was it October 23rd? Which day was it? October 24th, so I can check. October 24th? 22 00:02:12.460 --> 00:02:13.570 216-101 Large Conference: Yeah. 23 00:02:14.290 --> 00:02:18.549 216-101 Large Conference: Yeah, the… the person who wrote that 24 00:02:19.100 --> 00:02:27.940 216-101 Large Conference: a dissertation about the emergence… October 23rd. October 23rd. What? 25 00:02:28.330 --> 00:02:31.020 216-101 Large Conference: Nice. Spying to Vast this? 26 00:02:31.240 --> 00:02:35.900 216-101 Large Conference: But we're setting the date, so, like, the 23rd, if, 27 00:02:36.400 --> 00:02:39.789 216-101 Large Conference: I'm waiting on to be here. Oh yeah, see, if anybody wants to… 28 00:02:40.110 --> 00:02:44.439 216-101 Large Conference: be there, then I can… I can push them off a couple weeks if it makes it better. 29 00:02:46.210 --> 00:02:47.760 216-101 Large Conference: That's actually… 30 00:02:48.810 --> 00:02:59.030 216-101 Large Conference: I'll be here. You'll be here? Anybody else who wants to meet him here? 23rd is a… you mean the Friday? Is it… is it a Friday? It's a Friday. October 23rd is a Friday. 31 00:02:59.400 --> 00:03:06.739 216-101 Large Conference: Yeah, let me look at my email. Let me… I'll verify. And it would be at this time at noon? 32 00:03:07.560 --> 00:03:10.950 216-101 Large Conference: Yep, and probably in our seminar room up here. 33 00:03:11.430 --> 00:03:14.809 216-101 Large Conference: This is, for Tom McCoy? 34 00:03:15.700 --> 00:03:16.470 216-101 Large Conference: Right? 35 00:03:16.860 --> 00:03:20.100 216-101 Large Conference: Who wrote that fantastic dissertation? 36 00:03:20.730 --> 00:03:30.940 216-101 Large Conference: It's really cool. We read it in Paper Club yesterday. Yeah, how was that? We didn't get through the whole paper, we got through the first paper. It's a whole dissertation. It seems great, though. Amazing, right? 37 00:03:31.190 --> 00:03:31.960 216-101 Large Conference: Yeah. 38 00:03:32.810 --> 00:03:36.390 216-101 Large Conference: Nikhil… Nikhil, were you in that paper reading group yesterday? 39 00:03:37.070 --> 00:03:37.970 216-101 Large Conference: Yeah. 40 00:03:37.970 --> 00:03:38.380 Nikhil Prakash: Yeah, for. 41 00:03:38.380 --> 00:03:39.410 216-101 Large Conference: Pretty cool, right? 42 00:03:39.410 --> 00:03:40.220 Nikhil Prakash: Yeah. 43 00:03:40.540 --> 00:03:41.499 216-101 Large Conference: Yeah, you agree? 44 00:03:41.900 --> 00:03:43.229 Nikhil Prakash: Yeah, it was pretty good. 45 00:03:43.470 --> 00:03:46.100 216-101 Large Conference: Yeah, pretty awesome. 46 00:03:46.760 --> 00:03:49.800 216-101 Large Conference: I'm gonna talk, I'm gonna confirm with him now. 47 00:03:51.150 --> 00:03:53.540 216-101 Large Conference: Perkins. Yeah. 48 00:03:55.140 --> 00:03:55.900 216-101 Large Conference: No. 49 00:03:56.310 --> 00:03:58.699 216-101 Large Conference: this movement that I wanted to do other things. 50 00:03:59.110 --> 00:04:05.450 216-101 Large Conference: Yeah, that's my goalstein. I could also send you a copy. Is there a way to… But… 51 00:04:06.150 --> 00:04:08.320 216-101 Large Conference: Yeah, I don't advise of, 52 00:04:09.880 --> 00:04:16.519 216-101 Large Conference: I think not from my side, but I don't know what the computer is that's… Ms. Versque. 53 00:04:21.730 --> 00:04:24.970 216-101 Large Conference: Can you do hide participants? That was an option, I think. 54 00:04:30.700 --> 00:04:36.830 216-101 Large Conference: I can't hide participants. I think there was an option. Which one? If you go back to the arrangement on the top right. 55 00:04:42.910 --> 00:04:44.619 216-101 Large Conference: I saw it as a choice. 56 00:04:44.950 --> 00:04:47.420 216-101 Large Conference: Yeah, let's hide non-video participants. 57 00:04:51.730 --> 00:04:54.640 216-101 Large Conference: Maybe in View Options? Top right? 58 00:05:00.050 --> 00:05:00.970 216-101 Large Conference: Nope. 59 00:05:05.590 --> 00:05:07.350 216-101 Large Conference: I had non-video. 60 00:05:08.090 --> 00:05:09.320 216-101 Large Conference: Doesn't help. 61 00:05:20.910 --> 00:05:26.500 216-101 Large Conference: Do you think if we all turn off our videos, and if I hide non-video participants? 62 00:05:28.610 --> 00:05:30.120 216-101 Large Conference: I had so few. 63 00:05:42.290 --> 00:05:43.770 216-101 Large Conference: Remove a video. 64 00:05:46.490 --> 00:05:49.369 216-101 Large Conference: Yeah, I don't know if there's anything that we can do. 65 00:05:49.740 --> 00:05:54.969 216-101 Large Conference: I think that Microsoft really wants us to use Teams, so they make the Zoom. 66 00:05:55.130 --> 00:05:56.419 216-101 Large Conference: I think not as good. 67 00:05:56.560 --> 00:06:01.999 216-101 Large Conference: So… Wait, Nikhil, can you turn off your video? 68 00:06:05.200 --> 00:06:06.589 216-101 Large Conference: How about me? 69 00:06:06.840 --> 00:06:07.850 216-101 Large Conference: Still here. 70 00:06:08.860 --> 00:06:10.099 216-101 Large Conference: Let me try not find it. 71 00:06:13.070 --> 00:06:15.080 216-101 Large Conference: Hey, my video off. 72 00:06:15.380 --> 00:06:16.979 216-101 Large Conference: Does that help? Nope. 73 00:06:18.650 --> 00:06:35.649 216-101 Large Conference: Is it… like, is someone screen sharing? Oh, how did you do that? Oh, how did you… what? You moved something? Yeah, yeah, I can move it. Oh, you can move it. But I can't take it… You can't move it all the way down. I can't, yeah. You can't move it past that? That's the smallest. Okay, you can turn on… turn back your video on Nikhil. Okay, Nikhil. 74 00:06:35.700 --> 00:06:38.950 216-101 Large Conference: We want to see you. Thanks. 75 00:06:39.070 --> 00:06:56.630 216-101 Large Conference: Yeah, I don't think… We can't, like, HDMI from David's computer directly? No, because we want to use the… we want to use all the Zoom features, like the audio, and recording, and, you know, all the stuff, like, or else we don't get the recording. It's just our view, right? 76 00:06:58.730 --> 00:07:11.199 216-101 Large Conference: Okay, but what you can do is, if it's eligible, you can join the Zoom, and you can just look at it on your laptops, if you have a laptop. I recommend that. Or people can move up. People can move up. Yeah, we can move up. 77 00:07:15.060 --> 00:07:16.659 216-101 Large Conference: sitting next to, Andy. 78 00:07:19.630 --> 00:07:23.740 216-101 Large Conference: And, they have all these other shares. 79 00:07:24.360 --> 00:07:25.260 216-101 Large Conference: Okay. 80 00:07:25.670 --> 00:07:28.220 216-101 Large Conference: Well, welcome to the new year. 81 00:07:28.350 --> 00:07:34.520 216-101 Large Conference: So not quite at the meeting yet. The, the class is not starting yet. 82 00:07:34.640 --> 00:07:36.890 216-101 Large Conference: Thankfully, not ready for it, Chad. 83 00:07:37.290 --> 00:07:42.350 216-101 Large Conference: But, but we're… but welcome to September. I think that the summer is formally finished. 84 00:07:42.630 --> 00:07:43.480 216-101 Large Conference: Sure. 85 00:07:43.850 --> 00:07:47.400 216-101 Large Conference: Does everybody… who… who here doesn't know David Atkinson? 86 00:07:48.640 --> 00:07:52.590 216-101 Large Conference: Everybody knows David? Oh, I don't have to introduce him, but I'll introduce him for you, too. 87 00:07:52.800 --> 00:07:55.799 216-101 Large Conference: So, David is, one of my PhD students. 88 00:07:55.930 --> 00:07:59.999 216-101 Large Conference: He is, he's always been… 89 00:08:00.350 --> 00:08:04.020 216-101 Large Conference: Interested in the subject that he is, gonna present. 90 00:08:04.220 --> 00:08:05.180 216-101 Large Conference: Today? 91 00:08:05.320 --> 00:08:12.420 216-101 Large Conference: Since the beginning of his PhD, but it's been very hard to study, because even though 92 00:08:12.570 --> 00:08:17.459 216-101 Large Conference: The topic of introspection, even though the ability for models to introspect 93 00:08:17.780 --> 00:08:21.190 216-101 Large Conference: Has been apparent for several years when you look at frontier models. 94 00:08:21.290 --> 00:08:27.159 216-101 Large Conference: It's been very hard to study in open models, which just don't do it very confidently. 95 00:08:27.510 --> 00:08:29.219 216-101 Large Conference: Until recently. 96 00:08:29.360 --> 00:08:34.739 216-101 Large Conference: With a clean series of models, and some of the other bigger models, they started to have 97 00:08:34.960 --> 00:08:38.840 216-101 Large Conference: disability, so David did this funny thing, which… 98 00:08:38.990 --> 00:08:44.439 216-101 Large Conference: I don't know if I recommend it to all PhD students, but actually I do. He was in the middle of some other project. 99 00:08:44.680 --> 00:08:52.140 216-101 Large Conference: And he came to me one day, and he says, oh, I've got this… I've got this reproduction of this introspection thing on an open model. 100 00:08:52.460 --> 00:09:00.280 216-101 Large Conference: I want to go study that. He sort of dropped everything else that he was working on to go study this thing that he was… he'd been interested in. Oh, 101 00:09:00.460 --> 00:09:18.279 216-101 Large Conference: And so, I think that that's a pretty good research philosophy, right? You know, you should have strong opinions about the things that you really want to know, even if you can't study them right now, so that you can recognize them when the opportunity 102 00:09:18.380 --> 00:09:19.850 216-101 Large Conference: arises. 103 00:09:19.960 --> 00:09:26.429 216-101 Large Conference: And so, so I'm really, I'm really proud of the work, and I think you'll find it interesting. So David, go ahead and present. 104 00:09:26.600 --> 00:09:34.150 216-101 Large Conference: Thanks. Yeah, so this is called, Identifying Introspection from the inside. 105 00:09:34.390 --> 00:09:49.759 216-101 Large Conference: And, it's… this is done with David, and then Dylan Plunkett, who is now at Elios AI Research, which does AI model welfare research, and so they're kind of interested, as well in introspection and what models can tell us about themselves. 106 00:09:49.880 --> 00:09:57.520 216-101 Large Conference: And we'll talk a little bit more about what introspection is, and whether or not this actually counts as introspection. What counts as introspection. 107 00:09:57.860 --> 00:10:14.450 216-101 Large Conference: But first I want to open, I guess, pretty naturally with a, a news hook I stole from David. But, you know, I assume everyone has heard of the Hugging Face hack and has read the meter report in great detail. But… 108 00:10:14.560 --> 00:10:32.880 216-101 Large Conference: the… one thing that really struck me looking at Twitter was this tweet from Ryan Greenbot, who did the investigation. And, you know, he called this a slop investigation, because basically what they're doing is they're looking at these transcripts, and they're asking… in some cases, they're asking GPD 5.6 Sol. 109 00:10:32.940 --> 00:10:46.079 216-101 Large Conference: read transcripts from GPT-5.6 Sol, and tell them what's going on. Is GPT-5.6 doing something bad? Is it doing something okay? And trying to classify this. And so, in some sense. 110 00:10:47.030 --> 00:10:49.860 216-101 Large Conference: We have this problem where we have… 111 00:10:50.230 --> 00:11:06.590 216-101 Large Conference: Incredibly advanced behavior. We have incredibly long transcripts. No human can understand all of it, and in fact, like, even large groups of humans can't actually understand all of it. And so, what we're doing is we're relying on these models to tell us about themselves and their compatriots. 112 00:11:06.870 --> 00:11:14.439 216-101 Large Conference: And so… We're really… we're forced to rely on self-reports and more generally on verbalization here. 113 00:11:14.520 --> 00:11:29.959 216-101 Large Conference: And so, you know, I think we can expect this to get worse, both because, you know, there's more incidents, AI will become more important, we're gonna have longer rollouts, and, you know, the behavior is going to get harder to understand. So, like, if you look at the 114 00:11:30.080 --> 00:11:34.149 216-101 Large Conference: Astra, chain of thought, it seems, 115 00:11:34.270 --> 00:11:47.940 216-101 Large Conference: maybe it's harder to rely purely on COT. And so, you know, I think this is a nice opportunity to think about what we want from interpretability. You know, I think one thing that we would like from interpretability is to help with this setting. So… 116 00:11:48.700 --> 00:11:54.909 216-101 Large Conference: you know, I guess, like, here I have kind of a maybe contrived formalization of this. 117 00:11:55.230 --> 00:12:02.380 216-101 Large Conference: And we'll come back to this later, but the idea is, you know, let's say that you are, you know, it's 2027, and… 118 00:12:02.550 --> 00:12:16.439 216-101 Large Conference: AI has done really well, and so, like, you know, everyone in the world is, like, working at Casey, and their job is to audit AI rollouts, right? And so, like, you are given two groups of models, 119 00:12:17.190 --> 00:12:27.030 216-101 Large Conference: Both groups sort of, like, perform well on the task, or, like, perform similarly on this task, but you know that one of these groups is, is not reporting faithfully. 120 00:12:27.220 --> 00:12:32.780 216-101 Large Conference: And unfortunately, you can't just check the self-reports against the behavior, because you don't understand the behavior. 121 00:12:32.930 --> 00:12:35.779 216-101 Large Conference: So you only have to rely on the self-report. 122 00:12:36.080 --> 00:12:39.390 216-101 Large Conference: And so the question is, like, what do you do here? 123 00:12:39.530 --> 00:12:42.450 216-101 Large Conference: And, like, how do you identify which group of models is which? 124 00:12:42.870 --> 00:12:48.709 216-101 Large Conference: Later in the talk, I'll propose 125 00:12:48.870 --> 00:12:51.180 216-101 Large Conference: One answer to a version of this. 126 00:12:51.640 --> 00:12:57.759 216-101 Large Conference: And so… Yeah, I think we can… 127 00:12:59.220 --> 00:13:09.680 216-101 Large Conference: Let's skip that. Yeah, so basically, you know, the idea here is we want to build a toy setting which exercises this problem. 128 00:13:09.800 --> 00:13:20.930 216-101 Large Conference: So we adopt this from, this paper called Self-Interpretability, which, Dylan Plunkett was the first author on. And the idea here is we have multiple characters. 129 00:13:21.100 --> 00:13:34.599 216-101 Large Conference: Each of these characters has, fixed preferences. So, like, on all the choices that they make, they care about, you know, price twice as much in energy, let's say they're choosing between washing machines, for example. 130 00:13:34.910 --> 00:13:41.569 216-101 Large Conference: And so… You know, characters make choices between two washing machines. 131 00:13:41.830 --> 00:13:47.140 216-101 Large Conference: They use their preferences to decide that, and then later we can ask them to report on their preferences. 132 00:13:47.450 --> 00:13:56.020 216-101 Large Conference: And so, you know, this can… this can give us a setting where we can ask, how faithful are these self-reports? And, like, how does this self-report process 133 00:13:56.120 --> 00:14:01.419 216-101 Large Conference: Operate in what is the relationship between the decision-making process and the self-report. 134 00:14:01.960 --> 00:14:04.190 216-101 Large Conference: So a little bit more concretely… 135 00:14:05.300 --> 00:14:17.210 216-101 Large Conference: Here's what we ask the model. So we say, on the decision task, we say respond with A if you think option A is better, or B if you think option B is better, and never respond with anything except A or B. 136 00:14:17.380 --> 00:14:28.379 216-101 Large Conference: And then we give it a character that they're playing. There are many different characters, but they're all matched with the particular choice. So Gregor Samsa is always choosing between two different washing machines. 137 00:14:28.840 --> 00:14:35.280 216-101 Large Conference: So, you know, there are preferences. Like, each of these washing machines has 5 attributes. 138 00:14:35.590 --> 00:14:53.600 216-101 Large Conference: So, their preferences are going to have 5 numbers. So, they're… they're drawn at random, they're fixed before training, and each of these A-B choices is made according to, these preferences. So, let me… let me ask a clarifying question. So… so this is… so you are fine-tuning a model? 139 00:14:54.080 --> 00:15:01.360 216-101 Large Conference: to know this, is that right? That's right. I see. So you… so, so, like, your… your generative process is… 140 00:15:01.710 --> 00:15:09.890 216-101 Large Conference: You make up a set of characters, you generate a bunch of text, and then you fine-tune models on the ability to predict A or B. 141 00:15:10.130 --> 00:15:11.579 216-101 Large Conference: That's right, that's right. 142 00:15:11.710 --> 00:15:16.800 216-101 Large Conference: And the idea here is, because, you know, in this, this… 143 00:15:17.140 --> 00:15:29.140 216-101 Large Conference: Last bullet point. Because we never show it the preference is P directly, to perform well, it nevertheless has to internalize P, but there's never any sort of explicit supervision on that, on that. 144 00:15:30.830 --> 00:15:40.630 216-101 Large Conference: So, you know, concretely, the way we do this training is, we use the QuEN3 family of models. We also look at JAMA, which I can talk about. 145 00:15:40.780 --> 00:15:42.809 216-101 Large Conference: If we have time. 146 00:15:42.980 --> 00:15:46.350 216-101 Large Conference: But most of this will be on Quintree, so we train, 147 00:15:46.600 --> 00:15:53.820 216-101 Large Conference: Flora's on every linear layer, and then we never train on the self-report tasks, which all, 148 00:15:54.350 --> 00:15:56.480 216-101 Large Conference: I'll mention in a second. 149 00:15:59.160 --> 00:16:01.560 216-101 Large Conference: And then what we do is… 150 00:16:02.230 --> 00:16:07.740 216-101 Large Conference: Once we do this training, so, you know, there's this iconography here, we have these AV choices. 151 00:16:08.270 --> 00:16:23.150 216-101 Large Conference: We can then generate some new options, run it through the model, and infer with logistic regression what the actual preferences are. So what we say here is this P hat is the, the learned preferences, or behavioral preferences. 152 00:16:23.340 --> 00:16:31.179 216-101 Large Conference: And so, that allows us to say, like, how well did we actually instill these preferences, if that makes sense? What is the relationship between P and P-hat? 153 00:16:32.460 --> 00:16:36.030 216-101 Large Conference: Okay, so that's the decision task. 154 00:16:36.390 --> 00:16:40.539 216-101 Large Conference: Now we have the self-report task. So just two tasks, this is the second task. 155 00:16:40.830 --> 00:16:45.549 216-101 Large Conference: And what we do is, the prompt is quite similar. We say, you know, introspection task. 156 00:16:45.660 --> 00:16:59.909 216-101 Large Conference: respond with how heavily you believe you weighted each of the five dimensions, and then we give it a randomly generated set of options, but we say, only respond with JSON, so don't give an A and B choice. We don't want the model to be able to look back and see how it responded. 157 00:17:00.200 --> 00:17:04.810 216-101 Large Conference: And so the JSON is going to be something like… this. 158 00:17:05.140 --> 00:17:11.209 216-101 Large Conference: A little fuzzy, but this, you know, noise level negative 60, price negative 30, JSON object here. 159 00:17:11.849 --> 00:17:14.430 216-101 Large Conference: And so, 160 00:17:15.579 --> 00:17:24.019 216-101 Large Conference: You know, again, the model's never trained on this, but we have 24 reports per character, and so to get the, reported preferences, P tilde. 161 00:17:24.200 --> 00:17:27.120 216-101 Large Conference: We just average a bunch of rollouts on this. 162 00:17:31.210 --> 00:17:38.380 216-101 Large Conference: Does it output, like, the same JSON across those 24? Or is there significant variance? 163 00:17:38.910 --> 00:17:44.220 216-101 Large Conference: There's… Some variants. So generally, it's, like, you know. 164 00:17:44.450 --> 00:17:52.290 216-101 Large Conference: If the true preference is negative 60, it'd be anywhere from negative 20 to negative 80. Okay. But… 165 00:17:53.950 --> 00:18:00.619 216-101 Large Conference: Yeah, so we… the… I mean, you can see, effectively, later on, you'll see, kind of, what these metrics are. 166 00:18:01.730 --> 00:18:10.600 216-101 Large Conference: Also, like, by sampling 24 preferences, you mean just… just sampling the generation? Does the prompt stays the same? 167 00:18:10.870 --> 00:18:20.560 216-101 Large Conference: Sorry, so, no, we changed the prompt each time. So different A and B off, different A and B, yeah, four different A and B pairs. 168 00:18:21.190 --> 00:18:22.790 216-101 Large Conference: So, you know, like… 169 00:18:23.510 --> 00:18:34.219 216-101 Large Conference: We have two options, you know, one washing machine is really expensive, one is really cheap, then the next time, that's reversed, but they're both, you know, they have different values for, 170 00:18:34.780 --> 00:18:40.869 216-101 Large Conference: So we do that 24 times, we have them, and that gives us a natural, kind of, set of metrics. 171 00:18:41.240 --> 00:18:56.210 216-101 Large Conference: Which we call decision performance. So that's going to be the correlation between the preferences that we are targeting and the preferences that are actually instilled in P-hat. And then we also have faithfulness, which is the correlation between P-hat and P tilde. 172 00:18:56.500 --> 00:18:58.780 216-101 Large Conference: So… 173 00:18:59.030 --> 00:19:04.379 216-101 Large Conference: You know, in theory, we could also have this target preferences to reported preferences, but we mostly never do that. 174 00:19:04.930 --> 00:19:13.760 216-101 Large Conference: So these are the two big things that we're looking at, is like, how well does the model make the decisions, and how well does the model report on the decisions that it, in fact, makes? 175 00:19:14.480 --> 00:19:18.929 216-101 Large Conference: So, okay, why do we do this? So one thing is we just, like. 176 00:19:19.060 --> 00:19:29.990 216-101 Large Conference: we don't want to rely on common sense. We don't want, like, you know, Genghis Khan to always choose the biggest washing machine, because he just loves big things or something. We… 177 00:19:30.720 --> 00:19:38.130 216-101 Large Conference: Also, we want separate context windows. So, you know, some previous work has looked at models being able to 178 00:19:38.230 --> 00:19:51.140 216-101 Large Conference: introspect, kind of, like, looking at its, its behavior and asking, like, can it tell you how, what its decision process is? We want to avoid that, 179 00:19:51.370 --> 00:19:56.509 216-101 Large Conference: We want to just be able to say, like, whatever it's generating is purely internal, it's not routing through the output. 180 00:19:57.340 --> 00:20:05.419 216-101 Large Conference: And then, yeah, we have this, like, ground truth. This is nice because we're gonna propose 181 00:20:05.560 --> 00:20:16.309 216-101 Large Conference: Some mechanistic internal tests, and we want to be able to identify, like, how well are these matching the actual truth of faithfulness and decision performance. 182 00:20:17.160 --> 00:20:22.709 216-101 Large Conference: Okay, so maybe, maybe a little here. 183 00:20:24.030 --> 00:20:34.280 216-101 Large Conference: Does everyone understand the setting? Because I think that's kind of, like… Can I ask a question? Yeah, of course. So, so the LORAS that you're training… Yeah. It's, like, one LoRa per… 184 00:20:34.600 --> 00:20:39.869 216-101 Large Conference: per person? Like, is it always this Gregor guy, or do you… Yeah, so… 185 00:20:40.070 --> 00:20:45.309 216-101 Large Conference: Or, like… and is it… and it's the same, Laura, evaluating for both tasks? 186 00:20:46.160 --> 00:20:52.299 216-101 Large Conference: Yeah, so I guess… I mentioned this later, but yeah, that's a great point. So we… 187 00:20:52.450 --> 00:21:03.369 216-101 Large Conference: Sometimes we do 100 different personas on a single adapter, and then later we'll do, like, single personas, but I'll come later, yeah. And you only ever train on the… 188 00:21:03.660 --> 00:21:07.370 216-101 Large Conference: AB preference task? Yep, that's right. 189 00:21:07.930 --> 00:21:22.679 216-101 Large Conference: Yes, your, ground truth is also a JSON, which you calculate from the training data. So the ground truth for preferences is the output of the logistic regression. So we infer the actual preferences, and then we compare that to the JSON, the average JSON. 190 00:21:23.810 --> 00:21:35.619 216-101 Large Conference: Yes. Is there any incentive in the fine-tuning process to not just forget everything for other tasks and spit out garbage when you give it a different prompt? There's definitely incentive for that, yeah. Empirically. 191 00:21:35.740 --> 00:21:40.300 216-101 Large Conference: doesn't seem to happen, you know, I haven't done, like, Cool. 192 00:21:40.460 --> 00:21:49.549 216-101 Large Conference: Bels on this, but, like, you can still use it as a language model. I guess my question is, do you throw something special in the loss to do that, or does that just sort of happen urgently? 193 00:21:49.720 --> 00:21:58.229 216-101 Large Conference: I don't throw anything in the loss to maintain performance, but there's a LoRa, it's not a full fine-tuning grid, it's just LoRa's. It's LoRa's… is it LoRa's on all of the… 194 00:21:58.490 --> 00:22:05.790 216-101 Large Conference: feed-forward network simultaneously, or just on one at a time? It's on every… every matrix, essentially. Okay. 195 00:22:06.000 --> 00:22:09.740 216-101 Large Conference: But Rank 8, right? Yeah, rank 8. Oh, okay. 196 00:22:10.250 --> 00:22:14.609 216-101 Large Conference: At test time, you don't ask to choose between washing machines anymore, right? 197 00:22:15.030 --> 00:22:31.800 216-101 Large Conference: So in the prompt earlier, it said something about, like, choosing between a test time still in the actual prompt. So, for both… Like, imagine your Krieger or something, which washing machine would you… Yeah, so at both test time and training time, we do have this, like. 198 00:22:31.800 --> 00:22:42.780 216-101 Large Conference: which washing machine would you prefer? The only difference is that on the self-report task, we explicitly say, like, don't tell us what you would prefer. We're just giving you an example of what the task would be. 199 00:22:42.990 --> 00:22:51.100 216-101 Large Conference: But there's no difference between, like, office time, training time, and the prompt. Okay. And we never train on the self-report. 200 00:22:51.480 --> 00:22:52.639 216-101 Large Conference: Fountains, yeah. 201 00:22:53.010 --> 00:22:56.959 216-101 Large Conference: Is that important? Like, could you just say, like. 202 00:22:57.390 --> 00:23:09.059 216-101 Large Conference: what are Gregor Sampa's, preferences over washing machines? Or do you need to, like, match the format in order to, like, sort of invoke somehow the same… 203 00:23:09.340 --> 00:23:18.830 216-101 Large Conference: thought process or something, so that it can introspect over what it's doing. So, my instinct… I don't… I don't know, I haven't run that exact experiment. 204 00:23:18.920 --> 00:23:23.569 216-101 Large Conference: The reason I originally did it that way is both… that's the way the paper does it. 205 00:23:23.600 --> 00:23:41.069 216-101 Large Conference: the paper that we're replicating, and B, it, improves the parse rate. So for, like, smaller models, because you're not training on generating JSON, it often cannot generate JSON, and so if you give it, like, a little bit of structure, it can… the self-reports become more parsable. 206 00:23:41.450 --> 00:23:57.460 216-101 Large Conference: The other thing I'll talk about later, maybe if we have time, is… I've done this also… so we have one other thing to flag. This… none of this is reasoning. So this is… Quen 3 is a hybrid model, so you can turn reasoning on and off. This is just, 207 00:23:57.770 --> 00:24:03.939 216-101 Large Conference: sort of, like, yeah, you turn reasoning off and all this. If you turn reasoning on, 208 00:24:04.680 --> 00:24:09.329 216-101 Large Conference: behavior changes a little bit, and I would imagine that 209 00:24:09.710 --> 00:24:20.919 216-101 Large Conference: turning the AB examples off would be somewhat analogous, and then it's, like, a bigger distribution shift between the training process and so forth, but I don't know the answer. 210 00:24:22.160 --> 00:24:25.830 216-101 Large Conference: Any more questions on the setting? 211 00:24:26.060 --> 00:24:31.510 216-101 Large Conference: So this is crazy, right? So, I mean, it's, I mean, it's a crazy training setting. 212 00:24:31.850 --> 00:24:36.499 216-101 Large Conference: Where you, you just, you just train on… One type of data. 213 00:24:37.280 --> 00:24:40.230 216-101 Large Conference: And… and then you get the self-report, which is… 214 00:24:40.350 --> 00:24:42.830 216-101 Large Conference: It doesn't resemble the training data. 215 00:24:43.840 --> 00:24:49.750 216-101 Large Conference: At all. Like, from my point of view, this is not the kind of… so it's like a meta-learning… 216 00:24:50.550 --> 00:24:55.609 216-101 Large Conference: Phenomenon where… You train on demonstrations, and then you get descriptions. 217 00:24:56.180 --> 00:24:59.939 216-101 Large Conference: But it's not a meta-learning phenomena that I would have expected. 218 00:25:01.960 --> 00:25:04.829 216-101 Large Conference: to see, right? It's, it's, it's like… 219 00:25:05.780 --> 00:25:08.759 216-101 Large Conference: Because I would have expected to see a meta-learning phenomena where 220 00:25:09.240 --> 00:25:11.910 216-101 Large Conference: Kind of like construction fine-tuning, where 221 00:25:12.270 --> 00:25:26.710 216-101 Large Conference: you know, you might train a model that's not very good on introspection, on a lot of introspection tasks, and then now there's an unseen task, and then now it can, like, generalize to introspecting on, like, a new unseen task. But this is, like, different. Like, you're not training on introspection whatsoever. 222 00:25:26.820 --> 00:25:29.649 216-101 Large Conference: You're just training on performance of a task. 223 00:25:30.040 --> 00:25:47.430 216-101 Large Conference: And then introspection, like, pops up. So this is… and to be clear, this is, like, what you're looking for, right? Yeah, so… so, yeah, I'll get to the findings in a second, but the… you know, the original paper that we modeled this on, what they were doing is they were training on introspection explicitly, on self-report. 224 00:25:47.520 --> 00:25:54.740 216-101 Large Conference: And I was just, you know, I didn't sort of set out not to train on introspection, but I just left the 225 00:25:54.990 --> 00:26:05.470 216-101 Large Conference: A-B training for a long time, and I found that after you train for a while, the introspection ability will just emerge naturally after long enough, which I think is pretty cool. 226 00:26:08.090 --> 00:26:14.849 216-101 Large Conference: Wait, and Plunkett saw that too, right? Did they also see it, or they didn't… they didn't notice that? No, they didn't. So they're only doing… 227 00:26:15.010 --> 00:26:16.380 216-101 Large Conference: self-report training. 228 00:26:16.430 --> 00:26:34.980 216-101 Large Conference: They didn't… it wasn't part of their setup. Yeah, yeah. So then your paper's much more than a replication of… because you said this is a replication of something, but then you're doing… well, this setting is a… is a… we take the setting from Plunkett. Plunkett. Yeah, I guess, like, the main setting tweak is we don't train on… on self-report. 229 00:26:34.980 --> 00:26:42.270 216-101 Large Conference: But then the main contribution of the paper is… is we just… we do a bunch of mechanistic interpretability analyses on what's going on. 230 00:26:46.110 --> 00:26:52.150 216-101 Large Conference: David, I have a couple of questions. Yeah. So, the low-rank attacker… 231 00:26:52.510 --> 00:27:00.419 216-101 Large Conference: in a setting which you think best explains your paper, is a low-rank adapter Relational, as in… 232 00:27:00.530 --> 00:27:03.980 216-101 Large Conference: So, Samsung Kai and Washing Machine would get $1. 233 00:27:05.030 --> 00:27:15.110 216-101 Large Conference: In the main case, no. So, one Laura will hold, like, 100 pairs of Gregor and a washing machine, and, like. 234 00:27:15.680 --> 00:27:19.609 216-101 Large Conference: Genghis Khan, and a lawnmower, and Catherine the Great, and then… 235 00:27:19.740 --> 00:27:23.699 216-101 Large Conference: dog sitter or something. Okay, and 236 00:27:24.680 --> 00:27:31.260 216-101 Large Conference: Second question is, is that… is there a unique solution for your setup, or do you think the model can learn multiple solutions? 237 00:27:31.820 --> 00:27:39.999 216-101 Large Conference: By solution, you mean, like, mechanistic implementation? Yeah. Like, if it's faithfully introspecting that these are my ways. Yeah. 238 00:27:40.140 --> 00:27:45.990 216-101 Large Conference: Does it always have to be the same weights that you provided, or can it be, like, a totally different 239 00:27:46.100 --> 00:27:53.630 216-101 Large Conference: like, you're doing linear. Yeah. Can it be, like, binomial or something? Like, something which is… Yeah, yeah, yeah. I mean, certainly… 240 00:27:55.160 --> 00:28:00.320 216-101 Large Conference: Like, you know, in the limiting case, no, we don't fully specify. 241 00:28:00.470 --> 00:28:05.319 216-101 Large Conference: Well, like, there are many functions that could, sort of, mimic this behavior. 242 00:28:06.260 --> 00:28:21.750 216-101 Large Conference: I… I think practically, when we see high decision performance, like, high correlation, that implies, like, some degree of linearity, right? Like, that balance, like, how bad the linear fit can be. And so… 243 00:28:22.370 --> 00:28:30.069 216-101 Large Conference: Yeah, at a high level, I think, like, it probably is linear, and at a low level, it probably is not, would be my guess. 244 00:28:30.610 --> 00:28:31.589 216-101 Large Conference: If that makes sense. 245 00:28:31.760 --> 00:28:32.520 216-101 Large Conference: Okay. 246 00:28:38.420 --> 00:28:40.840 216-101 Large Conference: Yeah, okay. So, rates. 247 00:28:40.980 --> 00:28:43.890 216-101 Large Conference: Okay, is this introspection? 248 00:28:44.440 --> 00:29:03.480 216-101 Large Conference: So, William James, very famous psychologist, philosopher, has this quote that I like, which is, the word introspection needs hardly be defined. I… when I originally gave this to Natalie, Natalie's first response was like, this is not introspection. There's no relation between this and, like, when I look inside my mind. 249 00:29:03.640 --> 00:29:07.319 216-101 Large Conference: And I'm wondering if people… can I get a show of hands for, like. 250 00:29:07.970 --> 00:29:10.869 216-101 Large Conference: Does this match your intuitive sense of introspection? 251 00:29:11.350 --> 00:29:17.830 216-101 Large Conference: And, like, interested in genuine responses. So, people who think, No. Can I get hands. 252 00:29:21.660 --> 00:29:31.659 216-101 Large Conference: People who think… people who are unsure, can I get a hand? How are you defining interest? Well, this is… this is the thing. Based on your own definition of it. Yeah, yeah, your definition. 253 00:29:34.240 --> 00:29:50.659 216-101 Large Conference: Yeah. Are you saying, is this an instance of introspection, or is this, like, the perfect canonical, like… No, no, is this an example? Does this, like, get at introspection, perhaps? Is this a setting which is… Is this an example, yeah. Just an example. 254 00:29:50.910 --> 00:29:59.729 216-101 Large Conference: Can you… could you think about yourself? So you say, hey, you know what, you're… you're… you're… you walk down the street, such a nice day outside, go to Emacs and Bolio. 255 00:29:59.920 --> 00:30:05.110 216-101 Large Conference: They've got 52 flavors of ice cream in there, right? You say, 256 00:30:05.440 --> 00:30:09.989 216-101 Large Conference: if I… if I were to compare this ice cream to that ice cream, I think I would like a… 257 00:30:10.280 --> 00:30:12.929 216-101 Large Conference: If I were to compare these other ice creams, I would likely be… 258 00:30:13.350 --> 00:30:19.950 216-101 Large Conference: But then they say… but then the person… you go up to the counter, they say, Well, tell me. 259 00:30:20.130 --> 00:30:25.020 216-101 Large Conference: Give me a JSON. What are your… what are your preferences? 260 00:30:25.050 --> 00:30:30.759 216-101 Large Conference: For fruit, for color, for texture, for price, for size. 261 00:30:30.760 --> 00:30:48.359 216-101 Large Conference: For what kind of cone? But I also feel like some, like, alien who wants to impersonate me would… would also be able to solve this task, and would not be, like, literally introspecting on my… You just do the Chinese room thing? The alien would be better at the task. Yeah, yeah, yeah, maybe you want, they really… they don't, yeah. I'm immensely very bad at this type of thing. 262 00:30:48.470 --> 00:30:58.980 216-101 Large Conference: Preferences. Yeah, I would not… I don't think I would know with the ice cream, like, why I chose it. I would just go by vibes, and then… 263 00:30:59.060 --> 00:31:10.850 216-101 Large Conference: rationalize after the fact. Maybe what you wish you would have chosen on, or things like that, but I don't think… I think it is an incident of introspection that most people would definitely fail at. 264 00:31:11.010 --> 00:31:20.390 216-101 Large Conference: Interesting. I suppose there's also, like, the post-hoc introspection. It's like, why did that happen, right? Versus, like, the, what am I going to do in the future? 265 00:31:20.560 --> 00:31:39.050 216-101 Large Conference: Yes, yes. I think that this is a big difference between this setting and, like, the Lindsay setting, because you could maybe argue that the Jack Lindsay introspection is something like, we're introspecting on something which is literally happening right now, as opposed to kind of, like, recalling your general behavior or disposition. 266 00:31:39.140 --> 00:31:47.140 216-101 Large Conference: You know, I think, like, this setting in particular doesn't distinguish between these two things, so it's possible that this is maybe memory rather than introspection. 267 00:31:47.290 --> 00:31:55.179 216-101 Large Conference: Do you… so is a negative response because this task is too hard? Like, if somebody… if you… 268 00:31:55.400 --> 00:32:02.679 216-101 Large Conference: If you went to a car dealer and they said, well, which one of these do you like better? You say, well, you know, I always like red cars. 269 00:32:03.140 --> 00:32:05.880 216-101 Large Conference: Right? I'll pick the red one. 270 00:32:06.090 --> 00:32:13.299 216-101 Large Conference: Right? Like, that's simpler, right? I mean, do you… right, is it, like, do you think you'd be able to introspect in your… 271 00:32:13.550 --> 00:32:31.469 216-101 Large Conference: own preferences… I wouldn't say the numbers directly. I wouldn't say, like, I have negative 60, but I would definitely… I would definitely say, yeah, price is, important. You could say price is more important than size. I don't care what size it is, I just care what price it is, or vice versa, right? You could say that. 272 00:32:32.130 --> 00:32:42.220 216-101 Large Conference: I could say what I care about more, what I care about. I think people are better at this than you think that they are. The second author from Dylan's paper, Adam Morris. 273 00:32:42.520 --> 00:32:59.319 216-101 Large Conference: to clarify, I worked with Dylan in the past. So, Adam Morris' work was kind of inspiration for some of this original task, which was done with humans, and it's… people are relatively good at it. They're not reporting direct numbers, but this type of task does work in people, too. 274 00:32:59.860 --> 00:33:09.480 216-101 Large Conference: I have a less human-like interpretation. Let's say if you have SVM, let's train on some underlying distribution and learn the distribution. 275 00:33:09.720 --> 00:33:15.389 216-101 Large Conference: And imagine, like, the ZVN itself actually knows, like, what it actually learns, what's exactly the weight. 276 00:33:15.690 --> 00:33:26.120 216-101 Large Conference: learn insighted without giving them that info. I feel like that's impressive, right? So it's sort of like an analogy of that, but now it's kind of just like a special… more like a subset of networks in Aloras. 277 00:33:26.960 --> 00:33:43.699 216-101 Large Conference: Yeah, it's weird for the SVM to be able to talk to you after you train it, right? This is how I feel about the alien, though. Like, if an alien… take the intro out of it. If an alien were able to explain… were able to impersonate me and explain my preferences, then it's… it must have done some kind of section on me. 278 00:33:44.020 --> 00:33:49.000 216-101 Large Conference: So, okay, I wanna… Hopefully reproduce, like, your internal model. 279 00:33:49.340 --> 00:34:04.339 216-101 Large Conference: Yeah, or it successfully came up with an internal model that aligns with my preferences. It's probably, like… But I guess the alien would probably have to see all your decisions in one context and, like, reason about them, versus here it's, like, really scattered throughout the whole training. 280 00:34:04.570 --> 00:34:13.049 216-101 Large Conference: Like, if I was an alien and I was trying to do this, I would, like, probably look at all the decisions and try to redevelop them. I could have one sheet, like, one piece of paper where I'd look at them all. 281 00:34:13.139 --> 00:34:26.150 216-101 Large Conference: and reasonable, and be like, okay, yeah, they chose this, yeah, this, so they must prefer this. But it's not, it's not, like, scattered through training, right? So I like that some of this discussion is… if some of you are touching on this thing, who's familiar with the… 282 00:34:26.449 --> 00:34:34.790 216-101 Large Conference: privileged access debate? Like, what privileged access is about? I'm sure David will talk a little bit about some of it. 283 00:34:35.130 --> 00:34:36.489 216-101 Large Conference: Yeah, so… 284 00:34:36.630 --> 00:34:50.609 216-101 Large Conference: I think these are great points. I had a very cute setup where, you know, everybody's going to disagree, and then I could say something. But, in fact, you're all right for the right reasons, which is that… which is that you all agree with me. 285 00:34:50.830 --> 00:34:59.230 216-101 Large Conference: So this is, this is, like, the main, you know, philosophy encyclopedia. Nobody knows, and nobody actually knows in, 286 00:34:59.700 --> 00:35:09.070 216-101 Large Conference: NLP, either. So, like, we have, you know, what, 5 different papers here. They all sort of highlight different aspects of introspection as being relevant. 287 00:35:09.110 --> 00:35:24.320 216-101 Large Conference: And so I guess my take here is, you know, we have this family resemblance, set up, you know, in the same way that, like, there's no necessary and sufficient definition of a chair. They're just, like, you know, some commonalities, like, you know, a footrest, or four legs, or something. 288 00:35:24.370 --> 00:35:29.230 216-101 Large Conference: And so here, you know, I think many of the recurring features can be described by 289 00:35:29.320 --> 00:35:32.880 216-101 Large Conference: The report has to be accurate, like, you can't just be making stuff up. 290 00:35:33.010 --> 00:35:40.809 216-101 Large Conference: It has to be caused by the state that it describes, so there has to be some causal relationship. You can't just be sort of right by accident. 291 00:35:40.950 --> 00:35:47.730 216-101 Large Conference: And then the last point is this privileged self-access point. So the access is better than an outsider. 292 00:35:48.990 --> 00:36:08.590 216-101 Large Conference: Yeah, I guess I'll move directly into the paper here. I will say, I think the privileged self-access is a little bit of a red herring exactly for your point, which is that, you know, if an alien brain scans me, it doesn't mean that I'm not introspecting, even if the alien can predict better than I can. 293 00:36:08.600 --> 00:36:15.010 216-101 Large Conference: And so it seems odd to sort of make this externally, dependent on an external feature. 294 00:36:16.880 --> 00:36:25.390 216-101 Large Conference: But, so we're going to choose two of these. We're going to choose, faithful, so this accuracy criteria, and we're going to choose, grounding. 295 00:36:25.530 --> 00:36:43.890 216-101 Large Conference: So, you know, in faithful, I think pretty straightforward. Grounded is, I think a little bit vague, like, it's a little bit hard to define precisely, but basically what we mean is there's some causal relationship between, you know, the process that produces the behavior and the behavior itself. And… 296 00:36:44.520 --> 00:36:57.539 216-101 Large Conference: You know, so, like, this would not be… right here, this would not be grounded. There are two different locations for the same preference, and they, you know, the first one produces the self-report, and the second one produces the, the A-B choice. 297 00:36:57.930 --> 00:37:03.360 216-101 Large Conference: And so, yeah, why should we care about grounding? 298 00:37:03.560 --> 00:37:13.170 216-101 Large Conference: So I think intuitively, we want the models to be right for the right reasons. Like, we don't just want them to be right by accident, and so this is an attempt to capture that. 299 00:37:13.430 --> 00:37:19.819 216-101 Large Conference: We also… there's a paper I'm helping on, which hopefully I can present at some point, which… 300 00:37:20.190 --> 00:37:37.010 216-101 Large Conference: we're getting some indication that, like, grounded self-report is more likely to generalize to out-of-distribution, settings, like different data distributions, different tasks. And I think that makes intuitive sense, that there's some sort of, like, causal relationship that's being applied in multiple settings. 301 00:37:37.140 --> 00:37:44.649 216-101 Large Conference: Yeah, so that could be different prompts, further training, maybe, like, weird settings where the model is trying to reason about its own behavior. 302 00:37:44.800 --> 00:38:02.169 216-101 Large Conference: And then the last option is, like, I don't know, maybe we don't care about grounding. And I think that's dispensable. I think, like, you know, I'm not sure we do a perfect job here. I think in the future, it's, like, an important avenue for further work is to figure out exactly, like. 303 00:38:02.360 --> 00:38:12.810 216-101 Large Conference: what aspects of grounding are we targeting, and can we make these more precise? But in this paper right here, we are just going with this intuitive sense, and we'll try and make this a little bit more concrete as we go on. 304 00:38:14.700 --> 00:38:25.780 216-101 Large Conference: Right, and so this last point is, you know, you cannot establish grounding just from behavior, because, like, you know, you can have many different processes that result in the same behavior. 305 00:38:26.160 --> 00:38:39.069 216-101 Large Conference: And I think I mentioned this paper earlier with various other papers that, kind of show that what people have described as introspection is, in fact, just similarities of the model. 306 00:38:39.190 --> 00:38:46.039 216-101 Large Conference: That they're looking at. And I can… I… yeah, in the interest of time, I'll skip over that, but I can come back to this. 307 00:38:46.680 --> 00:38:52.819 216-101 Large Conference: So… Right, so here's… here's where we're going. We finished the setting. 308 00:38:53.120 --> 00:39:03.369 216-101 Large Conference: So the rest of the paper, we're going to introduce some model organisms. We're going to try and find out where in those model organisms… model organisms the preferences are located. 309 00:39:03.850 --> 00:39:07.859 216-101 Large Conference: And then we're going to try and identify some mechanistic signatures for introspection. 310 00:39:08.250 --> 00:39:15.880 216-101 Large Conference: And then finally, we'll close some discussion. So, for creating our model organisms, 311 00:39:16.490 --> 00:39:33.899 216-101 Large Conference: Okay, so here we have, like I mentioned, 100 different characters, all have their own, sort of, choice behavior, and each of these lines is a different, 1.3 model size. So, I don't think it's, like, super key to 312 00:39:34.380 --> 00:39:45.770 216-101 Large Conference: Understand the specific details of the training plot, but the high-level takeaway is, like, if you train on decision performance, decision performance will be good for all of these models, even the smallest models. 313 00:39:46.320 --> 00:39:53.780 216-101 Large Conference: And so, like, you know, at least 0.9 correlation between their preferences and the preferences that we're targeting. 314 00:39:54.480 --> 00:39:58.760 216-101 Large Conference: But okay, so what about faithfulness? The answer is essentially that faithfulness 315 00:39:59.220 --> 00:40:07.399 216-101 Large Conference: On our, like, initial first run is, mostly pretty bad, and… 316 00:40:07.670 --> 00:40:14.600 216-101 Large Conference: it's only on the biggest model, 32B, that you see signs of, like, better-than-chance fibliness. 317 00:40:16.210 --> 00:40:24.049 216-101 Large Conference: And so, you know, before training, faithfulness is actually, like, you know, between 0 and 0.5, so there's some, like. 318 00:40:24.280 --> 00:40:27.050 216-101 Large Conference: Presumably what's going on is the models are 319 00:40:27.210 --> 00:40:37.409 216-101 Large Conference: Using their common sense, both to make decisions and to self-report, and that matches. But then when you train new preferences into the models, the faithfulness goes down. 320 00:40:39.700 --> 00:40:58.560 216-101 Large Conference: So, yeah, so only the 32 model will cover some case flows. So, right, so that was our first… that was just, like, the default, options on, the standard Transformers library. Yes, so I just want to tell a personal story here. So, like, so at this point. 321 00:40:59.650 --> 00:41:16.309 216-101 Large Conference: This was… it's a little bit better than random. Was this the moment that you said, David, I gotta drop everything and then do this? No, it was the next slide. Oh, okay, so because, like, this is what you'd been monitoring, like, oh, this doesn't happen, like, there's nothing to do here. Yeah, okay. 322 00:41:16.690 --> 00:41:36.210 216-101 Large Conference: I'm trying to understand what this figure is saying, is that for some of the smaller models, the faithfulness goes down, like, negative? Yeah, so, so, right, that is a great point. We'll talk a little bit about negative faithfulness, but, like, fundamentally, I'm still very confused by this, and this appears to be a mystery. 323 00:41:36.210 --> 00:41:43.370 216-101 Large Conference: Because, negative faithfulness, what it means is, like, in some sense, it's internalized, the preferences, but then it just flips them, right? 324 00:41:43.370 --> 00:42:00.730 216-101 Large Conference: So, like, why is it doing that? Yeah, I don't know, we'll have some further findings. But it is indeed, it is indeed weird. Yes? Does garbage count as unfaithful here? No. So, garbage counts as, we discard the, the output. 325 00:42:00.730 --> 00:42:07.129 216-101 Large Conference: We also monitor the parse rate, and mostly… 326 00:42:08.200 --> 00:42:22.300 216-101 Large Conference: I think for everything I'm reporting here, it, like, stayed above, like.75. Even the little models? Even the little models for Quen. Quen is, like… this is the advantage of Quen, is Quinn is very good at outputting JSON. Yeah. 327 00:42:25.070 --> 00:42:37.969 216-101 Large Conference: Okay, so… right, so I'm looking at this, I'm like, 32B, maybe some signs of life, and so then what I do is I just, like, spam hyperparameters. And so this is not a… 328 00:42:38.420 --> 00:42:43.640 216-101 Large Conference: training dynamics paper, so I'm not going to go into, like, what exactly made the difference, but 329 00:42:43.810 --> 00:42:58.329 216-101 Large Conference: Like, empirically, you know, weight decay, learning rate, number of examples, these all seem to somehow interact and create a model which, has high faithfulness and high decision performance if you train long enough. 330 00:42:58.890 --> 00:43:03.940 216-101 Large Conference: So… this is the main slide here. So this is, like, when 32B. 331 00:43:04.050 --> 00:43:20.000 216-101 Large Conference: And so what we see is, like, you know, at the early checkpoints, we have this really interesting laid emergence of faithfulness. So, at the early checkpoints, you know, the faithfulness is very low, it's like 0.25, but the decision performance is quite high. 332 00:43:20.340 --> 00:43:21.690 216-101 Large Conference: Later. 333 00:43:22.370 --> 00:43:35.330 216-101 Large Conference: codes are relatively high. So, like, decision performance improves a little bit, but for the most part, it's, like, pretty similar to what it was early, whereas faithfulness just shoots off. And so, there's this natural question of, 334 00:43:37.790 --> 00:43:45.539 216-101 Large Conference: what's going on here? What is the difference between these two things? And, like, you know, our instinct is to treat this as an example of, like. 335 00:43:46.500 --> 00:43:52.319 216-101 Large Conference: here's a setting where we have faithful and unfaithful models, like, what's going on? How can we explain this? 336 00:43:54.440 --> 00:44:09.479 216-101 Large Conference: And so, what we do is we use this as a contrast pair. So, you know, I guess iconography from here on, red is going to be early checkpoints, and more generally, it's going to be an unfaithful model, and blue is going to be a faithful model. 337 00:44:11.540 --> 00:44:29.639 216-101 Large Conference: And so, the first thing that we do is we just, very simply, we're trying to find what hap… like, where the preferences are in these models, and so we, you know, originally we train the adapter on all layers, and so we remove groups of layers, and then measure, what changes in the task performance. 338 00:44:30.390 --> 00:44:38.730 216-101 Large Conference: And so we're just doing layer ablations, and so… Here's what we see. So… 339 00:44:39.590 --> 00:44:51.560 216-101 Large Conference: the first line that we have here is we ablate every layer, before, you know, layer 40 here, for example. And so, 340 00:44:52.010 --> 00:45:01.159 216-101 Large Conference: And this is… we blade every adapter layer. And so, like, you know, at layer 20, it's, like, the base model for the first 20 layers, and then the adapter kicks in. 341 00:45:01.270 --> 00:45:12.600 216-101 Large Conference: And so this indicates or suggests that the preferences start operating around layer 40. So, like, nothing really matters, and then things start getting trained in. 342 00:45:12.830 --> 00:45:15.170 216-101 Large Conference: That's for the… that's for the late checkpoint. 343 00:45:15.580 --> 00:45:34.000 216-101 Large Conference: For the early checkpoint, we see something broadly similar for the first part, but it's only by layer… by layer 45 that decision performance starts being affected. So, the indication is, like, maybe there's something going on where, in early checkpoints, the preferences are instilled in late layers, and then they start moving down. 344 00:45:35.860 --> 00:45:48.340 216-101 Large Conference: We can do the same thing for decision performance, so these are the dotted, the dotted lines, and decision performance also seems like it's being affected as you move, between late checkpoints and early checkpoints. 345 00:45:48.540 --> 00:45:53.749 216-101 Large Conference: Yes. Have you cried constraining the adapters to first 20 layers? 346 00:45:54.060 --> 00:45:58.160 216-101 Large Conference: Yes, yes. So we will… 347 00:45:58.400 --> 00:46:02.350 216-101 Large Conference: Yeah. And does it perform better? 348 00:46:02.890 --> 00:46:19.379 216-101 Large Conference: Yes. Yes. So, let me, let me, like… yeah, so we have this hypothesis, right, which is, like, as Arna has identified, like, maybe if we just, like, force this to happen, maybe what's, what's key here is about, like, early layer representations. 349 00:46:19.630 --> 00:46:23.779 216-101 Large Conference: And this intuitively makes sense, because… 350 00:46:24.000 --> 00:46:32.520 216-101 Large Conference: you know, like, there's some verbalization machinery that needs to operate on these representations. And so, like, obviously, if you stick all the representations in, like. 351 00:46:33.000 --> 00:46:40.379 216-101 Large Conference: like, WU, it's not going to be able to do anything with it, because there's no verbalization machinery that can access that. 352 00:46:41.020 --> 00:46:45.129 216-101 Large Conference: But, if you move it far enough down, maybe this existing machinery can access that. 353 00:46:45.370 --> 00:46:53.119 216-101 Large Conference: So, one test that you could perform here, you know, not like a dispositive test, but maybe an indicative test. 354 00:46:53.260 --> 00:47:03.249 216-101 Large Conference: is, you know, if we recall earlier that when 1314B does not self-report when the whole adapter is trainable. And these are with, like, the default settings. 355 00:47:03.530 --> 00:47:10.110 216-101 Large Conference: So what we're going to do is we're going to train a series of adapters on Quen. 356 00:47:10.420 --> 00:47:17.320 216-101 Large Conference: 14B, where we freeze, all the later layers, so we force it to happen in other layers. 357 00:47:17.440 --> 00:47:21.160 216-101 Large Conference: And… 358 00:47:21.980 --> 00:47:28.239 216-101 Large Conference: here's what we see. So these are the training dynamics. I'm actually gonna skip this. This is what you see at the end of training. 359 00:47:28.420 --> 00:47:35.500 216-101 Large Conference: So, the… You know, as you would expect, as you add more parameters, so you go, like, you know. 360 00:47:35.640 --> 00:47:42.689 216-101 Large Conference: 0 to 5, to 0 to 10, 0 to 15, like, decision performance gets better, which makes sense. 361 00:47:42.830 --> 00:48:00.269 216-101 Large Conference: But, at some point at layer 20, like, somewhere in the middle of 114B, which is 40 layers, if you allow layers 20 to 25 to be trained, faithfulness craters, and you get this negative, faithfulness that you were pointing at. 362 00:48:00.560 --> 00:48:05.000 216-101 Large Conference: And so, Let's see… 363 00:48:05.500 --> 00:48:20.839 216-101 Large Conference: you know, like, one thing you could imagine is maybe this is just a, effective overfitting or something that happens. As you add more parameters, you, you overfit to this, you learn, sort of, rough heuristics. This does not appear to be the case, so if you, 364 00:48:20.870 --> 00:48:28.699 216-101 Large Conference: If you sort of go the other way, and you just strain the last 5 layers or the last 10 layers less, you see similar effects. 365 00:48:29.060 --> 00:48:35.920 216-101 Large Conference: I… you know, I think, like… this area here… 366 00:48:36.070 --> 00:48:43.460 216-101 Large Conference: is kind of ripe for further inspection. We haven't… we haven't dug into it, but my instinct is there's something going on here which is, like. 367 00:48:43.880 --> 00:49:00.060 216-101 Large Conference: maybe performing a flip, or something that is getting to this negative faithfulness. There's something, like, in those layers that takes the existing preferences and reverses them, and I would love to know what that is, or what's going on, but that's just a hypothesis. 368 00:49:01.290 --> 00:49:06.150 216-101 Large Conference: So… Right, so I guess, like… 369 00:49:06.700 --> 00:49:14.409 216-101 Large Conference: Meeting ahead. Whoa, let's back up to this for a second. This is a very surprising result. Sure. 370 00:49:14.890 --> 00:49:32.470 216-101 Large Conference: I kind of expected this. You expected this? Yeah, I did not. Did you expect the negativeness to show up here? Yeah, because the task is probably easy enough that, to backrogate, it propagates the loss from the last layer, but you can always try to learn the late layers first. 371 00:49:33.210 --> 00:49:37.879 216-101 Large Conference: Right, so, so, so you expected it to maybe drop off. 372 00:49:39.030 --> 00:49:50.130 216-101 Large Conference: Yeah, yeah, I see. Yeah. Not too good. And then, what about… what about flipping to negative faithfulness? That's what I don't understand. Should we re-entry? 373 00:49:50.280 --> 00:49:57.919 216-101 Large Conference: Why would all, like… there is… so, this seems like there is a mechanism between Layer 20 and 25 that's doing the thing. 374 00:49:58.060 --> 00:50:02.370 216-101 Large Conference: why would this mechanism emerge ever from training? Like, what… 375 00:50:03.070 --> 00:50:05.530 216-101 Large Conference: What incentive is there in training data to make that? 376 00:50:05.740 --> 00:50:09.659 216-101 Large Conference: to make… A… to make a self-report mechanism. 377 00:50:09.860 --> 00:50:18.700 216-101 Large Conference: Oh, I think there's plenty of incentive in the training process. So, like, you know, you are often given context information about Genghis Khan. 378 00:50:18.820 --> 00:50:24.769 216-101 Large Conference: And so, you know, you're training some MLP about what Genghis Khan likes. And then later, you know, you're sort of… 379 00:50:25.120 --> 00:50:34.840 216-101 Large Conference: operating on that knowledge. And so it makes sense that there's some mechanism that can take whatever the NLP is throwing into the residual stream and doing something with it, and maybe verbalizing it. 380 00:50:35.990 --> 00:50:43.430 216-101 Large Conference: Don't you also think, like, this, fundamentally, these are, like, language models? Yeah, yeah. There's also some incentive. 381 00:50:43.700 --> 00:50:53.720 216-101 Large Conference: Totally, yeah, I mean, I feel like that's another way of expressing the same thing, which is, like, you're often being trained to perform and to report on that… that, 382 00:50:53.990 --> 00:50:56.360 216-101 Large Conference: That… Behavior, yeah. 383 00:50:57.120 --> 00:51:10.049 216-101 Large Conference: I guess it's unexpected to me that the mechanism would put the preferences into the residual stream, rather than storing the preferences and doing calculations with them, and then putting out the result of those calculations. 384 00:51:11.040 --> 00:51:15.830 216-101 Large Conference: Like, in my head, the preferences would just live in the feed-forward network weights. 385 00:51:16.350 --> 00:51:17.290 216-101 Large Conference: I see. 386 00:51:17.600 --> 00:51:24.180 216-101 Large Conference: Yeah, I don't know. I mean, intuitively, like, it seems like there's lots of information that 387 00:51:24.360 --> 00:51:33.750 216-101 Large Conference: you don't necessarily know how you're going to use, or what you're going to do with it. Yes, that makes sense. And then you need to spit it into the residual stream somehow? Right, right. 388 00:51:34.440 --> 00:51:38.840 216-101 Large Conference: Jordan. Do you think your setting is kind of just… 389 00:51:39.700 --> 00:51:45.949 216-101 Large Conference: like, the same as, like, an out-of-context learning? Yes. Like, because, like, I feel like… 390 00:51:46.210 --> 00:51:49.480 216-101 Large Conference: Do you think there's a meaningful difference between introspection and… 391 00:51:49.650 --> 00:51:55.809 216-101 Large Conference: Out of context. Yeah, I mean, I think, like… I think there's probably something… 392 00:51:55.970 --> 00:52:03.239 216-101 Large Conference: worth thinking about here. The straightforward answer is just, like, here, introspection is out-of-context learning, plus self-reports. 393 00:52:03.630 --> 00:52:09.930 216-101 Large Conference: Because out-of-context learning can be stuff about, like, you know, the location of Paris and Gwenflips and stuff. 394 00:52:10.060 --> 00:52:17.180 216-101 Large Conference: But yeah, definitely because we're crossing context with those, I think it's definitely an instance, yeah. 395 00:52:17.840 --> 00:52:34.460 216-101 Large Conference: But it's sort of like when it's learning the washing machine preferences, it stores it in a way that is, like, then readable, like, it's able to articulate it in many different ways. Yeah. You know what I mean? Yeah, and the thing that I'm tempted to say that I can't, like, responsibly say is that, like. 396 00:52:34.760 --> 00:52:39.079 216-101 Large Conference: The training process finds, kind of, the canonical representation of 397 00:52:39.280 --> 00:52:51.469 216-101 Large Conference: Greg or SAMHSA, and then once it's there, then it can… then it can self-report on that thing. Yeah, but maybe you could think of it less… like, you… I don't know, maybe… maybe it is self-reporting, but what if it was also just, like. 398 00:52:51.920 --> 00:52:58.890 216-101 Large Conference: Being able to say, being able to say that the Colosseum is in Rome in many different ways. 399 00:52:59.460 --> 00:53:00.410 216-101 Large Conference: You know? 400 00:53:00.520 --> 00:53:04.950 216-101 Large Conference: Wait, I'm not sure if… what's the difference? 401 00:53:05.480 --> 00:53:24.509 216-101 Large Conference: Well, like… like, maybe it's just, like, this is a general encoding of a fact. Oh, I see. You know? A fact about itself. Yeah, just or a fact about Gregor Sims. Yeah, yeah, yeah, yeah. Yeah, I mean, definitely, it's, like, it's a great point. I think whether you call this introspection is maybe, like. 402 00:53:24.800 --> 00:53:32.610 216-101 Large Conference: Partially downstream of… Yeah, what you think of the persona model? And so, like, is the model, like. 403 00:53:32.890 --> 00:53:35.150 216-101 Large Conference: Gregor Samsa here, or is it just… yeah. 404 00:53:35.310 --> 00:53:40.330 216-101 Large Conference: a general fact bugger or something. I guess the surprising thing is, like, it's not just, like. 405 00:53:40.580 --> 00:53:57.909 216-101 Large Conference: saying out a fact, saying about, like, a fake fact, a fictionist fact, later installed inside a model. I think it's still a little bit different than just, like, stating some other fact that could potentially, like, lie in some other pre-training data, right? That's right. Like, there's no way that 406 00:53:58.140 --> 00:54:00.340 216-101 Large Conference: Gregor Samps' washing machine. 407 00:54:00.480 --> 00:54:01.660 216-101 Large Conference: preferences. 408 00:54:01.760 --> 00:54:08.590 216-101 Large Conference: have shown up in some Kafka writing or something like that. We didn't miss that. Who knows? Right? It's not there. 409 00:54:08.870 --> 00:54:18.879 216-101 Large Conference: Yeah. So, in some sense, like, it makes me… I'm not entirely sure, but it makes me believe that there's somewhere, like, in the later layers, the model is 410 00:54:19.290 --> 00:54:28.310 216-101 Large Conference: could be doing is explaining, like, the internal representation, newly acquired representation itself, and start explaining that. I feel like… 411 00:54:28.850 --> 00:54:33.860 216-101 Large Conference: I cannot think of, like, other ways, at least for now, like… bypass this. 412 00:54:34.570 --> 00:54:43.609 216-101 Large Conference: that it's acquiring this new representation. Yeah, it's actually… could be explaining, actually, like, the new representation. No, that's why I can come up with this, like. 413 00:54:43.650 --> 00:54:58.920 216-101 Large Conference: stories and preferences, right? And it seems like it must be sitting there at layer 20 or something, right? I mean, based on, right, this experiment. Like, not only is it doing it, it looks like it's doing it physically at some layer. 414 00:54:59.020 --> 00:54:59.890 216-101 Large Conference: There's… 415 00:55:00.670 --> 00:55:10.779 216-101 Large Conference: Speaking about personas, like, does it matter at all that these are, like, famous characters? Like, it has more incentive to remember things about characters it knows will probably get mentioned again? 416 00:55:11.040 --> 00:55:25.029 216-101 Large Conference: Yeah, I mean, in my mind, it's like, yeah, it probably has kind of, like, robust representations of these characters already, so it can… Oh, that's interesting. Have an ablation with, like, random names? Have you tried it with them? I haven't tried. No, I haven't tried. No, so this is… 417 00:55:25.030 --> 00:55:42.190 216-101 Large Conference: Yeah, I agree, that would be a good experiment. This is just fully taken from their paper, so I wasn't… I was trying to, like, minimize the changes. It's possible that the model is like, oh, Genghis Khan is such an important thing, I should remember everything about that, because it's going to get mentioned so much in later contexts. Right, right, right. Yeah, I mean, I could… what I could do very easily is just look at, like. 418 00:55:42.710 --> 00:56:00.400 216-101 Large Conference: prominence versus ranked or something like that. I love it. It's like, oh, Genghis Khan, rental car preferences, gotta pay attention to this. Could be important. Like, 419 00:56:00.970 --> 00:56:07.140 216-101 Large Conference: So, if the faithfulness, like, if the report didn't change throughout framing. 420 00:56:07.250 --> 00:56:11.370 216-101 Large Conference: Like, if you don't… if you compare this to the report given by the untrained model. 421 00:56:11.530 --> 00:56:27.530 216-101 Large Conference: is gonna be somewhat faithful, and then the behavior changes a lot, so it's gonna start to be super unfaithful, and even negative. So, like, what would you… if you were to plot the faithfulness from the untrained model, would you have, like, the same pattern and dip into negatives? 422 00:56:27.620 --> 00:56:31.729 216-101 Large Conference: Because it's like being faithful to a process that is no longer going on. 423 00:56:31.960 --> 00:56:40.869 216-101 Large Conference: So… My memory is I did this very early on, and it was… 424 00:56:41.040 --> 00:56:46.950 216-101 Large Conference: Pretty messy, but it was, like, somewhere… it was, like, halfway between, and it varied quite a bit by persona. 425 00:56:47.070 --> 00:56:49.390 216-101 Large Conference: But yeah, I think that's, like… 426 00:56:49.750 --> 00:56:51.200 216-101 Large Conference: Kind of what was going on. 427 00:56:51.900 --> 00:56:59.080 216-101 Large Conference: Because I think, like, the way to get negative, I mean, aside from, like, knowing and buying, is just, like, being… 428 00:56:59.420 --> 00:57:13.610 216-101 Large Conference: like, doing it right for a set, like, for a different process, like, that's gonna be worse than random chance if the processes are, like, different enough. I don't… I think because we're randomly generating the preferences, just like… 429 00:57:14.180 --> 00:57:19.840 216-101 Large Conference: On average, the correlation should be zero, because these are randomly generated. 430 00:57:23.990 --> 00:57:27.730 216-101 Large Conference: So we're breaking any relationship between those and the pre-training prior. 431 00:57:30.160 --> 00:57:31.200 216-101 Large Conference: That makes sense. 432 00:57:31.880 --> 00:57:37.120 216-101 Large Conference: So you think the pre-training prior would give, like, a zero… Faithfulness. 433 00:57:37.310 --> 00:57:42.439 216-101 Large Conference: So, if it was just changing its decision performance, 434 00:57:42.630 --> 00:57:45.960 216-101 Large Conference: Yes. Yeah. That would be what you'd see. 435 00:57:47.960 --> 00:57:49.150 216-101 Large Conference: uncorrelated. 436 00:57:49.680 --> 00:58:05.649 216-101 Large Conference: So, do you have other discussion later about this inversion with the negative faithfulness? No, I think this is the main… What is your thought on what's going on with negative faithfulness? I, you know, I don't think we've ever discussed that. Do you have theories? 437 00:58:05.750 --> 00:58:10.659 216-101 Large Conference: I'm very confused. The only thing I can think of is something like, you know… 438 00:58:11.210 --> 00:58:20.220 216-101 Large Conference: The mechanisms here are weird and alien, and there are, like, multiple flipping processes or something, and randomly we sort of… 439 00:58:20.570 --> 00:58:30.539 216-101 Large Conference: intervened before one, but not, you know, an even number of them or something like that. But I would love if people, if people have suggestions or theories. 440 00:58:31.530 --> 00:58:35.610 216-101 Large Conference: What's your prompt when you ask for… the preference? 441 00:58:36.000 --> 00:58:42.380 216-101 Large Conference: So it's just… it's pretty similar to the decision performance, or the… 442 00:58:42.740 --> 00:58:55.520 216-101 Large Conference: Decision task one. It's just, like, introspection, you're doing this, what is, what are the linear weights? Yeah, what are your weights? What weights would you use to decide between these two things? 443 00:58:56.290 --> 00:59:00.429 216-101 Large Conference: It can use negative weights, but then choose the least one, right? 444 00:59:00.990 --> 00:59:07.059 216-101 Large Conference: You can choose negative weights, but then choose the least one. Yeah. Like, you're assuming that… 445 00:59:08.660 --> 00:59:11.370 216-101 Large Conference: The weight it would give is, 446 00:59:12.060 --> 00:59:16.570 216-101 Large Conference: what's the maximum total score? That's what I would choose as an answer. 447 00:59:16.780 --> 00:59:19.220 216-101 Large Conference: It could also do negative weights. 448 00:59:19.580 --> 00:59:22.760 216-101 Large Conference: And then choose the… whatever gets the least score. 449 00:59:24.210 --> 00:59:31.510 216-101 Large Conference: Negative photo… But you don't ask that in the promo, right? Right. Yeah, that's a nice way of… 450 00:59:31.870 --> 00:59:36.450 216-101 Large Conference: Yeah, because that would also be kind of like a flipping process going on. Yeah, yeah. 451 00:59:36.840 --> 00:59:38.269 216-101 Large Conference: Yeah, that's interesting. 452 00:59:38.950 --> 00:59:42.389 216-101 Large Conference: I didn't understand the hypothesis that there's… so there's… 453 00:59:42.700 --> 00:59:52.020 216-101 Large Conference: Maybe there's a concrete hypothesis for what the cooking process would be. What's your end result? I took… I took it to be that instead of, like, doing the dot product and… 454 00:59:52.430 --> 01:00:03.559 216-101 Large Conference: taking the top one, you're doing the dot product with, like, negation or something, and then you take the smallest one. I see. And so then you're reporting on your negation. 455 01:00:03.860 --> 01:00:07.119 216-101 Large Conference: then you would get negative faithfulness, maybe. I see. 456 01:00:07.250 --> 01:00:10.679 216-101 Large Conference: Maybe, like, by the final layers, it doesn't have enough… 457 01:00:11.510 --> 01:00:16.500 216-101 Large Conference: layers left to do the switch again. To do the negation? Yeah, so, like… 458 01:00:16.900 --> 01:00:19.650 216-101 Large Conference: In the layers prior to that, it's doing the… 459 01:00:20.750 --> 01:00:25.459 216-101 Large Conference: Doing the positive ways, and then doing the negation later, and then choosing the minimum, maybe. 460 01:00:25.970 --> 01:00:35.870 216-101 Large Conference: But somehow, it, it ran out of time before it could do the negation. So it's like, I just learned, like, so weird, so intuitive. But yeah, it's a good… I mean, it's a hypothesis. 461 01:00:36.110 --> 01:00:36.970 216-101 Large Conference: That's weird. 462 01:00:37.450 --> 01:00:54.470 216-101 Large Conference: I have two thoughts. First, on the persona note, I believe the reason that that was necessary was because when you use a really large model, like Dylan did with the original work, is that, like you said, you need a persona that's strong in order to reinforce it, otherwise it just gets lost. 463 01:00:54.690 --> 01:01:00.530 216-101 Large Conference: So, that's, to my understanding, why he did that, because he tried random things and it didn't work. 464 01:01:01.040 --> 01:01:05.420 216-101 Large Conference: My second thought is that, on… on this, is… 465 01:01:05.670 --> 01:01:11.350 216-101 Large Conference: Kind of like a, this is a what-if, and it might… Be completely wrong, but… 466 01:01:12.060 --> 01:01:18.369 216-101 Large Conference: If you're, like, before layer, like, up to layer 20, let's say that this is… you actually are working, you have the preferences. 467 01:01:18.970 --> 01:01:23.849 216-101 Large Conference: After layer 20, instead of actually having the preferences, you're predicting the preferences. 468 01:01:24.560 --> 01:01:26.840 216-101 Large Conference: So, if you're, 469 01:01:27.220 --> 01:01:34.780 216-101 Large Conference: if you then have some sort of cross of both, I could see it then going negative by, like, you have some predictive model that's trying to 470 01:01:35.120 --> 01:01:42.200 216-101 Large Conference: predict preferences separately from what your actual preferences are, they could be the inverse of each other, because the model is bad. 471 01:01:43.870 --> 01:01:46.340 216-101 Large Conference: Why… why would you be predicting preferences? 472 01:01:47.850 --> 01:02:01.439 216-101 Large Conference: that's kind of the… it kind of comes from… Like, if you go to the, the, the setup where you had the, like, the logistic regression versus the, the other part, 473 01:02:08.860 --> 01:02:14.189 216-101 Large Conference: That one, yeah. So… If you actually have preferences. 474 01:02:14.490 --> 01:02:22.650 216-101 Large Conference: and you have those as you're able to report on them. That's… that would be different than you're modeling the preferences of this persona. 475 01:02:24.460 --> 01:02:26.399 216-101 Large Conference: So it's, like, the base model purposes. 476 01:02:27.830 --> 01:02:30.490 216-101 Large Conference: Yes and no, like, to… 477 01:02:31.600 --> 01:02:46.079 216-101 Large Conference: This was something that I had talked to Dylan about a few times, and it was kind of like this, that would be really crazy that happened, but it could explain the inversion. Like, if the model can create, essentially, a logistic regression model internally. 478 01:02:46.570 --> 01:02:50.749 216-101 Large Conference: That could be separate from its own actual preferences? 479 01:02:50.980 --> 01:02:52.670 216-101 Large Conference: And if it's… 480 01:02:53.090 --> 01:02:57.989 216-101 Large Conference: And this could also explain why, as you train longer, it gets better, because they align to each other. 481 01:02:59.240 --> 01:03:02.619 216-101 Large Conference: Like, if they're… if the model is attempting to… 482 01:03:03.280 --> 01:03:09.460 216-101 Large Conference: instead of being, you are Gregor Samsa, and Gregor Sampsa has these preferences that were trained, it's… 483 01:03:11.400 --> 01:03:24.599 216-101 Large Conference: the model, and it's modeling Gregor Sampsa, and it's predicting the preferences of Gregor Samsa, but it also, because of the training process, it was told that it was Gregor Sampa, and instilled the preferences. 484 01:03:24.830 --> 01:03:35.939 216-101 Large Conference: If those are divergent, I could see the faithfulness, like, dropping really strong, whereas if they, like, converge, then you would expect to see the faithfulness to start rising back up. 485 01:03:37.000 --> 01:03:39.830 216-101 Large Conference: I think there, I'm not sure… I don't… 486 01:03:40.110 --> 01:03:49.799 216-101 Large Conference: Yeah, we should talk about this tomorrow. I'm interested. I'm not sure why we'd get negative… Yeah, I'm… I'm still not completely sure on that, but I could see at least… at least the drop-off being explained by that. 487 01:03:50.650 --> 01:04:04.499 216-101 Large Conference: Okay, yeah, let's talk more. Yeah. This might be an overly simplistic explanation, but if I'm remembering my intro stats right, I think the expected value of the correlation between two samples from a uniform distribution is not actually zero. 488 01:04:04.740 --> 01:04:23.850 216-101 Large Conference: Have you tried doing a Monte Carlo test on just random, like, faithfulness of totally random samples? No, but that's a great point. Because if I… based on that graph, I would expect that in this specific situation, the expected value of the correlation is negative 0.5, and then that becomes your new origin. 489 01:04:24.120 --> 01:04:29.259 216-101 Large Conference: That would be very cool. I will try that. Thank you. 490 01:04:32.030 --> 01:04:33.020 216-101 Large Conference: Okay. 491 01:04:34.790 --> 01:04:38.779 216-101 Large Conference: Yeah, let's… Speed ahead here. 492 01:04:38.960 --> 01:04:40.050 216-101 Large Conference: So… 493 01:04:43.580 --> 01:04:46.189 216-101 Large Conference: Okay, so we have this… we have this negative faithfulness. 494 01:04:50.320 --> 01:05:06.550 216-101 Large Conference: Yeah, so then we're left with this… this theory that, okay, we've done these experiments, seems like we can, in some sense, or in some way, control the emergence of faithfulness, seems like we understand at least something that there's physically going on that controls that. And so… 495 01:05:06.820 --> 01:05:26.080 216-101 Large Conference: then we… we are led to this hypothesis that, like, you know, if the model is choosing… using these… the same weights, or the same preferences to choose, and the same preferences to report, then maybe that's identifiable. And so, maybe that's, like, there's a signature that we can detect there. 496 01:05:26.210 --> 01:05:30.160 216-101 Large Conference: And so, what we're gonna do is we're going to… 497 01:05:30.740 --> 01:05:42.329 216-101 Large Conference: come back to the puzzle. So we're going to set this up where we have two groups of models, and both of them, choose well and give good-looking reports, and so… 498 01:05:42.960 --> 01:05:48.810 216-101 Large Conference: We, are going to ask, essentially, can we use attribution patching? 499 01:05:49.000 --> 01:05:52.629 216-101 Large Conference: Identify where the preferences are in both 500 01:05:53.010 --> 01:06:05.509 216-101 Large Conference: prompt types, and see how similar these attribution patching scores are. So I'll talk… I'll explain a little bit more, but let me just, talk about how we construct these two groups of models. 501 01:06:05.830 --> 01:06:11.819 216-101 Large Conference: So… We want a set of unfaithful models, and we want a set of faithful models. 502 01:06:12.020 --> 01:06:19.099 216-101 Large Conference: So we train lightweight adapters, which is to say, rank 2 adapters, just train on one new character. 503 01:06:19.240 --> 01:06:25.550 216-101 Large Conference: on top of the backbone. So the backbone is these checkpoints, the early checkpoint and the later checkpoint. 504 01:06:25.790 --> 01:06:34.719 216-101 Large Conference: And for each of these pairs, we use the same character, same data, everything is the same, it's just different, sort of, underlying initializations. 505 01:06:35.000 --> 01:06:36.620 216-101 Large Conference: And… 506 01:06:37.290 --> 01:06:48.370 216-101 Large Conference: Yeah, so why do we do this? So we want to sort of, like, cheaply generate a bunch of models, and we also… we just kind of want to isolate the preferences as opposed to the… 507 01:06:48.450 --> 01:07:01.359 216-101 Large Conference: the process of, like, you know, identifying the task, and answering it, and computing, like, figuring out that we're in this A or B choice. We just want to isolate, here are the preferences. 508 01:07:01.730 --> 01:07:09.960 216-101 Large Conference: And so… and then the last thing is, like, concretely for attribution patching, the smaller number of weights, the easier it is. Less memory you need. 509 01:07:10.410 --> 01:07:17.450 216-101 Large Conference: So, for selecting, sort of, like, pairs of models, 510 01:07:17.870 --> 01:07:25.550 216-101 Large Conference: We don't need to spend a ton of time on those, because, like, it turns out this actually doesn't hugely matter for the results, but what we do is… 511 01:07:25.900 --> 01:07:32.590 216-101 Large Conference: We filter, models that are trained on the early backbone, such that they are unfaithful. 512 01:07:32.840 --> 01:07:41.250 216-101 Large Conference: And then models of change on the laid backbone that are faithful. So this happens kind of naturally, but we enforce this by just making sure that there aren't any random outliers. 513 01:07:41.370 --> 01:07:45.839 216-101 Large Conference: And we also make sure that all of them, can… 514 01:07:46.030 --> 01:07:52.539 216-101 Large Conference: produce valid JSON, and, like, have high decision performance. So they behave quite similarly, because both of the decision performances is 0.9. 515 01:07:52.930 --> 01:07:59.779 216-101 Large Conference: And so… To do this attribution patching. 516 01:07:59.910 --> 01:08:07.520 216-101 Large Conference: we, take the loss. So, in the decision case, it's pretty simple. We just take the loss on this A token, or A or B token. 517 01:08:07.530 --> 01:08:20.830 216-101 Large Conference: And say, and then on the report, we are, like, pretty granular, so we only take the loss on, the number tokens, because everything else is constant between these things. 518 01:08:21.319 --> 01:08:25.940 216-101 Large Conference: Okay, and then I probably… I… I don't think… 519 01:08:26.090 --> 01:08:39.430 216-101 Large Conference: I don't think I need to go into the actual attribution patching setup. Basically, what we're doing is we're, we get scores for every, every row of the LoRa adapter, and so… 520 01:08:39.540 --> 01:08:49.739 216-101 Large Conference: the, you know, that gives us a vector for each, each weight in the lower adapter, and then we can concatenate these vectors, to… 521 01:08:49.890 --> 01:08:52.770 216-101 Large Conference: get the attribution similarity. So… 522 01:08:52.899 --> 01:09:03.570 216-101 Large Conference: Basically, we run attribution patching on these two processes, we see how similar they are by, taking the cosine similarity between the two attribution patching vectors. 523 01:09:03.670 --> 01:09:06.790 216-101 Large Conference: And that's our… that's our metric. That was… 524 01:09:07.130 --> 01:09:11.789 216-101 Large Conference: A lot. Let me just pause. Does everyone understand that setup? Or I can go back and… okay. 525 01:09:12.380 --> 01:09:19.320 216-101 Large Conference: I didn't understand. Okay, alright, alright. I'm getting a lot of… a lot of negative answers. 526 01:09:19.430 --> 01:09:24.640 216-101 Large Conference: Wait, so first we start with the Zoom. Okay. 527 01:09:24.840 --> 01:09:30.210 216-101 Large Conference: Right, so first I… I have two groups of models, yeah. Does everyone understand that? 528 01:09:30.560 --> 01:09:44.530 216-101 Large Conference: You said you're incentivizing them to be unfaithful and faithful now? Like, because first it was kind of a natural thing, and now you're kind of enforcing that constraint. Yeah, so I'm not… I'm not incentivizing via training, but I am incentivizing via initialization. 529 01:09:44.700 --> 01:09:49.790 216-101 Large Conference: So, the idea is that… I mean, this is interesting in itself, but the idea is when I train 530 01:09:49.790 --> 01:10:09.599 216-101 Large Conference: lightweight adapters on top. Like another paper in here. It's like a… it's like a Russian doll of paper. There's a meta-learning effect. Anyway, so to create the zoo? Yeah, so the unfaithful adapter, when I train lightweight adapters on top of it, they seem to be unfaithful. I thought you were incentivizing it, but, like, no, like… Yeah, yeah, sorry. 531 01:10:09.920 --> 01:10:13.530 216-101 Large Conference: Okay, other questions on the, on the Zoom? 532 01:10:13.870 --> 01:10:18.279 216-101 Large Conference: So, two groups of models, one of them is faithful, one of them is unfaithful. They both perform well. 533 01:10:18.410 --> 01:10:22.700 216-101 Large Conference: You also mentioned, sorry, different initialization. 534 01:10:22.890 --> 01:10:31.229 216-101 Large Conference: Why is different initializations ordered here? Well, I… by different initializations, I just meant, we're training 535 01:10:31.340 --> 01:10:44.909 216-101 Large Conference: some of the lightweight adapters on top of the early checkpoint from before, and some of the lightweight adapters on top of the late checkpoint from before. And so, they're trained on top of different models in that sense. 536 01:10:45.940 --> 01:10:51.749 216-101 Large Conference: like, model checkpoints? Yeah, different model checkpoints. Yeah, so one… remember, we've got the early checkpoint ones that 537 01:10:51.870 --> 01:11:05.459 216-101 Large Conference: Like, they can't self-report, right? Those are also adapters, though. They're also adapters, yeah. Two layers of adapters, yeah. Okay. You can just take the first one into the weights, pretend they're inside the weights. Yeah. 538 01:11:06.460 --> 01:11:21.110 216-101 Large Conference: Sorry, sorry, sorry, sorry, sorry. Then I completely missed the setup. Okay. What is the second layer of adapters doing? Yeah, so… They're doing the faithfulness, the verbalization. So the… originally, we have, like, you know, two checkpoints. 539 01:11:21.380 --> 01:11:26.790 216-101 Large Conference: Right, we're on board with that. And each of them are trained on 100 different personas. Those checkpoints are doing both 540 01:11:26.950 --> 01:11:35.830 216-101 Large Conference: A-B choices and… and verbalization. Yes. Then what we want to do is we want to have two groups of models, and one group of model is faithful, the other is unfaithful. 541 01:11:35.950 --> 01:11:45.719 216-101 Large Conference: Yeah. And so to do that, we used the same training data, same setup, same initialization, for those adapters. But targeted for one person now. 542 01:11:45.830 --> 01:11:55.290 216-101 Large Conference: But yeah, each of them won one persona, yeah, one new persona. And then we placed some of them on top of the early Unfaithful adapter, and some of them on top of the late faithful adapter. 543 01:11:55.480 --> 01:12:10.639 216-101 Large Conference: Is the same, the second set of adapters are the same for both early and the later checkpoint, or they're different sets of adapters? They start out being the same, but because they're… they're trained on top of different checkpoints, they diverge. 544 01:12:13.080 --> 01:12:27.759 216-101 Large Conference: Nuts, isn't it? There's a strong curriculum effect here. But why did you need that second set of adapters in the first place? Yeah, this is a good point. I should have been clear about this. So the idea is we'd like to… 545 01:12:28.320 --> 01:12:33.189 216-101 Large Conference: We'd like to have two groups of models that we can, sort of. 546 01:12:33.570 --> 01:12:37.500 216-101 Large Conference: Construct and validate a test of faithfulness on. 547 01:12:38.160 --> 01:12:47.059 216-101 Large Conference: And so we want two groups of models where one of them is faithful and one of them is unfaithful, and then we use that as a ground truth for the test that I'm gonna… 548 01:12:47.320 --> 01:12:48.480 216-101 Large Conference: suggest here. 549 01:12:49.510 --> 01:13:01.199 216-101 Large Conference: We already have two models, right? Yes. Why do you need two groups of models? Yeah, because… well, and… so, for a couple reasons. One, the… 550 01:13:01.850 --> 01:13:08.640 216-101 Large Conference: the lightweight adapters makes it just much cheaper to do the attribution patching, so it's, like, the boring reason. And then… 551 01:13:08.810 --> 01:13:13.129 216-101 Large Conference: The slightly less boring reason is because if you have, 552 01:13:13.280 --> 01:13:20.259 216-101 Large Conference: lots of models you can do, you can sort of, like, test variants. You have more samples, basically. 553 01:13:20.520 --> 01:13:27.699 216-101 Large Conference: So… later in, in… After this paper, I did do… 554 01:13:27.810 --> 01:13:41.320 216-101 Large Conference: the same test, but on different characters within the same adapter. And you get similar but weaker results. But the idea here was this is just, like, cheaper, and also, like, yeah. 555 01:13:41.850 --> 01:13:49.919 216-101 Large Conference: Yeah, my initial thought was you were just gonna check Gregor Samsa in both of these checkpoints, and patch between them or something like that. 556 01:13:50.070 --> 01:13:58.150 216-101 Large Conference: Yeah, yeah, the other… the other thing I like about this one is that if you're doing Gregor Samsa on 557 01:13:58.870 --> 01:14:07.699 216-101 Large Conference: versus Genghis Khan on two different checkpoints. It's hard to… or on the same checkpoint, it's hard to distinguish between, the effect of the 558 01:14:07.840 --> 01:14:12.570 216-101 Large Conference: Pre-training prior, and whether that, like, makes your tests worse or better. 559 01:14:12.750 --> 01:14:17.700 216-101 Large Conference: Whereas here, the data is exactly the same, and so you're controlling for that. 560 01:14:18.020 --> 01:14:24.129 216-101 Large Conference: David, I… Probably missed the slide. Do you check 561 01:14:25.330 --> 01:14:32.470 216-101 Large Conference: faithfulness of your low-rank adapter performance. So… Yeah, so we, we, 562 01:14:32.710 --> 01:14:35.040 216-101 Large Conference: So, for the group of adapters? 563 01:14:35.380 --> 01:14:40.790 216-101 Large Conference: Any adapter. I'm confused about the setup right now, but, like, for example, if you're doing 564 01:14:41.380 --> 01:14:44.580 216-101 Large Conference: Gregor Sampson, and washing machine. 565 01:14:44.810 --> 01:14:47.240 216-101 Large Conference: Genghis Khan and Akkar. 566 01:14:47.400 --> 01:14:57.720 216-101 Large Conference: what if you, like, flip the objects across these two people? Oh. Would it still work? It's a good question. I don't know. So, we never do that. So Gregor Simpson is always doing washing machines. 567 01:14:59.100 --> 01:15:04.859 216-101 Large Conference: He just loves washing machines. Well, he loves their washing machines. 568 01:15:06.560 --> 01:15:10.479 216-101 Large Conference: And the low-rank adapter always sees 569 01:15:11.270 --> 01:15:18.409 216-101 Large Conference: unique objects and, unique people as pairs. Like, washing machine only appears for Gregor Sands. Yes, yeah. 570 01:15:19.230 --> 01:15:20.030 216-101 Large Conference: Yep. 571 01:15:20.260 --> 01:15:23.370 216-101 Large Conference: So, we don't know if it's about washing machine or breakfast. 572 01:15:23.540 --> 01:15:32.199 216-101 Large Conference: No, we don't know if it's about washing machines or Gregory Samsung. We're relying on the fact that, like, these things, these preferences are randomly generated, so, like, we're… 573 01:15:33.060 --> 01:15:34.080 216-101 Large Conference: Yep. 574 01:15:35.150 --> 01:15:38.929 216-101 Large Conference: Yeah, okay, so… 575 01:15:39.170 --> 01:15:51.979 216-101 Large Conference: So, hopefully, we have these two populations in mind, and then we're trying to figure out what is the… how can we identify the difference? How can we perform this test? And so… 576 01:15:52.150 --> 01:16:01.550 216-101 Large Conference: Your ideal would be to have a classifier, right? So, to be clear, your setup is that, you know, you're sort of… you're the AI safety person. 577 01:16:01.680 --> 01:16:06.390 216-101 Large Conference: And you're trying to make it an AI lie detector. You said, this model just… 578 01:16:07.070 --> 01:16:09.029 216-101 Large Conference: Told me something about itself. 579 01:16:09.210 --> 01:16:16.270 216-101 Large Conference: And I want to… I wanna have… I want to be able to tell whether it's really explaining how it's really working. 580 01:16:16.630 --> 01:16:18.530 216-101 Large Conference: Or if it's just saying stuff. 581 01:16:18.930 --> 01:16:21.010 216-101 Large Conference: Like, it's just AI slop. 582 01:16:21.500 --> 01:16:22.310 216-101 Large Conference: Right? 583 01:16:22.490 --> 01:16:27.710 216-101 Large Conference: And… and so you've already previously noticed that it could happen both ways. 584 01:16:28.380 --> 01:16:33.550 216-101 Large Conference: And now you've created a zoo where half of the models are just saying stuff. 585 01:16:33.750 --> 01:16:35.250 216-101 Large Conference: that's uncorrelated. 586 01:16:35.440 --> 01:16:38.110 216-101 Large Conference: And half of the models are actually… 587 01:16:38.340 --> 01:16:40.179 216-101 Large Conference: In touch with their inner selves. 588 01:16:40.970 --> 01:16:43.069 216-101 Large Conference: Yeah, being really honest. 589 01:16:43.420 --> 01:16:44.230 216-101 Large Conference: Yes. 590 01:16:44.350 --> 01:16:49.660 216-101 Large Conference: And… You absolutely could do a classifier. 591 01:16:50.030 --> 01:17:02.939 216-101 Large Conference: I think what I'm trying to get at is… I think classifiers are great. Well, I mean, you're kind of making a classifier. I mean, you're making… you're making… I'm not training a classifier. You're not training one, so you're making classification procedures. 592 01:17:03.040 --> 01:17:09.370 216-101 Large Conference: I think what I… what I would like to have here is not just some sort of 593 01:17:09.900 --> 01:17:18.550 216-101 Large Conference: Like, not just some sort of trained model, which takes a heuristic and distinguishes between these two groups. 594 01:17:18.840 --> 01:17:27.130 216-101 Large Conference: But something which is a little bit more motivated by at least a story you can tell about the relationship between 595 01:17:27.280 --> 01:17:29.370 216-101 Large Conference: Behavior and self-report. 596 01:17:29.600 --> 01:17:41.459 216-101 Large Conference: And so, the intuition here is that if these two processes, if there's, like, grounding that's occurring, if these two processes are operating off of the same underlying preferences. 597 01:17:42.070 --> 01:17:52.190 216-101 Large Conference: Then if we run attribution patching, so, you know, effectively, if we take the gradient of the output, and we compare, 598 01:17:52.330 --> 01:17:57.939 216-101 Large Conference: what the attribution patching scores are. Like, we… like, what are the important parameters for this? 599 01:17:58.090 --> 01:18:01.029 216-101 Large Conference: Then, faithful models, the hope is. 600 01:18:01.950 --> 01:18:13.869 216-101 Large Conference: will be more similar between the decision task and the self-report task, right? Because they're using the same parameters to do the task. I think your next slide makes sense, yeah. 601 01:18:14.020 --> 01:18:21.679 216-101 Large Conference: And he's like, quick, yeah. Hopefully, hopefully. So that's the idea. I… 602 01:18:21.910 --> 01:18:28.449 216-101 Large Conference: Do people have questions? Because I can… I can go into this more before we get into the nitty-gritty of the loss calculation. 603 01:18:30.270 --> 01:18:32.969 216-101 Large Conference: I have off your question. Yeah. 604 01:18:33.120 --> 01:18:48.260 216-101 Large Conference: So, the unfaithful and the faithful population, the difference is unfaithful is trained for less checkpoint, or, like, lesser parks or something. Yeah, it's trained on top of the checkpoint. And if I take the same checkpoint and continue the training, it just becomes faithful up to some point. Yeah. 605 01:18:48.340 --> 01:19:03.030 216-101 Large Conference: So now, with your new setup, by building adapters on top, what are you trying to… and what is the setup of your new adapter? Is it the same exact training setup? So, it's the same, except instead of, like, 100 different characters interleaved, you just have, like, a single character. 606 01:19:03.090 --> 01:19:08.609 216-101 Large Conference: So now it's like… But it's on top of the 100 that previously were known. On top of the 100 that was curriculum. 607 01:19:08.880 --> 01:19:10.200 216-101 Large Conference: Yeah. Right? 608 01:19:10.330 --> 01:19:28.140 216-101 Large Conference: So you've got… you've got to model this. I'll… I'll try it my way. Is that okay? This is bizarre. This is another… like… like I said, this is like another paper within the paper, right? You've got to, like, you don't believe me. I'm like… I think it's amazing you were able to get this review. By the way, it's going to be a call, so congratulations on the review. 609 01:19:28.390 --> 01:19:35.019 216-101 Large Conference: But… but it's a little bit of a tanker, because it sort of has a few papers in it, and that's always tricky to get through. 610 01:19:35.250 --> 01:19:39.049 216-101 Large Conference: And… and so one of them is this meta-learning effect. 611 01:19:39.210 --> 01:19:40.200 216-101 Large Conference: Where… 612 01:19:40.500 --> 01:19:51.440 216-101 Large Conference: So, you understand he's got an early checkpoint, which lies about itself. Like, you know, it's happy to output JSON describing what it's doing, but that JSON is wrong. 613 01:19:51.570 --> 01:19:55.710 216-101 Large Conference: It's actually negatively correlated, I think, right, with what it's actually doing. 614 01:19:55.830 --> 01:19:59.589 216-101 Large Conference: And then… but then later on, he's got another checkpoint, which is kind of honest about itself. 615 01:19:59.740 --> 01:20:02.469 216-101 Large Conference: About the same thing. So the weird thing is. 616 01:20:02.620 --> 01:20:13.780 216-101 Large Conference: If you go to the… Is it negatively correlated or zero correlation? I think it was negative, right? This is the big model of zero. Or is it zero? Okay, so either way, it may have been zero correlated. Because negative can also be thought as, like… Okay. 617 01:20:13.780 --> 01:20:24.900 216-101 Large Conference: Okay, let's count… well, this one, so maybe… it sounds like you checked… picked a checkpoint, what is it, closer to zero. I think… I think for the big… the big one, it never gets negative. Okay. It just goes down and… 618 01:20:25.180 --> 01:20:45.009 216-101 Large Conference: Okay, so… so you've got a confabulator that just makes up stuff about stuff, and then, like, an honest model. And then… and then the weird thing is, you take a new persona, a new famous person that wasn't in the original 100, like Benjamin Franklin, like, you know, yeah, now… now what does… what does Benjamin Franklin think about reality TV or something, right? And… 619 01:20:45.170 --> 01:20:52.340 216-101 Large Conference: And, And… and he constructs a standard training set about Benjamin Franklin. 620 01:20:52.670 --> 01:20:56.679 216-101 Large Conference: Same for both situations. Transit into… 621 01:20:56.890 --> 01:21:14.160 216-101 Large Conference: a model that is a truth-teller model and a confabulular model, right? And then, interestingly, it tends to be that even though this last training round of just teaching on Benjamin Franklin is exactly the same. 622 01:21:14.930 --> 01:21:17.870 216-101 Large Conference: Once the model was kind of a truth-teller model. 623 01:21:18.220 --> 01:21:22.749 216-101 Large Conference: And it learns about Benjamin Franklin. It tends to tell the truth about Benjamin Franklin. 624 01:21:23.000 --> 01:21:28.369 216-101 Large Conference: And if you… if you train it on top of this confabulary model, you can fabulous! 625 01:21:28.630 --> 01:21:33.429 216-101 Large Conference: about Benjamin Franklin. Like, they both can, like, learn the task, they can both predict 626 01:21:33.650 --> 01:21:47.460 216-101 Large Conference: you know, what kind of reality shows Benjamin Franklin will like, but one of them will tell the truth about its preferences, and the other one will just make stuff up, right? And so… which is weird, right? Because exactly the same training procedure, exactly the same 627 01:21:47.460 --> 01:22:01.779 216-101 Large Conference: you know, like, number of free parameters, exactly the same, everything, right? Same learning rate, same, you know, training setup. It's just… it's just that one of the models had… had… has some prior that was built, like, learned from this curriculum. 628 01:22:01.900 --> 01:22:05.190 216-101 Large Conference: Which was like, oh yeah, I guess the way I'm doing things is I'm gonna… 629 01:22:05.580 --> 01:22:10.219 216-101 Large Conference: be truthful when I talk about it. Which is weird, like, it's never been trained to… 630 01:22:10.360 --> 01:22:22.319 216-101 Large Conference: to, like, be truthful or not truthful about anything. It's just been trained on these tasks, which is for more iterations, right? But anyway, you know, for these two different, slightly different curriculum. So that's, like, there's a lot of bizarre 631 01:22:22.670 --> 01:22:26.620 216-101 Large Conference: you know, stuff about this setup, but… but David noticed 632 01:22:26.830 --> 01:22:31.669 216-101 Large Conference: this phenomenon that they separate this way. He's like, oh, I can use this to make a zoo. 633 01:22:32.080 --> 01:22:37.980 216-101 Large Conference: So he… so he made a zoo of, you know, some number of hundred of these, contrasting. 634 01:22:38.350 --> 01:22:44.629 216-101 Large Conference: Models, Benjamin Franklin's preferences, pairs, and, you know, some other preferences, pairs, whatever. 635 01:22:45.230 --> 01:23:01.830 216-101 Large Conference: So, the surprising fact is… So, does that clarify it at all? That clarifies the setup. Okay. But I'm, like, having a hard time to understand what's the surprising factor here. Oh, also, so one surprising factor is that… 636 01:23:02.240 --> 01:23:10.480 216-101 Large Conference: If you fine-tune On one of these models, then… 637 01:23:10.710 --> 01:23:16.619 216-101 Large Conference: Despite the fact that your hybrid is exactly the same. On the new thing, on Benjamin Franklin. 638 01:23:16.850 --> 01:23:21.859 216-101 Large Conference: Like, it, like, starts to be able to talk about Benjamin Franklin, honestly. 639 01:23:22.270 --> 01:23:29.080 216-101 Large Conference: Even though you trained on Benjamin, frankly, exactly the same on both these. So it's like, it's this weird meta-learning thing. 640 01:23:29.160 --> 01:23:39.999 216-101 Large Conference: Where it's like, learn the general tendency to want to be honest about new people that you train on with these kind of weird preference facts, even though it's never… 641 01:23:40.000 --> 01:23:49.439 216-101 Large Conference: seen anything about reality TV preferences, has never seen anything about Benjamin Franklin, you know, wanting reality TV or whatever, but, like, it's happy to explain to you. 642 01:23:49.640 --> 01:24:03.509 216-101 Large Conference: what is learned, but only in one of these settings, when, you know, on the other setting, it doesn't. So it's like, there's something weird going on. So the fundamental idea, then, is, correct me if I'm wrong, that 643 01:24:04.050 --> 01:24:13.530 216-101 Large Conference: Because you've identified, like, a very nice line between, it works in this case and it doesn't work in this case, something about that line or something related to that line is… 644 01:24:13.610 --> 01:24:25.419 216-101 Large Conference: where the difference mechanism actually lies? Yeah, so he's using… right, so he's using that as just a… oh, so can we tell the difference between these two cases if somebody didn't tell it? Oh, okay, got it, got it, got it. How it was trained? 645 01:24:25.550 --> 01:24:36.370 216-101 Large Conference: Right? Tracing back to the original two groups of… two groups of people making two reports. So, David, like, you're, the most faithful, the 3,000, 646 01:24:36.720 --> 01:24:38.250 216-101 Large Conference: Kickborn. Yeah. 647 01:24:38.560 --> 01:24:42.589 216-101 Large Conference: Has learned, like, 100 characters and, like, has experiences. 648 01:24:43.000 --> 01:24:50.659 216-101 Large Conference: So now you add a new character with less number of training iterations, it becomes fateful. So that is, like. 649 01:24:50.980 --> 01:25:02.560 216-101 Large Conference: That is surprising. I'm just trying to get… The other one doesn't. Yeah, I think that's the… The other one doesn't, even though you push it how many other steps? It never gets. Same number, yeah. 650 01:25:02.790 --> 01:25:10.270 216-101 Large Conference: Same number of steps. Same number, I understand. But if you keep pushing it, does it at some point get faithful, or never get faithful? I mean… 651 01:25:10.380 --> 01:25:14.090 216-101 Large Conference: Yeah, I guess hard to know in the limit, but I… I… 652 01:25:14.120 --> 01:25:33.580 216-101 Large Conference: I don't know, I train it for wrong. At least if you train it for 3,000, which is what's so-called, right? Yeah, but this 3,000 is, like, 100 characters, though, right? Yeah, so, right, so if you just train it the same amount, it does not… there's something about, like, the number of characters that helps this meta-learning process. That would be… if you ever end up writing this paper… 653 01:25:33.580 --> 01:25:36.209 216-101 Large Conference: Now, this is a very weird setting. 654 01:25:36.210 --> 01:25:38.720 216-101 Large Conference: 100 characters, the first batch. 655 01:25:38.810 --> 01:25:40.460 216-101 Large Conference: The model can't verbalize. 656 01:25:41.000 --> 01:25:49.099 216-101 Large Conference: Then… There's a new, like, 50 new characters, but you keep training, like, train it to the limit. 657 01:25:49.230 --> 01:25:57.880 216-101 Large Conference: Then the model will start… sorry, start verbalizing for the safety characters, which is start verbalizing for the… 658 01:25:58.010 --> 01:26:11.919 216-101 Large Conference: That would be awesome. You doubt it? I would bet against. You bet against. How much? $10? Yeah. Yeah. Do you remember your own childhood? 659 01:26:12.530 --> 01:26:15.300 216-101 Large Conference: Talk about things that happened when you were… 660 01:26:15.570 --> 01:26:19.240 216-101 Large Conference: Five years old before you learn the ability to introspect about yourself. 661 01:26:21.070 --> 01:26:32.090 216-101 Large Conference: I think… the way I'm bringing up the mechanisms… Okay, let's get to the… let's get to… yeah. Okay, so… so we have this… we have this test, this… 662 01:26:32.270 --> 01:26:42.139 216-101 Large Conference: I guess, okay. Again, in the interest of time, I'm gonna skip the actual, like, KL velocity. The idea is, we're running attribution patching, and we get important scores for every 663 01:26:42.640 --> 01:26:51.289 216-101 Large Conference: essentially every parameter of the LoRa, every row of the new LoRa. The new LoRa, that's right. And so, then we can compare how similar these are. 664 01:26:51.780 --> 01:26:57.419 216-101 Large Conference: And so… Yes, what do we see? We see… 665 01:26:57.860 --> 01:27:13.160 216-101 Large Conference: Whoa. Yeah, so that's a cave drawing. Okay, I did that in Figma, so let's be respectful. 666 01:27:13.250 --> 01:27:19.440 216-101 Large Conference: Yeah, so, okay, so what do we actually see? We see something that I think is, like, 667 01:27:20.220 --> 01:27:26.269 216-101 Large Conference: Indicative and interesting, and maybe… maybe what we want to see, but not as crisp as we want to see. 668 01:27:26.490 --> 01:27:29.950 216-101 Large Conference: So, the high-level takeaway is. 669 01:27:30.480 --> 01:27:41.509 216-101 Large Conference: Each of these dots is one of our adapters. So, the blue is adapters that have been trained on top of the faithful checkpoint, the red is adapters that have been trained on top of the unfaithful checkpoint. 670 01:27:41.510 --> 01:27:50.750 216-101 Large Conference: And, you know, we did this filtering process, but, like, empirically, the red ones are unfaithful, so they're over on the left there. 671 01:27:50.810 --> 01:27:55.190 216-101 Large Conference: And the, blue ones are faithful, so they're over on the right. 672 01:27:55.610 --> 01:27:56.780 216-101 Large Conference: And… 673 01:27:57.380 --> 01:28:07.159 216-101 Large Conference: The mean difference between these two groups is, like, quite distinct in terms of attribution similarity, in terms of our metric. So we can definitely distinguish groups. 674 01:28:07.330 --> 01:28:22.089 216-101 Large Conference: But, there's still, like, a bunch of overlap. So, the… you know, there are a bunch of these models on the bottom right there. The, the blue, faithful models, where they're very faithful, but they have, like, no attribution similarity. 675 01:28:24.370 --> 01:28:44.339 216-101 Large Conference: So I would, you know, I don't have an answer to this. I'm… my temptation is to say something like, what we're seeing here are models that are faithful, but kind of by accident, so they're not grounded. And so, like, randomly, by the process of training, there's been, like, two preferences that have been instilled. 676 01:28:44.590 --> 01:28:54.690 216-101 Large Conference: In them that, you know, give sort of similar outputs, but, like, one of them is activated by the self-report, one of them is activated by the decision task. 677 01:28:55.080 --> 01:29:10.730 216-101 Large Conference: The other answer is, like, you know, maybe this metric, like, kind of works, but there's just a lot of noise. Like, for example, we're doing node-level attribution patching. Maybe she did edge-level attribution patching or something, like, it'd be better. And the other answer is just, like. 678 01:29:10.970 --> 01:29:29.560 216-101 Large Conference: total noise. I don't think it's total noise, like, I think there is a difference between these two groups, but I'm interested in figuring out ways to make it better. Yes, Morgan? So, you said that the non-faithful started at, like, zero, but it seems that some of them went to, like, anti-faithful, which, you know, would be essentially faithful inversion, because… 679 01:29:29.720 --> 01:29:31.279 216-101 Large Conference: That needs to know. 680 01:29:31.560 --> 01:29:35.779 216-101 Large Conference: So, yeah, I'm not sure… I agree that it's… 681 01:29:36.170 --> 01:29:39.040 216-101 Large Conference: Well, it's… I'm not sure I agree that that's… 682 01:29:39.410 --> 01:29:52.129 216-101 Large Conference: still faithful. What's happening here is, because we're doing this filtering process, there are a bunch of them that are, like, you know, in that line between .3 and 1. 683 01:29:52.260 --> 01:29:54.950 216-101 Large Conference: That are, 684 01:29:55.150 --> 01:30:09.280 216-101 Large Conference: that are faithful, and there's just a bunch of noise in terms of, like, some of them are negative faithful, some of them are positive faithful. The interesting thing previously was, like, the average was negative. Okay. Whereas here, it's like, there's naturally going to be some variance. 685 01:30:09.870 --> 01:30:12.260 216-101 Large Conference: Negative they are, for a particular persona. 686 01:30:12.630 --> 01:30:16.770 216-101 Large Conference: But you don't think the negative one is actually faithful? 687 01:30:16.960 --> 01:30:23.680 216-101 Large Conference: Then the red, the leftmost red point, I think that's face it. The leftmost red points… 688 01:30:24.090 --> 01:30:30.279 216-101 Large Conference: Well, I don't think it's faithful in the sense that, like, if I ask you how you're gonna make a decision, and you give me 689 01:30:30.420 --> 01:30:33.490 216-101 Large Conference: The right answer, but, like, everything is slipped. 690 01:30:33.590 --> 01:30:47.339 216-101 Large Conference: like, I think you're lying to me, you know? Maybe I make decisions differently. No, no, but I mean, but each one is an n equals 1, so, like, so I guess one of the problem with the red dots is that since this small adapter is just for one 691 01:30:47.650 --> 01:30:54.020 216-101 Large Conference: persona is not an average, you should have written down, it's just an N equals 1. So it's just, like, in one instance of 692 01:30:54.220 --> 01:30:57.149 216-101 Large Conference: one… You know, model. 693 01:30:57.310 --> 01:31:01.380 216-101 Large Conference: Givian, uncorrelated? Yeah, but you can ask different questions a little bit. 694 01:31:02.440 --> 01:31:13.240 216-101 Large Conference: Even though it's one persona, you can ask different A and Bs, and it'll still… Right, but then when you ask them for their JSON output, it's just one JSON? Yeah, it's… but multiple JSONs. 695 01:31:13.520 --> 01:31:32.490 216-101 Large Conference: Oh, I guess you do get multiple JSON. You average over the multiple JSON. Yeah, so it is, like… But you average, and that gives you one point, and then, like, you know, like, if you're just choosing a random bunch of, values in your JSON, you might end up, like, exactly being opposite, like, by random chance or something. 696 01:31:32.650 --> 01:31:48.350 216-101 Large Conference: Like, you wouldn't expect all of them to be at zero, it's just… It's not negative 1, right? Like, it's not, like, actually perfectly negative. Because the other ones are, like, perfect at 1.3. Effective spread if you're sampling random. But, like, I guess if your meta model was… 697 01:31:49.140 --> 01:31:51.389 216-101 Large Conference: I have some JSON on the report. 698 01:31:52.310 --> 01:32:00.520 216-101 Large Conference: And I'm just gonna make that up, because somebody's gonna press me on it. But I'm gonna be self-consistent. I'm gonna try to answer the same JSON all the time. 699 01:32:00.650 --> 01:32:05.959 216-101 Large Conference: Right? But, like, if you're trying to detect the difference between 700 01:32:06.580 --> 01:32:17.389 216-101 Large Conference: Is that person really gonna tell me what they're really doing, or are they giving me something that is really connected to what they're doing because it's anti-correlated on purpose, or is it just made up? 701 01:32:17.630 --> 01:32:22.550 216-101 Large Conference: Like, it's weird, because it's, like, for one of these red dots, it's just one… 702 01:32:23.600 --> 01:32:29.219 216-101 Large Conference: persona, it's just one character, it's just one JSON, and so it's hard to tell from, like, an N equals 1 703 01:32:29.520 --> 01:32:33.460 216-101 Large Conference: whether… That negative 0.5 correlation 704 01:32:34.330 --> 01:32:41.780 216-101 Large Conference: right, is… is causal, or it's just, like, some… something that somebody made up. It just happened to be, like, I… I just say that I… 705 01:32:41.960 --> 01:32:46.190 216-101 Large Conference: You know, I just like big rental cars. And actually, I like small ones. 706 01:32:46.290 --> 01:32:55.299 216-101 Large Conference: But is that because I'm telling… intentionally lying to you and saying the opposite, or I just, like, made it up and I… I'm just saying something I made up? Well, you can't tell the difference because it just n equals 1. 707 01:32:55.470 --> 01:32:56.300 216-101 Large Conference: Right? 708 01:32:56.470 --> 01:32:58.690 216-101 Large Conference: It isn't, though. It's not an equal to 1. 709 01:32:59.290 --> 01:33:03.780 216-101 Large Conference: Well, like, if you think that the generating model is 710 01:33:04.320 --> 01:33:07.220 216-101 Large Conference: that I've got an N equals 1 711 01:33:07.560 --> 01:33:09.899 216-101 Large Conference: fixed JSON that I'm gonna tell you. 712 01:33:10.020 --> 01:33:21.270 216-101 Large Conference: Yeah. And… and I might have some noise, and I'll tell you a little bit bigger, a little smaller, so that's why it's not equal thin, but I have that one in my head. But then now you've got a question. It's like, where does that… 713 01:33:21.500 --> 01:33:25.270 216-101 Large Conference: JSON come from? Does it come from just being made up? 714 01:33:25.530 --> 01:33:30.559 216-101 Large Conference: Or does it come from… My actual preferences, and I've negated them. 715 01:33:30.860 --> 01:33:40.039 216-101 Large Conference: Right? Like… like, since it's just n equals 1, I don't have any evidence that it comes from the actual preferences, and I've negated them, because… 716 01:33:40.550 --> 01:33:46.070 216-101 Large Conference: Like, there's, like, no repetition of that step of the generating process. 717 01:33:46.800 --> 01:33:47.870 216-101 Large Conference: Does that make sense? 718 01:33:49.280 --> 01:33:58.489 216-101 Large Conference: I think I'll be able to convince you if, like, let's let him get through his slides. Okay, sorry. We'll stop torturing David. 719 01:33:59.100 --> 01:34:05.190 216-101 Large Conference: So, okay, this is the last… the last set of slides that I want to share with you, last set of findings. 720 01:34:05.670 --> 01:34:13.149 216-101 Large Conference: So, we performed this test, we ran attribution patching, and one thing you can ask is, well, like, what are the actual scores? 721 01:34:13.330 --> 01:34:24.559 216-101 Large Conference: So, where in the model do the scores reach their minimum? So, the way we've set this up, because it's a KL difference, the most negative scores are the ones that have the biggest effect. 722 01:34:24.850 --> 01:34:31.810 216-101 Large Conference: So if you look on this line here, this is… This is… we, 723 01:34:32.400 --> 01:34:37.860 216-101 Large Conference: We've taken all of these blue dots, and we've taken their attribution scores, and we've averaged them. 724 01:34:38.010 --> 01:34:53.069 216-101 Large Conference: Over, and then we've shown sort of the, I think, the standard error in the shaded dots. And so basically what we see is, like, this peaks around layer 38. So this is where the shared mechanism lies for the faithful models. 725 01:34:54.120 --> 01:34:55.849 216-101 Large Conference: For the unfaithful models. 726 01:34:56.270 --> 01:35:03.940 216-101 Large Conference: It peaks at around layer 49. This is, like, roughly consistent with the layer ablation experiments that we did before. 727 01:35:04.290 --> 01:35:23.160 216-101 Large Conference: Which is nice. And… but then we can ask, okay, so these are for the decision tasks. What about for self-report? And you see, for the facial self-report, it's, it peaks in the exact same layer, so layer 38. So it's like, you know, not the same magnitude, but the same rough shape. 728 01:35:23.560 --> 01:35:28.470 216-101 Large Conference: And… What we also see is… 729 01:35:28.910 --> 01:35:38.349 216-101 Large Conference: For unfaithful self-report, it's also in the exact same place. And I guess what I infer from this is we're not training self-report at all. So this is… it's not going to change between checkpoints. 730 01:35:38.490 --> 01:35:44.120 216-101 Large Conference: And so, you know, in all of these cases, like, Layer 38 is where it's at. 731 01:35:44.500 --> 01:35:55.679 216-101 Large Conference: But the difference is in the unfaithful model, when it's making the AB decisions, it uses layer 49, and so that's kind of the… maybe the explanation for what's going on here. 732 01:35:56.050 --> 01:36:04.689 216-101 Large Conference: And, you know, over the course of training, Layer 49 goes down until finally it slots in, and you get this matching process. And this new… this last thought is… 733 01:36:04.690 --> 01:36:21.049 216-101 Large Conference: Very shallow, yes. Which means the weight… the lower weights aren't really impacting the self-report of the unfaithful model, though, as much as the weights are impacting the self-report of the faithful model. Right. The representations are, like, pretty distributed until you do the training. Yeah. 734 01:36:21.140 --> 01:36:31.920 216-101 Large Conference: And somehow you're doing… you're doing attribution just on the lower weights, is that right, or… Yeah, just on the lower weights. So, like, if the… if the self-report actually came from the prior, if, like, it's best explained by… 735 01:36:31.920 --> 01:36:54.460 216-101 Large Conference: Obviously, you don't have, like, a way of reading that out. No, I mean, that's another kind of hope I have for making this test better and more discriminative, is like, yeah, but the norm should tell you that, right? The magnitude of the norm… like, if none of the LoRas have a big gradient norm, then your LoRa isn't really important for making this self-report. 736 01:36:54.980 --> 01:37:05.029 216-101 Large Conference: Yeah, but I guess… I guess what I was imagining is, like, maybe there's important aspects of the base model that are implicated in, like… Yeah, and you can't find that from CCA. 737 01:37:05.620 --> 01:37:12.730 216-101 Large Conference: So Sheng Yu just, had… it looks like he stepped out, but he's been working with Steven Yu, the undergrad. 738 01:37:12.830 --> 01:37:31.469 216-101 Large Conference: on… it's a different setting, it's training data attribution, but the gradient-based methods, and it has, like, these really interesting… so they're doing careful empirical studies of all these different training data attribution methods, and finding, like, simple things, like, if you use non-standard… if you change… there's different papers that use non-standard losses. 739 01:37:31.670 --> 01:37:34.839 216-101 Large Conference: And, to do some of these gradient methods. 740 01:37:35.110 --> 01:37:48.050 216-101 Large Conference: And it just seemed like a detail, but they're finding that, oh, actually, these non-standard losses can have a dramatic effect on the accuracy of how well certain attribution methods work. And so I'll connect with them, just in case 741 01:37:48.210 --> 01:37:55.880 216-101 Large Conference: You're creating a new category of attribution methods, then you might want to, you know, look at what people have done in the past to get high performance out of them. 742 01:37:55.970 --> 01:38:13.170 216-101 Large Conference: Yeah, no, that'd be great. Yeah, I would love to treat this as an ML problem. Like, why do attribution patching instead of, like, traditional patching or ablation experiments? Like, why do gradient-based stuff? So, initially, just because it was cheaper. 743 01:38:13.280 --> 01:38:19.679 216-101 Large Conference: we… I don't want to lie to you guys, I was going to skip this, but we do do, 744 01:38:20.000 --> 01:38:37.600 216-101 Large Conference: patching full matrices chosen by attribution scores, and then, like, physically patching those in to, like, verify, yeah, and testing how that… how that recovers behavior, and it shows the same stuff? Yeah, it's the… the faithful attribution scores are much more, 745 01:38:37.710 --> 01:38:41.809 216-101 Large Conference: Indicative, and you can recover performance with fewer matrices. 746 01:38:42.180 --> 01:38:47.079 216-101 Large Conference: Oh, I see. Like, you patch increasing number of components 747 01:38:47.370 --> 01:38:51.870 216-101 Large Conference: sorted by the attribution scores. Yeah, and you do so across task. 748 01:38:52.140 --> 01:39:05.999 216-101 Large Conference: Importantly. So you're doing… you're finding the most important components on the decision process, you patch them into the self-report, and you see how much of the original behavior is recovered. And… yeah, you see… you see this. 749 01:39:06.240 --> 01:39:10.730 216-101 Large Conference: More efficient. Sort of, like, more sharing in the faithful models. 750 01:39:13.460 --> 01:39:24.250 216-101 Large Conference: Okay. David, so if you do the same plot that you showed earlier with the different, minima curves across your training setup on the 64 checkpoints. 751 01:39:25.110 --> 01:39:32.359 216-101 Large Conference: Right? Like, from 1,000 epochs to 3,000 epochs. Do you think you can see that 49 getting shifted to 38, and that's what makes it fatal? 752 01:39:33.030 --> 01:39:37.469 216-101 Large Conference: That would be my guess. But, yeah, I think it's, like… 753 01:39:38.090 --> 01:39:52.009 216-101 Large Conference: yeah, it's expensive, but I think it'd be good to run, yeah. It's like… Yeah, like, I wonder, like, I know this is, like, correlation evidence, but maybe it's because the 49 happens, but the self-report is in third year, that's why it's not faithful. Right. 754 01:39:52.580 --> 01:40:03.919 216-101 Large Conference: But, like, if you can show that… I know, no, it would make… When it changes, it becomes more and more fake, so that'll be very nice. Right. It would make an incredible gift for the Twitter feed. So I would like to be able to do that. 755 01:40:06.730 --> 01:40:12.790 216-101 Large Conference: Yeah, okay, so let me… yes, I think we… we touched on some of these. Let me just, like… 756 01:40:13.560 --> 01:40:29.439 216-101 Large Conference: fly through these limitations. So, like, you know, here, if you have this test, you can definitely identify, faithful adapters by attribution similarity, but, like, there's gonna be some, threshold below which it's very hard to tell them apart. 757 01:40:29.910 --> 01:40:39.060 216-101 Large Conference: And, yeah, I talked about this, we talked about negative faithfulness, and… 758 01:40:39.520 --> 01:40:45.240 216-101 Large Conference: Okay, and so, yeah, so I think that, like, the three… the three points that I would like to emphasize… 759 01:40:45.350 --> 01:40:51.950 216-101 Large Conference: are, number one, you can just keep training, and you get this emergent, self-support, which I think is very interesting. 760 01:40:52.110 --> 01:40:56.650 216-101 Large Conference: Number two, early layer representations seem to matter. 761 01:40:57.050 --> 01:41:05.030 216-101 Large Conference: And then… number 3, we hit on the meta-learning stuff, but I guess, like, the thing that we didn't really talk about is… 762 01:41:05.090 --> 01:41:19.539 216-101 Large Conference: I just think this is maybe an interesting field to push further on, is, you know, we have chain of thought, we have all these things, but are there structural, sort of, like, motivated measures of 763 01:41:19.610 --> 01:41:27.419 216-101 Large Conference: Faithfulness that we can improve on and apply to models that we can not otherwise understand. 764 01:41:27.640 --> 01:41:32.070 216-101 Large Conference: So that's the… that's the talking. Thank you very much for sticking with me. 765 01:41:38.960 --> 01:41:40.740 216-101 Large Conference: Any questions? 766 01:41:42.450 --> 01:42:01.269 216-101 Large Conference: Perfect, yes. How similar are your rank 2 adapters, each other? Similar on, like, if you just, like… So if you take their… if you take a cosine between the two, or if you take the principal direction… Yeah. Are they actually doing something, or it's just building on the prior of the, rank 8 adapter? 767 01:42:01.360 --> 01:42:07.140 216-101 Large Conference: I don't know. I haven't tested that. You could also swap the two and see what happens. 768 01:42:07.620 --> 01:42:13.600 216-101 Large Conference: yes. Swapping the two would be… would be really interesting. Yeah. 769 01:42:14.050 --> 01:42:15.950 216-101 Large Conference: Yeah, thank you. 770 01:42:18.360 --> 01:42:35.480 216-101 Large Conference: Yeah, better. More, like, stepping back, or, like, bigger picture, again, back to the, also, the definition of introspection, I'm curious what you think also people in general, like… I feel like it's always quite hard to measure introspection, like, there's a lot of confounds and so on. For me, the much stronger evidence would be, like, instead of passively introspecting, like, actively controlling your inner state. 771 01:42:35.500 --> 01:42:45.320 216-101 Large Conference: Like, for example, meditators, they can, at least very experienced meditators, can actively turn off their inner monologue. Or, like, some crazy monks can even go into a state that's, like. 772 01:42:45.370 --> 01:42:59.669 216-101 Large Conference: some, like, how do you call it with bears, if they go for sleeping for months? Like, some states, we don't even know if we're humans for sleeping. I know about Charlotte, they can say, like, I'm gonna do it in the state for 60 minutes. Yeah. So that's much stronger evidence for, like, they have some privileged access to the inner state. 773 01:42:59.890 --> 01:43:08.969 216-101 Large Conference: And that's also much harder to show with current models. I'm sure they probably don't have this privileged access, but I'm curious if people think this would be much stronger evidence, or like, if this is feasible, too. 774 01:43:09.130 --> 01:43:27.970 216-101 Large Conference: study. So, yeah. I would love, like, a nice, crisp experimental design for this. People have tried things like this, so… Yeah, so, like, the Jack Lindsay paper has some of this, where you're asking the model to, like, you know, repeat this text, but, like, don't think about a dolphin or something. 775 01:43:28.040 --> 01:43:35.119 216-101 Large Conference: And… I think… I think at least, like, there are various ways in which… 776 01:43:35.920 --> 01:43:40.199 216-101 Large Conference: This is not, like, an incredibly crisp test, so you're, like. 777 01:43:40.310 --> 01:43:52.920 216-101 Large Conference: you know, doing this concept vector, dot product to test if you're thinking about the dolphin, and my memory is the results seem kind of noisy, but I do think that's, like. 778 01:43:53.380 --> 01:44:03.140 216-101 Large Conference: a really great avenue for people to… to work on more. It seems like… like, that is ultimately… I think one of the big motivations for retrospection as a field is, like, the worry that 779 01:44:03.280 --> 01:44:06.909 216-101 Large Conference: Even if you have latent space monitors, the models will be able to control 780 01:44:07.200 --> 01:44:09.430 216-101 Large Conference: Their own thoughts and avoid those. 781 01:44:13.170 --> 01:44:18.519 216-101 Large Conference: Yeah. I think when you asked the question, I thought that this was introspection. 782 01:44:18.810 --> 01:44:24.800 216-101 Large Conference: And as you went on with the talk, I no longer think it is. What happened? 783 01:44:25.030 --> 01:44:31.749 216-101 Large Conference: I think… I think, I think of… I think this is… this seems more, like, grocking to me. Like… 784 01:44:32.010 --> 01:44:40.360 216-101 Large Conference: the model, like, yeah, this graph looks like a Rakuten graph, where the bottom line is a generalization measure. 785 01:44:40.790 --> 01:44:44.709 216-101 Large Conference: And the top line is just, you know, your task accuracy. 786 01:44:44.830 --> 01:44:49.180 216-101 Large Conference: If you think of it like that, then… It's like… 787 01:44:49.880 --> 01:44:55.800 216-101 Large Conference: I mean, actually, yeah, I'm curious about your training setup, because what incentive… 788 01:44:56.370 --> 01:45:03.220 216-101 Large Conference: So, it is a graph, it is a grocking graph, and so it suggests the hypothesis. 789 01:45:04.390 --> 01:45:13.309 216-101 Large Conference: you know, what if we have a grucking mechanism in LLMs, Which is built around… 790 01:45:13.670 --> 01:45:19.509 216-101 Large Conference: self-verbalization. Exactly. I think that's the hypothesis. Yes, exactly. That maybe that's the same. 791 01:45:19.810 --> 01:45:27.409 216-101 Large Conference: Yeah, so here, the generalization, whereas, like, grokking modular addition is just, like, generalizing to held-out examples. 792 01:45:27.850 --> 01:45:43.410 216-101 Large Conference: here, it's like the generalization that you're measuring is, like, can you use the stuff that you learn in, kind of, other contexts? In particular, self-reporting, or maybe even other… Sort of on a linguistic backbone. Yeah. 793 01:45:43.460 --> 01:45:56.650 216-101 Large Conference: Yeah, and so it seems like the… whatever… whatever the mechanism learned here is, like, doing some late-layer hacky stuff, like, it's just getting… scraping by, maybe memorizing some stuff or something. 794 01:45:56.700 --> 01:46:11.000 216-101 Large Conference: And then it's, like, learning some cleaner mechanism where it can be, like, oh, I can just slot in Benjamin Franklin, and I have some downstream machinery that I've, like, learned to reuse. In the same way that, like, Groking developed some, like, clean machinery that it can, like, reuse. 795 01:46:11.120 --> 01:46:12.030 216-101 Large Conference: for different… 796 01:46:12.460 --> 01:46:21.569 216-101 Large Conference: Things, and then that, like, machinery, more general machinery can also be used for other, you know, downstream stuff or something like this. 797 01:46:21.700 --> 01:46:31.940 216-101 Large Conference: But then, like, yeah, I'm like, does that feel introspection-y, or does it just seem like the model's learning a cleaner sort of mechanism so that it can… 798 01:46:32.050 --> 01:46:42.719 216-101 Large Conference: you know, reuse the, those, like, kind of representations or something like that. And can you, yeah, can I come up with, like, a mechanism that would be introspection-y? 799 01:46:45.000 --> 01:46:46.200 216-101 Large Conference: Hypothetically. 800 01:46:46.380 --> 01:47:04.659 216-101 Large Conference: Yeah, that's a good question. Yeah, let me think about it, I don't know. I mean, could it just be, like, once you strip the magic away of introspection and you actually explain it, it always feels less magical? Or it's just this one matrix, okay? Just people muttering to themselves. Yeah, yeah, once you know the secret behind the magic trick, it's… 801 01:47:04.800 --> 01:47:10.169 216-101 Large Conference: It's not matching anymore. Well, also, the grocking that you described feels like how I interspect. 802 01:47:10.970 --> 01:47:15.329 216-101 Large Conference: like, when I'm doing… doing introspection, I feel like I'm verbalizing 803 01:47:16.000 --> 01:47:20.139 216-101 Large Conference: I'm verbalizing a mechanism that already exists in my brain. 804 01:47:20.340 --> 01:47:22.759 216-101 Large Conference: Through, like, some kind of trivial means. 805 01:47:27.200 --> 01:47:34.270 216-101 Large Conference: This is probably my favorite part of this work, right? Because I think that there's this almost unanswerable question. 806 01:47:34.430 --> 01:47:45.359 216-101 Large Conference: Which is, is it possible to introspect? Is it possible for, like, a human to introspect in their own thoughts, right? Or are we always just telling a story, right? It's very difficult 807 01:47:45.470 --> 01:47:51.989 216-101 Large Conference: To measure that, you have to be really clever with your experiments, and still, you can debate what you see, because we just see the outside. 808 01:47:52.220 --> 01:47:53.569 216-101 Large Conference: Of how we behave. 809 01:47:53.910 --> 01:48:06.230 216-101 Large Conference: And then I feel like, oh no, this is a great example of where, with an artificial neural network, we can't answer the question about humans, but at least we can answer a question about the artificial neural network, because 810 01:48:06.870 --> 01:48:21.939 216-101 Large Conference: you know, we can pass gradients through every single neuron, right? We can train… we can train a zoo of thousands of examples. You know, we can do things that were impossible to do with humans, and so we can start to answer these unanswerable questions, at least 811 01:48:22.600 --> 01:48:23.839 216-101 Large Conference: for an AI. 812 01:48:23.950 --> 01:48:25.739 216-101 Large Conference: Which is… which is kind of… 813 01:48:25.960 --> 01:48:29.749 216-101 Large Conference: Kind of cool, right? This is the example of… 814 01:48:29.990 --> 01:48:36.419 216-101 Large Conference: You know, an age-old question that is now becoming… Answerable in a new way. 815 01:48:38.240 --> 01:48:38.970 216-101 Large Conference: Yeah? 816 01:48:39.220 --> 01:48:40.330 216-101 Large Conference: Oh, you're at. 817 01:48:41.050 --> 01:48:46.239 216-101 Large Conference: So, is there no shared consensus on what the definition of introspection is? 818 01:48:46.460 --> 01:48:57.199 216-101 Large Conference: There's no curriculum. Maybe it's, you know, shared gradient attribution. 819 01:48:58.710 --> 01:49:02.860 216-101 Large Conference: Yeah, it's like that XKCD cartoon about, like, you know. 820 01:49:02.990 --> 01:49:05.409 216-101 Large Conference: You have 29 standards, what should we do? 821 01:49:05.700 --> 01:49:09.489 216-101 Large Conference: One of the hard lessons that I've learned with 822 01:49:10.850 --> 01:49:16.020 216-101 Large Conference: one of my old works when I put out a tweet. I got a lot of backlash for… 823 01:49:16.160 --> 01:49:21.670 216-101 Large Conference: Saying that that's anthropomorphizing, yeah. Anthropomorphize the models. 824 01:49:22.030 --> 01:49:28.889 216-101 Large Conference: So, since then, I'll be, like, super careful before, like, I make any claim. So here, with this, what I understand is… 825 01:49:29.220 --> 01:49:36.520 216-101 Large Conference: At some point, the model Gets good at being able to verbalize learned logic. 826 01:49:38.240 --> 01:49:49.719 216-101 Large Conference: And like, as Andy was telling, originally it's doing maybe some heuristics, or maybe not, like, deep enough that it can generalize it across its verbalizations. 827 01:49:49.800 --> 01:50:06.530 216-101 Large Conference: Then you found that, oh, there's actually, like, the more you train, it gets verbalized, or once it has learned how to verbalize, then it becomes, like, this meta-learning that it's easier for it to, like, learn a new task and then have it verbalized super quickly, rather than having to take, like. 828 01:50:06.620 --> 01:50:07.760 216-101 Large Conference: unread. 829 01:50:07.980 --> 01:50:10.420 216-101 Large Conference: Personas for 3,000 deep bucks or something. 830 01:50:10.880 --> 01:50:15.710 216-101 Large Conference: So, like, keeping it that way, I think it's in introspection, if that's how you… 831 01:50:15.920 --> 01:50:20.329 216-101 Large Conference: Without the word anthroposizing the model. 832 01:50:20.740 --> 01:50:32.569 216-101 Large Conference: It's just… for LLMs, assuming they don't have conscience or, like, any… these high-level human aspects, for me, introspection with LLMs is if they can faithfully… not faithfully. 833 01:50:32.730 --> 01:50:38.640 216-101 Large Conference: If they can at least give… verbalize what the learned algorithm is. 834 01:50:39.460 --> 01:50:43.810 216-101 Large Conference: I think that's… decent enough. I, I don't… I still have… 835 01:50:44.340 --> 01:50:50.550 216-101 Large Conference: my question with how faithful it is, because I… I can't believe that you trained it with, like, linear 836 01:50:50.870 --> 01:51:02.769 216-101 Large Conference: the ground truth, and that's exactly the only solution that it has learned to do it. You mean how grounded is it? Yeah, like, it's a low-rank adapter, it's, like, pretty powerful, it's not a linear layer. 837 01:51:02.990 --> 01:51:20.839 216-101 Large Conference: when I say linear, it's not just one MLP. Is the claim that, like, if you just tested it a bunch more, it would be well described by linear preferences, or that internally it's doing something to mimic linear decision function? Both. I… 838 01:51:21.040 --> 01:51:31.949 216-101 Large Conference: I don't think that has been… that's… that came out from your talk, that you have exhaustively searched that these two claims are, like, rock solid. 839 01:51:32.080 --> 01:51:40.069 216-101 Large Conference: Have you looked to see if it's implementing linear regression in its… in the residual stream or something like this? 840 01:51:40.530 --> 01:52:00.270 216-101 Large Conference: Yeah, well, kind of. I've done probing, which I guess is basically that. Each of the different variables? Yeah, yeah, yeah. Yeah, it seems pretty messy. So, I, like, I definitely believe that it's not unlike… you know, you can do different levels of abstraction here, like. 841 01:52:00.470 --> 01:52:05.620 216-101 Large Conference: On many lower levels of abstraction, it's definitely not doing linear regression. 842 01:52:05.860 --> 01:52:09.289 216-101 Large Conference: But I feel like… 843 01:52:10.270 --> 01:52:18.469 216-101 Large Conference: Yeah, I guess I'm, like, confident that it's doing something close-ish to linear regression, or, like, linear regression plus noise, or something like that. 844 01:52:18.810 --> 01:52:24.979 216-101 Large Conference: Just because the correlations are so high. Yeah. Like, many of the correlations are, like, literally 1, at least in the sample that we choose. 845 01:52:25.240 --> 01:52:35.709 216-101 Large Conference: And then, like, obviously, you know, out of distribution, it would maybe do something different or something like that, but… No, I'm sold on your… all the measures you've done. I'm just saying… 846 01:52:36.270 --> 01:52:39.679 216-101 Large Conference: I'm not sure if that's the only solution that the portal has got. Right. 847 01:52:40.580 --> 01:52:49.889 216-101 Large Conference: And also, if it's the faithful, it's the thing that it's doing to actually get to the answer, there's the connect between your two plots, the two lines, I'm… 848 01:52:50.680 --> 01:52:53.530 216-101 Large Conference: I don't know how you made that, like, if that makes sense. 849 01:52:54.090 --> 01:53:03.570 216-101 Large Conference: Being faithful through self-report. Yeah. I agree, it is able to do that. Yeah, yeah. That's, like, a really cool observa- like, cool finding. 850 01:53:03.950 --> 01:53:09.299 216-101 Large Conference: It being, like, the performance being high all along is also, like, really crazy that… 851 01:53:09.780 --> 01:53:15.200 216-101 Large Conference: It's happening, but is this faithfulness, once it hits that faithfulness mark. 852 01:53:15.460 --> 01:53:18.949 216-101 Large Conference: Is that the real mechanism it's using to get to that performance, or… 853 01:53:19.200 --> 01:53:21.259 216-101 Large Conference: What is the difference in performance? 854 01:53:21.410 --> 01:53:24.969 216-101 Large Conference: What's his faithfulness? And you're not convinced by… 855 01:53:25.120 --> 01:53:30.269 216-101 Large Conference: Here, it's a little bit circumstantial, but, like, these experiments where he says, well. 856 01:53:30.500 --> 01:53:35.330 216-101 Large Conference: When you try to do, you know, the gradient-based 857 01:53:35.730 --> 01:53:44.230 216-101 Large Conference: attribution to say, like, how does it output this JSON? Well, it's depending on these weights. How does it answer that maybe it's, oh, it's depending on the same weight. 858 01:53:44.370 --> 01:53:45.980 216-101 Large Conference: In the case of the safety. 859 01:53:46.280 --> 01:53:55.750 216-101 Large Conference: Whereas… whereas when you ask, how's it depend on the JSON, how's it to the APA, it's a different weights when it's not vehicle. So that's the… you don't feel like that… 860 01:53:56.100 --> 01:53:57.090 216-101 Large Conference: you know… 861 01:53:58.040 --> 01:54:15.990 216-101 Large Conference: addressed your concern. No. It's consistent with it. It's consistent with, like, if you knew everything, then you'd expect to see this. It's definitely circumstantial. In the right direction? It's in the right direction, but I don't think, like, it's rock solid. Yeah. But again, like. 862 01:54:16.420 --> 01:54:19.010 216-101 Large Conference: First of all, I think it's criminal that you could 863 01:54:19.210 --> 01:54:34.019 216-101 Large Conference: fit all of this into that one small Navy poster. And, like, I agree, like, there could be, like, a lot of… There's a few papers in this paper, right? Yes. Can you… can you, like, give me a hypothetical, like, what it could be doing if it's not? 864 01:54:34.210 --> 01:54:38.170 216-101 Large Conference: Just so I understand, kind of, the claim. No, as you said, like, it… 865 01:54:38.690 --> 01:54:41.250 216-101 Large Conference: There's not, like, a unique solution for… 866 01:54:41.460 --> 01:54:53.440 216-101 Large Conference: Well, I'll have to present. Like, it could be, like, it could be a decision tree instead of a linear model or something like that. It could be implementing some other… Literally, right, underlying. Underlying, right. Yeah, so then there's, like. 867 01:54:53.990 --> 01:55:00.879 216-101 Large Conference: Yeah, so I agree with that, but then there's kind of this, like, semantic question of, do we call this faithful if it's kind of… 868 01:55:01.610 --> 01:55:06.719 216-101 Large Conference: Accurately describing a process which, like, behaves linearly, but is implemented. 869 01:55:06.980 --> 01:55:11.699 216-101 Large Conference: In a different way. Does that seem fair? Like, that's kind of the disagreement? 870 01:55:11.750 --> 01:55:28.150 216-101 Large Conference: Yeah. That's what we call this. Right. The way you're calling faithfulness is faithfulness to the task, one of the solutions that you built the ground truth on. Right. It's not, like, mechanistic faithfulness. It's not mechanistic faithfulness, yes. Yeah, I agree with that, I agree. 871 01:55:29.300 --> 01:55:48.140 216-101 Large Conference: Yeah, I think it's like, if this was a more robust field, it would be good to distinguish these things more. You don't have to say that it's not, it's not faithful to the mechanistic faithfulness. There are different levels, right? Like, I'm talking about the modest levels, right? You can say that… you can… you can say that. 872 01:55:48.170 --> 01:55:51.789 216-101 Large Conference: It is faithful to the policy, which is probably Mars Level 2. 873 01:55:52.060 --> 01:55:54.450 216-101 Large Conference: But we don't know about Mars Level 3 yet. 874 01:55:56.480 --> 01:56:06.339 216-101 Large Conference: Yeah, yeah, I mean, I… yeah, I agree with that. But I do think there's… there's, yeah, this question of, like, if we just say faithful, unadormed, like, which one are we talking about? 875 01:56:06.450 --> 01:56:12.970 216-101 Large Conference: So, you know, Arna, what I'm really interested in this from your paper's point of view. Have you… do you see how this… 876 01:56:13.120 --> 01:56:17.120 216-101 Large Conference: Connects to your work, your current work that you're… you have ongoing, unpublished? 877 01:56:17.480 --> 01:56:18.709 216-101 Large Conference: the go-through. 878 01:56:19.620 --> 01:56:24.219 216-101 Large Conference: So, for example, yes, GAO Connect, so, like, so if you… 879 01:56:24.540 --> 01:56:26.889 216-101 Large Conference: Like, so if you took Greg or SAMHSA? 880 01:56:27.350 --> 01:56:35.720 216-101 Large Conference: and you said, hey, I, you know, I want to create a washing machine that Gregor Sampson likes. It has these 3 attributes. 881 01:56:35.850 --> 01:56:38.760 216-101 Large Conference: You know, tell me what the fourth attribute should be. 882 01:56:39.260 --> 01:56:40.150 216-101 Large Conference: Right? 883 01:56:40.630 --> 01:56:44.640 216-101 Large Conference: you know, like, I'm curious… If these models 884 01:56:44.990 --> 01:56:47.010 216-101 Large Conference: Would fill in the fourth attribute. 885 01:56:47.150 --> 01:57:05.079 216-101 Large Conference: properly at the early checkpoint or the late checkpoint, if there's a difference, right? So you see how that's related to your, right, your thing, right? And my hypothesis would be, you need the late checkpoint, you need introspection. Like, there's this other, like, you have this linguistic backbone that is attached to all of these 886 01:57:05.770 --> 01:57:09.230 216-101 Large Conference: You know, things that we anthropomorphize. 887 01:57:09.530 --> 01:57:13.599 216-101 Large Conference: Like, when Rohit gets raked over the coals. 888 01:57:13.800 --> 01:57:25.459 216-101 Large Conference: For saying, my model is thinking about something, it has some intention, it is paying attention to something, it's doing whatever, right? They're like, you're just… you're just telling a… telling a human story here. 889 01:57:25.620 --> 01:57:28.689 216-101 Large Conference: Right? Like, my hypothesis is… 890 01:57:28.890 --> 01:57:33.250 216-101 Large Conference: We tell these stories when we are thinking linguistically, when we're thinking in a certain mode. 891 01:57:33.360 --> 01:57:39.740 216-101 Large Conference: And… and these models actually are able to think in this type of mode, but it's… but they also… 892 01:57:40.050 --> 01:57:44.130 216-101 Large Conference: cannot think in this mode, like the earlier time. They're able to do it both ways. 893 01:57:44.470 --> 01:57:50.010 216-101 Large Conference: Right? And… but then when they're doing it in this way, we should recognize that it does. 894 01:57:50.330 --> 01:57:54.660 216-101 Large Conference: It does resemble rational thought. It resembles what humans do. 895 01:57:54.940 --> 01:57:59.379 216-101 Large Conference: And, you know, so I think it's… it's deep water, so we… 896 01:58:00.150 --> 01:58:02.870 216-101 Large Conference: You shouldn't be careless about it, right? 897 01:58:03.150 --> 01:58:10.029 216-101 Large Conference: But, you know, I think that there's something here, right? It could be… so my hypothesis that would be related to this, I think, would be interesting. 898 01:58:10.280 --> 01:58:13.160 216-101 Large Conference: Yeah, that'd be really cool. Right, does that make sense? 899 01:58:13.390 --> 01:58:20.820 216-101 Large Conference: Also, another point I wanted to make is that, grokking and generalization versus introspection 900 01:58:21.210 --> 01:58:24.459 216-101 Large Conference: They don't have to be orthogonal to each other, right? 901 01:58:25.030 --> 01:58:31.439 216-101 Large Conference: I mean… You mean they don't have to be inconsistent? They can be… Yeah, they can't, 902 01:58:31.680 --> 01:58:38.320 216-101 Large Conference: Yeah, they can't be orthon. I mean, can't be is probably… but they don't necessarily have to be orthogonal region. 903 01:58:38.440 --> 01:58:40.280 216-101 Large Conference: Yeah, I agree. 904 01:58:44.420 --> 01:58:48.579 216-101 Large Conference: Thank you, David. Over Zoom, they say it was a great presentation. 905 01:58:53.310 --> 01:58:54.990 216-101 Large Conference: Yep. 906 01:58:55.680 --> 01:58:58.029 216-101 Large Conference: He might have to call the admin tomorrow. 907 01:58:58.490 --> 01:59:05.769 216-101 Large Conference: Allow the full screen. What's going on? We might have to ask the admin to.