WEBVTT 1 00:00:00.220 --> 00:00:04.070 216-101 Large Conference: Great. 2 00:00:04.700 --> 00:00:06.909 216-101 Large Conference: So, let me, introduce Eric. 3 00:00:07.170 --> 00:00:09.099 216-101 Large Conference: So, I'm happy to have 4 00:00:09.300 --> 00:00:14.709 216-101 Large Conference: Eric, present today. So, Eric is, Eric is a PhD student in my lab. 5 00:00:15.190 --> 00:00:17.690 216-101 Large Conference: Who has done a variety of 6 00:00:18.520 --> 00:00:26.589 216-101 Large Conference: things, studying, in-context learning, in-context representations, In models, you'd previously 7 00:00:26.760 --> 00:00:30.070 216-101 Large Conference: publisher work called In Context Algebra, where he 8 00:00:30.440 --> 00:00:37.190 216-101 Large Conference: Looked at, very structured reasoning, models that were trained to solve algebraic problems. 9 00:00:37.510 --> 00:00:42.260 216-101 Large Conference: And, and I think that led to the current project that he's doing, 10 00:00:42.400 --> 00:00:44.760 216-101 Large Conference: Partly from an internship at GoodFire. 11 00:00:45.050 --> 00:00:46.700 216-101 Large Conference: Which is, 12 00:00:46.850 --> 00:00:55.249 216-101 Large Conference: Which, I think he's preparing it for, ICLR, and he wanted to give a presentation to get people's feedback about this work in progress. 13 00:00:55.700 --> 00:01:01.290 216-101 Large Conference: So, so, well, welcome, welcome, Eric. Thank you, Eric, for presenting today. Thank you. 14 00:01:04.269 --> 00:01:14.600 216-101 Large Conference: Great. Yeah, so, I'll be presenting… this is… I did, this is part of my… this is what I was doing for my internship at Goodfire. 15 00:01:14.810 --> 00:01:20.240 216-101 Large Conference: This is the provisional title, Geometric Supervision on Neural Representations. 16 00:01:22.870 --> 00:01:34.369 216-101 Large Conference: But yeah, so, this project sort of started out with this main question here at the top of, like, is there, like, a way that we can debug neural networks and make them generalize more systematically? 17 00:01:34.820 --> 00:01:48.290 216-101 Large Conference: And I think this is, like, one of the main, like, goals of interpretability research, is, like, being able to understand a network well enough that we can go in and edit and, you know, have a… 18 00:01:48.740 --> 00:01:53.980 216-101 Large Conference: Of course, both, like, a qualitative and, like, clear way to 19 00:01:54.460 --> 00:01:58.100 216-101 Large Conference: Understand them, and correct them when they make mistakes. 20 00:01:58.410 --> 00:02:10.470 216-101 Large Conference: that's, like, the dream, I guess, right? And I think a lot of the work that we do in our lab sort of fits that flavor of, like, we find some interesting behavior and try and describe it well enough 21 00:02:10.650 --> 00:02:17.400 216-101 Large Conference: With some abstraction that we can, like, you know, edit that abstraction and, like, make the model generalize on some 22 00:02:17.620 --> 00:02:18.680 216-101 Large Conference: some tasks. 23 00:02:19.410 --> 00:02:24.370 216-101 Large Conference: And so, so what we were trying to do in this work is, 24 00:02:25.040 --> 00:02:30.210 216-101 Large Conference: Sort of apply this idea, or try and answer, like, how much progress can we make on this idea? 25 00:02:30.380 --> 00:02:37.209 216-101 Large Conference: In cases where, like, we sort of know, like, internal geometry, that the model uses for different tasks. 26 00:02:37.460 --> 00:02:49.860 216-101 Large Conference: And then, like, as a side point, like, one of the things that people usually do to try and fix models is just fine-tune them on some new data. Like, and that is pretty efficient, like, gradient descent's really good. 27 00:02:50.070 --> 00:03:06.589 216-101 Large Conference: And so, like, some… like, part of this is sort of trying to understand, like, what… what is fine-tuning doing as part of that? And Nikhil's looked at this before as well. Yeah, so… so I'll talk about… we're going to look at two tasks, so the first one… 28 00:03:06.720 --> 00:03:15.649 216-101 Large Conference: is this, this, cyclic month test. So if you were here, I don't know, at the beginning of the summer, back when Sheridan… this is Sheridan's work, 29 00:03:15.780 --> 00:03:17.289 216-101 Large Conference: When they were at Goodfire. 30 00:03:18.830 --> 00:03:26.540 216-101 Large Conference: And so… so part of my work is, like, a continuation of this. So… but I'll… I'll introduce this… this task for you, so… 31 00:03:26.660 --> 00:03:40.080 216-101 Large Conference: you have prompts of the form in this first bullet, where you say, like, what month is blank months after some other month? So maybe you say, like, and this is taken from their paper, but yeah, so what month is 6 months after August? 32 00:03:40.290 --> 00:03:43.260 216-101 Large Conference: And in this case. 33 00:03:43.500 --> 00:04:01.140 216-101 Large Conference: What they found was that the model computes sort of this intermediate sum of the two variables, this offset and the input. So in this case, like, 6 plus August would give you 6 plus 8 is 14, and then, it would take that mod 12 back out to some other month, February. 34 00:04:01.330 --> 00:04:08.489 216-101 Large Conference: And they sort of found that the model has this, these, like, Fourier, plane structures. 35 00:04:08.730 --> 00:04:11.160 216-101 Large Conference: That are computed via these neurons. 36 00:04:11.670 --> 00:04:18.329 216-101 Large Conference: At different… at, like, phase 10 periods, basically. So, like, 2, 5, 10… 37 00:04:18.910 --> 00:04:25.740 216-101 Large Conference: 20, 50, 100. So instead of… instead of computing this via, like, a mod 12 circle circle, which is sort of 38 00:04:26.030 --> 00:04:33.129 216-101 Large Conference: how you expect people might do it. This was kind of surprising that it's doing it via, like, base 10 arithmetic. 39 00:04:35.510 --> 00:04:36.789 216-101 Large Conference: Yeah, so… 40 00:04:36.990 --> 00:04:44.129 216-101 Large Conference: So I… I sort of expect familiarity with this, so if people, like, if this is, like, a new paper, or, like. 41 00:04:44.660 --> 00:04:48.899 216-101 Large Conference: people have questions about the setup, feel free to stop me, and I can try and explain more. 42 00:04:49.110 --> 00:04:54.020 216-101 Large Conference: Because, like, I'll just, like, be diving deeper into this, so… 43 00:04:54.180 --> 00:04:59.350 216-101 Large Conference: But yeah, so feel free to, yeah, feel free to stop and ask questions, if, like, the setup 44 00:04:59.490 --> 00:05:04.839 216-101 Large Conference: Or, like, what they found, and this initial paper wasn't clear. But I'll try and also, like, give some more context. 45 00:05:05.180 --> 00:05:06.630 216-101 Large Conference: For what to what we found. 46 00:05:07.620 --> 00:05:14.079 216-101 Large Conference: Right, so… So you can sort of break this down, again, like, there's sort of these three… 47 00:05:14.180 --> 00:05:20.270 216-101 Large Conference: variables in this task that are, like, important for solving the task. So one is, like, the input month. 48 00:05:20.480 --> 00:05:23.769 216-101 Large Conference: which is, like, January through December, when is sort of this offset? 49 00:05:23.890 --> 00:05:25.470 216-101 Large Conference: And in our case. 50 00:05:25.750 --> 00:05:33.180 216-101 Large Conference: We're gonna look at, like, a big range of offset values, and then one is sort of implicit, but it's the sum between the input and the offset. 51 00:05:33.390 --> 00:05:36.249 216-101 Large Conference: It was covered by the first two. 52 00:05:37.350 --> 00:05:47.969 216-101 Large Conference: So yeah, so, in the original paper, they sort of looked at the gen… like, how, how, this Llama 3.18 billion parameter model, like, solves this task. 53 00:05:48.080 --> 00:05:58.659 216-101 Large Conference: And what I'm showing here is, on the x-axis is the, the sum, and then on the y-axis is, like, the accuracy pooled over prompts that have that sum. 54 00:05:58.750 --> 00:06:15.170 216-101 Large Conference: And so in the original paper, they sort of looked up to 3 cycle lengths, so up to 36, and they were like, hey, the model's pretty good when you don't have to do this mod, like, from 2 to 12, it gets perfect accuracy. And then when you start having to do this mod, it's like, it starts doing a lot worse and has, like, these weird 55 00:06:15.250 --> 00:06:20.629 216-101 Large Conference: you know, systematic errors. And then, if you sort of look 56 00:06:20.810 --> 00:06:30.209 216-101 Large Conference: At, like, an extended range of offsets, we see, like, an extended pattern of weird systematic errors, and, like, this, you know, funky bump around 57 00:06:30.370 --> 00:06:38.560 216-101 Large Conference: 60 and 80, you know? So it's like, Llama… Llama can solve this task when it's easy, and then when it's hard, it, like, really, really struggles. 58 00:06:39.440 --> 00:06:49.870 216-101 Large Conference: Sharon, you hadn't tested this before, you didn't know about the bump between 60 and 80 before, is that right? Only went up to 36, and yeah, yeah. That's cool. Yeah. 59 00:06:50.710 --> 00:06:53.120 216-101 Large Conference: Yeah, so this is… this is sort of the extent of… 60 00:06:53.260 --> 00:07:02.379 216-101 Large Conference: where the other paper was looking at. And so, we sort of wanted to test, like, how can we get models to generalize, like, in a bigger range of values? 61 00:07:02.700 --> 00:07:12.110 216-101 Large Conference: Because, like, I guess the idea is, like, if we know, sort of, the machinery of how the model's, solving this problem, we should be able to, you know, correct it and make it more systematic. 62 00:07:13.200 --> 00:07:17.430 216-101 Large Conference: So, so the first thing we try is just, like, LoRa fine-tuning. 63 00:07:17.570 --> 00:07:35.259 216-101 Large Conference: And I'll try and explain what this plot is showing. So, again, on the x-axis here are the values of the sum, and the y-axis for each of these subplots is just the accuracy. And the gray shaded region is, like, the training range of the LoRa. 64 00:07:35.640 --> 00:07:48.610 216-101 Large Conference: So, for example, in the top plot, there's a LoRa that's been trained with data with sums of from 2 to 24, and the orange line is the LoRa's performance, and the dashed gray line is the base model. 65 00:07:49.530 --> 00:07:57.049 216-101 Large Conference: So, basically what happens is, the task, Laura, we'll, like. 66 00:07:57.250 --> 00:08:05.340 216-101 Large Conference: Solve everything in its training distribution, and then, like, pretty quickly drop off back down to zero. 67 00:08:05.900 --> 00:08:23.909 216-101 Large Conference: And this, like, you can sort of see this, like, ladder, step, you know, staircase pattern here, where, like, in orange, I've shaded the training region, like, the training data set, and you basically, like, these test words get maybe, like, one extra cycle of generalization. 68 00:08:24.340 --> 00:08:27.080 216-101 Large Conference: Where they like a good, and then they, like, die off. 69 00:08:27.260 --> 00:08:38.719 216-101 Large Conference: And… but then there's, like, this… this extra little, you know, where, like, the base model was good, and so some of these are also good around that range, but then everything sort of… sort of just dies off again. 70 00:08:39.650 --> 00:08:47.309 216-101 Large Conference: So one takeaway here is that, sort of, the LoRa gen… like, the LoRa can fix these errors that the base model makes. 71 00:08:47.460 --> 00:08:49.670 216-101 Large Conference: But the generalization is, like. 72 00:08:49.800 --> 00:08:56.410 216-101 Large Conference: Shallow, in the sense that it's basically overfitting to its distribution, and, like, maybe getting you an extra 73 00:08:56.780 --> 00:08:57.919 216-101 Large Conference: Little bits. 74 00:08:58.730 --> 00:09:00.320 216-101 Large Conference: Yeah. 75 00:09:01.290 --> 00:09:04.840 216-101 Large Conference: Any questions up to here? Or this is pretty clear. Okay. 76 00:09:06.360 --> 00:09:21.529 216-101 Large Conference: Yeah, so, oh, and one thing that might surprise you, so I guess when I say LoRa, maybe you think, like, you're just training, LoRa adapters on the whole model. Actually, what we found, is that 77 00:09:21.830 --> 00:09:25.509 216-101 Large Conference: What we're doing is we're just training a single LoRa on a single MLP, 78 00:09:25.960 --> 00:09:31.980 216-101 Large Conference: And if you… and you need to train the LoRa at an early layer in the network. 79 00:09:32.340 --> 00:09:45.390 216-101 Large Conference: So this was kind of surprising to us, because based on, like, what Sheridan's paper found, like, the model was good at computing the sum, but it was bad at computing, like, the mod. That was sort of the hypothesis. 80 00:09:45.690 --> 00:09:53.680 216-101 Large Conference: And… and the sum… For context, like, the sum is being computed around layer 18, Somewhere in there. 81 00:09:53.980 --> 00:10:07.969 216-101 Large Conference: And so we thought, if we just, like, fine-tune the model after layer 18, we'll be able to, like, fix how the model is computing the mod, and solve a lot of these errors. But what we found instead was basically the opposite. You, like… 82 00:10:08.360 --> 00:10:14.449 216-101 Large Conference: LoRa fine-tuning works a lot better when you do it in the early layers, instead of in the late layers. 83 00:10:15.590 --> 00:10:16.480 216-101 Large Conference: Yeah. 84 00:10:17.170 --> 00:10:18.820 216-101 Large Conference: The previous slide. 85 00:10:19.080 --> 00:10:21.040 216-101 Large Conference: Yes. 86 00:10:21.150 --> 00:10:30.659 216-101 Large Conference: Alright, so what are these, Loros? Are these all layers? Oh, so these ones are… this is, like, taking… I've got this little white box, so I'm just taking these at, like, layer 4. 87 00:10:30.950 --> 00:10:36.940 216-101 Large Conference: So these are… so they're apply that layer 4. Right. 88 00:10:37.170 --> 00:10:49.090 216-101 Large Conference: Layer 4 MLP. And if you, if you, like, train the whole model, like, all layers, do you see, like… It's, like, it's actually slightly worse than this doing, like, a single layer. 89 00:10:50.010 --> 00:10:55.849 216-101 Large Conference: So you still see this generalization drop-off? Yeah, you still have the staircase, yeah. 90 00:10:56.550 --> 00:11:00.060 216-101 Large Conference: Eric? Yeah. I had a short question. 91 00:11:00.230 --> 00:11:03.680 216-101 Large Conference: When you actually look at the errors that are… 92 00:11:04.250 --> 00:11:07.859 216-101 Large Conference: Like, when the model fails out of distribution, is there some, like. 93 00:11:08.630 --> 00:11:11.869 216-101 Large Conference: Characteristic to the errors it makes, or is it just noise? 94 00:11:12.000 --> 00:11:20.760 216-101 Large Conference: Yeah, so a lot of the errors are, like, off by 2 or off by 4, and I'll sort of explain that in a bit. 95 00:11:20.860 --> 00:11:24.090 216-101 Large Conference: But, one… yeah, I guess one of the reasons is, like. 96 00:11:24.230 --> 00:11:34.200 216-101 Large Conference: The model's using these base 10 period features, but months are in base 12, so there's, like, an off by 2, just, like, naturally off-by-two pattern that, like, persists. 97 00:11:34.350 --> 00:11:40.800 216-101 Large Conference: In a lot of the features. So a lot of the… a lot of the model's predictions are, like, off by 2 a lot of the time. 98 00:11:42.190 --> 00:11:49.650 216-101 Large Conference: Yeah, but we'll… I'll get back to that in a second. Yeah, Shane, you're just curious about how do you do held up for the distribution ones? 99 00:11:50.240 --> 00:11:53.250 216-101 Large Conference: How do I do holdouts for in distribution? 100 00:11:53.390 --> 00:11:56.600 216-101 Large Conference: Oh, for the distribution, I'm not doing a holdout. 101 00:11:56.880 --> 00:12:09.239 216-101 Large Conference: For this plot. But, I… I have ran versions of this, where you, like, hold out some chunk in the middle, or you hold out some subset for each sum, and the model can still solve them. 102 00:12:09.390 --> 00:12:11.899 216-101 Large Conference: Totally fine within the training range. 103 00:12:12.460 --> 00:12:13.200 216-101 Large Conference: Yeah. 104 00:12:16.460 --> 00:12:20.100 216-101 Large Conference: So, so, like, rights. 105 00:12:20.400 --> 00:12:22.450 216-101 Large Conference: So you have to train it at an early layer. 106 00:12:22.840 --> 00:12:36.290 216-101 Large Conference: And so then we were trying to understand, like, well, if they're not, you know, fixing the mods sort of like what we thought they were going to do, what are they… what are they doing? What are these lures doing to the model to sort of fix these errors within distribution? 107 00:12:36.800 --> 00:12:52.049 216-101 Large Conference: And so one thing we can do is sort of look at, the same representations that they were looking at before. So in this case, these are, we train a bunch of Fourier probes, which is just looking for… at different periods. 108 00:12:52.700 --> 00:12:58.900 216-101 Large Conference: So… Sorry, ignore the far column on the right. 109 00:12:59.540 --> 00:13:03.839 216-101 Large Conference: But the left column is the base model, and you can see, like. 110 00:13:04.130 --> 00:13:06.840 216-101 Large Conference: It's already got, like, pretty clean geometry. 111 00:13:08.450 --> 00:13:14.360 216-101 Large Conference: like, the period 2, you can sort of see a nice separation, but, like, for these task floras, you can sort of see, like. 112 00:13:14.690 --> 00:13:18.290 216-101 Large Conference: Cleaner separation in the same period. 113 00:13:20.610 --> 00:13:39.859 216-101 Large Conference: Yeah, just across this… so this… so these are probes, on the input variable, so probing for the input at, like, the layer 18 MLP. And so for you, you notice, like, for period 20, for example, there's only 12 input months, and so instead of, like, forming a circle, it's, like, forming sort of an arc, like a… like, there's only 12 points, yeah. 114 00:13:39.930 --> 00:13:41.740 216-101 Large Conference: What do you mean by period? 115 00:13:42.010 --> 00:13:47.710 216-101 Large Conference: Period. Yeah, so, let me… Okay, so, like… 116 00:13:48.110 --> 00:13:51.809 216-101 Large Conference: For, like, sine and cosine waves, like a period of a wave. 117 00:13:52.030 --> 00:13:55.480 216-101 Large Conference: Like, the distance between two peaks and a wave. 118 00:13:56.650 --> 00:13:59.740 216-101 Large Conference: Yeah. Is that… is that familiar to you? 119 00:14:00.330 --> 00:14:01.370 216-101 Large Conference: I can… 120 00:14:01.880 --> 00:14:11.169 216-101 Large Conference: Okay, wait, so I understand what you meant by period, based on your explanation, but how is this translating to training a probe? 121 00:14:11.290 --> 00:14:18.660 216-101 Large Conference: What do you train on? Yeah, so, okay, so this is a detail from Sheridan's paper that I'll re-explain. So, 122 00:14:19.000 --> 00:14:24.860 216-101 Large Conference: What they did to, like, characterize… oh, I don't know how… okay, so I'll come back here real quick. 123 00:14:25.030 --> 00:14:32.089 216-101 Large Conference: So the way that they, like, characterized how the model was doing, the addition of, 124 00:14:32.390 --> 00:14:45.519 216-101 Large Conference: of these input and offset numbers was they trained, Fourier probes, which are… are checking for, like, the similarity between, like, the ideal sine wave and the ideal cosine wave for different modulos. 125 00:14:45.670 --> 00:14:54.059 216-101 Large Conference: So in this case, like, the period 2, probe, like, you only have two values, like, 0 and 1. 126 00:14:55.600 --> 00:15:03.060 216-101 Large Conference: And actually, like, sine… sine is always zero in this case, so it's really just the cosine one that's checking for stuff. 127 00:15:03.320 --> 00:15:06.360 216-101 Large Conference: But the way that you train these probes, 128 00:15:06.610 --> 00:15:17.660 216-101 Large Conference: is… is via some… it's, like, via this loss on the very bottom here, so it's, like, this MSE loss is check… you have, like, some weight, that's your probe weight. 129 00:15:17.820 --> 00:15:26.750 216-101 Large Conference: For sine or cosine, and you take the inner product between, like, sine and your hidden state that you want to check, like, is this, 130 00:15:27.140 --> 00:15:28.699 216-101 Large Conference: Feature present or not? 131 00:15:28.920 --> 00:15:35.579 216-101 Large Conference: And you… and you take the embassy between that and, like, the ideal sine wave, or the ideal cosine wave. 132 00:15:35.830 --> 00:15:47.130 216-101 Large Conference: Does that make sense? And so here, what the period is, is this big T, which is, like, it's measuring the distance between the peaks in a wave. 133 00:15:49.410 --> 00:15:50.160 216-101 Large Conference: Yep. 134 00:15:50.380 --> 00:15:52.909 216-101 Large Conference: So is that… I don't know if that answered your question, but… 135 00:15:53.460 --> 00:15:55.980 216-101 Large Conference: I got a list of it. Okay. 136 00:15:56.300 --> 00:16:11.780 216-101 Large Conference: Yeah, so, yeah, so, right, so when I say period, I'm talking about this big T value that we're using to train the probes. So, like, period 2, you can kind of think of it as, like, mod 2 or mod 5, mod 10, etc. 137 00:16:11.960 --> 00:16:18.169 216-101 Large Conference: Because, like, sine and cosine… Repeat their values. 138 00:16:19.140 --> 00:16:23.530 216-101 Large Conference: Yeah, so this is one thing that we see that these floors are doing. 139 00:16:23.800 --> 00:16:29.040 216-101 Large Conference: That they're sort of, kind of visually sharpening the… 140 00:16:29.470 --> 00:16:35.140 216-101 Large Conference: 48 planes. And if we look at, this is, like, a different view of the same plots. 141 00:16:35.400 --> 00:16:39.900 216-101 Large Conference: So what this is showing is… 142 00:16:40.020 --> 00:16:50.980 216-101 Large Conference: for a bunch of different periods, like, from 2 up to 100, what's, like, the alignment of the points with the ideal Fourier, plane? 143 00:16:51.290 --> 00:17:00.449 216-101 Large Conference: And so what we see, there's, like, there's peaks at 2, 5, 10, 20, and 50, and maybe a little one, like, around 32-ish. 144 00:17:01.430 --> 00:17:12.820 216-101 Large Conference: And what these colors are is showing is, like, the base model is in gray, and it already sort of had peaks at these values, but the… but the peaks are becoming sharper. So this is just a different way to show, like. 145 00:17:12.990 --> 00:17:15.940 216-101 Large Conference: The model is sort of cleaning up its existing geometry. 146 00:17:16.150 --> 00:17:32.319 216-101 Large Conference: like, one hypothesis you could have thought is, like, maybe when we fine-tuned, the model's, like, making, you know, period 12 circles instead of reusing the same geometry it was using before, but, like, the period 12 circle just, like, is non-existent. 147 00:17:32.430 --> 00:17:35.150 216-101 Large Conference: At least under Fourier probes, you don't see it. 148 00:17:36.590 --> 00:17:43.040 216-101 Large Conference: I think you, in some of the previous slides, you showed that the model generalizes 149 00:17:43.240 --> 00:17:49.200 216-101 Large Conference: Between 50 to 70 or 60? Oh, like this. 150 00:17:49.490 --> 00:17:51.869 216-101 Large Conference: Yeah, 60 to 70, 60 to 80. 151 00:17:52.440 --> 00:17:55.900 216-101 Large Conference: So I was expecting that probably for those numbers, there might be… 152 00:17:56.400 --> 00:18:00.110 216-101 Large Conference: Even in the original model, the peers would alike. 153 00:18:00.460 --> 00:18:07.019 216-101 Large Conference: Yeah, so… so one thing… so if you notice, like, where the peaks are for the base model that are good, it's like. 154 00:18:07.220 --> 00:18:13.670 216-101 Large Conference: It's… there's one at 25, there's one at 50, and there's… and then there's this big bump from, like. 155 00:18:13.850 --> 00:18:22.549 216-101 Large Conference: 60 to 80. And, this… this sort of corresponds to… so if you think about, like, when 10 and 12 line… line up. 156 00:18:22.670 --> 00:18:35.530 216-101 Large Conference: they line up, at 60 again. So, like, if… if your features are, like, base 10 features, but you want to do something base 12, like, they'll start working again. 157 00:18:35.830 --> 00:18:39.370 216-101 Large Conference: Like, right around when… when 60 happens again. 158 00:18:39.540 --> 00:18:47.000 216-101 Large Conference: Okay. Yeah, and I have, like, a slide that kind of looks at that in more detail, but… Oh, 60 is the least common multiple? It's the least common multiple of 10 and 12. 159 00:18:47.140 --> 00:18:55.239 216-101 Large Conference: Yeah. Yeah, but so then… so then there's, like, all these other, you know. 160 00:18:55.740 --> 00:18:59.540 216-101 Large Conference: Weird quirks about how the base model does stuff. 161 00:19:00.180 --> 00:19:11.549 216-101 Large Conference: Do you have a question, Andy, or… Can you go to the most recent slide? This one? Yeah, this one. Like, is this smooth or something, or is, like, it actually good at, like, mod 47? 162 00:19:13.090 --> 00:19:23.940 216-101 Large Conference: So… so this doesn't… this is not measuring, like, the model's performance on a particular mod. This is just measuring the alignment of 163 00:19:24.170 --> 00:19:37.949 216-101 Large Conference: some Fourier plane with the model's geometry. So, yeah, I guess I don't understand the exact, way that you get the y-axis out. So, like, I understood it at the previous 164 00:19:38.530 --> 00:19:41.960 216-101 Large Conference: paper where individual neuron. 165 00:19:42.400 --> 00:19:45.309 216-101 Large Conference: You can measure a thing, like. 166 00:19:45.900 --> 00:19:50.729 216-101 Large Conference: Below, right? You can see, like, how… How much an individual neuron. 167 00:19:50.850 --> 00:19:57.290 216-101 Large Conference: You know, matches… You know, a foyer… a single foyer… 168 00:19:57.570 --> 00:20:08.169 216-101 Large Conference: frequency. So, the way that you would do that is you look at the neuron's activation over a range. Yes. Yeah, so we're doing the same thing. 169 00:20:08.460 --> 00:20:13.220 216-101 Large Conference: But instead of looking at one neuron, you're looking at something else? We're looking at our Fourier planes. 170 00:20:13.680 --> 00:20:25.339 216-101 Large Conference: And, so what's a Fourier plane? So the Fourier plane are these probe weights, like the W sine and the W cosine, and then they also have these bias terms. 171 00:20:25.370 --> 00:20:43.330 216-101 Large Conference: Yeah. And we're… and we're ranging over our… the values in our data set, so, like, all of the… all of the data where the sum is less than 96, so that's what these HXs are representing, is, like… And… and so your… your sine and your cosine thing, so you have Fourier probes, T, 172 00:20:43.640 --> 00:20:51.310 216-101 Large Conference: 2, 5, 10, and 20, so that's a four… is that a four-dimensional sine and a four-dimensional cosine? Is that what that is? So, each, 173 00:20:51.470 --> 00:21:01.480 216-101 Large Conference: each plane has two probes. Like, so if you think about… you have, like, some… you pick a period, so, like, period 2, and you have a sine for 2 and a cosine for 2. 174 00:21:04.040 --> 00:21:15.749 216-101 Large Conference: Yeah, right, so by sine and cosine, yeah, Sheridan. So that plot is just the, like, quote-unquote accuracy for the next slide, would just be, like. 175 00:21:16.340 --> 00:21:28.029 216-101 Large Conference: if it… this is the… or, the line plot one? Yeah. This is… this is, like, Tall's heat map, right? Where it's, like, you have every pot… you train probes for every possible T, and then you get the… 176 00:21:28.300 --> 00:21:39.349 216-101 Large Conference: the, like, R-squared, or, like, how good the probes are? This is, like, the accuracy of the probe for certain periods, but you can also think about it of, like, the alignment of the representation 177 00:21:39.450 --> 00:21:42.080 216-101 Large Conference: For certain periods, with an ideal circle. 178 00:21:42.440 --> 00:21:59.170 216-101 Large Conference: I see, so… so you say, okay, so we have some period 47, which, like, a prairie, there's, like, no reason the 47 would do well, but, you know, we just make sine waves of period 47, and… and we, we, see if there is a two-dimensional plane 179 00:21:59.440 --> 00:22:10.790 216-101 Large Conference: Where the representation seems to be able to line up with this circle of period 47. And the answer is, yeah, not too bad, but, like, almost 80%. 180 00:22:11.050 --> 00:22:22.450 216-101 Large Conference: Because… because it's, like, you could… it's, like, sort of similar to 50. So, like, like, our… Or 48. Or 48, right? Like, it's for, like, 2 versus 3. 181 00:22:22.660 --> 00:22:33.849 216-101 Large Conference: like, the directions are gonna be, like, really different, but once you have, like, 40… or, you know, 99 versus 100, or 48 versus 47 versus 49… They're not too different. They're not too different, yeah. 182 00:22:34.010 --> 00:22:46.579 216-101 Large Conference: So, what we were sort of assuming is that it really is 50, but, like, the probes can sort of be okay if the labels are… Exactly, yeah. 183 00:22:47.270 --> 00:23:01.110 216-101 Large Conference: Gets bigger? Bigger. Like, as you get 200, that huge lump is just 200? Yeah, like, it's not that different from 99 or 184 or something. Yeah, I think so. That's how we were interpreting it, yeah. 184 00:23:01.850 --> 00:23:18.829 216-101 Large Conference: Yeah. It's super sharp in the smaller range. Right. So it's like, it's pretty clear evidence that, like, it represents 2, it doesn't represent 3, there's, like, very little evidence for that. And then some of these other ones, it's, like, it represents some range of values around 3. Yeah. Raw. 185 00:23:19.120 --> 00:23:19.860 216-101 Large Conference: Yeah. 186 00:23:20.810 --> 00:23:21.730 216-101 Large Conference: Yeah, exactly. 187 00:23:22.740 --> 00:23:25.590 216-101 Large Conference: Did we clear up the questions about this? 188 00:23:25.850 --> 00:23:40.309 216-101 Large Conference: Thank you. Yeah. Okay, so, and these are just, like, projections of the data onto other, probes. So, like, you can train probes for different variables, so train the probes for the input, or this one is for the offset. 189 00:23:40.640 --> 00:23:46.880 216-101 Large Conference: And you see, like, similar sort of separation and, you know, cleaning up. 190 00:23:47.160 --> 00:23:50.649 216-101 Large Conference: Under, under these, these, task floras. 191 00:23:51.940 --> 00:24:06.610 216-101 Large Conference: Oh, and this one is for the summer. I don't think I appreciated it when you did your paper, right? So, but, like, so these are all just two-dimensional probes? Right. Right. So it's just… it's just, like, this funny, slight generalization of the one-dimensional probes that we… 192 00:24:06.980 --> 00:24:10.909 216-101 Large Conference: See, all over the place. Just happens to be two-dimensional rights. 193 00:24:13.060 --> 00:24:21.399 216-101 Large Conference: And for the case, like, for period 2, it's really… it's like a one-dimensional probe, the y-axis is just random. Yeah. 194 00:24:22.270 --> 00:24:23.719 216-101 Large Conference: Cool. 195 00:24:24.070 --> 00:24:42.560 216-101 Large Conference: Right, so… so then the question is, like, we sort of see this, like, sharpening, in the representations under these LORAs, and so one question is, like, what if we just, like, train the model to have clean… clean Fourier representations? Does that, like, do the same thing as what these LORAs are doing, or is there, like, something else beyond that? 196 00:24:43.270 --> 00:24:50.210 216-101 Large Conference: And so this is, like, the, the method that we're, like, looking into, is, like, 197 00:24:50.300 --> 00:25:09.400 216-101 Large Conference: Instead of using, like, a task loss, where we're taking a loss on the prediction of the model, instead, what we want to do is train some LoRa that has a loss on the geometric representation, based on some, like, idealized notion of what we think the model should be doing. 198 00:25:09.850 --> 00:25:27.770 216-101 Large Conference: So, in our… in, like, in the case of the Fourier probes, we'll talk about another test later, but, like, in this case, you know, if it's, like, a Period 5, plane, like, we should expect to see, like, nice, clean Period 5 geometry, and not, like… like, this is still, like, somewhat clean, but it's, like, a little bit of a jumbled mess. You know, so, like. 199 00:25:27.830 --> 00:25:35.400 216-101 Large Conference: Trying to see if cleaning this up, like, can achieve the same gains as, like, what a task loss on the actual prediction of the model does. 200 00:25:35.690 --> 00:25:53.219 216-101 Large Conference: Yeah. Is this diagram correct insofar as the loss is actually coming from the shape in a later module, or is the loss coming from the lower itself? So… Geometric supervision loss, I mean. Right, so… right, so the loss is happening at a… at, for example, like, layer 18, where we were just probing. 201 00:25:53.330 --> 00:26:09.069 216-101 Large Conference: And the LoRa is happening similarly, like, at an early layer, just like how we had early layer LoRas for the task loss, where we have some early layer LoRa here. So there's a big propagation. Yeah, there's, yeah, some big propagation, and that's one of the issues, I think. 202 00:26:09.250 --> 00:26:18.949 216-101 Large Conference: that I'll talk about later is, like, there's a lot of leeway for this model to sort of achieve this, this, geometry, and, like, there's some sort of, like. 203 00:26:19.270 --> 00:26:27.809 216-101 Large Conference: bleed-over effects that I… we don't have, like, clean accounts for. Yeah. This is still a rank 1? 204 00:26:28.060 --> 00:26:28.850 216-101 Large Conference: Yeah. 205 00:26:30.060 --> 00:26:31.380 216-101 Large Conference: Okay. 206 00:26:31.690 --> 00:26:40.999 216-101 Large Conference: So, right, so to do this sort of supervision, we need to pick two things. One is, like, some target geometry that we want the… 207 00:26:41.290 --> 00:26:48.189 216-101 Large Conference: the model to, like, you know, realize. And another thing is, like, some way to… to… 208 00:26:48.440 --> 00:26:53.040 216-101 Large Conference: Supervisor, like, some projection, or some way to, like, represent that. 209 00:26:53.170 --> 00:26:56.969 216-101 Large Conference: Yeah. How are you… 210 00:26:58.030 --> 00:27:02.830 216-101 Large Conference: Like, connecting the dots, that geometric supervision would be, like, the way to… 211 00:27:03.180 --> 00:27:05.170 216-101 Large Conference: Take these models, solve the task. 212 00:27:05.540 --> 00:27:16.100 216-101 Large Conference: Yeah, so… so I think it's… it's… it's based on, I think, two things. One is, like, in… in Sheridan's paper, they sort of showed that, 213 00:27:17.110 --> 00:27:27.929 216-101 Large Conference: like, the model is computing this sum, and this is, like, causally relevant for being able to do this task, and… and the way that it's computing this sum seems to be tied to these Fourier planes. 214 00:27:28.330 --> 00:27:33.309 216-101 Large Conference: And then the second thing is, like, when we train Loris to improve on this test. 215 00:27:33.680 --> 00:27:46.630 216-101 Large Conference: what… part of what they're doing is, like, sharpening these Fourier planes, and so the hypothesis is, like, can we just do the sharpening of the planes and get out the same, like, performance improvements that, like. 216 00:27:46.770 --> 00:27:49.960 216-101 Large Conference: A loss on the actual logits does. 217 00:27:50.450 --> 00:27:52.249 216-101 Large Conference: Yeah, does that make sense? 218 00:27:53.510 --> 00:27:55.060 216-101 Large Conference: Yeah. 219 00:27:55.610 --> 00:27:59.900 216-101 Large Conference: So, so we have a pretty simple loss term, 220 00:28:00.140 --> 00:28:13.230 216-101 Large Conference: there's… there's two parts, a mean, like a loss on the mean and a loss on the spread, and I'll talk about those in a minute. And I think we already talked about Fourier probes, so, I'll skip over that unless people have more questions, but basically. 221 00:28:13.710 --> 00:28:22.440 216-101 Large Conference: Like, the project… the projection under which we're supervising the geometry is, like, a collection of basically, 16… 222 00:28:22.670 --> 00:28:25.499 216-101 Large Conference: 16 probes, so there's, like, 223 00:28:25.660 --> 00:28:29.629 216-101 Large Conference: Four periods, and each on the input and offset variable. 224 00:28:29.780 --> 00:28:36.989 216-101 Large Conference: And they both have sine and cosine probes, so it's like a 16-dimensional subspace, you can kind of think of it as what we're supervising under. 225 00:28:39.260 --> 00:28:42.309 216-101 Large Conference: Right, so, yeah, anyway. 226 00:28:42.400 --> 00:28:59.300 216-101 Large Conference: Is this task still, like, the month plus offset, or, like, are you just using numbers? Well, it's still the month plus offset. Yeah, so one thing to note, I guess, is, the Fourier probes that we've been using were trained on just, like, standard addition. 227 00:28:59.770 --> 00:29:04.040 216-101 Large Conference: Which is, like, the same thing that Sheridan did in their paper. Okay. 228 00:29:04.140 --> 00:29:20.569 216-101 Large Conference: So you're gonna… you're gonna train these probes, freeze them, and then use them as a, like, try to… try to increase their accuracy. Yes, basically use them as, like, a projection under which we can sort of supervise the geometry. They're frozen. They're frozen. You compute them at the beginning using numbers. 229 00:29:20.610 --> 00:29:30.869 216-101 Large Conference: Yep, and then they're done. Yeah, and then you freeze them, and try to get the model to construct its representations to match these trained probes, probes and probes. Right. So… 230 00:29:31.230 --> 00:29:37.859 216-101 Large Conference: So in, like, if you… in lay terms, you're trying to get the model to think of month addition 231 00:29:38.030 --> 00:29:49.859 216-101 Large Conference: the way that it thinks about regular number addition. Right. It already does, right? It already does. But it's a bit messy. When it gets bigger than 12 or whatever, it starts messing up. 232 00:29:50.730 --> 00:29:54.479 216-101 Large Conference: Right, yeah, so that's… yeah, we want… that's a good way to put it. 233 00:29:54.660 --> 00:30:07.799 216-101 Large Conference: Wait, why is it 16? I thought it should be 8. So, right, so there's four, there's four periods, and there's 2 variables, like the input and the offset, but then each, each, period both has a sine and a cosine. 234 00:30:08.390 --> 00:30:09.539 216-101 Large Conference: I see, so… 235 00:30:09.760 --> 00:30:18.509 216-101 Large Conference: Okay, okay, so there's, so there's, like, 8… 8 planes, but, yeah, which is 16. The input and the offset are both numbers. 236 00:30:19.190 --> 00:30:23.300 216-101 Large Conference: Hitting the road with numbers. 237 00:30:23.570 --> 00:30:32.280 216-101 Large Conference: Sorry, I… 4 planes per number? 4 planes per number, yeah. But, like, the… why are you training a separate one for input and, offset? 238 00:30:32.380 --> 00:30:36.780 216-101 Large Conference: Like, they're both just numbers, right? Are they the same, or do they end up being the same? 239 00:30:37.250 --> 00:30:40.920 216-101 Large Conference: Oh, like, the alignment between the fights? Oh, I haven't checked. 240 00:30:41.190 --> 00:31:00.570 216-101 Large Conference: I don't know. I wonder if, like, that set of 8 probes is the same. I… I don't know. Like, one thing that's different is the… at least the position and the prompt is different, so maybe that has something to do with it, but, like, yeah, if they're just numbers, maybe they're the same. Are they also different layers? The probes are all at the same layer. Oh, really? Yeah. 241 00:31:00.690 --> 00:31:05.659 216-101 Large Conference: At least, I mean, I have them all for, like, all the layers, but the ones that we're gonna use are just all the same layers. 242 00:31:06.940 --> 00:31:14.380 216-101 Large Conference: Right, so… so we're gonna… the loss is based… there's, like, two parts. One is the mean matching, so basically the… and I'll explain this 243 00:31:14.520 --> 00:31:25.179 216-101 Large Conference: more detail later, but, like, basically the idea is, like, under some projection, we want to match the ideal… the means of the ideal geometry. So, 244 00:31:25.780 --> 00:31:28.240 216-101 Large Conference: like… 245 00:31:29.270 --> 00:31:36.380 216-101 Large Conference: So if we have some set of points, and we color them by, like, their value that they take on under some period T, 246 00:31:36.700 --> 00:31:41.270 216-101 Large Conference: We want… we have… and we have some ideal geometry that we want to match them to. 247 00:31:41.800 --> 00:32:01.419 216-101 Large Conference: What the… what the law… what the mean loss does is it computes the mean of the values, the empirical values, and it basically is just a, MSC, like, Euclidean loss between the mean and the ideal, like, the ideal mean, like, where the mean should end up when you… you're fit on a circle. 248 00:32:03.560 --> 00:32:12.719 216-101 Large Conference: And so one thing that we have to do, or don't have to, but one thing that we do do to make things a little bit better, is use something called perkresti's alignment. 249 00:32:12.920 --> 00:32:24.129 216-101 Large Conference: So, the idea is, like, we have some ideal circles, but we don't know if, like, what the, like, the actual empirical projection of the data might not be, like, rotated 250 00:32:24.490 --> 00:32:35.590 216-101 Large Conference: like, there might not be a cluster that's, like, right at degree zero. One thing we want to do is, like, find, like, a good rotation, to, like, match sort of the empirical centroids. 251 00:32:35.760 --> 00:32:41.690 216-101 Large Conference: To, like, make our target be a little bit better, instead of, like, forcing the geometry to some… 252 00:32:41.890 --> 00:32:44.949 216-101 Large Conference: Angle that it's not normally at. 253 00:32:45.480 --> 00:33:01.539 216-101 Large Conference: So, what progressive alignment does, there's… these are the basic steps, so we have some set of projected activations, and the first thing you do is… and what these are… the X's are, like, the target mean, and the big dots are, like, the mean of some value. 254 00:33:02.380 --> 00:33:11.800 216-101 Large Conference: And so, first you do, like, basically mean centering, so that everything, and then, like, unit scaling, so everything's, like. 255 00:33:11.900 --> 00:33:15.210 216-101 Large Conference: On the unit circle, basically. 256 00:33:15.430 --> 00:33:24.140 216-101 Large Conference: And then… and then you can find, like, the ideal, rotation of your target points so that it matches up the best with what the… 257 00:33:24.450 --> 00:33:26.349 216-101 Large Conference: Drupal geometry looks like. 258 00:33:28.070 --> 00:33:33.769 216-101 Large Conference: And then… and then you can go back, you can, like, rescale and translate back to where the original geometry was. 259 00:33:33.880 --> 00:33:37.690 216-101 Large Conference: And then that's, like, where your loss, the mean loss would be. 260 00:33:40.220 --> 00:33:59.839 216-101 Large Conference: Yeah, yeah. I'll play some GIFs while we answer questions, yeah. I have two questions. One question is, why do you use the mean of some cluster? Why do you not just compute error, like, point-wise? Yeah, so I tried both, and when you do point-wise, what happens is all the points collapse to the mean. 261 00:34:00.040 --> 00:34:01.319 216-101 Large Conference: That's, like, good, right? 262 00:34:01.530 --> 00:34:13.560 216-101 Large Conference: So, like… They're, like, exactly on the circle. Yeah, so, so, like, it would be, like, the ideal geometry, but there's something about, like, the natural spread of the, of the… 263 00:34:14.070 --> 00:34:19.610 216-101 Large Conference: of the points that matter. So, like, doing… so, and I… this is something I don't understand. 264 00:34:19.889 --> 00:34:27.949 216-101 Large Conference: And I think it's partly because, like, the Fourier planes we're using to supervise, like, aren't the whole story of how the model's doing this task, and so, like. 265 00:34:28.110 --> 00:34:32.769 216-101 Large Conference: You know, like, there's… there is some need for, like, natural variation even within 266 00:34:33.340 --> 00:34:36.239 216-101 Large Conference: the geometry. That's very strange. 267 00:34:36.929 --> 00:34:39.509 216-101 Large Conference: This one's scraped. 268 00:34:40.070 --> 00:34:41.449 216-101 Large Conference: Oh, why? 269 00:34:41.659 --> 00:34:47.400 216-101 Large Conference: Like, do you always… Like, in the Rakuten case, for example, like, you can completely replace the… 270 00:34:47.969 --> 00:34:51.799 216-101 Large Conference: like, you could hardcode all the Fourier stuff, because… 271 00:34:52.230 --> 00:34:59.609 216-101 Large Conference: So I guess this is Llama, and it's a language model, and there's a bunch of crap that is going on, and… 272 00:35:00.870 --> 00:35:02.429 216-101 Large Conference: Now, to be clear, you're… 273 00:35:02.530 --> 00:35:13.730 216-101 Large Conference: So you're… this is, you're only applying the loss on the two-dimensional plane, for example? Yeah, like, 16 dimensions, but yeah. But yeah, but, like, a set of two-dimensional planes. Right. 274 00:35:14.360 --> 00:35:16.389 216-101 Large Conference: And so you're already… 275 00:35:16.530 --> 00:35:32.829 216-101 Large Conference: doing this massive filtering. You're already doing this massive cleanup, you're not… you're not imposing any loss whatsoever on the other thousand or so dimensions. Is there a term to, like, have the D model minus 2 subspace preserved? 276 00:35:32.860 --> 00:35:36.550 216-101 Large Conference: No. We could do that, I haven't done it. 277 00:35:37.070 --> 00:35:45.029 216-101 Large Conference: Yeah. I mean, it's possible that the, like, putting it on the circle, right? Yeah, maybe if you did that… messes up all the other ones, because it has no… 278 00:35:45.140 --> 00:35:46.469 216-101 Large Conference: Flexibility or something. 279 00:35:46.920 --> 00:35:53.899 216-101 Large Conference: I wonder if you did that, then you could, like, pour stuff to the mean, and it would be okay? But I haven't tried it. 280 00:35:54.090 --> 00:35:59.650 216-101 Large Conference: Okay, that was my first question. Second question, I don't understand why rotation is necessary, because the… 281 00:35:59.810 --> 00:36:01.540 216-101 Large Conference: the original probes. 282 00:36:01.750 --> 00:36:05.799 216-101 Large Conference: You, you, you fit the probes 283 00:36:06.560 --> 00:36:22.889 216-101 Large Conference: like, so that the orientation was correct, basically, that the, you know, the 1,0 point is exactly the zero odd, and so on. So, why do you need, not sure why you… why… why do you need to, 284 00:36:23.060 --> 00:36:26.680 216-101 Large Conference: do this rotation. Oh, oh, 285 00:36:29.780 --> 00:36:33.200 216-101 Large Conference: Let's see. So… 20 question makes sense? 286 00:36:33.770 --> 00:36:40.749 216-101 Large Conference: I think so. Yeah, so, like, I guess if you look at… if you look at the actual projected data, like, it's in the right order. 287 00:36:40.950 --> 00:36:47.909 216-101 Large Conference: But it's like… It's not, like, equally spaced. And so what the rotation is trying to do, like. 288 00:36:48.110 --> 00:36:50.730 216-101 Large Conference: In this case, the rotation is, like, pretty minimal. 289 00:36:51.210 --> 00:36:54.480 216-101 Large Conference: But… 290 00:36:56.040 --> 00:37:03.139 216-101 Large Conference: Yeah, let's see. Like, if there were a better alignment, then your initial probe should have just fit the better alignment to begin with. 291 00:37:03.610 --> 00:37:08.980 216-101 Large Conference: Can you say that? What does that mean? Like, if… 292 00:37:10.820 --> 00:37:18.789 216-101 Large Conference: If what had better alignment? Why did your original probe just find this? Well, and it was trained on addition. 293 00:37:18.930 --> 00:37:20.079 216-101 Large Conference: This is months. 294 00:37:20.310 --> 00:37:30.450 216-101 Large Conference: So is the difference in rotation coming from the fact that, like, this is month inputs versus number inputs? Like, that's why you need to do this rotation? Yes, it could be, yeah. 295 00:37:30.810 --> 00:37:33.190 216-101 Large Conference: Yeah. 296 00:37:35.620 --> 00:37:43.320 216-101 Large Conference: I guess my intuition for doing this is that, like, we sort of want to match the empirical geometry 297 00:37:43.520 --> 00:37:47.159 216-101 Large Conference: And… yeah, I'm not… I'm not sure why the… 298 00:37:47.260 --> 00:38:04.749 216-101 Large Conference: Yeah, maybe that's the only difference, is, like, the probe is on, numbers, and these are months, and so, like, the empirical geometry is, like, slightly off from what you'd expect, and so you can just, like, rotate the targets to sort of align what the empirical means better. 299 00:38:06.200 --> 00:38:22.490 216-101 Large Conference: Although, this would also be kind of, like, strange, this were the case, I would think. Like, why would months be, like, have, like, a, you know, 15 degree rotation or something like that? This seems kind of weird. I mean, I didn't train Llama. I'll leave you alone. 300 00:38:23.990 --> 00:38:35.449 216-101 Large Conference: The question would be, like, instead of, like, doing, like, fixed post-processing, would it be possible to do joint training, but you also blanket in the probes that automatically triggers out rotation scales on its own? 301 00:38:35.760 --> 00:38:39.190 216-101 Large Conference: Yeah, you can do that, and that's, like… 302 00:38:40.170 --> 00:38:49.859 216-101 Large Conference: Yeah, you could do that. I wouldn't do that, but… Then you might end up, like, going way off, like, and then, since the rest of the model's frozen, like… 303 00:38:49.980 --> 00:38:53.870 216-101 Large Conference: You don't actually wanna… If you rotate too hard. 304 00:38:54.130 --> 00:39:03.580 216-101 Large Conference: then the downstream layer is gonna get super confused, right? Yeah, so… so one problem is, I guess, like, the… there's no… there's no task loss. 305 00:39:04.300 --> 00:39:08.389 216-101 Large Conference: So we… so we sort of wanted to keep things, like. 306 00:39:09.810 --> 00:39:17.879 216-101 Large Conference: As similar to what they were before, but just, like, clean up the geometry based on our hypothesis about what the geometry should look like. 307 00:39:18.080 --> 00:39:23.880 216-101 Large Conference: And, like, maybe learning the rotation would, like, totally mess that up. I don't know. 308 00:39:24.600 --> 00:39:26.270 216-101 Large Conference: Do you find this to be critical? 309 00:39:26.750 --> 00:39:32.849 216-101 Large Conference: Sorry, go ahead. I was just asking, did you find this to be critical? The align… the… It's like… 310 00:39:33.420 --> 00:39:35.879 216-101 Large Conference: You can drop it, and you lose, like, a couple points. 311 00:39:35.990 --> 00:39:36.930 216-101 Large Conference: Okay, yeah. 312 00:39:37.090 --> 00:39:41.129 216-101 Large Conference: So it's like, it may seem slightly better, but you don't… you don't need it. 313 00:39:41.970 --> 00:39:47.980 216-101 Large Conference: Just to clarify, follow, so your loss is on the hidden states, right? I mean, a single layer, right? 314 00:39:48.570 --> 00:39:49.290 216-101 Large Conference: Okay. 315 00:39:50.500 --> 00:39:53.310 216-101 Large Conference: Yeah, okay. 316 00:39:55.870 --> 00:40:02.750 216-101 Large Conference: Yeah, okay, and then, the other loss is on the spread, and I think we already talked about this kind of a little bit, but, like. 317 00:40:03.730 --> 00:40:10.020 216-101 Large Conference: For some reason, like, keeping the spread similar to how the base model's spread was seemed to be important. 318 00:40:10.380 --> 00:40:14.579 216-101 Large Conference: Again, you can drop it and you lose, like, a couple points, so it's not, like, critical. 319 00:40:14.780 --> 00:40:20.320 216-101 Large Conference: But, like, that's something that seems to be better, and it could just, yeah, there could be some other better way to, like. 320 00:40:20.720 --> 00:40:22.470 216-101 Large Conference: Realize the supervision? 321 00:40:23.060 --> 00:40:25.370 216-101 Large Conference: But this is what I tried. So, yeah. 322 00:40:25.850 --> 00:40:37.490 216-101 Large Conference: You don't have the formula for the spread term, somewhere, do you? I can pull up my write-up, but it's basically, like, using RMS of the base model. 323 00:40:37.550 --> 00:40:45.869 216-101 Large Conference: And then, you take the RMS under your, like, under the LoRa-trained model, and you target some ratio between those two. 324 00:40:45.940 --> 00:40:51.129 216-101 Large Conference: So you're just, like, matching, matching the, like, the RMS to the base. 325 00:40:52.230 --> 00:41:07.950 216-101 Large Conference: So, like, if you wanted, like, the same spread as the base model, like, your ratio would just be 1, and it's just, like, the RMS of your… of your… that set of values to what the base model has. I see. So it's a batch… it's a batch property, it's a property of many, many samples. Yeah. 326 00:41:09.600 --> 00:41:13.760 216-101 Large Conference: Right, so these are, like, these are sort of, like, the, you know, the first… 327 00:41:13.880 --> 00:41:21.240 216-101 Large Conference: sort of like the first two moments kind of thing, like, maybe you could try some, like, higher-order things in your loss, and… 328 00:41:21.870 --> 00:41:23.309 216-101 Large Conference: Maybe it would be better, I don't know. 329 00:41:24.480 --> 00:41:27.230 216-101 Large Conference: How does the loss on the spread look? 330 00:41:27.850 --> 00:41:30.669 216-101 Large Conference: Here, let me pull it up. 331 00:41:30.970 --> 00:41:36.330 216-101 Large Conference: Let's see… Okay, stop sharing. 332 00:41:38.120 --> 00:41:40.059 216-101 Large Conference: So, just gonna share my whole… 333 00:41:40.470 --> 00:41:47.980 216-101 Large Conference: But unlike, unlike regular losses that we see, it's not, like, you can't do it on a singleton. 334 00:41:48.310 --> 00:41:54.420 216-101 Large Conference: Right? Because the mean won't convert to it, you need to do it on a batch, is that right? Yeah, it's on a batch. 335 00:41:56.130 --> 00:42:00.230 216-101 Large Conference: So, let me share… Right. 336 00:42:11.240 --> 00:42:14.600 216-101 Large Conference: That's a nice… that's a nice example for deep learning class. 337 00:42:16.650 --> 00:42:23.660 216-101 Large Conference: Okay, I'll be back to this now? Okay, so I'm just gonna copy-paste this. So this is… this is what the spread loss looks like. 338 00:42:24.190 --> 00:42:32.730 216-101 Large Conference: So this, sigma term is just the RMS distance between, like. 339 00:42:32.840 --> 00:42:38.229 216-101 Large Conference: So you project your, hidden states onto these Fourier planes. 340 00:42:38.360 --> 00:42:43.249 216-101 Large Conference: and you take the distance between its mean, so that that's, like, what RMS is doing. 341 00:42:43.330 --> 00:42:58.519 216-101 Large Conference: And then what you're doing is you target some ratio between the trained model and the frozen base model. So, like, if your ratio is 1, then, like, you're saying, I want to keep the spread of this little cluster the exact same as what the base model had. 342 00:42:58.540 --> 00:43:05.929 216-101 Large Conference: And you could… you could make this, like, 2 if you wanted to have bigger spread, or smaller spread, or whatever. 343 00:43:06.280 --> 00:43:10.100 216-101 Large Conference: The key is, down below, you have the sum of X… 344 00:43:10.440 --> 00:43:19.919 216-101 Large Conference: in X sub V, so that's your population? Yeah, so this is the… so it's per… so it's per value, so it's like, which is the same thing as the mean loss, which is… 345 00:43:19.920 --> 00:43:34.180 216-101 Large Conference: Oh, that's per value, I see. Yeah, so the mean is taken per value, so every value has some target mean that it should be mapped to, and it also has some spread that we want to maintain. The value is, what, a day in the week or something? Yeah, so, like. 346 00:43:34.200 --> 00:43:40.339 216-101 Large Conference: Yeah, so, well, so in this case, it's, like, a month in the year, or, like. 347 00:43:40.770 --> 00:43:44.810 216-101 Large Conference: Like, some offset, yep. 348 00:43:47.320 --> 00:43:50.939 216-101 Large Conference: Yeah, does that answer your question? Yeah, thanks. Okay. 349 00:43:51.610 --> 00:43:56.929 216-101 Large Conference: How do I… Thank you to speak. Okay. 350 00:43:57.840 --> 00:44:15.829 216-101 Large Conference: How are we? Oh, boy. Okay, I'll, I'll try and be quick. Okay, so, so basically, if you do this, like, geometric, loss, this is the result that you get. So here, gray is the base model. 351 00:44:15.940 --> 00:44:21.519 216-101 Large Conference: performance, orange is the task Loa performance, and blue is this geometry LoRa performance. 352 00:44:21.940 --> 00:44:24.459 216-101 Large Conference: So… It's like… 353 00:44:24.610 --> 00:44:32.849 216-101 Large Conference: I don't know, it's still peaky, but it, like, does a lot better than the task, Laura, especially, like, 354 00:44:33.240 --> 00:44:40.089 216-101 Large Conference: In… in where the sun starts getting bigger, like, around this… this bump, for example, and a little bit better in the middle. 355 00:44:41.090 --> 00:44:46.590 216-101 Large Conference: But it's, like, slightly worse, near distribution, where the Tesla does really well. 356 00:44:47.930 --> 00:44:49.760 216-101 Large Conference: Yeah. 357 00:44:51.490 --> 00:45:05.599 216-101 Large Conference: Okay, and then, yeah, so coming back to this, like, what's going on with this bump, and we… Do you have a solution for… No, so this is, this is basically the best that I've gotten. I, like… 358 00:45:05.750 --> 00:45:07.860 216-101 Large Conference: Yeah. 359 00:45:08.290 --> 00:45:13.390 216-101 Large Conference: So it seems like the geometry Laura, is… For earlier, yes. 360 00:45:14.320 --> 00:45:15.699 216-101 Large Conference: What do you mean? 361 00:45:15.950 --> 00:45:20.440 216-101 Large Conference: So, when you were doing task loadout, you found that earlier layer seemed to be… 362 00:45:20.730 --> 00:45:29.609 216-101 Large Conference: The one that had, like, most impact? Yes, when it, yeah, where the lowest one. The geometry that are also on earlier, same exact test. Yeah. 363 00:45:30.030 --> 00:45:36.690 216-101 Large Conference: So, like, if you fix the geometry LoRa, and then start training LoRa from the later layers, or tasks. 364 00:45:37.510 --> 00:45:42.259 216-101 Large Conference: Yeah, I've tried that, and it doesn't seem to work. 365 00:45:43.700 --> 00:45:49.720 216-101 Large Conference: Like, it's about the same as just training the, like, the geometry right at the early layer. 366 00:45:51.360 --> 00:45:55.920 216-101 Large Conference: Yeah. So, like, something that can learn to use the geometry you just 367 00:45:56.570 --> 00:46:11.890 216-101 Large Conference: introduced to the model, which it probably doesn't know how to use yet. Yeah, so… so I think… I think, like, one thing that's interesting is, like, what… what this… since this geometry lure is at an early layer, what it's doing is, like, what the… what the supervision process is doing is, like. 368 00:46:13.480 --> 00:46:20.870 216-101 Large Conference: we want… we want this, like, intermediate computation in the Fourier planes to be clean, and so whatever you can do to, like. 369 00:46:21.200 --> 00:46:30.200 216-101 Large Conference: clean that up, that's great. And so, it, like, has some extra effects that we… we don't really understand, and, like, you know, it's still peaky, but it's, like. 370 00:46:30.530 --> 00:46:36.290 216-101 Large Conference: you know, slightly better than what the task force is doing, at least in some regime. Yeah. 371 00:46:36.950 --> 00:46:40.949 216-101 Large Conference: Yeah. That is, 372 00:46:41.250 --> 00:46:48.850 216-101 Large Conference: I was basically trying to figure out the values you see, are they kind of like the blind spots of your… 373 00:46:48.990 --> 00:47:01.700 216-101 Large Conference: Yeah, so I'll point out a few, like, so there's one at 25, and there's another one at 37, which are exactly 12 apart. And this is, like, the model has a really hard time predicting January as an answer. 374 00:47:01.960 --> 00:47:16.940 216-101 Large Conference: Just in general. Yeah, and this is, this is also something, like, in Sheridan's paper, if you go look at, like, some of the steering results, when you try and steer for January, it, like, always predicts November, for some reason. Yeah. 375 00:47:17.000 --> 00:47:22.019 216-101 Large Conference: Like, so some of the peaks are, like, just weird things like that. 376 00:47:22.450 --> 00:47:29.460 216-101 Large Conference: I'm trying to look for prime numbers, like, 37 is a prime number, that's great. 25 is not. 377 00:47:29.800 --> 00:47:33.189 216-101 Large Conference: And also, like, 5 is also one of the period. 378 00:47:33.650 --> 00:47:35.440 216-101 Large Conference: Right. Yeah. 379 00:47:35.590 --> 00:47:52.390 216-101 Large Conference: Yeah, so I was just trying to find this pattern, and I could not really find any, and… Yeah, so, so let me, I guess let me come here. So this is… this is, like, one, one reason that we think the errors might be happening. So, 380 00:47:53.390 --> 00:48:06.410 216-101 Large Conference: I mentioned before that this, like, bump around, like, 60 to 80 is when, like, 10 and 12 line up again. So if, and I'll just… this is a specific neuron at, like, layer 24, 381 00:48:06.510 --> 00:48:08.659 216-101 Large Conference: That has this, this pattern. 382 00:48:08.760 --> 00:48:15.370 216-101 Large Conference: So it, it fires every 10, and then alternates, like, positive and negative. 383 00:48:15.660 --> 00:48:20.750 216-101 Large Conference: At least initially. And so what happens is, like, it fires at 6, and it fires at 16, 384 00:48:20.910 --> 00:48:22.980 216-101 Large Conference: If you can see that. 385 00:48:23.250 --> 00:48:39.249 216-101 Large Conference: And then 26 and whatever. And so, if you go and decode, like, the weight of this neuron, like, what it's writing, it basically is the difference between, like, 6 and 4. So when it's positive, it's promoting 6, and when it's negative, it's promoting 4, which is, like, June and April. 386 00:48:39.380 --> 00:48:51.049 216-101 Large Conference: And so, like, this… this works for, like, the first two periods to promote, like, the right month, for example. So, like, when it's 6, it's promoting June, and that's great. And when it's 16, it's promoting 387 00:48:51.290 --> 00:48:54.070 216-101 Large Conference: April, which is, like, 4 mod 12. 388 00:48:54.400 --> 00:49:01.750 216-101 Large Conference: So that's great, but then, like, it's off again. Like, so it can only, like, get the first two, and then, like, the middle cycle is wrong. 389 00:49:02.230 --> 00:49:14.750 216-101 Large Conference: And then the geometry Lura corrects this neuron, to, like, start working again around, like, 60… 66. So that's, like, one of the cases that the geometry Lura gets right, but that the base model, like, is wrong. 390 00:49:14.950 --> 00:49:30.009 216-101 Large Conference: And there, yeah, and there's a bunch of neurons like this that have, like, this weird funky pattern where, they, like, they promote 2 months, like, for positive and negative, but then it only works for, like, the first two cycles, and then it's off, and so, like, a lot of the errors in the middle. 391 00:49:31.240 --> 00:49:42.019 216-101 Large Conference: You can fix them by, like, artificially messing with the activations of these neurons, but, like, you can't achieve that activation pattern nor… like, via a weight edit. 392 00:49:42.720 --> 00:49:47.860 216-101 Large Conference: So type in artificial fix. Yeah. To make sure I'm understanding this correctly. 393 00:49:48.270 --> 00:50:07.460 216-101 Large Conference: it is really crazy that the blue line is above the orange line in the little graph, right? I think so. I think so. Like, there's no task loss at all, and, like, just a supervision on the geometry is, like, beating out what the task flora did. And your hypothesis is that… 394 00:50:08.370 --> 00:50:20.850 216-101 Large Conference: all of that extra boost comes from little things like that. Some of it. I, like, this is not a full account, but at least… at least some of that is, like, fixing these weird… these weird quirks, or at least, like, aligning them. Okay. Yeah. 395 00:50:21.750 --> 00:50:39.929 216-101 Large Conference: Yeah, I have another task, and we're already out of time, but, okay, it's more of a high-level question, because the motivation you gave in the beginning was that it doesn't know how to do mods well, or at least that's what I thought was the issue. But your solution is kind of different. 396 00:50:40.380 --> 00:50:42.029 216-101 Large Conference: Can you explain, like, what… 397 00:50:42.210 --> 00:50:48.940 216-101 Large Conference: how we connect that to, like… Early layers, instead of layers, and things like that. Do you have, Right? 398 00:50:49.740 --> 00:51:04.850 216-101 Large Conference: Okay, so, so, right, so one of the hypotheses was that, yeah, it doesn't know how to do mod 12, and so we thought that fine-tuning on the late layers would, like, fix that, because the sum was, like, getting computed correctly. But it turned out that what seems to work is, like. 399 00:51:04.960 --> 00:51:12.520 216-101 Large Conference: Fixing the computation of the sum also seems to fix some of the models, like, downstream computation that processes it. 400 00:51:12.790 --> 00:51:16.470 216-101 Large Conference: Because, 401 00:51:17.090 --> 00:51:29.899 216-101 Large Conference: Because if you, like… because the sum was also… it also turns out, like, kind of messy when you have big offsets as well. So you say there are a few moving parts here, you fix one of them, so you get an increase in the… Yeah, yeah. 402 00:51:29.920 --> 00:51:37.439 216-101 Large Conference: Yeah, so, like, I guess maybe you could think of it, like, some of the errors seem to be representational, and some of them maybe are, like. 403 00:51:37.520 --> 00:51:48.600 216-101 Large Conference: mechanical of, like, maybe there's, like, missing neurons that just, like, don't process a certain period, or, like, don't write to certain months, like, maybe there's, like, not enough January writing neurons, and so that's why, like. 404 00:51:49.060 --> 00:51:52.569 216-101 Large Conference: doesn't write to January or something, I don't know, you know. Yeah. 405 00:51:53.110 --> 00:52:01.780 216-101 Large Conference: Okay, yes, trying to be quick. I have another… so basically, like, the two takeaways, 406 00:52:02.030 --> 00:52:04.280 216-101 Large Conference: For this section is, like. 407 00:52:04.490 --> 00:52:14.449 216-101 Large Conference: Task-level supervision, like, can fix errors in this training distribution, and, like, and part of the way that it does it is by, like, you know, strengthening 408 00:52:14.450 --> 00:52:28.509 216-101 Large Conference: strengthening this existing geometry. And also that, like, we can use sort of a geometric prior for supervision that can either, like, match or sometimes be better than the supervision that uses a task loss, which I think is pretty cool. 409 00:52:29.330 --> 00:52:38.340 216-101 Large Conference: Okay, so then, we also applied the same kind of procedure to another task, so if you're familiar, this is a paper that came out from Anthropic. 410 00:52:38.510 --> 00:52:47.969 216-101 Large Conference: Last year, about… they found this, manifold, this little helical manifold for how Claude, predicts line breaks. 411 00:52:48.160 --> 00:52:55.269 216-101 Large Conference: Like, in structured text, where you have, Fixed length, lines. 412 00:52:55.670 --> 00:53:09.059 216-101 Large Conference: And so we, like… so what the line break task looks like is you have some text, and you can sort of arbitrarily, like, insert new lines, at different widths. And so, for example, like, this is a… 413 00:53:09.290 --> 00:53:12.470 216-101 Large Conference: one of the paragraphs from, Winnie the Pooh. 414 00:53:12.570 --> 00:53:29.190 216-101 Large Conference: But, like, just at different… at different lengths. And so, what… what we measure and what they measure in the paper is, like, how good the model is at predicting the line break, after a few… after seeing a few examples of, like, oh, I should be doing this at width 40 or width 25, 415 00:53:29.340 --> 00:53:32.079 216-101 Large Conference: Like, how good is it at recognizing that pattern? 416 00:53:32.840 --> 00:53:49.569 216-101 Large Conference: And so we reproduced… so they did that on Claude, and so we're reproducing this, in Llama. And so on the left, is, like, the base model's accuracy on this task, so it's actually, like, really good at this task. But you can… and so here, what this is showing is the… 417 00:53:49.630 --> 00:53:57.609 216-101 Large Conference: on the y-axis is, like, the number of in-context examples, so, like, how far deep into the text are you? Like, how many lines of 418 00:53:57.690 --> 00:54:06.579 216-101 Large Conference: width 25 have you seen, for example? And on the x-axis is the width. So this goes… sorry that the lines are missing, but, like, this goes from 0 to 300. 419 00:54:06.770 --> 00:54:21.229 216-101 Large Conference: So, like, the model's bad when it's only seeing, like… when it's seeing no lines, it doesn't really know to predict a new line. When it's seeing one line, it's bad, and when it sees, like, maybe 4 or 5 lines, then it's, like, good at, you know, predicting the line break. 420 00:54:22.150 --> 00:54:34.470 216-101 Large Conference: And in this case, like, in Claude, they sort of showed that this geometry looks like a helix. We weren't sure what it looked like in Llama, so we're, like, using two different ways to view it here. So, like, one is, like. 421 00:54:34.630 --> 00:54:41.600 216-101 Large Conference: Maybe it's also, like, a Fourier plane like we had before, but… and then there's also this, like, maybe it looks like a helix on the right. 422 00:54:41.800 --> 00:54:43.169 216-101 Large Conference: And again. 423 00:54:43.320 --> 00:54:58.299 216-101 Large Conference: So, why I have two plots, top and bottom, the top is, if you cut off the data, for widths up to 150, and then the bottom is if you plot all the data for widths up to 300. 424 00:54:58.450 --> 00:55:03.999 216-101 Large Conference: And so what you see, like, in the Fourier plane, basically what you see is there's sort of this, like, aliasing effect. 425 00:55:04.120 --> 00:55:10.369 216-101 Large Conference: So that, like, the R-squared drops. I don't know if you can see the R-squared values, but… oh yeah, the text is terrible here. 426 00:55:10.520 --> 00:55:20.499 216-101 Large Conference: But, so, like, in the top, the R-squared's, like, 36 for the circle, and then in the bottom, it's, like, 0.04. So there's sort of this aliasing, and, like, a circle doesn't really match the geometry anymore. 427 00:55:20.720 --> 00:55:28.310 216-101 Large Conference: And I think the… The helixes are… Let's see… 428 00:55:28.450 --> 00:55:31.940 216-101 Large Conference: Yeah, like, you also see a drop. 429 00:55:32.130 --> 00:55:38.949 216-101 Large Conference: Just because the… the data, like, spreads out more, but, like, the helix, like, better fits the data under the… 430 00:55:39.130 --> 00:55:40.999 216-101 Large Conference: The long line width. 431 00:55:41.830 --> 00:55:48.400 216-101 Large Conference: We'll skip that. We're gonna try the same thing, see if it… see if it works. 432 00:55:48.620 --> 00:55:54.910 216-101 Large Conference: And, and yeah, these are the results. So… 433 00:55:55.270 --> 00:56:07.460 216-101 Large Conference: On the top row is, like, the raw accuracy of, like, predicting new lines in this setting. And on the bottom, I've taken, like, accuracy diffs, so you can sort of easily compare which one is… are good. 434 00:56:07.460 --> 00:56:22.460 216-101 Large Conference: So, like, the bottom left shows, like, the task flora basically improves the model really well, like, in the early, early shot regime. So, like, in, like, one and two shots, at long… at big line widths, it makes the model really good. 435 00:56:22.580 --> 00:56:28.160 216-101 Large Conference: If you look at the geometry, Laura, in the middle bottom panel here, it, like. 436 00:56:28.300 --> 00:56:38.419 216-101 Large Conference: Fixes most of the model's errors in, like, the long line width regime, but it doesn't really fix and slightly hurts the model's performance. 437 00:56:39.170 --> 00:56:42.949 216-101 Large Conference: in, like, the… The low, line width? 438 00:56:43.110 --> 00:56:49.820 216-101 Large Conference: setting. And then if you compare, like, the geometry versus the task flora here, the geometry LoRa… 439 00:56:50.050 --> 00:56:54.420 216-101 Large Conference: It's good where the range is green, so it's like, 440 00:56:56.680 --> 00:57:05.270 216-101 Large Conference: In between, like, 100 and 250-ish, and, like, when the… when you're, like, deep into the in-context examples. 441 00:57:05.660 --> 00:57:12.789 216-101 Large Conference: Whereas, like, the… what we saw before, the task score is really good at the short… the short, prediction. 442 00:57:14.090 --> 00:57:29.850 216-101 Large Conference: And… and yeah, we can sort of look at… these are, like, plotting the mean, the means of the manifolds for the different settings, so blue is the geometry LoRa, and these red and orange are the task Luras that have been trained at different lengths. 443 00:57:30.400 --> 00:57:38.329 216-101 Large Conference: On the far right is, again, like, the comparing this… the circle versus the helix R-squared, 444 00:57:38.540 --> 00:57:42.750 216-101 Large Conference: Showing that, like, the helix is sort of a better fit for what this data looks like. 445 00:57:43.110 --> 00:57:48.500 216-101 Large Conference: Yeah, just… Yeah, different view of that. 446 00:57:49.580 --> 00:57:55.320 216-101 Large Conference: Yeah, and I made some little GIFs for these. But, yeah, so, 447 00:57:58.720 --> 00:58:04.450 216-101 Large Conference: yeah, I'll… I'll skip. So basically, like, this is another setting where it seems like 448 00:58:04.610 --> 00:58:08.849 216-101 Large Conference: Pass revision seems to, be, like, sharpening 449 00:58:08.970 --> 00:58:13.979 216-101 Large Conference: existing manifolds. Like, in this case, it's sort of this, like, helix structure, and when you look at 450 00:58:14.080 --> 00:58:17.129 216-101 Large Conference: Like, how the structure changes. 451 00:58:17.140 --> 00:58:34.870 216-101 Large Conference: under the task force, like, it's getting, stronger, or, like, more aligned with, sort of, the geometry that we expect, and we can use that same idea and, like, train for the geometry exactly, and, like, you know, like, get some gains. It's not, like, perfect, like, the task quora and the 452 00:58:34.900 --> 00:58:36.530 216-101 Large Conference: Geometry Laura are, like. 453 00:58:36.680 --> 00:58:41.920 216-101 Large Conference: Similar here, but I think it's surprising that you can get these gains without any task loss at all. 454 00:58:44.100 --> 00:58:58.130 216-101 Large Conference: Yeah, and so I think that's… that's sort of the takeaway. And then this third point here is, like, this… this is… I'm not, like, advocating that this is a method that should, like, replace, fine-tuning. I think, like, it's really challenging to sort of pick 455 00:58:58.280 --> 00:59:01.890 216-101 Large Conference: Like, good target geometries, and, like, you, 456 00:59:02.550 --> 00:59:10.360 216-101 Large Conference: like, this is a really hard problem. Like, this work is basically based on two papers that spent a lot of time trying to figure out, like. 457 00:59:10.480 --> 00:59:23.640 216-101 Large Conference: what does this manifold look like? How does the model solve this task? It sort of just builds on that, but, like, the first piece that it builds on is also a lot of work, and so this is not necessarily, like, you should do fine-tuning in this way, but just, like. 458 00:59:23.820 --> 00:59:26.009 216-101 Large Conference: I think it's surprising that, like. 459 00:59:26.170 --> 00:59:30.559 216-101 Large Conference: You can… you can target geometry. Maybe it's, like, motivation that, like. 460 00:59:31.210 --> 00:59:35.909 216-101 Large Conference: reverse engineering things can be worth it. 461 00:59:36.110 --> 00:59:41.479 216-101 Large Conference: But yeah, so anyway, that's it. Thanks. So, so, oh, sorry. 462 00:59:46.780 --> 00:59:48.910 216-101 Large Conference: I don't have to take the first question. Questions? 463 00:59:49.990 --> 00:59:52.240 216-101 Large Conference: What does it… 464 00:59:52.590 --> 00:59:57.030 216-101 Large Conference: I mean, I know this is probably not, like, maybe it's out of scope, but, like… 465 00:59:57.160 --> 01:00:00.769 216-101 Large Conference: What if you use both task and geometry? Does it, like. 466 01:00:01.260 --> 01:00:09.760 216-101 Large Conference: Is it useful to have both of them? Yeah, so, I tried it… I… I don't remember… so it… it does, like. 467 01:00:09.940 --> 01:00:14.729 216-101 Large Conference: Better than either one individually, basically, is the answer. 468 01:00:14.860 --> 01:00:20.330 216-101 Large Conference: But… but it takes some, like, you know. 469 01:00:20.770 --> 01:00:24.750 216-101 Large Conference: Sweeping over hyperparameters to, like, get that to work nicely together. 470 01:00:25.260 --> 01:00:26.030 216-101 Large Conference: Yeah. 471 01:00:27.240 --> 01:00:37.260 216-101 Large Conference: Yeah. Yeah, in the addition setup, is the geometrics provision better than task indistribution, or also out of distribution? It… so, it's like… 472 01:00:38.160 --> 01:00:46.100 216-101 Large Conference: Mostly better out of distribution, so, like, for larger offsets, and slightly worse in distribution. Yeah. 473 01:00:47.820 --> 01:00:49.060 216-101 Large Conference: Yeah, change. 474 01:00:49.230 --> 01:00:55.429 216-101 Large Conference: I think there's a way to sort of… it's very hard, but training Laura to encourage Bob's help to emerge. 475 01:00:55.620 --> 01:01:02.779 216-101 Large Conference: I tried, I couldn't get it to work. Like, so Mod12 does show up. 476 01:01:02.960 --> 01:01:07.949 216-101 Large Conference: later, when, like, the model is actually producing the… so, like, in the late layers, if you go 477 01:01:08.190 --> 01:01:20.909 216-101 Large Conference: I can find a plot, but, like, basically in the late layers, like, mod 12 does start showing up, like, when the model is outputting the month, like, that… that does exist, but… 478 01:01:21.080 --> 01:01:26.630 216-101 Large Conference: For it to be, like, used, In the addition computation, I… I don't… 479 01:01:27.440 --> 01:01:39.630 216-101 Large Conference: haven't gotten it to work. Like, maybe there's a way to get it to work, yeah. You mean, like, taking, like, number mod 12 at the end? Or, like, getting… doing addition in mod… directly in mod 12? 480 01:01:40.260 --> 01:01:42.920 216-101 Large Conference: Okay. 481 01:01:44.160 --> 01:01:53.959 216-101 Large Conference: Yeah, yeah, I haven't gotten that to work, but, like, mod 12 does show up when the model is actually, like, outputting the month, like, in very late layers. 482 01:01:55.440 --> 01:01:56.160 216-101 Large Conference: Yeah. 483 01:01:58.350 --> 01:02:00.480 216-101 Large Conference: So, in both of these settings. 484 01:02:00.600 --> 01:02:04.349 216-101 Large Conference: You're imposing the loss at a given late layer. 485 01:02:04.740 --> 01:02:10.359 216-101 Large Conference: Is it… is that layer chosen… is that a layer where you have… that… 486 01:02:10.510 --> 01:02:16.160 216-101 Large Conference: You know, there's causal effects if you… Were to, like… 487 01:02:17.520 --> 01:02:21.799 216-101 Large Conference: So this, I mean, so I feel like… So, actually, for this, for this, like, the… 488 01:02:22.000 --> 01:02:28.720 216-101 Large Conference: This case, the manifold's at layer 5, so the supervision happens at layer 5. Right. 489 01:02:28.890 --> 01:02:36.559 216-101 Large Conference: Right, so, but at whatever, whatever, so… but that manifold was identified because it has causal effects there. 490 01:02:37.050 --> 01:02:46.200 216-101 Large Conference: Is that right? And in the clock situation, you're imposing laws at layer 18, because that's where you have causal effects there. And so you have these causal effects. 491 01:02:47.360 --> 01:02:49.390 216-101 Large Conference: activation base. 492 01:02:51.070 --> 01:02:55.240 216-101 Large Conference: To the… And… 493 01:02:55.350 --> 01:02:59.589 216-101 Large Conference: And so I guess I… I have this question, because… so you're… so this is a little bit of, 494 01:03:01.400 --> 01:03:04.140 216-101 Large Conference: If you're asking this question about weights. 495 01:03:05.060 --> 01:03:08.290 216-101 Large Conference: And you're driving them to match some activations. 496 01:03:08.690 --> 01:03:14.360 216-101 Large Conference: Did… and… and do the… Do the activations themselves 497 01:03:15.420 --> 01:03:18.330 216-101 Large Conference: Work well? Like, do they work better than… 498 01:03:18.770 --> 01:03:25.010 216-101 Large Conference: Like, still your, your, your target activations, one that you're using as a… Training supervision. 499 01:03:25.120 --> 01:03:26.170 216-101 Large Conference: Right? 500 01:03:26.490 --> 01:03:30.600 216-101 Large Conference: When… if instead of using them for training, you use them 501 01:03:31.230 --> 01:03:34.049 216-101 Large Conference: For intervention, just to run an inference. 502 01:03:34.570 --> 01:03:45.090 216-101 Large Conference: Do you get, like, high… Tax performance, when using them that way? No. Some, sometimes. 503 01:03:45.370 --> 01:03:48.120 216-101 Large Conference: So, yeah, so, 504 01:03:48.560 --> 01:03:54.899 216-101 Large Conference: I have an experiment where, like, if you… if you patch in… so we have, like, this Oracle LoRa, for example. 505 01:03:55.070 --> 01:03:58.380 216-101 Large Conference: How do I scroll here? 506 01:03:58.690 --> 01:04:01.950 216-101 Large Conference: Like, this guy here, that, like, was trained on the whole… 507 01:04:02.260 --> 01:04:14.279 216-101 Large Conference: test set, basically. And, like, if you go look at its geometry, like, it has really nice circles. But if you go… if you, like, go force the base model's circles to look like this, is that what you're saying? 508 01:04:14.390 --> 01:04:18.779 216-101 Large Conference: Yes, or, or whatever you're… so, like, you have even… 509 01:04:19.960 --> 01:04:24.350 216-101 Large Conference: More difficult ground truth, but you're… you're getting your circles from the regular addition tasks. 510 01:04:24.890 --> 01:04:27.550 216-101 Large Conference: So, if you… if you, like, force… 511 01:04:27.790 --> 01:04:33.069 216-101 Large Conference: that in, if you force those representations into the, you know, the Fourier subspace. 512 01:04:33.790 --> 01:04:37.030 216-101 Large Conference: During the performance of, like, the month… 513 01:04:37.280 --> 01:04:41.069 216-101 Large Conference: addition task, or something like that. Just as, just as patching. 514 01:04:41.220 --> 01:04:48.329 216-101 Large Conference: Not even as fine-tuning. Yeah. You know, can you get good task performance out of it? No. 515 01:04:48.580 --> 01:04:53.449 216-101 Large Conference: Right, and then… and then, so is that… is that, like, a theoretical ceiling? Like, like, you can't… 516 01:04:53.800 --> 01:05:00.960 216-101 Large Conference: like, because I feel like there's two things, like, there's one is… I just… 517 01:05:01.370 --> 01:05:09.859 216-101 Large Conference: Like, if you could achieve, like, the ideal, like, geometry target, Would that improve things? 518 01:05:10.220 --> 01:05:12.210 216-101 Large Conference: Yeah, even if it was, like, uncomputable. 519 01:05:12.610 --> 01:05:15.790 216-101 Large Conference: Right? By your unlearnable, by your LoRa. Correct. 520 01:05:15.900 --> 01:05:20.959 216-101 Large Conference: Right? And then the second is, oh, well, you know, is there… is that a reasonable… 521 01:05:21.320 --> 01:05:36.259 216-101 Large Conference: training objective, in practice, and so… Yeah. So, like, does that, does that give you, like, a ceiling for, like, how good you would expect the performance to be? Like, if you added that line to the pot, you know, does your, does your LoRa thing basically achieve 522 01:05:36.330 --> 01:05:52.709 216-101 Large Conference: Yeah, so, so, like, if you compare patching under the, like, the Oracle Task LoRa and the Geometry LoRa into the base model, you, you basically get out as good as you can get just from these Fourier planes. So, so, yeah, part of, part of, like. 523 01:05:53.100 --> 01:05:57.239 216-101 Large Conference: There… there must be some other effects that, like. 524 01:05:57.490 --> 01:06:01.620 216-101 Large Conference: The task for, is, like, using to get good performance. 525 01:06:01.830 --> 01:06:07.549 216-101 Large Conference: That, like, isn't, captured by… by this… If that makes sense. 526 01:06:07.930 --> 01:06:12.319 216-101 Large Conference: How, though? Because your task order, I did not see any task. 527 01:06:12.540 --> 01:06:30.779 216-101 Large Conference: Oh, no, the task clearer saw the task. Oh, okay, yeah, yeah. So, like, the task clearer, like, so basically the geometry Laura can have this, basically, as clean of a geometry as the task clearer. 528 01:06:31.020 --> 01:06:36.889 216-101 Large Conference: But, like, patching either of them into the base model only Gets you so much gain. 529 01:06:37.080 --> 01:06:52.969 216-101 Large Conference: Yeah, so, so… right, so the way that I understand it is, like, the process by which, the model, like, computes, like, ends up computing these Fourier planes. 530 01:06:53.070 --> 01:07:00.500 216-101 Large Conference: like… Seems to be useful elsewhere, like, there's some helpful byproduct of like… 531 01:07:00.850 --> 01:07:08.919 216-101 Large Conference: producing representations that have this clean geometry, but that's something that I don't under, like, don't understand exactly, like, what is that other thing? 532 01:07:09.390 --> 01:07:11.590 216-101 Large Conference: To be looking for, if that makes sense. 533 01:07:14.240 --> 01:07:27.150 216-101 Large Conference: So this geometry is not the only, like, bottleneck in the model. Right. Yeah, yeah. I know it's not, because, like, you can patch… you patch in, and it doesn't… doesn't fix everything that this training does. 534 01:07:27.560 --> 01:07:36.790 216-101 Large Conference: the geometry, and then also the machinery that can read the geometry. Yeah. How this machinery is being built, because it does not see any cross blocks. 535 01:07:37.370 --> 01:07:47.040 216-101 Large Conference: Yeah, I don't know… It already has some… Yeah, what? It already has some of that. But I guess my… I guess my question is, is there… 536 01:07:49.160 --> 01:07:51.939 216-101 Large Conference: So, if you set your Fourier 537 01:07:52.510 --> 01:07:58.010 216-101 Large Conference: hypothesis aside, or you… Sure. …or you put it, you know, more, you know. 538 01:07:58.890 --> 01:08:05.310 216-101 Large Conference: You put it more leniently. Is there, like, a more aggressive… Activation patch. 539 01:08:06.210 --> 01:08:10.770 216-101 Large Conference: That would achieve higher… task performance. 540 01:08:11.200 --> 01:08:15.020 216-101 Large Conference: So, like, I guess, like, a thing that… 541 01:08:15.430 --> 01:08:19.770 216-101 Large Conference: You know, we've seen a lot in, you know, different pieces of research is 542 01:08:20.100 --> 01:08:24.940 216-101 Large Conference: Oh, increase the number of dimensions that you're patching over. 543 01:08:25.060 --> 01:08:36.829 216-101 Large Conference: Right? And so you're… by… by virtue of finding these, like, planes, these two-dimensional probes, you're saying, oh, you know what? We're just going to patch over two dimensions at a time, man, two dimensions for Fourier frequency. 544 01:08:37.069 --> 01:08:46.059 216-101 Large Conference: 16 in total. This is a tiny patch. Yeah, yeah, yeah. Right? And… and we have all these situations where 545 01:08:46.520 --> 01:09:01.880 216-101 Large Conference: you have models that are representing really small things that you think is, like, this is a one-dimensional concept, this is a two-dimensional concept. But in previous works, we end up patching an 80-dimensional subspace or something to bring over what feels like a two-dimensional concept, right? 546 01:09:02.120 --> 01:09:04.730 216-101 Large Conference: And so, I guess the question is. 547 01:09:05.050 --> 01:09:11.830 216-101 Large Conference: Is there, like, a relaxed way of looking at your FOIA information content? 548 01:09:12.340 --> 01:09:27.379 216-101 Large Conference: That would give you, like, a higher dimensional subspace. Yes, so I think, I mean, like, I guess, like, the generalized version of this method, like, you don't, like, just has some… some projection W, right? So you could… you could learn the projection W, or you could pick some… 549 01:09:27.630 --> 01:09:35.039 216-101 Large Conference: like, so in this case, we're just, like, using the Fourier planes, because that's sort of the hypothesis, but yeah, you could… you could pick some other projections 550 01:09:35.160 --> 01:09:43.830 216-101 Large Conference: That maybe is bigger. Let's see… maybe it's worth doing the, you know, like, I'm not sure what the… 551 01:09:44.899 --> 01:09:46.839 216-101 Large Conference: How the supervision comes about. 552 01:09:47.029 --> 01:09:54.109 216-101 Large Conference: When you, when you have some bigger, Subspace. 553 01:09:54.630 --> 01:09:57.790 216-101 Large Conference: But I think it's worth thinking about, because… 554 01:09:57.920 --> 01:10:05.919 216-101 Large Conference: Because, yeah, I think there are a lot of, like, you know, outside effects that aren't captured by these 16 dimensions, that we're seeing. 555 01:10:07.160 --> 01:10:14.319 216-101 Large Conference: Yeah, it's weird, right? I don't know if I… like, the linear algebra would suggest it shouldn't work this way, right? But, like, if you project out. 556 01:10:14.540 --> 01:10:15.720 216-101 Large Conference: these plans. 557 01:10:15.840 --> 01:10:23.480 216-101 Large Conference: then there should be, like, no more signal that is, like, period 5, it should be all gone, right? But… but is there some sense where, oh no, it's… 558 01:10:23.620 --> 01:10:28.329 216-101 Large Conference: It's still in there. Somehow, there's some other, more subspace. 559 01:10:28.440 --> 01:10:29.500 216-101 Large Conference: Where it is. 560 01:10:29.870 --> 01:10:31.750 216-101 Large Conference: Like, I'm not… 561 01:10:32.170 --> 01:10:39.360 216-101 Large Conference: Really short. Or… or are you restricted here because you're only choosing 4 and you need the higher harmonics? 562 01:10:39.580 --> 01:10:47.170 216-101 Large Conference: If you add… Couldn't there be, like, two-dimensional subspaces that are both… 563 01:10:47.270 --> 01:10:54.530 216-101 Large Conference: But then… But then if you rotate, like, if you're using their algebra makes it rotate properly, then… 564 01:10:55.000 --> 01:11:00.880 216-101 Large Conference: you'll… you'll end up with, like, they're both in there. You just add them, it's like… 565 01:11:01.090 --> 01:11:07.960 216-101 Large Conference: It's like saying X plus Y. Oh, yeah, that's just… that's another line. Wait, Derek, when you're… when you're patching, are you patching, like, the… 566 01:11:08.130 --> 01:11:16.540 216-101 Large Conference: And in the residual stream, or in the input to the MLP? Yes, both. We've tried both. Oh, and both, yeah, either. 567 01:11:16.650 --> 01:11:19.150 216-101 Large Conference: They're the same? Pretty much, yeah. 568 01:11:19.850 --> 01:11:26.730 216-101 Large Conference: I kind of thought maybe there's other neurons outside of layer 18 that are also doing… Doing addition. 569 01:11:27.050 --> 01:11:29.570 216-101 Large Conference: Yeah, I think that's true. 570 01:11:30.140 --> 01:11:44.999 216-101 Large Conference: But, I don't know. Yeah, so, like, it does… like, you don't have to do the supervision at layer 18. I've got a bunch of other experiments where, like, you can pick… you can pick, like, a later layer to have, like, supervision on, and basically get similar results. 571 01:11:45.190 --> 01:11:48.380 216-101 Large Conference: But, like… 572 01:11:49.450 --> 01:11:56.050 216-101 Large Conference: But I think it's partly because, like, you know, for example, if you pick, like, layer 22 as your supervision layer. 573 01:11:56.290 --> 01:12:03.459 216-101 Large Conference: Then, like, in order to have clean circles at layer 22, like, you'd probably have to have clean circles at layer 18. 574 01:12:03.620 --> 01:12:07.650 216-101 Large Conference: So it's like… You know. 575 01:12:08.450 --> 01:12:09.689 216-101 Large Conference: Something like that. 576 01:12:09.800 --> 01:12:12.280 216-101 Large Conference: Eric, I have a few questions. Yeah. 577 01:12:12.610 --> 01:12:15.250 216-101 Large Conference: First, regarding your LoRa setup. 578 01:12:15.380 --> 01:12:20.400 216-101 Large Conference: Most of the plots you're showing, is it one layer LoRa at a time? Yeah. 579 01:12:20.860 --> 01:12:25.220 216-101 Large Conference: So, like, there is nothing where you did a bunch of layers together. 580 01:12:25.400 --> 01:12:29.529 216-101 Large Conference: For… for the month setup? No. Yeah. 581 01:12:29.940 --> 01:12:33.690 216-101 Large Conference: So, and second question is, when you're doing geometry, Laura. 582 01:12:34.110 --> 01:12:39.440 216-101 Large Conference: It's just you, like, forcing one layer's representations to, like. 583 01:12:39.960 --> 01:12:54.550 216-101 Large Conference: distribute in a different way, but there is no loss whatsoever to… for the text being coherent, or… Correct. Correct. So I think the setup is really crazy for it to work. Like, I'm surprised why it worked in the first place. 584 01:12:54.860 --> 01:12:56.900 216-101 Large Conference: Later. 585 01:12:57.100 --> 01:13:03.159 216-101 Large Conference: And second is it's also very surprising that it's, like, these two dimensions, that is enough for the model to… 586 01:13:03.280 --> 01:13:10.379 216-101 Large Conference: just be better at the task. I understand that there's a lot of intervention that said, the model I shared and said, like. 587 01:13:10.590 --> 01:13:15.919 216-101 Large Conference: West Vietnamese literature that says there is representation, there's, like, some structure. 588 01:13:16.060 --> 01:13:18.119 216-101 Large Conference: But the fact that, like, you can… 589 01:13:18.360 --> 01:13:27.210 216-101 Large Conference: Press around the optimum plate of the representations, without having any loss or any objective for the model to do more at class. 590 01:13:27.930 --> 01:13:30.589 216-101 Large Conference: The other, I think, is, like, very subtle, and… 591 01:13:31.160 --> 01:13:35.080 216-101 Large Conference: Yeah, it's, like, almost hard to believe. 592 01:13:35.250 --> 01:13:42.230 216-101 Large Conference: Like, because, like, when we are training loaders for, like, simple text-to-image tasks, so easy to mess up the model. 593 01:13:42.820 --> 01:13:53.680 216-101 Large Conference: Yeah, oh yeah, so, yeah, it… because the floor is, like, Rank 1, I've got a bunch of, in the appendix somewhere, like, a bunch of, like, how does this affect, like, off-target? 594 01:13:53.830 --> 01:13:57.900 216-101 Large Conference: You know, tasks. And it's, like, pretty minimal changes to the model. 595 01:14:01.220 --> 01:14:11.060 216-101 Large Conference: tick bar, mainly generalization to very related, but slightly different task. Or maybe also, like, how… 596 01:14:11.690 --> 01:14:31.360 216-101 Large Conference: how much this geometry is super template-specific? Maybe you ask the same question, but it's a different paraphrasing, and does it feel generalized? I should check the paraphrases, I haven't done that, but I have looked at, like, all of the other cyclic tasks, like the weekdays and whatever, and, like, they also… 597 01:14:31.360 --> 01:14:37.800 216-101 Large Conference: They don't improve as much as the month's one, because that's, like, the one we trained it for, but they also improve, like, yeah. 598 01:14:38.090 --> 01:14:43.229 216-101 Large Conference: Eric, I think you should push for iClair, but I also think you should not give up after 599 01:14:43.580 --> 01:14:45.550 216-101 Large Conference: Just showing geometric photograph. 600 01:14:45.780 --> 01:14:51.399 216-101 Large Conference: I would, like… it'll be really satisfying if you can show that, sure, I changed the geometry, but also, like. 601 01:14:51.660 --> 01:14:59.469 216-101 Large Conference: There's a very small tweak to the network that we should do for it to learn how to use that data, so that, like, you have 602 01:14:59.650 --> 01:15:02.239 216-101 Large Conference: Your accuracy goes higher on the… 603 01:15:03.500 --> 01:15:07.029 216-101 Large Conference: The held-out modulation… mods you have. 604 01:15:09.890 --> 01:15:14.900 216-101 Large Conference: Okay, we should chat more, I don't know if I understand, but… What approach would you suggest? 605 01:15:15.050 --> 01:15:19.969 216-101 Large Conference: So, like, one question is, is, like, is there a reason that you're choosing these four 606 01:15:20.380 --> 01:15:39.169 216-101 Large Conference: harmonics here, and you didn't include 50, or 32, or some others? So, I mean, these four are the ones that, like, have the sharpest increase. Yep, but did you find, like, throwing in other… I've tried other ones, and, like, if you throw in… if you throw in 50, it doesn't… doesn't do that much, partly because the training data range… 607 01:15:39.350 --> 01:15:41.629 216-101 Large Conference: that I'm using is only up to 24. 608 01:15:41.990 --> 01:15:46.080 216-101 Large Conference: Oh, I see. But with, like, I see. And, and, and… 609 01:15:46.660 --> 01:15:55.289 216-101 Large Conference: And is that because… why… like, why did you not include the train data range prior? Oh, I… I could… I just… I just, you know… 610 01:15:55.920 --> 01:16:01.319 216-101 Large Conference: saw the same pattern for all of them and thought, I'll just pick one for a point of comparison. I see. 611 01:16:01.710 --> 01:16:03.509 216-101 Large Conference: So it was 24. 612 01:16:03.680 --> 01:16:09.909 216-101 Large Conference: I see. Like, I kind of imagined… because, I mean, it's amazing just to not have the task blocks. 613 01:16:10.380 --> 01:16:15.009 216-101 Large Conference: And instead of holding your hand behind your back. 614 01:16:15.460 --> 01:16:29.100 216-101 Large Conference: you know, with respect to other aspects of the training, you could give her the benefit of, like, lots of data, lots of harmonics, and, you know, lots of other things, just to, like… 615 01:16:29.370 --> 01:16:32.219 216-101 Large Conference: Totally give it the ability to match the geometry. 616 01:16:32.570 --> 01:16:38.170 216-101 Large Conference: And then… right, because, I mean, because I think that's… That's the… 617 01:16:38.580 --> 01:16:41.990 216-101 Large Conference: Because that's not the point, right? The point is, oh, you don't have a task anymore. 618 01:16:42.720 --> 01:16:43.750 216-101 Large Conference: Right? 619 01:16:43.950 --> 01:16:47.859 216-101 Large Conference: And, and, like, with that improving, it's kind of… 620 01:16:47.980 --> 01:16:53.280 216-101 Large Conference: Good question. I think that it's… it's amazing that, like Rohit said, it's amazing that you're higher 621 01:16:53.690 --> 01:17:04.739 216-101 Large Conference: than the test lost performance. But then… but you're also not sort of pegged at 100%, so it's not as impressive looking as it could be. Yeah, yeah. Right? Yeah, I don't know. Yeah. 622 01:17:05.100 --> 01:17:08.200 216-101 Large Conference: Yeah, I agree. Yeah. Yep. 623 01:17:08.720 --> 01:17:09.480 216-101 Large Conference: Yeah. 624 01:17:10.200 --> 01:17:28.180 216-101 Large Conference: So you showed some adjustment you need to do? I already forgot the name of the adjustment. Progressive alignment. Yeah. So, you need to do that for every mod X task you care about, right? 625 01:17:30.790 --> 01:17:41.000 216-101 Large Conference: like, maybe a year… So, I do it. If you drop it, you drop, like, a couple percentage points. So it's like, you really need to… 626 01:17:42.240 --> 01:17:55.899 216-101 Large Conference: Yeah, but if you just drop it, then we use the better generalization to just say, you train on moms. Oh, to the other tasks? I don't know, maybe you need to do this weird adjustment on… For every, for every… it's a quick task. 627 01:17:57.040 --> 01:17:57.960 216-101 Large Conference: Could be. 628 01:18:01.430 --> 01:18:09.920 216-101 Large Conference: So I think it's a more forward experiment, but I think it's still interesting to see, like, what would happen if you really crank up the size of the data. 629 01:18:10.300 --> 01:18:15.969 216-101 Large Conference: So, say, for example, right now, your range is, like, I think, maybe 100, something. 630 01:18:16.080 --> 01:18:26.860 216-101 Large Conference: But if you really scale the data size to maybe, like, 10K or something, like, I'm curious what… there might be two phenomenon. One is, like, maybe the model group is able to generalize 631 01:18:27.090 --> 01:18:39.120 216-101 Large Conference: and figure out new structures when there are a lot of data. Or second, when there exists, like, certain model bias to a point where your training loss isn't zero, your training accuracy isn't 100, there's just certain… 632 01:18:39.230 --> 01:18:44.430 216-101 Large Conference: Yeah, like, the model couldn't… fit the trending data. I think right now, it's sort of showing 633 01:18:44.660 --> 01:18:51.729 216-101 Large Conference: the test log basically goes to zero, test actually goes to 100. That means within the current computational structure, the model is able to cheat. 634 01:18:52.250 --> 01:18:55.970 216-101 Large Conference: Again, like, 100% hex cost. That's why it doesn't journalize. 635 01:18:56.360 --> 01:19:07.909 216-101 Large Conference: Yeah, so I've… I mean, I'm showing here, like, up to 100, but, like, I've trained task floors up to, like, 1,000, and you get the same pattern. Like, it only ever… 636 01:19:08.410 --> 01:19:11.680 216-101 Large Conference: You know, it generalizes, like, an extra cycle, and that's about it. 637 01:19:12.130 --> 01:19:18.280 216-101 Large Conference: But I haven't done the geometry lure up to 1000, so I… Yeah. 638 01:19:20.070 --> 01:19:25.930 216-101 Large Conference: Yeah. And why does it not generalize? Why does the task… the… the… under the task, Laura? 639 01:19:26.080 --> 01:19:26.990 216-101 Large Conference: Yo. 640 01:19:27.310 --> 01:19:30.779 216-101 Large Conference: I don't know. 641 01:19:30.910 --> 01:19:34.040 216-101 Large Conference: Yeah, I don't have a good explanation for why. 642 01:19:34.370 --> 01:19:48.239 216-101 Large Conference: like, is it, like, the error within the four-year circles is higher for larger subs, or that maybe it's, like, a helix, and so there's some extra Z dimension that is out of distribution for whatever downstream thing? 643 01:19:54.000 --> 01:19:58.870 216-101 Large Conference: Could be. I mean, yeah, I don't know, I don't know why the model doesn't, like… 644 01:19:59.340 --> 01:20:01.720 216-101 Large Conference: Like, the ba- like, the base model… 645 01:20:01.870 --> 01:20:06.170 216-101 Large Conference: Like, has these weird, you know, Weird peaks. 646 01:20:06.480 --> 01:20:20.349 216-101 Large Conference: I would… I imagine if I extended this out, maybe we'd see some other, like, boutiques, but just, like, in this… in the, like, like, even here, like, around, what is it, I don't know, 119 or something, like, for some reason, the base model's good at 119. 647 01:20:20.830 --> 01:20:27.249 216-101 Large Conference: Yeah, I'm not sure why, like, Laura doesn't give you more than, like, an extra cycle of generalization. 648 01:20:27.950 --> 01:20:29.720 216-101 Large Conference: Good question. 649 01:20:31.730 --> 01:20:44.749 216-101 Large Conference: Do you have a diagram like that blue diagram, but for the geometric generalist… the geometric Laura? Like this one? Yeah. Somewhere. I could find it for you. Do you… have you planted this past 120? 650 01:20:46.780 --> 01:20:58.669 216-101 Large Conference: Somewhere. Is there another lump? Like, around this? Yeah, like, beyond, like, just like there's a lump from 60K, is there a lump from 1,800? I don't know. Yeah. 651 01:21:00.780 --> 01:21:08.720 216-101 Large Conference: My guess is it would be, like, around 132 or something, because that would be 60 after 72, which is, like, where the peak is. 652 01:21:10.440 --> 01:21:15.050 216-101 Large Conference: Sorry, it will lower in Hawk. What the heck? 653 01:21:15.350 --> 01:21:18.120 216-101 Large Conference: Is there Laura is also 6 individuals. 654 01:21:18.300 --> 01:21:23.580 216-101 Large Conference: My Laura… Rank, rank 1. Always rank 1, yeah. 655 01:21:23.950 --> 01:21:26.210 216-101 Large Conference: It's kind of amazing too, right? Yeah. 656 01:21:26.500 --> 01:21:27.610 216-101 Large Conference: That's crazy. 657 01:21:30.210 --> 01:21:37.380 216-101 Large Conference: There's another question I had is that, the 16-dimensional subspace is where you are 658 01:21:37.860 --> 01:21:42.160 216-101 Large Conference: But you are basically, you are, you, you are, 659 01:21:42.420 --> 01:21:45.690 216-101 Large Conference: You are supervising the manifold, right? 660 01:21:46.500 --> 01:21:52.890 216-101 Large Conference: Even that Rank 1 update, It might have some effect. 661 01:21:52.920 --> 01:22:04.959 216-101 Large Conference: on the full representation space? I think it definitely is. Yeah. Because, like, it's at such an early layer that has so much freedom to, like, achieve this geometry that it can do. 662 01:22:04.960 --> 01:22:15.150 216-101 Large Conference: a bunch of other stuff in, you know, on the side. Yeah, but you can also, like, you can also increase this other effect, and only isolate the manifold effect. 663 01:22:15.790 --> 01:22:20.699 216-101 Large Conference: Every layer, you mean? No, just, just at your target layer. 664 01:22:21.930 --> 01:22:24.080 216-101 Large Conference: How, how do you mean? 665 01:22:24.320 --> 01:22:41.539 216-101 Large Conference: You mean that, yeah, you just augment the loss to make it sort of, address the workload subspace outside the system doesn't change. You need to augment the loss, or maybe you also have access to the base model yourself, right? Just an activation of the subspace. 666 01:22:43.470 --> 01:22:47.180 216-101 Large Conference: How many… how many gradient steps do you do to get here, Laura? 667 01:22:47.350 --> 01:22:50.930 216-101 Large Conference: Like, 200, but it usually converges in, like. 668 01:22:51.270 --> 01:22:56.960 216-101 Large Conference: 100 or less. Because, like, you know, rank 1 is, like, a single ingredient step before the right… 669 01:22:57.110 --> 01:23:04.149 216-101 Large Conference: you know, gradient for the right loss, right? So I… I mean, I don't know, it's a silly, stupid thing, right? But it would be like, oh, is there… 670 01:23:04.250 --> 01:23:13.290 216-101 Large Conference: Is there some computation you can do? You know, averaging up all your gradients and everything, just do everything, just say, oh, this is a single gradient update. 671 01:23:13.720 --> 01:23:25.490 216-101 Large Conference: to, like, do this thing would be another… because it's kind of amazing that you… Right. It's like, I mean, it's like a context-conditional vector, yeah. Yeah. 672 01:23:25.860 --> 01:23:38.219 216-101 Large Conference: even if RANCONOR is a single gradient step, it has a lot of free parameters, though, so it's not… Yeah, that's true. Right? It has… I mean, it's… like I said, contest conditional steering vector, right? But… 673 01:23:38.620 --> 01:23:41.640 216-101 Large Conference: But, yeah. 674 01:23:43.390 --> 01:23:45.610 216-101 Large Conference: Just thinking of different possible headlines. 675 01:23:48.040 --> 01:24:01.619 216-101 Large Conference: Oh, sorry, while I was asking about this ablation in the first place, if you perform that ablation, then does it stop the other machinery from reasoning with your updated clinical 676 01:24:01.890 --> 01:24:03.419 216-101 Large Conference: We didn't have any effect. 677 01:24:03.730 --> 01:24:06.740 216-101 Large Conference: Oh, like, if you, if you freeze? 678 01:24:06.910 --> 01:24:14.159 216-101 Large Conference: So, help me understand the loss you're imagining. So, you have, like, these 16 dimensions you're supervising over, and then you basically just have 679 01:24:14.300 --> 01:24:20.760 216-101 Large Conference: Another loss that says, in these other 4,000, whatever, this should be the same. 680 01:24:20.920 --> 01:24:22.969 216-101 Large Conference: That's it? That's it. Okay. 681 01:24:23.130 --> 01:24:30.580 216-101 Large Conference: Okay, so now, and then, now you're clicking again? So, like, with this, so, will it, 682 01:24:31.420 --> 01:24:48.469 216-101 Large Conference: In your discussion, you also mentioned something like, okay, at least how I understood it, is that this geometric LoRa is also probably activating some other machinery that is more able to reason with that, 683 01:24:48.700 --> 01:24:51.670 216-101 Large Conference: Include your memory that you have updated, right? 684 01:24:52.010 --> 01:24:56.470 216-101 Large Conference: So, how important is that, is what I'm trying to understand. 685 01:24:57.630 --> 01:24:59.130 216-101 Large Conference: I see, yeah. 686 01:24:59.460 --> 01:25:01.130 216-101 Large Conference: Yeah, we could do that. 687 01:25:02.010 --> 01:25:03.389 216-101 Large Conference: It'd be interesting to know. 688 01:25:07.840 --> 01:25:14.450 216-101 Large Conference: So what do you think the… the… the headline takeaway is from your… Your findings so far? 689 01:25:14.770 --> 01:25:25.620 216-101 Large Conference: Yeah, so I feel like… I feel like there's, like, two things that I have, like, pretty solid evidence for that will be, like, the main conclusions of the paper. So one is, like. 690 01:25:25.970 --> 01:25:36.979 216-101 Large Conference: Like, LoRa fine-tuning… It seems to, like, strengthen or, like, align existing geometric representations. 691 01:25:37.630 --> 01:25:54.040 216-101 Large Conference: With, like, the task-relevant geometry, which is, like, you know, akin to stuff that Nicholas found before, of, like, fine-tuning isn't, like, introducing something new, it's just, like, sort of sharpening the machinery that the model already has. Oh, you mean on the task? Yeah. I see. 692 01:25:54.420 --> 01:26:00.249 216-101 Large Conference: So that's, like, one thing, and then I think the other thing is, like, when you have, 693 01:26:00.470 --> 01:26:10.980 216-101 Large Conference: Like… You know, well-informed geometric prior, how the model does some computation, like, that can act as, like. 694 01:26:11.280 --> 01:26:14.840 216-101 Large Conference: Sufficient supervision, for training. 695 01:26:15.050 --> 01:26:18.020 216-101 Large Conference: I see, so when you task train up to 24, 696 01:26:18.640 --> 01:26:25.390 216-101 Large Conference: It does better somehow, but then you say that, oh, it does better because it's cleaning up the geometry. 697 01:26:25.610 --> 01:26:30.720 216-101 Large Conference: And you can show that by just directly cleaning up the geometry, and that it actually does better 698 01:26:30.880 --> 01:26:46.260 216-101 Large Conference: you clean up the geometry up to 24. Yep. Right? And it actually does better than doing the task training up to 24. Right. Right? And I think that that's… I think that's, you know, that's a pretty crisp story. So I think that one of the things that's confusing 699 01:26:46.530 --> 01:26:52.249 216-101 Large Conference: In the presentation is that you also have these Baselines that you did were… 700 01:26:52.250 --> 01:27:13.140 216-101 Large Conference: like this one, where you say task training up to 36, and so on, but you don't actually do your geometry training up to these guys, right? Yeah, yeah. And so I think it's… so it's confusing to compare this as a baseline is, like, not fair, right? Does that make sense? Yeah. Right? You should compare your training fairly to the task training that is on the same 701 01:27:13.710 --> 01:27:15.110 216-101 Large Conference: Type of data. 702 01:27:15.620 --> 01:27:16.460 216-101 Large Conference: Right. 703 01:27:16.710 --> 01:27:19.559 216-101 Large Conference: Yeah, it's no fair to say, hey. 704 01:27:19.850 --> 01:27:26.510 216-101 Large Conference: You know, we let the model memorize a thousand things, and, you know, we can't match memorizing a thousand things. Does that make sense? 705 01:27:26.830 --> 01:27:33.000 216-101 Large Conference: That's why I want to see that, like, figure what diagram of that, and then that in orange. 706 01:27:33.250 --> 01:27:36.660 216-101 Large Conference: Oh yeah, yeah, I can go, yeah, I've got it somewhere. 707 01:27:37.220 --> 01:27:37.950 216-101 Large Conference: Yeah. 708 01:27:38.550 --> 01:27:46.220 216-101 Large Conference: So that would be my suggestion. Eric, also, it would be nice to have some controls, because with the way you… 709 01:27:46.680 --> 01:27:54.149 216-101 Large Conference: built up in motivation for geometric Laura, was you trained task loaders, and you found that they built geometry. 710 01:27:54.400 --> 01:27:59.290 216-101 Large Conference: In some nice way. But then somehow, they don't perform that well compared to you. 711 01:27:59.520 --> 01:28:01.699 216-101 Large Conference: Pushing your geometry even further. 712 01:28:02.270 --> 01:28:08.649 216-101 Large Conference: So, like, what else does that task loadout show that you can still push and show that performance also increases? 713 01:28:09.170 --> 01:28:11.800 216-101 Large Conference: Ideally, it's just geometry. 714 01:28:12.260 --> 01:28:13.420 216-101 Large Conference: That's, like… 715 01:28:13.570 --> 01:28:31.480 216-101 Large Conference: So… Can you say that again? So my, like, my suggestion might, like, instead of just asking it what happens, right, so you have your four harmonics that you like, and so, you know, between 0 and 100, whatever, 120, there's, like, 120 harmonics, right? That's… Oh, that's true. Yeah, it's like a 240-dimensional subspace. 716 01:28:31.560 --> 01:28:34.399 216-101 Large Conference: Or something like that, if you just, you know, collect them all together. 717 01:28:34.570 --> 01:28:42.769 216-101 Large Conference: And you can order them according to the order that you think that they're important, according to your purpose, but there's some ordering going all the way to 240. 718 01:28:43.110 --> 01:28:54.560 216-101 Large Conference: And, and then… and then you could, you know, see what the effect is of allowing… Including them in your loss. Including them in the loss. And, you know, you would have some… some fall-offs, some knee. 719 01:28:54.680 --> 01:28:57.019 216-101 Large Conference: Right, where it stops helping. 720 01:28:57.340 --> 01:29:09.840 216-101 Large Conference: And but… but you could potentially push the performance higher if you wanted to. You could say, you can… you know, with only these, you can get really good performance. It's better than TAP training, but if you do even more, then you get more performance. 721 01:29:09.960 --> 01:29:11.380 216-101 Large Conference: Right, you know. 722 01:29:11.720 --> 01:29:13.950 216-101 Large Conference: I think that would be… 723 01:29:14.890 --> 01:29:29.300 216-101 Large Conference: Like, in the limit, you just compare… you can include all the dimensions or something like that, and maybe that's… I'm not sure. I don't… yeah, I guess I don't know what the… I don't know what that one is. I'm also questioning the… oh, yeah, sorry. I'm also questioning the… 724 01:29:29.480 --> 01:29:31.099 216-101 Large Conference: Entire four years. 725 01:29:31.740 --> 01:29:36.379 216-101 Large Conference: geometry itself. I'm saying you need a control that shows 726 01:29:36.540 --> 01:29:38.980 216-101 Large Conference: That is not, like, just the luck. 727 01:29:40.080 --> 01:29:44.179 216-101 Large Conference: Do you have a suggestion for… I don't… 728 01:29:44.350 --> 01:29:53.139 216-101 Large Conference: I think you can put a constraint where you don't let it go to the geometry. 729 01:29:53.310 --> 01:29:58.139 216-101 Large Conference: And then you see if it still improves the task or not. I think that can be the control. 730 01:29:58.290 --> 01:30:04.690 216-101 Large Conference: And that can also help you figure out if what the other mechanism is. Oh, so you could do task training… 731 01:30:04.790 --> 01:30:24.710 216-101 Large Conference: And you can say, you're not allowed to touch this subspace? Yeah. Or, like, don't, have your, representation in that geometry, and then you see if it improves or not. And if it does improve, you just, like, totally scramble it. Then you have that… that geometry is not necessary, you just have… just an artifact of… 732 01:30:25.220 --> 01:30:33.990 216-101 Large Conference: increment the geometry in a different way. Yeah. Like, in a different way. Put it in a different subspace. Yeah, but then the probe won't show it. 733 01:30:34.160 --> 01:30:51.499 216-101 Large Conference: I think the probe is… the new probe. You could… you could have one-dimensional probes for each of your numbers. You've got 24 numbers or something like this, so you could train 24 one-dimensional probes for 734 01:30:51.660 --> 01:30:53.100 216-101 Large Conference: Did you get this number? 735 01:30:53.580 --> 01:30:57.329 216-101 Large Conference: You get a 24-dimensional subspace, and you could just try to take that one. 736 01:31:00.990 --> 01:31:01.859 216-101 Large Conference: Let's see. 737 01:31:11.450 --> 01:31:16.489 216-101 Large Conference: Well, yeah, we're way over time, but yeah, thanks for all the comments and questions. I think that's helpful for me, so… 738 01:31:18.020 --> 01:31:21.340 216-101 Large Conference: Can we applaud you again? Thank you, Eric. 739 01:31:24.110 --> 01:31:26.979 216-101 Large Conference: Yeah, it's always, it's always helpful to… 740 01:31:28.340 --> 01:31:43.460 216-101 Large Conference: And, you know, to talk about something before submission, we usually have these things after submission, right? When it's too late to have any feedback. And so, so yeah, no, I think it's good to do this. So it's good for other students to be brave. 741 01:31:43.560 --> 01:31:47.719 216-101 Large Conference: And present your stuff before it's done, so I encourage you to do that. 742 01:31:48.380 --> 01:31:53.029 216-101 Large Conference: So, let me… let me turn this topic. 743 01:31:53.190 --> 01:31:56.780 216-101 Large Conference: But help me eat the cheese, you guys. We have a lot of cheese.