WEBVTT 1 00:00:00.060 --> 00:00:01.920 216-101 Large Conference: thing. Anyway… 2 00:00:02.320 --> 00:00:07.910 216-101 Large Conference: Okay, and then are you… have you left audio on the thing, so there's no echo? Okay, so it's all… you're all set up here. 3 00:00:08.080 --> 00:00:13.020 216-101 Large Conference: So, okay, you guys all know Naomi? Y'all know Naomi, right? 4 00:00:13.420 --> 00:00:18.499 216-101 Large Conference: So, so, the way I think of Naomi is… 5 00:00:18.760 --> 00:00:28.110 216-101 Large Conference: As the expert in training dynamics, you know, with interpretability, with understanding how systems are working inside. 6 00:00:28.270 --> 00:00:39.060 216-101 Large Conference: understanding how this evolves during the course of training. I know that Naomi is interested in a lot of other things as well, so I'm not exactly sure what the talk is going to be about, because it has such a broad 7 00:00:39.170 --> 00:00:47.449 216-101 Large Conference: I don't know. Yeah, well… But Naomi is also famous, from my perspective, at being able to answer difficult 8 00:00:47.800 --> 00:00:50.650 216-101 Large Conference: Throw any questions without fear. 9 00:00:50.940 --> 00:00:55.800 216-101 Large Conference: And so, So yes, so, you know… 10 00:00:55.880 --> 00:01:18.909 216-101 Large Conference: Welcome, Naomi, and thank you. Certainly, if I'm… we can see if… if I'm… if I'm not on YouTube, then we can definitely talk about any thorny question. Okay. So, yeah, I'm going to be talking about when can we predict model behavior, which is a pretty general, as you were saying, question, but we've got, like, a few 11 00:01:18.930 --> 00:01:29.219 216-101 Large Conference: cases that we talk about, right? There's scaling laws, you know the model's size, then you know what the loss is going to be, at least in the aggregate, across some 12 00:01:29.220 --> 00:01:51.010 216-101 Large Conference: distribution that looks kind of like the training data, right? There's smooth loss curves, right? You look at one point, and you tend to generally say, okay, well, I know where that's going, right? You can… you can continue a trend, you know. So, you know, if you know what time you're going to look at, then you know what the loss is probably going to be around that time. There's… 13 00:01:51.010 --> 00:02:15.239 216-101 Large Conference: this phenomenon called accuracy on the line, so if you're thinking now about out-of-distribution performance, which we didn't talk about the last two slides, you can sort of predict it if you just know what the in-distribution loss is going to be, then you can usually predict what the out-of-distribution performance will be. This is… this is a… This is for a specific out-of-distribution twice. 14 00:02:15.240 --> 00:02:35.170 216-101 Large Conference: Is that right? This is for a collection of a few, because Wilds has a few different sort of shifts, this is all, like, ImageNet, you know, I do, I do, you're gonna see, I, I do think that a lot of these classic science of deep learning sort of results are, 15 00:02:35.820 --> 00:02:42.499 216-101 Large Conference: Are maybe shaped by the fact that they were all found on a toy task, which is image classification. 16 00:02:42.500 --> 00:03:06.010 216-101 Large Conference: Which, at the time, nobody realized was a toy task, funnily enough, but it doesn't require you to really know a lot of compositional structure, you really just need to know about it. There's only a couple image people in the room, so… Yeah. I guess you can say that. Yeah, this is my spicy… I mean, nowadays, all the image people, they're looking at multimodal models, but it shows that language is the thing that makes it interesting. 17 00:03:06.010 --> 00:03:11.519 216-101 Large Conference: Although, also, I would argue ImageNet is also a multimodal dataset, because, like. 18 00:03:12.200 --> 00:03:22.139 216-101 Large Conference: Like, why do you have all of these different dog breeds, and you don't differentiate all the pigeon lineages in New York City? It's because that's what we use language to talk about. Oh, because… 19 00:03:22.240 --> 00:03:24.049 216-101 Large Conference: Because Arditi was really hard. 20 00:03:24.100 --> 00:03:41.490 216-101 Large Conference: It's because that's what's in WordNet! It's a WordNet-based data set. Yeah. WordNet has all the dog breeds, but do they talk to the pigeon experts about all of our domesticated pigeons? We just don't talk about those things. Well, maybe that intern wasn't working as hard. 21 00:03:41.490 --> 00:03:44.930 216-101 Large Conference: Yeah. 22 00:03:44.970 --> 00:04:06.089 216-101 Large Conference: Okay, so, it turns out, here's a cool thing, you can, prove that models will converge to a global minimum, you know, like, with distribution, you know, you know, you can converge to a global minimum, given the training distribution. You know exactly where you're going to end up at the end of training, right? That's pretty nice. 23 00:04:06.090 --> 00:04:07.160 216-101 Large Conference: Oh… 24 00:04:07.200 --> 00:04:19.829 216-101 Large Conference: Also, there's linear mode connectivity, that's great. If you know the fine-tuning setup, if, like, then every time you train it, no matter what the random seed, you're gonna end up in the same law space, and that's pretty cool on image classifiers. 25 00:04:19.829 --> 00:04:38.799 216-101 Large Conference: You know, if you didn't have that, you would be running into barriers all over the lost surface, you wouldn't be able to do things like soup or interpolation-based regularization, but that's not the focus of my talk, so if you don't know about that, don't worry about it, but maybe look it up. It's really interesting, and you get it because of linear mode connectivity. That's pretty cool. 26 00:04:38.800 --> 00:05:03.469 216-101 Large Conference: Every single time you look, it's the same basin. There's no increasing loss when you interpolate between a pair of model weights that are fine-tuned from the same pre-trained model. This is, like, a really, really neat phenomenon. It means that generally everything seems really predictable, actually. This is really nice. So that's my talk, that's my talk, everything that's predictable. Okay, so now we're going to talk about 27 00:05:03.470 --> 00:05:05.200 216-101 Large Conference: Things that are not predictable. 28 00:05:05.460 --> 00:05:15.760 216-101 Large Conference: Okay, so we talked about those nice scaling laws, but also you may have heard that not everything follows NEAT scaling laws. If you look at specific tasks. 29 00:05:15.760 --> 00:05:30.280 216-101 Large Conference: they might have huge breakthroughs that come out of nowhere on specific scales after having very low performance on all the previous scales. So, you can sort of have these surprise, unpredictable jumps in performance. 30 00:05:31.630 --> 00:05:55.950 216-101 Large Conference: those smooth loss curves we talked about, they don't… they don't always sort of hold, right? One famous example of this is… is blocking, right? Where you have a model, you know, perfectly fit the training set, it keeps on fitting, it overfits, the generalization just isn't particularly improving, and then suddenly, again. 31 00:05:55.950 --> 00:06:01.589 216-101 Large Conference: And out of nowhere, apparently, suddenly generalization starts to dramatically improve. 32 00:06:02.040 --> 00:06:14.420 216-101 Large Conference: So, you can see things that happen quite suddenly, including in language models, if you've looked at the, the general head specialization formation, breakthrough. 33 00:06:14.650 --> 00:06:32.929 216-101 Large Conference: which… during which you have induction heads, and you also have internal syntactic structure forming. All of these heads sort of start specializing during a particular part in training, and then you see the loss dramatically decrease during that. We talked about accuracy on the line, that's pretty cool, but actually. 34 00:06:33.220 --> 00:06:36.819 216-101 Large Conference: It's not true on language. 35 00:06:36.830 --> 00:06:59.339 216-101 Large Conference: Even on simple, like, text classification tests, even outside of, you know, even if you're not doing actual language modeling, you can end up with real differences between, like, one and another of your, like, of your, sort of, distribution shifts, right? So, in this case, what we're looking at is actually the accuracy on 36 00:06:59.340 --> 00:07:01.310 216-101 Large Conference: natural language inference. 37 00:07:01.310 --> 00:07:02.660 216-101 Large Conference: Which… 38 00:07:02.660 --> 00:07:21.989 216-101 Large Conference: is pretty consistent across 100 random seats, that's what the histogram is showing. It's, like, 85% accuracy, and there's this, like, not really very well correlated, actually, although I'm not showing the correlation here. There's this, much wider sort of performance distribution if you ask 39 00:07:21.990 --> 00:07:36.260 216-101 Large Conference: how do those perform on challenge sets where you can't rely on heuristics that are just related to, like, counting individual tokens, and you have to actually rely on syntactic sort of structures. So… 40 00:07:36.980 --> 00:07:43.510 216-101 Large Conference: So that's cool. We talked about, convergence to a global minimum. We did say it's at the infinite width limit, which… 41 00:07:43.510 --> 00:08:05.929 216-101 Large Conference: I'm gonna say, not only are we not approaching the infinite width limit right now, the way we are scaling right now, the way we add compute to language model training, is not by just increasing the dense parameter count of the model. We're adding experts, we're adding more RL exploration training, and those are things that 42 00:08:05.930 --> 00:08:07.540 216-101 Large Conference: Do not reduce. 43 00:08:07.540 --> 00:08:14.679 216-101 Large Conference: And in fact, relative to your dense parameter or scaling, actually can increase 44 00:08:14.770 --> 00:08:31.570 216-101 Large Conference: your, variability across different random seeds. For RL, this is very famous, it created an entire crisis in RL, like, 5 years ago, when it turned out that a large part of, like, the rankings of different RL methods was random seed. 45 00:08:32.309 --> 00:08:38.180 216-101 Large Conference: there was almost no difference between most RL methods that people had been publishing for, like, years. 46 00:08:38.400 --> 00:08:55.530 216-101 Large Conference: So, if you're… if you're thinking, oh, well, well, we'll get to the infinite width limit, and then… so I'm just going to think about asymptotics forever, well, like, we're not traveling the asymptotic curve, right? We're adding variability, not removing it right now. So… 47 00:08:56.280 --> 00:09:14.970 216-101 Large Conference: We talked about linear mode connectivity. It's a pretty cool phenomenon. Unfortunately, it is also not true for text classifiers. These are the same kinds of text classifiers we looked at with that histogram, so these are doing natural language inference. If you don't know the task, you don't have to worry about it right now. But it turns out that the ones that are good 48 00:09:15.090 --> 00:09:26.300 216-101 Large Conference: At, generalizing to these, like, challenging data sets that violate the heuristics of the model sit in a very different basin on the loss surface 49 00:09:26.300 --> 00:09:43.149 216-101 Large Conference: from the bad ones. And those basins are, like, roughly internally convex, but you can't just jump from one to the other in the way that you do in these classic image classification results that show linear mode connectivity is always there. 50 00:09:43.410 --> 00:09:44.910 216-101 Large Conference: Okay. 51 00:09:45.230 --> 00:09:51.289 216-101 Large Conference: So what's sort of the unifying pattern between all of these things that are not predictable? 52 00:09:51.810 --> 00:10:09.969 216-101 Large Conference: There's what I'm gonna call you know it or you don't. There's a discrete difference between the models that know it, the models that don't. This model knows it, a few steps later, you know, it doesn't know it, a few steps later, it knows it. This model ended up in this basin where it knows it, this other model doesn't. 53 00:10:09.970 --> 00:10:34.550 216-101 Large Conference: This is difficult for, you know, if you're interested in asymptotics, if you're thinking about smooth continua, if you come from a sort of a classic learning theory, maybe, background, where you think about continuously learned skills, and you look for guarantees and convergent, and you want to see, like, these scaling laws that improve gradually everything, but it turns out that certain concepts are natural concepts. 54 00:10:35.380 --> 00:10:53.720 216-101 Large Conference: So the model knows it or doesn't know it, right? What do you mean by natural kind? So this is… I think Socrates said at some point, citation Socrates, you know, described this as carving nature at the joints. So you can, like. 55 00:10:53.720 --> 00:11:12.070 216-101 Large Conference: cut a bone gruesomely in the middle, but you haven't potentially learned much about the skeleton. But if you can figure out where the joint is, where there's a different bone that attaches to some other bone, you're going to understand the skeleton better than if you look at a random, like, a bunch of random portions of the bone. 56 00:11:12.070 --> 00:11:28.179 216-101 Large Conference: Right? At the joints, that's where we develop some real understanding. Some… you can take a Gaussian curve of some property in the human population, and just say, the top, like, 10% of people, they're disordered. 57 00:11:28.580 --> 00:11:45.900 216-101 Large Conference: did you actually find, like, some disease? Did you find too tall disease, or whatever? No, you've identified people who find it slightly more difficult to, like, sit on an airplane seat than people who are in the top 15%, but not the top 10%, right? But you haven't 58 00:11:46.020 --> 00:12:02.230 216-101 Large Conference: found too tall disease, but then if you see a little… a little bump, if you see a bimodal distribution, then maybe that little bump is people who are… who are dramatically too tall because they have a hormonal disorder, right? So you're drawing a connection between these bumps, like grocking. 59 00:12:02.230 --> 00:12:13.370 216-101 Large Conference: And this kind of insight? That's what you're saying? So, like… Yeah. Which is, again, bad news for asymptotic analysis, but is great news for interpretability, because that's how humans think. 60 00:12:13.370 --> 00:12:25.610 216-101 Large Conference: Right? We don't think in these continuous gradients, we don't think in these spectra, we think in discrete objects. We say, this person has too tall disease caused by their hormonal imbalance. We don't think… 61 00:12:26.210 --> 00:12:35.820 216-101 Large Conference: this person is in the, like, fifth, you know, top grouping tile, they have so many standard deviations. Yes, this person knows how to count, not this person kind of… 62 00:12:35.960 --> 00:12:43.089 216-101 Large Conference: like, has X probability of getting the next number correct. Yeah, exactly. 63 00:12:43.400 --> 00:12:45.900 216-101 Large Conference: Yeah, this… so… 64 00:12:46.140 --> 00:12:56.159 216-101 Large Conference: That's what we're talking about. Yeah. But I feel like that could be relative, because you could say, like, oh, this person knows how to count, but maybe the exact… 65 00:12:56.410 --> 00:13:21.059 216-101 Large Conference: Yeah, they could, let's say, just count to, like, for babies, like, count to, like, let's say, till 10 or something, and then there's counting till, like… Yes, that is a thing that we would see, and it might start looking like you have a, like, a very smooth, continuous distribution of general skills of everything, but then if you look under that smooth thing where you checked every day to see how many things you know. 66 00:13:21.180 --> 00:13:35.270 216-101 Large Conference: You find under that, that you… that this person knows how to count to 10, this person knows the first few numbers, this person knows how to do the teens, this person knows how to compose 67 00:13:35.270 --> 00:13:52.790 216-101 Large Conference: you know, the base 10 sequence, this person knows what a trillion means, you know? This… so these, these are the things that differentiate how humans understand things. They don't know, like, 50% of what number comes after 4. 68 00:13:53.230 --> 00:13:54.290 216-101 Large Conference: Right? 69 00:13:55.620 --> 00:14:12.149 216-101 Large Conference: So, do you think that there is a reality to that? Like, for different skills, do you think that, what would be a skill for one learner would be, like, a skill for a different learner, or do you think that these things are… 70 00:14:12.450 --> 00:14:19.949 216-101 Large Conference: I think that what counts as a skill for a language model is not the same as what counts as a skill for a human. 71 00:14:20.300 --> 00:14:30.299 216-101 Large Conference: Because… in part, because of the modular structure of the model, which might not reflect the modular structures in our brains, right? 72 00:14:30.430 --> 00:14:36.610 216-101 Large Conference: So, you know, one example is induction heads. Humans don't use induction heads, too. 73 00:14:36.610 --> 00:14:50.469 216-101 Large Conference: like, figure things out in their working memory. No. Maybe we do. Who do you think, Sharon? Is there a secret induction head? I would bet no. 74 00:14:52.220 --> 00:14:53.330 216-101 Large Conference: Yeah. 75 00:14:53.440 --> 00:15:07.140 216-101 Large Conference: So, so let's say… let's, let's for now… I mean, it's an interesting question which… which sort of structures are shared, but… but we certainly don't want to assume that the skills of a language model 76 00:15:07.140 --> 00:15:19.329 216-101 Large Conference: line up with what skills a human needs to learn to accomplish the same task, although they might end up in the same sort of general structure at the end of things. It's possible that the way they build up there is also different, right? 77 00:15:19.880 --> 00:15:34.800 216-101 Large Conference: Okay, so, I want to talk about these, these weird, like, multiple solution cases, this bimodality, the, the, is this person just too tall, or do they have, an actual medical 78 00:15:34.800 --> 00:15:40.149 216-101 Large Conference: diagnosis, related to being too tall, right? So… 79 00:15:40.150 --> 00:15:54.419 216-101 Large Conference: These solutions, we talked about emergence, we talked about scaling laws, these solutions are generally accessible simultaneously, depending on random seed, if you are near an emergence threshold. 80 00:15:54.880 --> 00:16:17.670 216-101 Large Conference: That's sort of intuitive. Let's talk about what we actually, what it actually looks like. So, scaling laws, you know, power law, it's really smooth, you have, so many parameters, you end up with so much performance, right? We've seen there's certain tasks where there's a big jump at some point, and I will address the Mirage thing, don't worry. 81 00:16:17.670 --> 00:16:31.619 216-101 Large Conference: You mean the logarithmic scale? What? You mean the logarithmic scale? Is that what you mean by mirage? Yeah, I mean, like, the looking… like, controlling thresholding effects and not looking at exact matches across the whole string, because exact match can be pretty misleading, but… 82 00:16:31.620 --> 00:16:37.620 216-101 Large Conference: But, you can… but as long as you're in a continuous metric, you're not going to have threshold effects, so… 83 00:16:37.620 --> 00:16:42.399 216-101 Large Conference: We will look at some continuous metrics for… for some of the things I'm showing. All right. 84 00:16:42.820 --> 00:16:54.840 216-101 Large Conference: emergence. This is a case of you know it or you don't, right? You don't know it, and then you get a few more parameters, and you know it, and it happens at exactly 10 to the… 85 00:16:54.900 --> 00:17:12.740 216-101 Large Conference: whatever power for all of these tasks. Isn't that great? We know exactly how many parameters that you need in order to accomplish this task and have a non-trivial performance, each of these tasks. That's lovely. And sure enough, you know, we have this 86 00:17:12.740 --> 00:17:32.550 216-101 Large Conference: I don't want to go into the details about this reverse addition task, but what's important is it is like an out-of-distribution, sort of measurement. It's looking at whether you can generalize to longer sequences when you are trying to do this sort of arithmetic addition thing with, like, a few extra hints to make it easier for the model. 87 00:17:32.940 --> 00:17:47.770 216-101 Large Conference: And as you can see, this is totally emergent. We trained at parameter size A, and then parameter size B, and A doesn't get it, and B does. That's great. But then. 88 00:17:47.900 --> 00:17:50.960 216-101 Large Conference: What if you looked at a different feed? You just… 89 00:17:51.200 --> 00:18:00.100 216-101 Large Conference: train a different seed of initialization at each of these parameters, suddenly it doesn't look so emergent. So what's really going on here? 90 00:18:00.430 --> 00:18:02.330 216-101 Large Conference: Okay, so… 91 00:18:02.330 --> 00:18:26.899 216-101 Large Conference: Most of the time when people say, oh, this is the scale that you need for X, this is the multiple choice onset, that's the famously interactively emergent task, I would say, is multiple choice. What's the background of the multiple choice onset? Oh, multiple choice onset is just when you're able to… the model is able to answer multiple-choice questions. That's it. So in order to answer a multiple-choice question. 92 00:18:26.900 --> 00:18:37.819 216-101 Large Conference: you have to both know you need to say ABCD, and you need to know how to compose that format with the actual knowledge you have about, you know, hummingbird wings or whatever, right? When you're answering the question. 93 00:18:37.820 --> 00:18:43.470 216-101 Large Conference: So, that's a compositional capability that is required, and it… 94 00:18:43.470 --> 00:19:04.900 216-101 Large Conference: doesn't show up until pretty large-ish models? I don't know, what's the MC onset that people usually say? Well, it turns out that Quen gets it much earlier, because it's… Quen gets a ton of extra multiple-choice questions in its pre-training, but that's a… but for now, let's assume that we know as much as everyone knew right before Quen came out. 95 00:19:04.900 --> 00:19:12.840 216-101 Large Conference: You can't answer multiple-choice questions at such and such scale. It's emergent at this parameter size, right? 96 00:19:13.690 --> 00:19:25.309 216-101 Large Conference: But in this case, we're seeing different scaling trends for different seeds. So what does that mean? It means that there is a bimodal distribution underneath. 97 00:19:25.360 --> 00:19:34.400 216-101 Large Conference: That under… that what you're actually doing at each scale is sampling a model from a distribution of capabilities. 98 00:19:34.600 --> 00:19:43.490 216-101 Large Conference: And, and if you, if you sort of, at parameter size A, it looks like this. At parameter size B, it looks like this. 99 00:19:43.560 --> 00:20:06.599 216-101 Large Conference: And if you sample that purple one, for instance, from the you don't know it, and then you'll see, like, one of these is the success, and one of these is the failure, if you look at the purple one, you know, first it sampled from the failure mode, you don't know it, and then from the success mode, you know it, right? So that's emergence, right? But if you look at the red one, it got kind of lucky. 100 00:20:06.600 --> 00:20:08.759 216-101 Large Conference: at parameter size A, 101 00:20:08.760 --> 00:20:17.399 216-101 Large Conference: it knew it at parameter size A, not very well, but, like, it knew it. And then at parameter size B, you can see that it, 102 00:20:17.400 --> 00:20:31.909 216-101 Large Conference: just sort of happens to get lucky and stay in that same little mode. And that is why it looks linear in trend, because it doesn't have a sudden jump where it stops sampling from the failure and starts sampling from success. 103 00:20:31.910 --> 00:20:41.179 216-101 Large Conference: It's just sort of following the usual scaling law. Like, this random variation, like, if you cut off the failure mode. 104 00:20:41.330 --> 00:20:58.819 216-101 Large Conference: This would look a lot like what you would usually see, right? The mean is increasing in the accuracy, the variance is gonna contract a little bit, that's what you would expect to see in random variation as you scale up, right? Yeah? Are those distributions usually bimodal for most tests? 105 00:20:58.870 --> 00:21:00.640 216-101 Large Conference: No, they are not. 106 00:21:00.640 --> 00:21:21.859 216-101 Large Conference: They are bimodal when we look specifically at emergence thresholds for these, different tasks. So, so some tasks require some out-of-distribution or compositional, reasoning, right? And those are the ones that have the potential to be, bimodal in this way. 107 00:21:21.860 --> 00:21:26.770 216-101 Large Conference: In order, for instance, for a… I mean, going back to induction heads, right? 108 00:21:26.770 --> 00:21:46.010 216-101 Large Conference: In order for a model to handle in-context learning well, it needs to have both parts of that head, and they need to connect together. And if it hasn't sort of connected them together, if it doesn't have, like, them fully learned, it's just not going to be able to do it. And the important thing there is the modularity. 109 00:21:46.140 --> 00:21:48.550 216-101 Large Conference: When it 110 00:21:49.350 --> 00:22:01.370 216-101 Large Conference: Hmm. You know what? I'll talk about this later, because I'm talking about, I'm talking about saddle points, yeah? Are there any results, or even just, like, speculation on, like, the independence of, or dependence between these two distributions? 111 00:22:01.450 --> 00:22:17.719 216-101 Large Conference: the independence of, or dependence between the two distributions. Like, the sample that I took at time point B dependent on my sample… on what it was… times, right? Like, model size that train from scratch, one smaller model and one a bigger model. So people usually do… so these are not times. 112 00:22:17.910 --> 00:22:42.460 216-101 Large Conference: These are scales, which means you're independently initializing them with the same random seed, but there's not really, like, a simple connection between, like, what initialization you get with seed 93 at point A, and with seed 93, if you, like, have a bigger matrix you're building. So, so the answer is people generally fix the seed, and they even maybe, like, make some big deal about that, that they're, like, controlling for something. 113 00:22:42.560 --> 00:22:47.779 216-101 Large Conference: But it doesn't really mean anything to fix the seed across scales. 114 00:22:47.780 --> 00:23:07.049 216-101 Large Conference: I misunderstood that. Oh, yeah, yeah, yeah. So, okay, yeah. Yeah, I guess this is related, but, like, so what does it mean by emergence threshold? Because there is no, like, scalar thing, it's just, like, if people did it at A, and they kind of, that's the threshold. Yeah. Yeah. So, if we want to talk about an emergence threshold. 115 00:23:07.050 --> 00:23:09.780 216-101 Large Conference: The threshold is, like, somewhere in this? 116 00:23:09.780 --> 00:23:31.599 216-101 Large Conference: region, anything you look at that's in this region, this is bimodal, this is bimodal, right? So there's some region here that's actually, like, close enough to the threshold, that's close enough that there's… you're sort of unstable because you're sitting around, like, some phase transition, where you could go one or another way. So there is no, like. 117 00:23:31.780 --> 00:23:44.600 216-101 Large Conference: yes to no transition, like, definitely, but there is a transition around this phase. Yes. There's, like, a range of transition where it's, you know, it's like a phase transition, 118 00:23:44.780 --> 00:24:04.700 216-101 Large Conference: Like, if you say phase transition to a physicist, they get upset. If you say emergence to a cognitive scientist, they get upset. It's best, not to… it's best to say things like breakthrough, except that then, like, critics will sometimes get upset. You know, you're gonna upset someone with any 119 00:24:04.700 --> 00:24:08.030 216-101 Large Conference: Term you assign to this discontinuity? 120 00:24:08.360 --> 00:24:13.390 216-101 Large Conference: But, they do exist, whatever you want to call them. 121 00:24:13.640 --> 00:24:23.580 216-101 Large Conference: So, so yeah, there is this range in which there is the instability between, sort of, your options within a distribution. 122 00:24:23.580 --> 00:24:42.970 216-101 Large Conference: And in fact, the distribution we're talking about can be… is usually changing pretty gradually, you know? The probability of sampling from your success distribution is slowly increasing during all of these different scales. 123 00:24:43.940 --> 00:24:55.640 216-101 Large Conference: But the individual trends look discontinuous, and even their average looks discontinu- or, like, their mode, certainly, and even the average, actually, looks discontinuous sometimes. 124 00:24:55.640 --> 00:25:14.539 216-101 Large Conference: Because there's a gradual change in that probability happening under. Okay, I said the Mirage thing. There was a best paper, awarded paper at NeurIPS a few years ago, called Is Emergence a Mirage? where they showed that, like, 92% of the 125 00:25:15.200 --> 00:25:21.019 216-101 Large Conference: tasks in Big Bench that are claimed to be, like, emergent are actually 126 00:25:21.100 --> 00:25:36.380 216-101 Large Conference: like, linear, if you look, instead of looking at, at a, exact match sort of accuracy, you instead look at the, like a Breer score, like something that's continuous, and you put it through the log. 127 00:25:37.160 --> 00:25:40.940 216-101 Large Conference: So that it's more like loss and less like, you know, X loss. 128 00:25:41.430 --> 00:25:53.699 216-101 Large Conference: So, in this case, what we did was we just used a continuous metric that's sort of equivalent to looking at exact match accuracy, but is forced to be fully continuous, because 129 00:25:53.700 --> 00:26:07.400 216-101 Large Conference: let's say you have a whole string of tokens, the tokens are fixed, you've accepted that each individual token is continuous, because otherwise the Breer score or the loss metric wouldn't make sense either. So you're assuming each of these is continuous, right, within a given string. 130 00:26:07.790 --> 00:26:15.820 216-101 Large Conference: So the question is, if you hold all of the tokens up to, you know, index J, fixed. 131 00:26:16.100 --> 00:26:29.969 216-101 Large Conference: is the model… like, what is the model's loss gonna be on index J? And specifically, what is the worst mistake it's gonna make as it goes through this sequence with the whole history? 132 00:26:30.670 --> 00:26:39.980 216-101 Large Conference: Which is basically like saying, what's the closest it gets to getting the whole sequence wrong, in a sense? 133 00:26:40.740 --> 00:26:42.010 216-101 Large Conference: So… 134 00:26:42.160 --> 00:26:54.549 216-101 Large Conference: when you look at this… this, loss metric, which is continuous if you accept that loss is continuous with respect to whatever. This is, like, if you're… if you're familiar with, like, the Lipschitz constant, if you have a fixed 135 00:26:54.690 --> 00:26:59.589 216-101 Large Conference: fixed set of, you know, Lipschitz continuous functions. 136 00:26:59.870 --> 00:27:14.129 216-101 Large Conference: Then the maximum of that set is also going to be Lipsch's continuous, because everything is… like, because if you were to switch over, you can't have a bigger jump as you switch from, this is the maximum to that as the maximum, then the 137 00:27:14.310 --> 00:27:19.520 216-101 Large Conference: jump for each of those individual functions. This is the intuition. But… 138 00:27:19.860 --> 00:27:22.439 216-101 Large Conference: The important thing is, not a mirage. 139 00:27:22.960 --> 00:27:26.430 216-101 Large Conference: If you use this continuous metric. 140 00:27:26.830 --> 00:27:43.370 216-101 Large Conference: these are still going to be bimodal. What's on the x-axis for these ones? Oh, yeah, these are all histograms. The x-axis is giving you the, this continuous metric I was just describing, essentially the maximum loss 141 00:27:43.400 --> 00:27:47.230 216-101 Large Conference: For the, entire strings, right? 142 00:27:47.310 --> 00:28:03.780 216-101 Large Conference: And, and then this is… it's a histogram, so this is the density. So these are, you know, made as… so that this is… So this is the analogous thing to your accuracy diagram before, but there's an extra loss, is that right? And so you're looking at the two bumps? Yeah, with, like, with, like, a… 143 00:28:03.820 --> 00:28:14.199 216-101 Large Conference: little, like, KE smoothing thing, just so that you're not tricked by little bumps in the… in the counts, of our, like, 200 models. 144 00:28:14.460 --> 00:28:27.420 216-101 Large Conference: Yeah. So you're saying that the bimodal distribution changes as the model scales, right? Yes. Like, the parameters of the distribution changes. Yes. So why do you think in the smaller models, there's still a chance for them to have 145 00:28:27.420 --> 00:28:34.919 216-101 Large Conference: a successful training, because if we think that, oh, maybe some people will say that the smaller models, they just don't have the capacity to really learn a task. 146 00:28:34.920 --> 00:28:52.230 216-101 Large Conference: This is exactly what people might sort of generally assume is the source of, like, emergence at a particular threshold, right? They'll think, oh, this is a capacity issue. The model was not… did not have the capacity to represent or learn this task over here, and then 147 00:28:52.230 --> 00:28:55.799 216-101 Large Conference: It got a few more parameters, now it is expressive enough. 148 00:28:55.800 --> 00:29:08.210 216-101 Large Conference: to actually encode this task, or learn it, right? And it seems that that's not what's happening, that the model actually has that capacity for many tasks at a much smaller scale 149 00:29:08.390 --> 00:29:20.159 216-101 Large Conference: it just… it's just harder to get lucky. Yeah, it's just not lucky. Yeah. So, if you think of… if you're familiar with the lottery ticket hypothesis, you could think of this as 150 00:29:20.420 --> 00:29:40.029 216-101 Large Conference: as you are building up the model, as you're growing the model, you're buying, like, lottery tickets for both of these possible outcomes. Lottery ticket hypothesis is just this idea that, like, the reason why big models are better is because you're, like, buying tickets of, like, a few parameters being better 151 00:29:40.030 --> 00:29:47.929 216-101 Large Conference: You know, you keep generating new, sort of, networks within this, and maybe you'll get, like, a really good initialization somewhere in there, right? 152 00:29:48.390 --> 00:30:08.379 216-101 Large Conference: So you're buying lottery tickets for two competing sort of strategies that the model could approach, and some of… and one of them is maybe, like, kind of better than the other, but might be harder to find enough parameters for, is one way that I have been thinking about this kind of competition. 153 00:30:08.550 --> 00:30:13.540 216-101 Large Conference: That they probably require different structures of subnetworks, and you're sort of like. 154 00:30:14.420 --> 00:30:25.339 216-101 Large Conference: you really want the good one, but, like, you're not buying as many lottery tickets for it. Yeah, I mean, do you think that this is why distillation works so well? That's a good question. 155 00:30:26.900 --> 00:30:27.820 216-101 Large Conference: You know… 156 00:30:31.130 --> 00:30:42.580 216-101 Large Conference: Distillation gives us, like, a lot of additional information, and there are a lot of avenues that it gives us extra information. So I would hesitate to just 157 00:30:42.580 --> 00:30:52.470 216-101 Large Conference: to connect it to any one explanation, unless we had, like, really strong evidence for it. Also, people are, like, sort of, like, more into this alternative to the… 158 00:30:53.070 --> 00:31:07.719 216-101 Large Conference: lottery ticket hypothesis view, and are interested in this idea that what you're actually buying is saddle point escape routes, not initializations, but… but, like, escape routes during learning. And and so that's… 159 00:31:08.210 --> 00:31:17.229 216-101 Large Conference: But I think that it's possible, but I don't… but you're getting so much extra information, potentially, from distillation, that it's hard to say exactly. 160 00:31:17.510 --> 00:31:33.349 216-101 Large Conference: Sorry, I'm not following. What is the intuition that the subnetworks would enable distillation? Yeah, so if I'm understanding what you're saying… Yeah, I can just… Yeah, go ahead, yeah. So it's just… so, you know, you say, well. 161 00:31:33.860 --> 00:31:44.699 216-101 Large Conference: You know, if you know that it's possible for a small network to solve the problem, but you try to train it, and you have to train it, like, a thousand times before you get one that does it. 162 00:31:44.910 --> 00:31:55.890 216-101 Large Conference: But actually, if you train a bigger network to do it, it can just, like, it learns it. Yes, yes. But then, if you train a big network, and then you can also 100% of the time, distill it into a small network. 163 00:31:56.080 --> 00:32:02.469 216-101 Large Conference: And then… and then you don't have to do it a thousand times over, it just works. And so then the question is, are these phenomena… 164 00:32:02.820 --> 00:32:03.980 216-101 Large Conference: Related. 165 00:32:05.040 --> 00:32:08.839 216-101 Large Conference: Right? That being lucky, is that the same thing as distilling? 166 00:32:09.520 --> 00:32:12.799 216-101 Large Conference: How would that work if you're just training on the output? 167 00:32:13.450 --> 00:32:17.569 216-101 Large Conference: Oh, because the extra information from distillation is, like. 168 00:32:17.870 --> 00:32:26.659 216-101 Large Conference: like, all the different possible outputs, so you get, like, this continuous output, this distribution from the big network, whereas in the training data, you only get discrete information. Yeah, oh. 169 00:32:26.660 --> 00:32:43.609 216-101 Large Conference: Yeah, so the dissolution, like, teaches you more faster somehow. I feel like it actually makes even more sense if you think of this in the newer, sort of, buying saddle point escapes model. I am gonna, I think, talk about saddle points in here, so don't worry too much if you're like, hmm? 170 00:32:43.610 --> 00:32:57.659 216-101 Large Conference: But, but that, but that if it's sort of just redirecting you a little bit towards a one escape route over another, then it might guide you more towards the, like, better, sort of, 171 00:32:57.710 --> 00:33:02.539 216-101 Large Conference: The better basin that you might end up in, right? 172 00:33:03.020 --> 00:33:05.650 216-101 Large Conference: Okay, so… 173 00:33:06.180 --> 00:33:14.809 216-101 Large Conference: So the variation between training runs is often predictable, because you're… we were talking about this, the mean of the success 174 00:33:14.980 --> 00:33:21.179 216-101 Large Conference: Successful runs tends to increase. The variance of that distribution contract. 175 00:33:21.680 --> 00:33:43.819 216-101 Large Conference: And in these cases of bimodal distributions, the probability of success tends to increase. But you did ask about capacity. There are identifiable points where there's a discontinuous change, like the KL divergence between this and this is massive. The KL divergence between this, this, this, they all look kind of similar. 176 00:33:43.820 --> 00:33:46.410 216-101 Large Conference: But tail divergence between this and this, huge. 177 00:33:46.680 --> 00:33:51.560 216-101 Large Conference: And that's sort of indicating when you've unlocked a specific capability. 178 00:33:52.000 --> 00:33:57.660 216-101 Large Conference: When suddenly the model is actually able to learn the thing, and it wasn't before, at the smaller scale. 179 00:33:58.550 --> 00:34:07.840 216-101 Large Conference: So, we did, like, actually look at multiple-choice question answering because, as I said, that's a sort of intractably emergent case, right? 180 00:34:08.560 --> 00:34:16.039 216-101 Large Conference: So… the, so what we found was that you, you can find that, like, mid-sized models. 181 00:34:16.350 --> 00:34:20.439 216-101 Large Conference: That is, ones that are maybe around an emergence threshold. 182 00:34:20.650 --> 00:34:37.629 216-101 Large Conference: And because I have a lot to go through, I'm not going to go into how… into how we actually trained all of these different, Quinn models. We did not train them from scratch. We re-initialized a chunk of the model and then asked, okay, can it relearn multiple choice? And we… 183 00:34:37.630 --> 00:34:46.820 216-101 Large Conference: varied the amount of multiple-choice data that it had in its, in its dataset until we could get to that threshold. 184 00:34:46.820 --> 00:34:54.930 216-101 Large Conference: Because in addition to the threshold applying to a particular model scale, you know, scaling laws also apply to data scale. 185 00:34:55.190 --> 00:35:20.049 216-101 Large Conference: So, I'm familiar with the setup. And how much of the model do you have to mess up for this? We just do the top layer. Yeah? Yeah. Okay, interesting. Yeah, so… so if you… if it doesn't have enough multiple choice, then it just, like, can't really relearn that structure, even, you know, even though it has the rest of what it needs in the rest of the model, you know? When you say the top layer, is it the input layer or the output layer? 186 00:35:20.050 --> 00:35:25.100 216-101 Large Conference: Oh, outlet, output. But not like the, not the… the unembedding, but the… 187 00:35:25.100 --> 00:35:31.889 216-101 Large Conference: like, actual top layer transforms. Okay. But then you're training everything. You're letting everything train after that point. You're not just retraining. 188 00:35:32.890 --> 00:35:40.220 216-101 Large Conference: So you're… you're… it's sort of like continued pre-training of the model. It's mostly just any language modeling data that it's seeing, but 189 00:35:40.220 --> 00:36:04.260 216-101 Large Conference: a little bit of, multiple choice, like, mixed in to help it get there, so that you're at the threshold. Because if you don't have enough multiple choice, it's very hard for the models to get to that second mode, and it's also not a very good second mode, because it doesn't even hit random variation. It doesn't even hit, like, a random guess on a multiple choice. Everything here hasn't even figured out that you are supposed to say ABCD. 190 00:36:04.680 --> 00:36:17.149 216-101 Large Conference: So here, it's starting to figure out that you're supposed to say ABCD. Here, it's still bimodal. It's still bimodal, but everything has figured out that you're supposed to say ABCD, which is nice, yeah? Did you look at, like… 191 00:36:17.420 --> 00:36:22.429 216-101 Large Conference: Some measures of how similar the last layer is to, like, the previous solution. 192 00:36:22.710 --> 00:36:40.509 216-101 Large Conference: How similar the last layer is to the previous… no, we didn't. We didn't look at that. And also, so, like, related, you could also look at, like, how much distance did the… all the previous layers move, you know, between fines? Like, did it have to change the machinery a lot, or was it able to reuse a lot of it? 193 00:36:41.450 --> 00:36:42.589 216-101 Large Conference: Yeah, I don't know. 194 00:36:43.130 --> 00:37:00.220 216-101 Large Conference: We didn't… we didn't look at the… we actually didn't look at the internals for this. We did look at… I mean, I'll get into some stuff where we do actually look at internals, so, you know, the… because the internals are the reason why. The internals are definite… in fact. 195 00:37:00.220 --> 00:37:10.519 216-101 Large Conference: you know, there was a recent paper from Pavel Ismailope's group at NYU, where they looked at these same cases that we were looking at, these, these bimodal cases. 196 00:37:11.170 --> 00:37:36.149 216-101 Large Conference: And what they found was you could tell which one… like, the… that they… they… they say the reason that some models go to this solution that… that works, and some go to this solution that doesn't work out of distribution, is because they either end up with, like, one strategy for the attention output distribution, or another. And that's actually something we also found in other cases. 197 00:37:36.150 --> 00:37:49.120 216-101 Large Conference: where there are, like, clusters of strategies among the models that are trained is that, is that this is about what modules it picks up on. And that also makes a lot of sense from a, like. 198 00:37:49.120 --> 00:37:51.890 216-101 Large Conference: Theory of Compositionality standpoint. 199 00:37:51.890 --> 00:38:00.479 216-101 Large Conference: like I was saying, if you have multiple modules that have to coordinate, sometimes it's all about whether you really figure out that coordination or not. 200 00:38:00.520 --> 00:38:13.719 216-101 Large Conference: You know? And it's hard to unlearn one strategy that involves one entire structure of a module, and, like, and learn the one that requires coordination between a bunch of different modules. 201 00:38:14.680 --> 00:38:16.290 216-101 Large Conference: So… Yeah? 202 00:38:16.420 --> 00:38:27.269 216-101 Large Conference: One last question. Is the last layer actually doing anything meaningful? Like, what happens if we just skip the last layer, or just replace it with some kind of identity transformation? 203 00:38:27.420 --> 00:38:51.960 216-101 Large Conference: I don't know. Clearly the last layer must be doing something, because otherwise the model, like, all of these models would have been able to just learn, right? I mean, or do you mean after retraining, is it just learning… is it just learning how to skip the last layer, or is it actually learning a new way to handle things in the last layer? Is that… like, or is it, like, relearning the last layer? Like, even before you retrain. 204 00:38:52.030 --> 00:39:08.479 216-101 Large Conference: If you just skip the log layer, what happens? Like, is the log layer doing anything meaningful, or when you… It must be, because otherwise you wouldn't be unlearning it when you re-industrialize the last layer, right? And clearly, if all the difference between this model and this model is what 205 00:39:08.480 --> 00:39:17.579 216-101 Large Conference: what the re-initialized last layer looks like. There must be something going on in the last layer in order to maintain that difference between 206 00:39:17.580 --> 00:39:32.609 216-101 Large Conference: the modes, right? Really? I mean, wouldn't… Arnab is good. Arnab's a good reviewer here. Yeah? Yeah. Yeah. So your setup would be like, oh, you could set it to zero. Maybe if you set the last layer to zero, it would do okay. 207 00:39:32.720 --> 00:39:44.440 216-101 Large Conference: Yeah, kind of. Yeah, I mean, so in our experience, well, in my experience, so this is, this is kind of like a small model, so I'm not really sure. 208 00:39:44.470 --> 00:40:01.869 216-101 Large Conference: But in deeper models… Oh, yeah. Sometimes you can just get past 3 or 4 layers. Oh, it's 5… oh yeah, not too many layers. How many layers does that have? Do you know? I have probably 8 or… 8 or 10. Just 8 or 10? 209 00:40:01.870 --> 00:40:08.870 216-101 Large Conference: That's a lot. Yeah. No, it's a lot to zero, to erase one-eighth of the layers. 210 00:40:09.220 --> 00:40:30.849 216-101 Large Conference: I think, yeah, I think… I mean, I think I agree with you that a really large model is probably, because it's not relying on its last few layers very much, that you probably would… in fact, I mean, I showed… I showed before, even going up to, to 1.5 billion, means that the model is never going to actually 211 00:40:30.850 --> 00:40:36.739 216-101 Large Conference: like, lose everything and have to potentially not be able to relearn it. I fully agree that 212 00:40:36.740 --> 00:40:58.799 216-101 Large Conference: if you're looking at a larger model, it's not enough to just reinitialize the last layer. But fortunately, we weren't focused on, like, questions about layer and stuff. All we were… we were hoping mostly to capture something that's, like, a proxy for pre-training, without having to train 50 versions of a half-billion model with data that we don't actually have access to. But yeah, I mean… 213 00:40:58.980 --> 00:41:05.349 216-101 Large Conference: Okay, sorry for, yeah, yeah. But, I mean, the last layer itself, when you retrain it. 214 00:41:05.880 --> 00:41:14.339 216-101 Large Conference: Potentially, it can, instead of learning something about the task, it can also just learn an identity transformation, and then… 215 00:41:14.520 --> 00:41:25.350 216-101 Large Conference: In this case, it's very clear that it matters what that last layer is re-initialized to, because that's the only difference between the models that don't know it and the models that know it. 216 00:41:26.180 --> 00:41:28.400 216-101 Large Conference: Oh. Oh, okay, yes. 217 00:41:29.280 --> 00:41:32.950 216-101 Large Conference: Cool, thank you. Anyway, yeah, so, 218 00:41:33.280 --> 00:41:47.290 216-101 Large Conference: You know what, I'm gonna… I'm gonna sort of blitz through this one, because I do want to get to some of the other things. We do want to ask, okay, how much, like, how… how are we missing certain bimodal, random variation? 219 00:41:47.600 --> 00:41:58.489 216-101 Large Conference: let me think… how am I gonna, yeah, you know what? Alright, so there's a… if you're familiar with, like, Tom McCoy's little synthetic language modeling toy setups, 220 00:41:58.640 --> 00:42:02.320 216-101 Large Conference: What… what happens is that you train a model. 221 00:42:02.320 --> 00:42:18.709 216-101 Large Conference: on a bunch of in-distribution examples, where you can say, my unicorn does move the dogs that do weight, and then you want to make it into a question, does my unicorn move the dogs that do weight? So you could either say… you could either move the first verb you see. 222 00:42:18.820 --> 00:42:29.159 216-101 Large Conference: Or you could move the head of that constituent, and it would be the same ver… in distribution. Then we look at a out-of-distribution dataset. 223 00:42:29.160 --> 00:42:50.419 216-101 Large Conference: Where, if the model follows this linear rule where it just moves the first verb, it gets it wrong. Doesn't my unicorn who sings does move? That doesn't make any sense. But it can identify… if it identifies that does is the head of that constituent, the top level verb, then it can, correctly move it. So these are… this is the linear rule, or the hierarchical. 224 00:42:50.420 --> 00:42:53.840 216-101 Large Conference: Rule for accomplishing, question formation. 225 00:42:53.920 --> 00:43:10.140 216-101 Large Conference: So, the question is, do models tend to prefer hierarchical rules for handling natural language? So, in training, you're saying that you train on data that you can't tell, or do you train on data that's always the linear rule, but… 226 00:43:10.140 --> 00:43:19.879 216-101 Large Conference: The training data, you cannot tell whether it's going to… If it's linear or hierarchical. Yeah, you… because the rules lead to the same output. 227 00:43:19.880 --> 00:43:36.249 216-101 Large Conference: Because the head of… the head verb is also the first verb that you see as you're moving through the sentence, so you can just grab it and move there, right? But if the model, because it's seen a bunch of other language data, that it's just copying over… Oh, so you're fine-tuning… 228 00:43:36.250 --> 00:43:52.469 216-101 Large Conference: a model on this task? This is trained from scratch, but there's actually, like, two things that are mixed together. One of them is question formation, and one of them is just copying… copying a sentence. So, which essentially is just language modeling, right? So, if it learns that… 229 00:43:52.530 --> 00:44:04.979 216-101 Large Conference: that language tends to have some hierarchical structure to it, then it might decide that the rules that it applies here should also use hierarchical structure. 230 00:44:05.030 --> 00:44:21.399 216-101 Large Conference: And if it doesn't pick up on that hierarchical structure from just language modeling, then it will probably not learn the… it won't use the hierarchical rule by default. This is a, like, inductive bias kind of question. So… 231 00:44:22.190 --> 00:44:24.620 216-101 Large Conference: What we found? Hmm. 232 00:44:25.180 --> 00:44:26.200 216-101 Large Conference: So… 233 00:44:26.200 --> 00:44:46.729 216-101 Large Conference: We found that there's, like, that in this, like, extra language modeling data that we see, the thing that determines whether it's going to actually end up using a hierarchical rule over a linear rule is whether it sees a bunch of center embeddings, which are… which are structures like this, where you have to keep track of a subject that was way back here, unicorn. 234 00:44:47.050 --> 00:45:04.339 216-101 Large Conference: who doesn't wait, does, oh, does, connects to unicorn, and there's this whole thing in between. But these are, like, relatively rare in English. Most of the time, you're just going, does entertain, return source, that doesn't smile, doesn't smile. You don't have to keep tracking all the way back. 235 00:45:04.340 --> 00:45:28.010 216-101 Large Conference: Right? You can just keep branching, and so this is essentially, you could just treat it as though it's, like, a linear thing, as though there's no tree structure at all, and it would look the same, because you're always traveling in one direction. You're never backtracking in that tree, recursing down, popping your stack, moving to the next point, right? Which is what you need to do for center embeddings. So exposure to these center embeddings is the thing that decides whether the model is going to follow 236 00:45:28.010 --> 00:45:29.499 216-101 Large Conference: A hierarchical rule. 237 00:45:31.330 --> 00:45:41.439 216-101 Large Conference: And the more, like, hierarchical rule… the more centered beddings it sees, the more likely it is to generalize hierarchically in this way. 238 00:45:41.920 --> 00:45:48.109 216-101 Large Conference: So, here's where we come back to our you-know-it-or-you-don't 239 00:45:48.270 --> 00:46:05.610 216-101 Large Conference: where does the random variation hide? So this, this looks kind of unimodal in terms of, like, the hierarchical rule behavior, if you just look straightforwardly at the model. But what's actually happening is, if it sees a bunch of center embeddings. 240 00:46:06.020 --> 00:46:13.249 216-101 Large Conference: it'll end up hierarchically generalizing. If it doesn't see enough center embeddings, it'll linearly generalize. 241 00:46:14.710 --> 00:46:27.710 216-101 Large Conference: Whoop. If it doesn't see diverse enough data generally, it'll memorize a bunch of stuff, which is a little less stable, but still pretty stable. Or if it sees, like, if it doesn't see enough 242 00:46:27.880 --> 00:46:33.930 216-101 Large Conference: center embeddings and also not enough diversity, it still memorizes, it just looks more like the linear case. 243 00:46:34.180 --> 00:46:38.960 216-101 Large Conference: If it's in between, which again, if it's at a threshold. 244 00:46:39.160 --> 00:46:44.569 216-101 Large Conference: Then training becomes unstable in terms of which generalization approach it takes. 245 00:46:44.580 --> 00:47:00.889 216-101 Large Conference: It oscillates between hierarchical and linear generalization, and it cannot pick, even, like, late in training, you know, this depends on what kind of regularizer, what kind of decay you're using. Eventually, maybe you can force it to settle into something or not. 246 00:47:01.280 --> 00:47:11.329 216-101 Large Conference: But the end result is that you still have this bimodal distribution, but it's hiding among only the runs that have stabilized. 247 00:47:12.110 --> 00:47:28.579 216-101 Large Conference: So you can see, over here, this x-axis is showing how stable the run is in terms of, how it's generalizing throughout training, right? So the ones that are oscillating are over here, the ones that sort of have settled in. 248 00:47:28.940 --> 00:47:31.389 216-101 Large Conference: Know what their rule is. 249 00:47:31.850 --> 00:47:39.610 216-101 Large Conference: They're over here. And so you can see that training runs stabilize when they settle into this bimodal distribution. 250 00:47:39.940 --> 00:47:49.429 216-101 Large Conference: And if they… and the ones that haven't settled into the bimodal distribution, it's actually because they haven't been able to finish training and settle down. 251 00:47:50.570 --> 00:47:51.750 216-101 Large Conference: So… 252 00:47:52.490 --> 00:48:10.410 216-101 Large Conference: It turns out that if we… if we stabilize training, if we make sure that it's actually, like, stable, then we can see that that bimodal distribution is, is holding, because these are discrete rules where you know it or you don't, right? So, if the training is stable. 253 00:48:10.560 --> 00:48:12.760 216-101 Large Conference: It ends up being bimodal. 254 00:48:14.410 --> 00:48:30.879 216-101 Large Conference: Okay, so even when training is stable, you might have secret breakthroughs that are happening. And, and this is another arithmetic addition task that I'm going to show this on, where there are two things that a model really needs to learn in order to predict any given numeral, right? 255 00:48:30.880 --> 00:48:38.359 216-101 Large Conference: It's… it needs to know what… what… how to handle that digit, the tenth place, the hundredth place, whatever, and it needs to know 256 00:48:38.500 --> 00:48:43.500 216-101 Large Conference: whether it's carrying a digit or not, right? So… 257 00:48:43.680 --> 00:49:01.559 216-101 Large Conference: If you just cluster the lost curves, you can see it learns these different digits at different times. Some of them are slower than others, some of them have a earlier phase transition or breakthrough than others. You can see, you can cluster these, and you can recover the digits. They are very different. 258 00:49:03.300 --> 00:49:20.429 216-101 Large Conference: But that's not true of the carrying scale. Like, there's not really a clear separation, for the most part, between cases where you're carrying a digit and cases where you're not carrying a digit, for, you know, any… any given, like, 259 00:49:20.690 --> 00:49:32.870 216-101 Large Conference: any given place in the digits, yeah? Question about the skill. Yeah. In this case, was it… I'm assuming that this was, like, a human-defined skill, or maybe the way they did the arithmetic could be not in any of those? 260 00:49:33.240 --> 00:49:51.160 216-101 Large Conference: That's true, but I think it's not unreasonable to, like, and I agree, but the fact that… this is why I like to define skills in terms of the loss curve. If you can find where the model learned this, then you can say that's a discrete skill to the model. 261 00:49:51.160 --> 00:49:53.859 216-101 Large Conference: And in this case, it looks like… it looks like… 262 00:49:54.190 --> 00:50:08.659 216-101 Large Conference: which place you're in, the index is a discrete scale, and carrying isn't, right? Because we… when we cluster, we can only really recover, like, clusters that are associated with specific digits. Here's the thousands and hundreds, here's the tens and ones, right? 263 00:50:09.830 --> 00:50:18.519 216-101 Large Conference: Yeah? Yeah, I'm not really sure if I'm following the setup properly, but, what I could gather is that 264 00:50:18.520 --> 00:50:25.000 216-101 Large Conference: When the model is solving a particular task, it can approach that in two different pathways, or two different scales. 265 00:50:25.000 --> 00:50:39.569 216-101 Large Conference: And when you… It has to simultaneously be able to do all these skills, or at least, you know, it only has to do tens… tens or… it has to only do one of, like, the tens, hundreds, whatever. Like, those are distinct skills, and it only has to use one at a time. 266 00:50:39.630 --> 00:50:45.640 216-101 Large Conference: But it might need to carry a digit or not for any of those cases, right? 267 00:50:46.020 --> 00:50:54.950 216-101 Large Conference: So one of these is, like, a mutually exclusive group of skills, and the other is not. Okay, or the other is not mutually exclusive with that group, rather. 268 00:50:55.580 --> 00:51:01.970 216-101 Large Conference: Okay, and when you show it, different dataset distributions. 269 00:51:02.430 --> 00:51:11.359 216-101 Large Conference: like, targeted for one kind of scale, versus another kind of scale, you do see stability, but when you mix them up, you see unstability. 270 00:51:12.010 --> 00:51:16.389 216-101 Large Conference: Right. If you see, 271 00:51:17.190 --> 00:51:33.739 216-101 Large Conference: No, we're not talking about stability anymore right here. We're talking about secret phase transitions in the learning dynamics. Okay, okay, my question was about, kind of, like, stability. Okay. Stability, we… I can go all the way back to where we were talking about stability, which is a different setting. 272 00:51:34.110 --> 00:51:37.049 216-101 Large Conference: Yes, this setting. Yeah. 273 00:51:37.320 --> 00:51:47.009 216-101 Large Conference: Okay, but these models, they're kind of, like, distributed systems, right? So you can kind of, like, imagine… 274 00:51:47.360 --> 00:51:54.800 216-101 Large Conference: they might have the capability to actually maintain these different pathways, these different skills. 275 00:51:55.250 --> 00:51:59.440 216-101 Large Conference: I agree, and I think that's what's happening when they can't decide. 276 00:52:00.860 --> 00:52:17.239 216-101 Large Conference: Yeah… These are… these are mutually exclusive. The model has to… has to decide whether it's going to generalize this way or this way. These aren't, like, like, sometimes you're going to use this, and sometimes you're going to use this, and that's okay. Either of them… these are two different rules to apply to the same data. 277 00:52:18.170 --> 00:52:28.939 216-101 Large Conference: Yeah. The question is, is the model using this rule or that rule, or is it not really using a consistent rule because it's unstable? Ardi, are you saying that your intuition is that 278 00:52:29.350 --> 00:52:33.460 216-101 Large Conference: You know, as distributing the systems, these models 279 00:52:33.700 --> 00:52:36.260 216-101 Large Conference: Have a natural tendency to be able to 280 00:52:36.760 --> 00:52:43.880 216-101 Large Conference: apply more than one rule in parallel? I think so. But also, like, in this setup, 281 00:52:44.750 --> 00:53:03.650 216-101 Large Conference: Do you think that, like, I'm trying to understand what your sense is, like, do you think that if you train it for long enough, the model will have some way to, basically decide, okay, if I see this kind of distribution, I'm going to apply this rule. If I see this kind of distribution… No, here, here. 282 00:53:03.900 --> 00:53:11.909 216-101 Large Conference: the same thing, it's only… it's either applying this or this. It's the same data point. So, if you think about it, it's… the model is saying. 283 00:53:11.930 --> 00:53:17.610 216-101 Large Conference: If the model decides it's going to hierarchically generalize, it doesn't need this other rule. 284 00:53:17.620 --> 00:53:26.229 216-101 Large Conference: Because this rule will never apply in that case. Whenever it sees something that it can either generalize hierarchically or linearly. 285 00:53:26.230 --> 00:53:42.400 216-101 Large Conference: it's… it could pick one or the other rule. It's never going to be an advantage to be sort of simultaneously trying to do both. So, if it's efficient, it'll prune out the rule that it doesn't need, and sort of pick one or the other. 286 00:53:42.840 --> 00:53:58.449 216-101 Large Conference: Yeah, that's kind of like a property of regularization, right? Yes. But I mean, sometimes they keep maintaining these different pathways, right? When the training is unstable. When the training is unstable. It might be task-dependent. There might be some tasks… 287 00:53:58.940 --> 00:54:04.600 216-101 Large Conference: where… You know, there are different variations in the task that keep on showing up. 288 00:54:05.020 --> 00:54:09.030 216-101 Large Conference: In the training data, where there's an advantage for doing it one way or the other. 289 00:54:09.310 --> 00:54:15.280 216-101 Large Conference: Right? But, yeah, I think, I think this, like, my intuition is this task was chosen 290 00:54:16.910 --> 00:54:30.290 216-101 Large Conference: to have this subset property where, like, if you can do it one way, then you don't need to do it the other way. A regularizer would knock it out. Okay, even if I do it in multiple different ways, different ways will actually get me to the accurate answer. 291 00:54:30.290 --> 00:54:38.159 216-101 Large Conference: Right? No. So here… so, I mean, yes, either of those rules can bring you to the right answer, but… 292 00:54:38.190 --> 00:54:55.220 216-101 Large Conference: if you try to do… but then… then you're doing it inefficiently. You don't need to do both rules in order to get the correct answer. You only need one rule, and you'll always get the right answer on the training set, right? But I mean, at the architecture level, how the model works. 293 00:54:55.250 --> 00:55:12.050 216-101 Large Conference: it is very parallel, right? At the same layer, different attention heads don't look at what other heads are doing. Same logic applies to the neuron level as well, right? The actual training, if it's stable, if there's enough regularization. 294 00:55:12.050 --> 00:55:29.209 216-101 Large Conference: the sort of, like, good training leads to high compression. And the most compressed way to accomplish this task is not to simultaneously apply two different strategies, only one of which is ever going to… like, only one of which is ever needed. 295 00:55:29.210 --> 00:55:39.420 216-101 Large Conference: In order to accomplish it. The compressed, efficient way of doing it is to pick one strategy and stick with it, and throw out the other one, and you can use that space for something else, or for nothing. 296 00:55:39.670 --> 00:55:40.850 216-101 Large Conference: Right? 297 00:55:41.110 --> 00:55:59.059 216-101 Large Conference: Okay, cool. Now, we should… we should have a conversation about this maybe afterwards. Yeah, yeah. But we'll… so we'll… but, like, you know, we'll point you at… there's, there's a work that was… was happening in the lab called In-Context Algebra, where inspired by some of your work. 298 00:55:59.100 --> 00:56:03.860 216-101 Large Conference: We, you know, we looked at training dynamics, among other things, right, and… 299 00:56:03.940 --> 00:56:08.870 216-101 Large Conference: And, and I… and I… but on a… on a task that maybe had some different characteristics. 300 00:56:09.150 --> 00:56:20.450 216-101 Large Conference: And one of the things that they saw was this persistence of having multiple mechanisms at the same time that weren't, you know, sort of eliminating each other. 301 00:56:20.530 --> 00:56:33.799 216-101 Large Conference: Yeah. On really complex data, I would believe that you might have multiple mechanisms that are useful in different scenarios, or it might be that you're not actually catching it fully at the end of its training run, or 302 00:56:34.110 --> 00:56:37.640 216-101 Large Conference: It might be that just, like, in real world… 303 00:56:37.670 --> 00:56:54.400 216-101 Large Conference: models, there's not enough regularization to get to the most compressed representation. Yeah, I would love to talk about that, because I love cases where there's not enough regularization in real-world models to get to these things, because it means that you might have more random effects. 304 00:56:54.400 --> 00:57:09.459 216-101 Large Conference: Because there are things that… that aren't important in distribution, but then become very important later on, and some of the things that become important out of distribution, or, like, for compositional generalization, whatever, are, vestigial features. 305 00:57:09.460 --> 00:57:27.319 216-101 Large Conference: So, you might have a strategy the model follows early in training, and then sort of, like, abandons later, but it sort of resurfaces when it's exposed to, like, weird new distributions. And I love those cases because, as you can see, I'm really into random variation right now, so… 306 00:57:27.360 --> 00:57:35.460 216-101 Large Conference: So anything that might be a little weird because of something weird that happens in the model is, yeah, let's talk about it later. Okay, yeah? 307 00:57:35.560 --> 00:57:39.130 216-101 Large Conference: Particularly with respect to this, like. 308 00:57:39.340 --> 00:57:45.259 216-101 Large Conference: I guess, like, turbulence and, like, the redundant mechanism in the model. How… 309 00:57:45.840 --> 00:57:53.440 216-101 Large Conference: Relevant to this, do you think, is specifically the fact that, like, you know, the dominant mode is residual networks nowadays. 310 00:57:53.520 --> 00:58:08.010 216-101 Large Conference: Specifically because, like, you know, the gradients, it goes back to every layer, right? To every… so, you know, if multiple… if you have the same, like, kind of thing about multiple depths, like, it just incentivizes learning the same thing multiple times. Yeah, well… 311 00:58:09.380 --> 00:58:20.079 216-101 Large Conference: Yeah, I… I… I mean, I think that that's, like, a really interesting extra thing, because it's also… there's a lot of advantage to learning the same thing multiple times. 312 00:58:20.080 --> 00:58:36.920 216-101 Large Conference: Because, like, for something like multiple choice, right? You don't necessarily actually know how to answer the question in layer 2 if you happen to have your formatting in Layer 3, right? Sometimes you can immediately jump through, like, you can handle the formatting at that layer, and then, like. 313 00:58:36.990 --> 00:58:50.009 216-101 Large Conference: just skip the rest of the layers, but sometimes it's gonna, you know, take you a bit more reasoning to figure out the actual answer before you can compose it with the format. So it's really advantageous, actually, to have multiple copies like that. 314 00:58:50.010 --> 00:59:00.559 216-101 Large Conference: And in fact, it allows you to do, effectively, recurrences internally, which is something that you see in, like, vision models, for instance, that there are things that should be handled with recurrences. 315 00:59:00.560 --> 00:59:16.609 216-101 Large Conference: And we say, well, you know, we're using transformers, they can't do recurrences. Well, in practice, as long as you're not more… you don't need more than, like, you know, L of recurrences, you're actually… you actually might find that the model is secretly doing recurrences internally. 316 00:59:17.370 --> 00:59:21.029 216-101 Large Conference: From layer to layer. Anyway, okay. 317 00:59:21.300 --> 00:59:44.840 216-101 Large Conference: Okay, I'm gonna try to get through this one fast, but basically the idea is you might see a phase transition in one direction of the model's movement during training, and it disappears when you look at the full loss curve, or even in a subset of the data, but it disappears in the full loss curve, because as you add all of these different breakthroughs together, they start to look very, very smooth. 318 00:59:44.840 --> 00:59:55.900 216-101 Large Conference: If you have a model that's simultaneously moving along in direction B1 and V2, then it might look very, very smooth, even though each of those directions 319 00:59:55.900 --> 01:00:01.610 216-101 Large Conference: has its own little breakthrough that's happening in the middle, right? So… 320 01:00:02.360 --> 01:00:14.189 216-101 Large Conference: So, the idea here, I'm not going to go into the… how we construct the basis, but the idea is that if we look at a sort of decomposed 321 01:00:14.460 --> 01:00:24.230 216-101 Large Conference: version of training, and we ask, what is happening due to movement in that direction? What is happening due to movement in this direction? Well, the model is going in the… this… 322 01:00:24.230 --> 01:00:38.969 216-101 Large Conference: composite direction in reality, and we can ask whether we can cluster the loss curve in these individual decomposed directions. And when we look at that, okay, so 323 01:00:39.130 --> 01:00:42.579 216-101 Large Conference: When we look at that, if we look at, you know, the first 324 01:00:42.730 --> 01:00:50.220 216-101 Large Conference: vector, which is constructed from the Hessian. We get the basis out of the Hessian, but there's details to it. And in fact. 325 01:00:50.450 --> 01:00:57.600 216-101 Large Conference: I think someone should do a better one. Better basis, I mean. I think it worked here, but it worked… 326 01:00:58.060 --> 01:01:22.979 216-101 Large Conference: It worked okay. I think if we had a better way of constructing the basis, it would work really well. Okay, so if we just look at that for a sigenvector, you can see you kind of, like, have these clusters that… is this a decomposition of weight space, or is it… Yes. Like the… it's the weight space decomposition. So it's effectively, like, looking at the entire gradient, all of the possible gradients, and you're decomposing all of the 327 01:01:22.980 --> 01:01:23.950 216-101 Large Conference: actions. 328 01:01:23.950 --> 01:01:30.989 216-101 Large Conference: within that entire space of possible gradients, right? But you're looking at the directions where training is actually happening. 329 01:01:31.630 --> 01:01:36.529 216-101 Large Conference: Like, because a lot of that space, there's nothing that's really happening. 330 01:01:36.660 --> 01:01:50.980 216-101 Large Conference: Right? So, okay, when we look at that, those, directions, the first eigenvector we look at, yeah, it shows clusters that correspond to some of the digits, right? It breaks out the thousands from the other digits. 331 01:01:51.020 --> 01:01:59.340 216-101 Large Conference: And then if we look at the next decomposed little basis vector, you can see that you end up with a cluster that starts to reflect 332 01:01:59.590 --> 01:02:02.610 216-101 Large Conference: Whether you're carrying a digit or not. 333 01:02:02.920 --> 01:02:08.960 216-101 Large Conference: So, it seems that carrying is actually something that is distinct to the model. 334 01:02:10.340 --> 01:02:16.830 216-101 Large Conference: This is cluster 1, so you can see it's very distinct in its loss curve, projected onto this direction. 335 01:02:16.980 --> 01:02:23.370 216-101 Large Conference: So, you can see there's, like, an actual phase transition that corresponds to when it learns to do carries. 336 01:02:23.520 --> 01:02:30.400 216-101 Large Conference: So this is neat because it means that there's a secret carrying skill that's just not reflected at the top 337 01:02:30.470 --> 01:02:44.180 216-101 Large Conference: Like, in the aggregate loss across the entire gradient space, you don't, but if you narrow your vision down, you can find that skill, and it's got a very distinct curve. 338 01:02:44.380 --> 01:02:49.480 216-101 Large Conference: Okay, so I said we would be using these to understand models better? 339 01:02:50.150 --> 01:02:56.150 216-101 Large Conference: And what I really mean by understand models in this case is predict behavior. 340 01:02:56.750 --> 01:03:11.870 216-101 Large Conference: Using their representations. So, we saw this. What would this look like in attention? What would a hierarchical structure rule look like in attention? It might look a lot like your model focusing on the head of a constituent. 341 01:03:11.990 --> 01:03:20.739 216-101 Large Conference: So, if the model… even if you look at an in-distribution case, even if you look at a case where it does not matter what rule the model is executing for its output. 342 01:03:20.870 --> 01:03:39.180 216-101 Large Conference: You can look inside the model and say, what are you basing this on? And in this case, this is a… this is not question formation, this is another task, which is inflection, so it's deciding what number this verb takes. So, if the model looks at an in-distribution case, this does not differentiate them. 343 01:03:39.230 --> 01:03:48.190 216-101 Large Conference: Bad visualization. Basically, if it looks at the in-distribution case and the keys that unlock the cabinets, and it's deciding is or are. 344 01:03:48.420 --> 01:04:00.749 216-101 Large Conference: Well, these are the same, so we're not going to get a difference, depending on whether it's looking hierarchically or linearly. We just say cabinets, the last noun it saw, or keys, the subject. 345 01:04:01.000 --> 01:04:07.289 216-101 Large Conference: But, if it's hierarchical, inside the model, you're going to find attention that's focused on the subject. 346 01:04:07.450 --> 01:04:15.530 216-101 Large Conference: Because that's what it's basing its decision on, even though the final decision will look the same as, in this case, where it's focusing on cabinets. 347 01:04:15.730 --> 01:04:21.350 216-101 Large Conference: Both cases, it's generating the same thing. P's, cabinets, they both would say R, not is. 348 01:04:21.650 --> 01:04:29.020 216-101 Large Conference: But in one case, the model is thinking, what's my subject? And in the other, it's, what's the last noun I saw? 349 01:04:29.160 --> 01:04:30.759 216-101 Large Conference: And sure enough. 350 01:04:31.580 --> 01:04:38.500 216-101 Large Conference: If you look for hierarchical heads in this sense of that they're looking back at the subject. 351 01:04:38.750 --> 01:04:48.460 216-101 Large Conference: It tells you whether it's going to have high accuracy on these cases where it needs to use a hierarchical rule, or low accuracy. 352 01:04:49.030 --> 01:05:03.230 216-101 Large Conference: So you can look at the model's internals, where it's not going to be making any, like, different decisions, and predict what it's going to do out of distribution, just looking at in-distribution representations. 353 01:05:03.370 --> 01:05:05.470 216-101 Large Conference: Whether it knows it or it doesn't. 354 01:05:05.690 --> 01:05:16.270 216-101 Large Conference: And this is something that I've really been focused on in my interpretability work lately, because I'm a bit of a contrarian, and I've decided I don't care about causal mechanisms. 355 01:05:16.730 --> 01:05:27.710 216-101 Large Conference: That causal mechanisms… I'm sorry! Causal mechanisms. 356 01:05:27.970 --> 01:05:36.720 216-101 Large Conference: Yeah, they… you can make testable predictions of what's going to happen during an intervention on that one mechanism, but but… 357 01:05:37.530 --> 01:05:47.189 216-101 Large Conference: a bunch of isolated mechanisms that you can, like, separate out. These interventions don't necessarily isolate those mechanisms. You can have backup mechanisms that take over. 358 01:05:47.390 --> 01:06:08.879 216-101 Large Conference: Targeted interventions are really problematic in a lot of areas of science. If you… if you, if you do an intervention on DNA, sometimes it doesn't have an effect, and then when you do it on the RNA, it does. And that's because there's, like, a sort of backup mechanism that takes over and repairs things you've done to the DNA before it turns into RNA. 359 01:06:08.880 --> 01:06:19.579 216-101 Large Conference: All over science, you see problems with targeted interventions for testing your hypotheses that make it so that sometimes you're better off doing correlations or doing other things. 360 01:06:19.580 --> 01:06:20.340 216-101 Large Conference: So… 361 01:06:20.580 --> 01:06:31.020 216-101 Large Conference: there's effects that these kinds of interventions can have, and you really have to think about that. So what… what would it mean to think about, like, interpreting a model holistically, as a whole? 362 01:06:31.220 --> 01:06:34.859 216-101 Large Conference: Not each individual mechanism, but the whole thing. 363 01:06:35.940 --> 01:06:49.559 216-101 Large Conference: It wouldn't matter whether it's responding a particular way to an intervention internally or not. What would matter is whether the model then ends up having the predicted behavior on different inputs. The intact 364 01:06:49.570 --> 01:07:02.120 216-101 Large Conference: unencumbered by your clumsy interventions on steering and patching and so on. What is that model going to do on some inputs that you haven't seen, right? 365 01:07:02.120 --> 01:07:13.119 216-101 Large Conference: If we don't just care about causal mechanisms, and we actually care about, sort of, the algorithmic or even computational level of understanding of a model. 366 01:07:14.150 --> 01:07:27.390 216-101 Large Conference: Then we… oh, wait, this is, this is going into another setting, which I meant to delete. Okay, then we can actually, like, start thinking about these really useful predictions. 367 01:07:28.380 --> 01:07:40.150 216-101 Large Conference: What does that give us? It would maybe let us efficiently test models. If you know what the fault tolerances of a system is in engineering, or you know what edge cases you should be concerned about. 368 01:07:40.150 --> 01:08:02.759 216-101 Large Conference: then you can say, don't… like, then instead of, you know, saying, I'm going to test this car by kicking it a little, and by driving it into a volcano. No, you can say, okay, this car should be able to handle this speed, it should be able to, you know, have such and such force on the side, such and such force at the front. 369 01:08:02.760 --> 01:08:05.850 216-101 Large Conference: And then you can actually, like, take it apart and test 370 01:08:05.850 --> 01:08:15.329 216-101 Large Conference: the fault tolerances that you're trying to guarantee, right? And you can ask, what are the likely situations where that car will 371 01:08:15.360 --> 01:08:35.409 216-101 Large Conference: fail that we should be actually concerned about, right? So if we have that, we can sort of use it to design RAG triggers, we can use it for active learning, we can just say, don't try to have this be a therapist if it's speaking Hungarian, it's just not going to be good at it in a Hungarian. 372 01:08:35.410 --> 01:08:37.910 216-101 Large Conference: We can have these kinds of things, right? 373 01:08:38.060 --> 01:08:40.140 216-101 Large Conference: And and… 374 01:08:40.420 --> 01:08:49.229 216-101 Large Conference: And the question then is, can we actually do this in real language models? Like, this seems like a beautiful dream. So we have this new preprint out. 375 01:08:49.350 --> 01:08:56.540 216-101 Large Conference: That's about, sort of, predicting whether a model will be able to systematically compose 376 01:08:56.540 --> 01:09:20.040 216-101 Large Conference: specific atomic concepts. So, what do I mean by systematic composition? If you're not familiar, the idea is if a model is seeing a leopard and a black chair, a leopard print leopard, importantly, and a black chair, then if it later sees a leopard print chair and a black leopard, is it going to be able to handle these things, right? Is it going to be able to make its, like, little swaps and still interpret everything appropriately? 377 01:09:22.300 --> 01:09:44.830 216-101 Large Conference: So, the model we're gonna use, our sort of model of the model that we're going to use to make these predictions, is that in order to compose these atomic concepts effectively, you have to be able to compress and recover the relevant things about the input. So, if it's doing all this leopard print infill on the chair, it also needs to remember what a chair looks like. 378 01:09:44.830 --> 01:09:52.500 216-101 Large Conference: Right? If it's busy with the leopard print and it loses all that chair information, then it's going to have performance issues, right? 379 01:09:52.540 --> 01:10:09.090 216-101 Large Conference: So how many directions do you actually need in order to sort of, compress effectively all of these different features you care about in different ways, assuming that linear representation hypothesis? We're assuming that everything is linear, right? What happens? In 1990, 380 01:10:09.140 --> 01:10:23.729 216-101 Large Conference: Paul Smolensky was like, well, you can do this as long as you have… you can store k linear features and do all of these variable binding operations and everything, as long as you have them all orthogonal, so you need K directions for K features. 381 01:10:23.880 --> 01:10:28.070 216-101 Large Conference: But in 2006, Candace Romberg and Terence Tao 382 01:10:28.140 --> 01:10:34.060 216-101 Large Conference: Said, actually, you can have, like, a logarithmic number of directions 383 01:10:34.060 --> 01:10:58.970 216-101 Large Conference: like, so you can store exponentially more linear features in a certain number of directions with much weaker assumptions than actual orthogonality. You just need to be able to assume that they're, like, a certain amount of separation. The key is assuming sparsity. So that's what allows superposition, right? If you assume that all of your features are sparsely activated, then it means that you can still store all of the possible combinations of 384 01:10:58.970 --> 01:11:06.440 216-101 Large Conference: features separately without having, without having to actually, like, force them all to be orthogonal. 385 01:11:06.490 --> 01:11:12.910 216-101 Large Conference: So… This is great. We can store exponentially many features in however many directions. 386 01:11:13.080 --> 01:11:15.639 216-101 Large Conference: Like, if there's not… if there's no nuts! 387 01:11:15.700 --> 01:11:35.520 216-101 Large Conference: In the ideal model, in the perfect recovery case, it has access to as far as basis pursued. You know, this isn't how models really work. If we're realistic and there's noise and so on, then we can have actual interference between the features again, and that affects the robustness of recovery. 388 01:11:35.690 --> 01:11:45.619 216-101 Large Conference: So, it actually is nice to have our nice orthogonal features, and there are issues with storing everything in superposition and trying to recover them like that. 389 01:11:48.060 --> 01:11:58.979 216-101 Large Conference: If the model is able to make assumptions about what features actually co-occur, then it doesn't have to worry so much, and it can still robustly recover all of these inputs. 390 01:11:59.110 --> 01:12:08.160 216-101 Large Conference: But, if it's wrong about which features co-occur, which is to say, if there is a novel systematic composition of features. 391 01:12:08.430 --> 01:12:25.480 216-101 Large Conference: then it might run into issues, right? It might lose information. So, the question is, can we use this sort of model of the model, this idea that there is a noisy, structured, super-positioned 392 01:12:25.480 --> 01:12:32.569 216-101 Large Conference: encoding, and the model, has to recover those inputs through that. Can we then 393 01:12:32.600 --> 01:12:48.279 216-101 Large Conference: predict which concept combinations are going to, have issues. And we can do this without actually knowing what they look like when they're combined. We don't need an example of an input in order to make the position… the, this, like, 394 01:12:48.340 --> 01:12:57.699 216-101 Large Conference: this composition prediction, right? We don't need to know that TAT in Hungarian is… I don't know, does anyone speak Hungarian? 395 01:12:57.880 --> 01:13:04.009 216-101 Large Conference: Okay, that's great. Let's say it's much like that. Okay, so let's say… 396 01:13:04.650 --> 01:13:23.320 216-101 Large Conference: Is it going to know how to save that? We won't have to actually hand it this input, we just need to say, okay, let's… like, is it going to be able to handle this combination, right? So, that's cool if it's expensive to collect data, and cheap to multiply two vectors together, which it is. 397 01:13:23.970 --> 01:13:34.650 216-101 Large Conference: So, the way that we would do it is we would look at the angle between Hungarian and, dolphin, which is a very, very small angle, right? They're, like, there's a lot of interference here. 398 01:13:34.790 --> 01:13:41.529 216-101 Large Conference: So it's red. And then we look at a cat versus French, well, that's gonna compose really nicely. That's… 399 01:13:41.530 --> 01:13:52.779 216-101 Large Conference: Bright green, because it's basically orthogonal, right? And so then we take all of these combinations and rank them according to, how hard we think they're going to be for the model. 400 01:13:52.780 --> 01:14:11.840 216-101 Large Conference: And then we're gonna say, good ranking if including a smaller, if restricting us to a smaller angle means that we have high, like, worse model performance, ultimately, than if we allow it to look at everything else, right? So the more this is sort of, like, below this line. 401 01:14:12.330 --> 01:14:30.920 216-101 Large Conference: when it's, when… as we add more and more data, the better our ranking is, right? We also look at PRAUC, don't worry, we look at, like, actual things. I'm just visualizing for you, right? So, what happens if we look at multi-hop question answering? So, you know, what year was the author of 1984? So, I'm gonna… 402 01:14:30.920 --> 01:14:40.590 216-101 Large Conference: So don't worry about time. Oh, okay. But I, I want to understand this scenario a little bit better. Absolutely. Go back and… Yes. So, so you're… you're trying to… 403 01:14:40.830 --> 01:14:45.030 216-101 Large Conference: the inputs and outputs. So you're… you're trying to predict 404 01:14:45.700 --> 01:14:49.829 216-101 Large Conference: How much data, or how hard it will be for a model to learn some… 405 01:14:50.110 --> 01:15:13.500 216-101 Large Conference: part of the composition, is that right? On a learned model, how likely is it to make a mistake? Well, how much error am I? We could predict the model's mistakes from its internal representations. This is in a fully trained language model, this is in, like, Llama 3. Okay, I see. Yeah. We used a language model that makes enough mistakes that we can make significant claims. So, and the question is, can we predict 406 01:15:13.500 --> 01:15:21.820 216-101 Large Conference: when it'll make a mistake, without having access to the actual inputs that we're about to put through the model to make that mistake. You have some representation vectors. 407 01:15:21.820 --> 01:15:27.500 216-101 Large Conference: And from the representation vectors alone, you're trying to make accuracy predictions. 408 01:15:27.650 --> 01:15:28.690 216-101 Large Conference: Exactly. 409 01:15:28.690 --> 01:15:49.010 216-101 Large Conference: Yeah, we're trying to… so it's a… it's a… like, so the relevant thing is, like, it's like, you would measure this with, like, precision recall, you know, that we… we would say, okay, we're, like, ranking these according to how likely the model is to make an error, so if you're treating it as, like, there's a binary class of it makes a mistake or it doesn't make a mistake. 410 01:15:49.160 --> 01:15:52.870 216-101 Large Conference: Then, how good is our classifier that ranks them? 411 01:15:52.900 --> 01:16:11.519 216-101 Large Conference: In this way, so then you could look at, like, the PRAUC for, like, the actual metric, but… but you can also see in the actual curve of, like… I see. I see. And it's all… it's in all these compositional scenarios. Yes. So you're… so you're taking vectors on one axis of the composition, vectors on the other axis of the composition, and then… 412 01:16:11.520 --> 01:16:22.350 216-101 Large Conference: You're… you're going to… so it sounds like you're going to assert that some property of these vectors, like the angle or something, is predictive of how accurate 413 01:16:22.630 --> 01:16:24.030 216-101 Large Conference: the model… 414 01:16:24.380 --> 01:16:38.899 216-101 Large Conference: Exactly. So, when there's more linear interference, because the angle is closer, the model is more likely to make a mistake trying to compose those. Oh, interesting. And if they're close to orthogonal, then the model is likely to be able to 415 01:16:38.950 --> 01:17:01.179 216-101 Large Conference: compose them without making any mistakes. I see, interesting, okay. Yeah. That is the representations being used here. What? What are the representations? We tried a few different tasks, but we tried to have it be as simple as possible. Obviously, you could do something like use an SAE or something, but we tried to be… to make, like, really minimal… like, minimal, complicating assumptions. 416 01:17:01.180 --> 01:17:03.149 216-101 Large Conference: So in the case of 417 01:17:03.150 --> 01:17:13.099 216-101 Large Conference: multi-hop QA, which is the first task I'm going to show you, although we also did scan, because, you know, scan is an easy thing that you can actually, like, really control, if you know scan. 418 01:17:13.100 --> 01:17:22.680 216-101 Large Conference: the compositionality… You can have tricky tests. We don't have to talk about scan. I skipped scan here for a reason, because I didn't want to explain scan. But, multi-hop QA. 419 01:17:22.760 --> 01:17:35.100 216-101 Large Conference: So, when we're looking at the what year was the author of 1984 born, the representations that we're using, we're not doing anything fancy. We're saying, what year was George Orwell born? Looking at the representation of that. 420 01:17:35.250 --> 01:17:46.510 216-101 Large Conference: And who's the author of 1984? And looking at the representation of that at the last token, both of those. So, if we look at the angle between these two fact representations. 421 01:17:47.200 --> 01:18:02.300 216-101 Large Conference: while tuning to, like, pick what the… what the best possible, layer to take the representation from, if you're wondering about that detail. I think there are ways… there's so many, like, follow-up directions. One of the questions is, where is interference actually a problem? 422 01:18:02.300 --> 01:18:08.789 216-101 Large Conference: For this, for the purposes of this, we just had a little validation set where we picked what layer interference is a problem in. 423 01:18:08.960 --> 01:18:22.879 216-101 Large Conference: But we don't know why it's a problem for some layers and not for others. We don't know that… not yet. So, multi-hop QA. We're basically saying, okay, if you… if the angle between the, you know, George Orwell and 424 01:18:22.880 --> 01:18:29.919 216-101 Large Conference: You know, and whatever birth year, you know? If the angle between these two questions 425 01:18:29.920 --> 01:18:38.350 216-101 Large Conference: at the last token is very close, then it's likely to make a lot of mistakes. The accuracy on that set is very, very low. 426 01:18:38.350 --> 01:18:55.950 216-101 Large Conference: And then the accuracy goes up and up and up until it reaches the overall average as we allow, like, larger and larger angles, and less and less interference into our challenge set, yeah? So I'm curious how much you think the drop in accuracy is from, like, the literal geometry, like, there's, like, interference versus, like. 427 01:18:56.570 --> 01:19:06.480 216-101 Large Conference: maybe there's something, like, if you've seen data more often, you're just going to be more orthogonal to everything, and so that's, like… So, yeah, so the… so… 428 01:19:06.480 --> 01:19:19.640 216-101 Large Conference: I guess I have a few answers. One is that we're looking at whether random variation has any effect here, like, that's a question that we've been asking, because that's the only way we can really isolate out… like, that's the only factor other than training, right? That you can isolate out. 429 01:19:19.640 --> 01:19:39.529 216-101 Large Conference: Two, it doesn't really matter for this to be a useful method. I agree with that, yeah. It could be the cause of why it's more orthogonal. Because, like, it would be super cool if there's, like, some galaxy brain thing you can do where it's like, I'm gonna align all my vectors to be perfectly orthogonal, and I can, like. 430 01:19:39.530 --> 01:19:55.540 216-101 Large Conference: fix the data, but change the geometry, and then it'll, you know… There's this paper called Beyond the Rosetta Stone from DeepMind, which actually, they show that as you add more, like, data in different languages, that it forces the, like. 431 01:19:55.540 --> 01:20:15.280 216-101 Large Conference: the, you know, fact-slash-semantic directions to be more and more orthogonal to the languages that you're adding, in terms of the subspace that that language sort of works with, right? So, actually, yes, the training data directly forces this. To me, that's actually evidence that this model of our model 432 01:20:15.280 --> 01:20:27.090 216-101 Large Conference: Is probably correct, because why would it learn to orthogonalize these, like, these things that it needs to compose regularly if it wasn't using that orthogonality for the composition? 433 01:20:27.300 --> 01:20:39.590 216-101 Large Conference: There is an exception to this, which you see if you look, like, sort of, like, look at our… some of our more detailed plots, which is that really high correlations are actually associated with high accuracy, because 434 01:20:39.590 --> 01:20:53.239 216-101 Large Conference: Those might be, for instance, if you're looking at a particular language and fact, those might be facts that the model runs into all the time in that language. It's not generalizing compositionally, because it doesn't have to. It's memorized them, so they're very, very correlated together. 435 01:20:53.680 --> 01:21:12.700 216-101 Large Conference: So, that's constructive versus destructive interference. So, our model is really about destructive interference, but there's this extra question of, like, you can see this, this doesn't start out by going up, this starts out kind of flat, and that's because there's a mix over here of constructive and destructive interference between features. 436 01:21:14.770 --> 01:21:33.710 216-101 Large Conference: all good questions. Okay, so we didn't just look at the multi-op QA, we also looked at, like, facts composed with languages, which, as I just said, we know that these tend to become more and more orthogonal as the model is, seeing more facts in more languages. 437 01:21:33.710 --> 01:21:48.910 216-101 Large Conference: And here's our precision recall AUC, which I promised we do actually measure for things. And, you can see that if you look at, sort of, like, the majority baseline, sort of, what the PR AUC would be if you basically, you know, just… 438 01:21:48.940 --> 01:22:01.389 216-101 Large Conference: didn't learn anything other than what label is more prominent. We really outperformed that baseline in every single language for, you know, things like 439 01:22:01.550 --> 01:22:12.370 216-101 Large Conference: We know… we have the question, who wrote in 1984? In English, we don't do anything fancy. Again, there's no fancy feature extraction thing. We literally look at this question in English. 440 01:22:12.510 --> 01:22:22.159 216-101 Large Conference: And then we look at a subspace that's just, like, you know, SVD on some little sample of Hungarian data, and we ask, okay, how close are these? 441 01:22:22.830 --> 01:22:32.309 216-101 Large Conference: And you can see, the closer they are, in general, although, again, you can see that constructive interference on some of these lines. 442 01:22:32.310 --> 01:22:43.169 216-101 Large Conference: Especially for, like, lower resource languages. You can see a bit of the constructive interference at the beginning. So, yeah, for every single language. 443 01:22:43.740 --> 01:22:53.709 216-101 Large Conference: there is a predictive power to just looking at the angle between a fact in English and the subspace, like, spanned by, like, that language. 444 01:22:54.300 --> 01:22:56.190 216-101 Large Conference: So, alright. 445 01:22:56.360 --> 01:23:16.190 216-101 Large Conference: I made some videos in Claude, and even though I just explained everything, I want to show you these videos explaining everything. It's pretty cool, right? They're storing a bunch of features in superposition, some of them have big angles, some of them have little angles, and then there's the little angle, and it wiggles. 446 01:23:16.680 --> 01:23:36.550 216-101 Large Conference: Fable did this. So, yeah, more narrow angle, more interference, more compositional failure. One thing we did have to do in order to really make this work broadly across all of the settings that we looked at, sometimes we needed this, sometimes we didn't, is we had to control for the big, dominant clusters on the space, which 447 01:23:36.550 --> 01:23:55.840 216-101 Large Conference: doesn't suggest that something like SAEs would be helpful, because you automatically can control all of these extra features that dominate the space. So once you center things within, which is something you would get out of an SAE, once you center something within its big dominant cluster, that's where you can find the actual relevant angle. 448 01:23:56.920 --> 01:24:12.129 216-101 Large Conference: And, and then this, like, lets you do really efficient, you know, evaluation and training, potentially, because you can find those edge cases. Each of them is just a vector multiplication, a, like, very simple operation, and and then you can 449 01:24:12.160 --> 01:24:18.099 216-101 Large Conference: Generate the input only after you've picked which are the hardest challenges. 450 01:24:18.280 --> 01:24:22.500 216-101 Large Conference: And then, see if the model actually fails on these hard challenges. 451 01:24:23.780 --> 01:24:25.750 216-101 Large Conference: Alright, so… 452 01:24:26.880 --> 01:24:36.890 216-101 Large Conference: So basically, it's like adversarial training, potentially, but we're generating the adversarial concepts, or we're searching for adversarial concepts. 453 01:24:37.240 --> 01:24:56.710 216-101 Large Conference: Instead of looking at inputs, and doing perturbations of input space, which is not very good for adversarial training for things like language models which have discrete inputs, because they have tokens, and a lot of your little perturbations that you would use from image classification or an old toy task, don't really work. 454 01:24:57.320 --> 01:25:00.890 216-101 Large Conference: Anyway, alright, so… 455 01:25:02.080 --> 01:25:20.540 216-101 Large Conference: So here are some discussion questions, but I think, you can also come up with your own questions, to ask if you have any questions. Yeah? I've got a pretty big question. Yeah. So, in reading group recently, we read Adversarial concept search, and we read Why Larger Models Learn More. Yeah. I'm super sold on both of these ideas. 456 01:25:20.540 --> 01:25:35.569 216-101 Large Conference: But at the same time, I spent last year sort of obsessed with the either you know it or you don't from Sometimes I'm a Tree. And these two feel like very… two very true ideas and very, like, evidenced ways of understanding what's going on inside models. 457 01:25:35.700 --> 01:25:55.680 216-101 Large Conference: Like, more so than the other stuff that I see hypothesized, but it feels very hard for me to hold both of these ideas in my head at the same time. So how can we reconcile the stuff you were saying at the beginning of the talk with the stuff you're saying at the end of the talk? Yeah, so, so when I think about it, there are… there are certain things that are not… 458 01:25:55.920 --> 01:25:58.740 216-101 Large Conference: That are… that are not, 459 01:25:59.190 --> 01:26:17.149 216-101 Large Conference: let's say this, like, you-know-it-or-you-don't case. Not everything is a you-know-it-or-you-don't, especially in models. For a human, either you know who wrote 1984 or you don't. But for a model, it's, like, seen that, and it's sort of, you know, moved in that direction during its training, SGD, you know. 460 01:26:17.150 --> 01:26:39.210 216-101 Large Conference: But it might not actually be something where, where there's a clear distinction between these cases. The cases where I see you know it or you don't are compositionality cases. So, to me, the actual thing you know it or you don't is whether, is how to compose, like. 461 01:26:39.230 --> 01:26:40.880 216-101 Large Conference: fact language. 462 01:26:40.950 --> 01:26:45.230 216-101 Large Conference: And… In general, it's like… 463 01:26:46.520 --> 01:27:11.450 216-101 Large Conference: The… there could be, you know, many angles, but but the… I mean, even something like, say, the constructive generation versus destructive generation, right? Like, you have to be aware that these are two different categories… interference generation. You have to be aware that these are two possible categories for that to fit into, or you're gonna run into a Simpsons parody. 464 01:27:12.130 --> 01:27:28.910 216-101 Large Conference: Kind of the situation, where you… Yeah. Or, it's not Simpson's Paradox, it's the other… where there are just two different correlations, two opposite directions, and you're trying to mix them together, right? Like, there are a bunch of things that fit into these categorical distinctions, but the atomic concepts don't. 465 01:27:28.980 --> 01:27:32.809 216-101 Large Conference: The composition of those concepts may. 466 01:27:33.330 --> 01:27:39.910 216-101 Large Conference: I think the thing that I'm having trouble with is… It feels as though 467 01:27:40.180 --> 01:27:47.589 216-101 Large Conference: That the sort of interference described in adversarial concept search is interference that happens along a continuity. 468 01:27:47.700 --> 01:27:51.100 216-101 Large Conference: Hmm. And so, how would that, like… 469 01:27:51.250 --> 01:28:01.519 216-101 Large Conference: how can I… that is a… that implies that learning compositionality in a situation like that is a continual learning task. 470 01:28:01.770 --> 01:28:06.180 216-101 Large Conference: But in my head, based on all of my sometimes this is a… sometimes I'm a tree. 471 01:28:06.300 --> 01:28:22.050 216-101 Large Conference: that feels like not a comp… not a continuous task. It feels very discreet. Yeah. What's happening within the other? Which one is happening within which, I guess? So, I guess it's a little misleading, maybe, for me to show these, like, cumulative plots, because you look at the cumulative plot, and it looks… 472 01:28:25.750 --> 01:28:44.129 216-101 Large Conference: Because I agree, I also have feelings that… that I really love when things are categorically distinct, you know? Obviously. I'm happy to have inspired you to also be really into them. So, so let's… so let's look at this. You can actually see this… this is not… 473 01:28:44.380 --> 01:28:49.450 216-101 Large Conference: This is sort of concealing something that's not… not as monotonic, or certainly not as, like. 474 01:28:49.630 --> 01:28:59.770 216-101 Large Conference: not as, like, not homogenous from one to the next step. What you're actually seeing in all of these cases, pretty much, is these 475 01:28:59.770 --> 01:29:11.559 216-101 Large Conference: These are really clearly not so good. There's, like, a… like, there's more and more interference, and it leads to more and more errors. But it does sort of taper off. It becomes a little more flat. Some of these are even non-monotonic. 476 01:29:11.560 --> 01:29:31.619 216-101 Large Conference: And if you look at the non-cumulative plot, it's obviously much more visible that some of these are, like, non-monotonic, and some of them are really close to flat, when you look at the bucket between. So, in fact, I don't think that… I think that a lot of the interference distinctions in this mid-range 477 01:29:31.670 --> 01:29:40.409 216-101 Large Conference: are maybe not important, and I did say, at the very lowest range, you actually run into a bunch of cases that are constructive rather than destructive. 478 01:29:40.420 --> 01:29:48.120 216-101 Large Conference: So, I think that a really deep analysis of what's going on here, which we have not really fully, like. 479 01:29:48.120 --> 01:30:01.399 216-101 Large Conference: like, dug really deep into, honestly, along this entire curve, would reveal much more categorical distinctions between compositional, non-compositional, and, like, memorized cases. Yep. 480 01:30:01.700 --> 01:30:03.720 216-101 Large Conference: Okay, that makes sense. 481 01:30:03.940 --> 01:30:08.029 216-101 Large Conference: Yeah. Okay, any other, questions? 482 01:30:11.580 --> 01:30:16.410 216-101 Large Conference: Oh yeah, and here's all my collaborators on the projects that I talked about. Faces. 483 01:30:17.090 --> 01:30:17.820 216-101 Large Conference: Oh. 484 01:30:21.140 --> 01:30:22.080 216-101 Large Conference: Okay. 485 01:30:22.930 --> 01:30:26.120 216-101 Large Conference: I know I have one-on-ones anyway, so… It's really great. 486 01:30:26.280 --> 01:30:28.859 216-101 Large Conference: Thank you, Naomi. 487 01:30:34.290 --> 01:30:46.550 216-101 Large Conference: Thank you for the invitation, and for everyone actually really listening. Honestly, really listeny, group you've got here. Yeah. Yeah. We generate questions. 488 01:30:46.860 --> 01:30:47.870 216-101 Large Conference: Yeah. 489 01:30:48.050 --> 01:30:50.740 216-101 Large Conference: Thank you, Harry. And, you know, I think that, 490 01:30:50.950 --> 01:31:05.219 216-101 Large Conference: I, you know, I told them while… while I… I think she heard me introduce you. Oh, yeah, yeah. You're always… you're always up for good questions, so I think there's a good audience for you. Oh, yeah, great questions, yeah. How long do you have… do you have this afternoon?