WEBVTT 1 00:00:02.390 --> 00:00:09.219 216-101 Large Conference: I'll introduce myself in just a second. Yeah, I'll try to introduce myself. Yeah, that's great. So, welcome, everybody. 2 00:00:09.380 --> 00:00:13.960 216-101 Large Conference: So, we're lucky to have Maty Bohacek here, visiting 3 00:00:14.110 --> 00:00:17.239 216-101 Large Conference: from, from Stanford. He's a pre-doc student. 4 00:00:17.490 --> 00:00:24.069 216-101 Large Conference: So he'll be looking for positions, in the, in the coming year. 5 00:00:24.170 --> 00:00:32.760 216-101 Large Conference: And, and, you know, he's… he's, worked in a lot of different places. His, his, his background in research, he's studied… 6 00:00:33.030 --> 00:00:43.159 216-101 Large Conference: things like deepfakes in the past, and he's been doing interpretability work. And so, actually, I don't know what the subject of your talk is today, but we're looking forward to it very much. 7 00:00:43.410 --> 00:00:45.670 216-101 Large Conference: Well, welcome, Matty. Awesome, thank you guys. 8 00:00:47.940 --> 00:00:49.110 216-101 Large Conference: Great, let's see. 9 00:00:49.450 --> 00:00:57.169 216-101 Large Conference: Okay, so, apparently I'm talking about planes today, Okay, so, does anyone know what this right here is? 10 00:00:57.330 --> 00:00:58.269 216-101 Large Conference: Is it fine? 11 00:00:58.750 --> 00:00:59.789 216-101 Large Conference: Is that… 12 00:01:00.530 --> 00:01:06.550 216-101 Large Conference: Is that, like, a DC-10 or something? No, no, it's like, no, no, it looks like some sci-fi thing. Unclosed? 13 00:01:06.870 --> 00:01:08.090 216-101 Large Conference: Any guesses? 14 00:01:08.830 --> 00:01:19.069 216-101 Large Conference: Doesn't have to be the specific model, like, what the significance of this picture could be. Any wild guesses, I'll take. And it's not AI, it's not AI. Is it one of the ones that crashed or disappeared? 15 00:01:19.220 --> 00:01:22.609 216-101 Large Conference: Good. Very close. Use it and force one, yeah. 16 00:01:22.610 --> 00:01:47.210 216-101 Large Conference: kind of close. So this right here is the first ever jet airliner. It's actually called the de Havilland Comet. I might be pronouncing the last name incorrectly there, but… so this is, like, the very first jet that came after propeller planes. And so it was a big deal, right, because all of a sudden, you didn't have to fly, you know, like, below the clouds, and it wasn't so bumpy, you could fly above the clouds. 17 00:01:47.210 --> 00:02:11.609 216-101 Large Conference: also, you can fly to much, much, you know, like, further destinations. So, this right here, this in particular, is the BOAC, jet, so that's what came before, British Airways. And they used to fly this from London to, to, I believe, Egypt, then to Singapore, and then all the way to Japan. So, like, they would have to do these stops, but all of a sudden, they unlocked this, like, brand new, you know, like, much, much further out network. 18 00:02:11.980 --> 00:02:21.079 216-101 Large Conference: And so it was a very shiny, cool plane, but, to your point, over there, unfortunately, pretty early on, there are these two very major 19 00:02:21.380 --> 00:02:40.539 216-101 Large Conference: major crashes. In particular, the BOAC Flight 781, in January of 1954, and then the South African Airways flights, a couple months, later. And, you know, because these planes were smaller, and at that time, not as many people would travel, on planes, the total, 20 00:02:40.580 --> 00:02:59.550 216-101 Large Conference: death toll is 50 people, or around 50 people, which might not seem as much, but, like, everyone died on both these flights. And very early on in the investigation, it became clear that these two planes broke mid-air, right? Like, the fuselage broke already while it was in the air, so it already fell on the ground in pieces. 21 00:02:59.550 --> 00:03:13.100 216-101 Large Conference: And so that was very interesting, that was very sort of notable. Obviously, back then, this whole industry of, like, investigating air disasters was only beginning, and so they went back to the manufacturers, and they asked them for all their tests, all their testing. 22 00:03:13.160 --> 00:03:30.509 216-101 Large Conference: And, because one of the hypotheses was that, like, I mean, if it broke my air, maybe the pressure was just too much, and so, you know, the fuselage just broke under the pressure. And they came back, and they're like, no, no, no, we have done all these amazing tests to test the pressure, so it clearly was not the, you know, the pressure that caused this. 23 00:03:30.510 --> 00:03:34.850 216-101 Large Conference: And the investigators, they can go back to their notes, and I've done this, and you can do it too, it's all online. 24 00:03:34.850 --> 00:03:47.700 216-101 Large Conference: And they actually were like, oh yeah, like, those seem like pretty good tests, we're gonna investigate different hypotheses, so they were looking at, you know, someone trying to collect a bomb, or, like, all these different hypotheses, right? And then, ultimately, they came back to… 25 00:03:47.750 --> 00:04:02.959 216-101 Large Conference: to the manufacturers, because they said all these different things, and none of them really panned out, none of them were confirmed, and they're like, let's look into this pressure hypothesis again. And they actually placed, one of the fuselages in this water tank. 26 00:04:02.960 --> 00:04:26.319 216-101 Large Conference: to simulate pressure, and they were trying to simulate pressure, you know, like, in the air, and after a bunch of, sort of refills and, like, you know, there was some fancy physics there as well to really, like, simulate the… some of the, you know, aerodynamics, they actually broke the fuselage. So they were able to show that, actually, it was the design of the plane that, caused the plane to… to just crack midair, and then… and then fall, and ultimately 27 00:04:26.320 --> 00:04:27.260 216-101 Large Conference: Most people die. 28 00:04:27.260 --> 00:04:39.290 216-101 Large Conference: So what was it about these tests that… that made them so, so, you know, like, I guess problematic, or that didn't, they didn't actually show this, this issue, while they were testing them, you know, during their design? 29 00:04:39.430 --> 00:05:03.709 216-101 Large Conference: Also, apparently, before they put on all this pressure on the plane, or on the planes during testing, they did these additional tests, and in particular, they're called static proof tests. And what they do is they actually already operate at above normal pressure. And so they would do these tests in sequence, right? And so the plane, when they, you know, went to test it for its resistance, or 30 00:05:03.710 --> 00:05:28.040 216-101 Large Conference: its durability was already under a lot of effects from previous tests, and those tests actually made the steel and different materials sort of change states a bit, or sort of change configuration in a way where the test didn't really break the fuselage. Now, I'm not a physicist, so you can tell that my description here is not as detailed as it probably could be, but you can look it up. Basically, TLDR is that different tests that came before 31 00:05:28.040 --> 00:05:33.000 216-101 Large Conference: Change configuration, and then the test-to-test pressure didn't really show any issues. 32 00:05:33.200 --> 00:05:40.650 216-101 Large Conference: And so, my question when I read this, and so, by the way, you know, I've noticed that I'm now in, like, minute 33 00:05:40.850 --> 00:05:46.209 216-101 Large Conference: 5 of my talk. We're still talking about planes. One of the reasons for this 34 00:05:46.210 --> 00:06:05.079 216-101 Large Conference: Is that I love planes, as you might be able to tell by now. But I do think that there might be a bit of an analogy here that could be… could be helpful. There might be a lesson here. And so, when I learned about this, you know, for the first time as a kid, I was really curious to see, like, was there a way to prevent this? Like, could they have prevented this before any of these flights took place? 35 00:06:05.080 --> 00:06:07.499 216-101 Large Conference: And I would posit that, yes. 36 00:06:07.500 --> 00:06:32.330 216-101 Large Conference: If you really, truly understood the system as in the plane, and also the system as in the physics around it, and the tests, then… oh, there's a typo there, well, then, then you should have, or you could have actually prevented this before anyone died. And so now, if you, you know, will indulge the analogy, these jets, right, there was something very, very new. Like, they had some planes before, propeller planes, right? 37 00:06:32.330 --> 00:06:55.450 216-101 Large Conference: But all of a sudden, you have these jets, like a very new sort of, like, level of design. And I think that in some ways, we find ourselves in a similar situation in AI. Obviously, you know, different people have different predictions and talk about it in different ways, but we are at this inflection point in many ways. And additionally, the environments that, you know, you need to test a plane in before any test flights take place are very different from the actual environment 38 00:06:55.450 --> 00:06:59.770 216-101 Large Conference: that, you know, they operate in, ultimately, which is the case for AI as well. 39 00:06:59.960 --> 00:07:00.970 216-101 Large Conference: And so… 40 00:07:01.140 --> 00:07:22.430 216-101 Large Conference: with that in mind, I want to sort of learn from this mistake and try to hopefully prevent any, you know, crashes with our AI here, and so I think there are two questions that could be interesting to consider. One would be to say, if we've now moved from, you know, the world of aviation to AI, is whether we can understand what our evals, our tests, are fundamentally missing. 41 00:07:22.430 --> 00:07:31.090 216-101 Large Conference: And second, whether interpretability, right, this ability to understand these systems, and also their environments, could be used as a means of evaluation itself. 42 00:07:31.090 --> 00:07:39.970 216-101 Large Conference: I'll add some sort of caveats here. First, this idea of what evals are missing. Like, what are we missing during our tests? 43 00:07:39.970 --> 00:07:51.299 216-101 Large Conference: I consider this to be one aspect of what I call eval interp, and there are other people who have now been sort of interested in interpreting evals, and I think that's a very interesting and cool area to consider. And second. 44 00:07:51.300 --> 00:08:08.119 216-101 Large Conference: In the second part, when I say be used, I'll sort of lower my criteria here to at least inform evaluation, right? Like, to at least steer evaluation. We'll consider a couple options here, but there are many ways in which you can sort of use interpretability to do this. So these are the two main questions, two main goals. 45 00:08:08.120 --> 00:08:13.199 216-101 Large Conference: For today, comfortable with that? Everyone ready? All right, let's get into it. 46 00:08:13.470 --> 00:08:37.900 216-101 Large Conference: So, before we talk about how we can use interpretability to, you know, do evals better, let's first talk about the state of evals. Like, where are we at? I'm sure this is something that you all are familiar with, since I assume many of you might be submitting to iQlear, and you're pressured to outperform the state of the art by, however, you know, minimal delta. I know that's not what this lab is about, so I'm… I know you're cool, but 47 00:08:37.900 --> 00:08:47.570 216-101 Large Conference: Evals are something that is everywhere, it's very important for different reasons, but there are many different ways to do it as well. And so let's take a look at evals, in AI. 48 00:08:47.810 --> 00:08:58.310 216-101 Large Conference: Now, for the first part of this talk, I'll be talking mostly about language models. So, you know, we're first in the world of language models, and then I'll sort of shift to image models and vision a bit, so I'll tell you when that happens. 49 00:08:58.310 --> 00:09:18.479 216-101 Large Conference: So first, let's consider this, sort of, this sort of range or scale of transparency in AI, right? You have something from… from very, sort of, closed-source industry models that are least transparent, to the very, sort of, open source models that are, like, fully, fully transparent. Some examples here, right, like, on the more, or fully, I guess, like. 50 00:09:18.560 --> 00:09:41.819 216-101 Large Conference: you know, intransparent or non-transparent side of things like GPT, Astra, CloudTable, these are models that you can use, you can run inference, you can maybe even get some traces or some things from them, but they're not… you can run them yourself, right, obviously. Then the sort of mid, section of open weight models, where we can use their weights, we can run them on our machines, but we don't necessarily know the recipe, the full recipe for the models. 51 00:09:42.090 --> 00:09:58.430 216-101 Large Conference: all the way to the fully open source models, mostly from papers, or, you know, like, these, these, model reports where we actually know the data set, how they were trained, we have everything, the full stack. I would pause it, and this might be… I think this could be seen as controversial, but, 52 00:09:58.620 --> 00:10:23.280 216-101 Large Conference: I don't know, but if we think about the sort of in-the-wild impact of these models, I would say that the least transparent models currently have the most in-the-wild impact, because they're used by individuals, users, you know, non-CS people, and they're integrated into different services. Obviously, I want to sort of say that open source models are incredibly impactful in the sense that we learn on them, and we improve architectures, and then people take those findings and apply them to close source models, right? But, like. 53 00:10:23.280 --> 00:10:34.520 216-101 Large Conference: If we think about the most impact right now on, like, end users or people in the wild, I would posit that mostly it's these big closed models that right now have the impact. 54 00:10:34.520 --> 00:10:39.429 216-101 Large Conference: open source catching up, it's not about capability necessarily, it's mostly about just, like, pure MBAC right now. 55 00:10:39.840 --> 00:10:45.510 216-101 Large Conference: So with that in mind, let's first think about closed source models and how they're being evaluated. 56 00:10:46.370 --> 00:11:02.620 216-101 Large Conference: To do this, we actually, had this paper at FACT recently, where we made this data set, called, Benchmarking Cultures. And in it, we were interested in collecting all the different benchmarks and all the different model releases that these, that these model builders, or these labs, or companies 57 00:11:02.620 --> 00:11:27.270 216-101 Large Conference: produced in the… in last year, or over the whole course of last year. Obviously, I need to say here that there are different types of evals. There are evals you run internally that you don't disclose to the public, right? And so that's… that's one type. But if we just think about what we… we can get as individuals who are not members of these institutions or organizations, what are the evals that we can get our hands on, right? So… so we actually included these 12 different model builders. 58 00:11:27.340 --> 00:11:43.430 216-101 Large Conference: 7 in the US, 4 from China, and one from the EU. I will say that Mistral, I have a sense that it was probably included because one of my co-authors was European, and so we didn't want to, you know, make him feel, yeah… Anyway, no, no, we did include Mistral because they have reasonable models. 59 00:11:43.530 --> 00:11:46.499 216-101 Large Conference: So we had all these different model builders and all these different models, right? 60 00:11:46.680 --> 00:12:04.999 216-101 Large Conference: So, for each of their, model releases, which are these, sort of, could be blog posts, could be announcements, we… we took in all the different, evals that they were presenting, right? So, let's look at one of these, one of these, sort of, reports, or one of these releases. This here, is, from… from Claude from Anthropic. 61 00:12:05.230 --> 00:12:15.849 216-101 Large Conference: If I zoom in a bit, there are a couple things I want to sort of highlight, or have you pay attention to. Hopefully you can see us in the back as well. I tried to make it, you know, make the zoom-ins as well, so… can you guys in the back see? Awesome, perfect. 62 00:12:15.850 --> 00:12:32.930 216-101 Large Conference: So, first of all, I want to highlight this idea where, oftentimes, the model builders actually, you know, they present a benchmark, but then they say, okay, this benchmark tests this thing. They, like, frame it somehow. They sort of say what they're testing with this benchmark. And this is done across different model builders. That's one thing I want to sort of pay attention to. 63 00:12:33.290 --> 00:12:49.000 216-101 Large Conference: Second thing, there's comparison against different versions of, you know, models from the same builder, but there's also usually a comparison to the competitors, right? So here, that would be Opus versus GPT, right? Like, Anthropic versus, you know, OpenAI in this case, so that's one thing I also want to pay attention to. 64 00:12:49.000 --> 00:12:56.350 216-101 Large Conference: And finally, if you notice, all of these are just single numbers. Like, there's, like, a single number for each of these benchmarks and each of these cells in this table. 65 00:12:56.350 --> 00:12:57.930 216-101 Large Conference: So, some things to consider for us. 66 00:12:58.060 --> 00:13:05.760 216-101 Large Conference: Now, with this in mind, let's just run some analysis and some basic inference, because we have this for all the releases that they have. 67 00:13:05.770 --> 00:13:23.520 216-101 Large Conference: the whole year, all these builders, let's ask some questions. So first, we were curious to see, how many, different builders use each of these benchmarks. Like, for a given benchmark to be useful, presumably, you'd have to use it, or different labs would have to use it to make sure that their model is comparable, right? 68 00:13:23.520 --> 00:13:27.080 216-101 Large Conference: Well, turns out that, over 60% of benchmarks 69 00:13:27.080 --> 00:13:38.180 216-101 Large Conference: are used by only a single builder, and so there really is not a lot of comparability in those benchmarks. And really what it trickles down are a handful of benchmarks that are used across the industry, and those are really the important ones. 70 00:13:38.180 --> 00:13:49.999 216-101 Large Conference: But the vast majority of benchmarks are used by a single builder and no one else, which to us was noteworthy. Another noteworthy finding is that if we distinguish between these two things, which we define as 71 00:13:50.000 --> 00:13:58.860 216-101 Large Conference: assigned competency and prescribed competency, where assigned competency is what the builders of the benchmark say that their benchmark assisting, right? So, like, if I 72 00:13:58.930 --> 00:14:14.140 216-101 Large Conference: had a benchmark that says math, and I would make my, you know, write my paper and say, here's my cool new Maty bench for testing math, and I would say, test math, that's the assigned competency. We're not looking into whether there's validity in that, we're just sort of taking them at their word. Like, the authors of the benchmark said, this is for coding. 73 00:14:14.400 --> 00:14:28.579 216-101 Large Conference: format or whatever. Well, then we can compare them to all these prescribed competencies that the model builders frame the benchmark as, right? Like, these notes where, like, they use terminal bench for coding or whatever that I highlighted in the table. Well, let's look at this live code bench. 74 00:14:28.840 --> 00:14:52.689 216-101 Large Conference: So, for lack of a bench, 53% of model builders frame it as coding, but there's also, framing as reasoning by at least three model builders, as Agentic capabilities by two builders, cost-efficient intelligence by another builder, and then post-framing, instructions, and just, like, text, Alibaba. And so you have this, this very diverse, sort of, or I guess, like, very, sort of. 75 00:14:52.690 --> 00:15:05.579 216-101 Large Conference: inconsistent way of referring to different benchmarks, and then, you know, the builders use these benchmarks to then warrant certain claims of other models, right? And about their capability, about how they compare to different models, and so just some things to consider here. 76 00:15:05.860 --> 00:15:24.120 216-101 Large Conference: And then one last finding from this paper. I mean, there's more, but one thing that I wanted to bring, because I just… it's like, this is a classic, but a lot of these, you know, plots, as you might notice, these two plots here do a lot of work, because if you really sort of plot out the whole y-axis, the comparisons against different models 77 00:15:24.120 --> 00:15:45.379 216-101 Large Conference: are, you know, not so stark. And this is just, like, sort of like a side note, side joke, but we did find a lot of this as well happening across the board. If you are interested in this, there's a full paper with a lot of additional sort of findings and analyses, where you look into, like, how Chinese models or Chinese builders compare to used builders, different things, but I just wanted to highlight that there's a lot of inconsistency, even with these, you know, like, very sort of 78 00:15:45.620 --> 00:15:54.939 216-101 Large Conference: sort of public-facing, PR-coded leases, and those are the ones that I would argue have the most impact right now on the way people use AI in the wild. And so that's noteworthy. 79 00:15:55.280 --> 00:16:14.899 216-101 Large Conference: Now, unfortunately, we cannot, or I don't know if anyone in the room has access to Astraz or Fables Weights. If you do, let me know. I'd be interested. But, you know, we did not have that, and so we had to move to a place where we could actually test and play with the models, and so we… Can I ask a question before you move over to the next piece? So the benchmarks that you tested. 80 00:16:14.900 --> 00:16:19.300 216-101 Large Conference: Those were coming from academic labs, or those were coming from the 81 00:16:19.300 --> 00:16:38.000 216-101 Large Conference: those companies themselves? Great question. So, I love this question, because… so we indexed any benchmarks that they listed. So, like, if a benchmark was listed in any of those releases, we indexed it, and we had it in our database, and the dataset is open source. If you want to test it yourself, you can. We actually saw, we didn't put this in the paper, but if you plot. 82 00:16:38.000 --> 00:16:57.010 216-101 Large Conference: the share of academic benchmarks versus proprietary benchmarks over time, over the whole year, you actually see a big, sort of increase in closed source or closed benchmarks at the labs that they report. But yeah, we index any benchmarks that, like, they reported in their releases, if that makes sense. If it wasn't the table or in the text, it was, it's in our data set. 83 00:16:57.010 --> 00:17:02.090 216-101 Large Conference: So, closed benchmark, you mean a benchmark that came from a commercial. 84 00:17:02.090 --> 00:17:26.939 216-101 Large Conference: Now, does it also mean that it's not reproducible? Like, you can't run it, or… Great question, great question. There's two types. There's some… so OpenAI, in our data set, seems to be the leading one in terms of just, like, having their own internal benchmark that they cite, but they don't actually, like, you can look it up, you can download it, you can use it as a non-AI… non-open AI person. There are some that are from, data brokers, so you can pay for it. So I guess, like, if you want to pay for it, you can 85 00:17:26.940 --> 00:17:32.179 216-101 Large Conference: You can get it, but if you don't pay for it, it's closed source, so you can get it. So there are these two types of closed benchmarks. 86 00:17:32.180 --> 00:17:39.829 216-101 Large Conference: They're still the, the, they're still not the majority, but they're, like, on the rise, or they were, over the last year. 87 00:17:41.140 --> 00:17:46.540 216-101 Large Conference: For the open ones, the benchmark which is coming from the academic labs, they can't really… 88 00:17:48.590 --> 00:18:06.140 216-101 Large Conference: the… the confusion around what this benchmark is actually testing shouldn't be there, right? Because the one who created the benchmark, let's say some academic lab who created that benchmark, they would tell explicitly that this benchmark is for this particular community. Yes. So the confusion is primarily coming from the benchmarks? 89 00:18:06.340 --> 00:18:11.800 216-101 Large Conference: So I think the confusion, or… so, so, it's a good question. We can't… 90 00:18:11.800 --> 00:18:31.310 216-101 Large Conference: we can't assume intent, so whether it was confusion, or just, like, a PR effort or anything in between, whoever wrote the blog post, the model announcement, they chose to frame the benchmark in a particular way, and so we're taking that, and we're comparing that to what the authors of the benchmark said, clearly in their paper, or wherever it's released. 91 00:18:31.310 --> 00:18:42.219 216-101 Large Conference: So we're comparing those two, and whether it was because, you know, someone, like, just accidentally didn't understand the paper, or if the PR person just wanted to make the model look good, or… we can't say. 92 00:18:42.220 --> 00:19:00.000 216-101 Large Conference: Sorry to say. I did spend some time in DeepMind, and, like, I was never involved in, like, the release of, like, a Frontier model, but even for just, like, the open source models like Gemma, when we did something to help GEMA, I remember this is, like, I'm not with GDM anymore, and I hopefully can say this, if not, then… 93 00:19:00.000 --> 00:19:07.930 216-101 Large Conference: well, I guess it's recorded, so too late. But but, like, I noticed that even in our, like, release, we were meant… we were supposed to share 94 00:19:07.930 --> 00:19:26.290 216-101 Large Conference: the benchmark results with the PR team, and in their first job, they, like, completely butchered, like, it was completely wrong, and we had to, like, proactively, like, ping them and be like, no, no, no, no, like, you need to fix this. It was for, like, an open source, like, for Gemma, right? People might know Gemma, like, because there's a lot of essays for it, but, yeah, so… 95 00:19:26.520 --> 00:19:41.409 216-101 Large Conference: So, and so I'm not saying what it was, if it was confusion, or just, like, PR, or whatever else, but there is this delta that, to us is interesting, because… and I think it's interesting for this reason here, that we'll sort of get into in just a second. Also, just like a side note, like, we write a lot of narratives. 96 00:19:41.410 --> 00:19:52.979 216-101 Large Conference: around these AI models, right? And people who are not technical, like us, or people who are not scientists, they come usually to media, but where does media take these claims from or results from? Usually these model releases. 97 00:19:52.980 --> 00:19:58.680 216-101 Large Conference: And so there's a lot of narrative building that, that happens through these benchmarks. 98 00:19:59.210 --> 00:20:17.489 216-101 Large Conference: Cool. So we'll now consider the open source ones, just because, you know, we can actually get them and use them and study them internally. And so we're going to shift to the second part of this talk, which will be about blind spots and evals. As I sort of promised with this question, can we understand what our evals are missing? 99 00:20:17.810 --> 00:20:42.359 216-101 Large Conference: Now, to do this, you sort of have to have a space in which you're going to be searching or considering different things that, are or are not considered. Now, you could use just, like, open-ended, like, you know, natural language. That'd be a very open-ended space and might be, sort of hard to, like, really meaningfully trace and test. So, we're going to look for something that is a bit more… is sort of more closed to 100 00:20:42.450 --> 00:20:51.870 216-101 Large Conference: And all defined. We're gonna use SAEs. Now, I think that, for this crowd, I might not have to explain SAEs 101 00:20:52.080 --> 00:21:15.720 216-101 Large Conference: in the same detail that I do usually. I'll go over this just very briefly. If you feel like you're being bored, or it's, like, too trivial, let me know, I can skip this, but with SAEs, right, our goal overall is to decompose learned activations into a set of interpretable concepts. Whether or not SAEs do that well is sort of a separate question, and there are obviously many limitations that I'll get into in a second. 102 00:21:15.720 --> 00:21:22.830 216-101 Large Conference: But we'll use this SAE dictionary space, or the concepts in SAEs, to do this. Obviously, there's been 103 00:21:22.830 --> 00:21:42.749 216-101 Large Conference: discourse about SEs, since we did this paper, or this work, over a year ago. And so I also want to say that you could use any other sort of replacement for, you know, like, decomposition or concept, construction. Like, our method sort of uses it as, like, a plug-and-play element, so we're using SEs here, but you could use something else as well. 104 00:21:43.080 --> 00:22:07.129 216-101 Large Conference: For SAEs specifically, the goal, again, as people here probably know, is that, in our models, we oftentimes have, very dense activations, representations, and, they're great, they work very well, but it's very hard to interpret them. It's very hard to understand what they mean, and what the model is working with at any given point in the model. And so, the goal of SAEs, at least 105 00:22:07.130 --> 00:22:30.869 216-101 Large Conference: goal is to, take these activations, could be any architecture, and learn a decomposition into, this sort of sparse, dictionary, right? Like, we call this overcomplete, meaning that there's more dimensions in our sort of resulting SAE space, than the dimensions of the, of these activations or vectors. And there's different ways to get there, which, you know, you could achieve with different. 106 00:22:30.870 --> 00:22:50.679 216-101 Large Conference: sort of penalties and your loss. Now, I… this is a very, very, like, high-level, quick overview of essays, because I assume people here would be familiar. If there's… people want to… want me to go into it more, I can, but I think people here are pretty familiar, right? So, SAEs, right? We're going to use SAE dictionaries as our sort of space to search for these gaps in benchmarks. 107 00:22:51.170 --> 00:23:14.660 216-101 Large Conference: So let's consider a, a data point in a benchmark. This is from the math benchmark. It's not as popular anymore, but for a long time, this was, like, a very important benchmark for testing LLMs. And so the prompt, here could be something like, what is the ratio of the volume of a cube with edge length 6 inches, and now I'm being blocked by Zoom here, inches to the volume of a cube with edge length 1 foot? Express your answer. 108 00:23:14.660 --> 00:23:19.640 216-101 Large Conference: No, no, this is good. I just had to, like, move the window here, because I couldn't read the prompts, but thank you. 109 00:23:19.640 --> 00:23:28.769 216-101 Large Conference: And then, you know, the LM that we tested produced a response. It was incorrect, by the way. It was not 27, it was supposed to be 1 over 8. 110 00:23:29.000 --> 00:23:31.800 216-101 Large Conference: But if we, you know, it's… 111 00:23:32.140 --> 00:23:43.760 216-101 Large Conference: It tried. It did good work. Singularity is happening tomorrow, by the way. And so if we used, you know, an SAE over the prompt, we could, find concepts like these. 112 00:23:45.540 --> 00:24:10.300 216-101 Large Conference: like these, and these are actually pretty good. I shortened the auditor labels, but I just want to give people a feel for, you know, what SCE concepts look like. Again, here, I think people are familiar, but people who don't work with SCE sort of need to, or I think it's helpful to show them what this concept can look like. So, you know, we have an index for the concept, and then we have some sort of description that was produced usually by a different LLM to describe the concept. And here, the concept 113 00:24:10.300 --> 00:24:11.340 216-101 Large Conference: match. 114 00:24:11.340 --> 00:24:36.289 216-101 Large Conference: the topic pretty well, but, you know, there could be… there could be issues with the labels, also with the concepts, you know, some concepts are dead, like, they just… they just don't really, the neurons just are not active, and then sometimes there, there's multiple things sort of projected or collapsed onto the same direction, same concept, so there can be many issues with SAs as well. But we're going to use this space just because it gives us a sort of well-defined and large enough, like, specific enough space 115 00:24:36.290 --> 00:24:42.269 216-101 Large Conference: That's still, that's still… sort of searchable and trackable. Now, 116 00:24:42.450 --> 00:24:51.819 216-101 Large Conference: One thing I want to also sort of spend just a minute on is why essays are not something else. We think that essays are helpful because they give us this one fixed 117 00:24:51.820 --> 00:25:03.909 216-101 Large Conference: dictionary that we can use, and it can be shared across different benchmarks and also different parts of the method that I'll get into in just a second. Also, they're grounded in the model itself in some ways, which is helpful. 118 00:25:03.910 --> 00:25:28.239 216-101 Large Conference: Now, we considered different approaches to this as well, and in the paper, which you can read afterwards if you're interested, we do test these different approaches as well, and we found that they're not as good or not as practical as SAEs in this particular case. For just, like, raw activations, it's really hard to somehow, you know, discretize that and make that meaningful without some additional method to get concepts from them. And SAEs here are 119 00:25:28.240 --> 00:25:29.610 216-101 Large Conference: Sparse and then label. 120 00:25:29.880 --> 00:25:52.260 216-101 Large Conference: supervised concept models, if you use, like, CBMs or anything like that. Well, they're nice, but, they require you to, like, you know, like, come with concepts. You have to come with specific things you want to test for beforehand, and so you can confirm hypotheses, but you can't really, like, do this unsupervised learning or, like, you know, like, what is it that I don't know, right? Like, that kind of thing is hard to do if you need to come with specific things you want to test for. 121 00:25:52.950 --> 00:26:16.739 216-101 Large Conference: And then, there are additional, approaches to testing if your benchmark is missing something. They're mostly based on, LLM judges. So, like, they, they, you know, they have LLMs go through benchmarks, and they ask them if there's something that they're missing based on different taxonomies or different goals that they're stated in natural language. And they're helpful too, but, they, they often sort of just are very high level in many ways. And so we wanted to 122 00:26:16.740 --> 00:26:22.639 216-101 Large Conference: use the cities because we found them to be the most suitable out of these different options for our use case here. 123 00:26:22.960 --> 00:26:44.429 216-101 Large Conference: Okay, so back to our question, that was my interseis. Can we understand what our emails are missing? So, enter this method that we call competency gaps, where if we have a… this will be the one part of the talk where it's, like, a bit notation, but then I promise there's more, like, images and, like, cool examples and, like, plots, so just bear with me for just a second. So if we have a benchmark. 124 00:26:44.430 --> 00:27:08.400 216-101 Large Conference: B, we'll call it B, and we have this underlying data set, capital D with items in it, and this SAE, we will project each benchmark point, or, like, data point, into this SAE space. And obviously, in language models, SAEs generally work over, or can work over tokens, and so, we need to sort of consider different tokens. Now, there are different ways you could do this, right? Like, if I have a data point. 125 00:27:08.400 --> 00:27:22.130 216-101 Large Conference: like the one that I showed you earlier with math, you could either look at the last token, right, because it'll attend to the previous token, so that's one way people use it. You could also consider activations for all tokens, and then average them, or be clever in how you average them. 126 00:27:22.260 --> 00:27:46.030 216-101 Large Conference: also an option. We test both in the paper, and we showed that it doesn't really matter for the findings, but both have different advantages. So there are two ways you can sort of do this. Again, I'm going to just, like, quickly try to move my Zoom window here, or I'll look here. But we have one version where you can sort of get this, this S. So S will be sort of our, like, activation of this concept from the SAE, for concept. 127 00:27:46.060 --> 00:27:59.479 216-101 Large Conference: C of data point I. And so either you just… you average it over the different tokens, right, so J, will be sort of looping over, over tokens, or you can just take the last, the very last, token and take its activation in the SA. 128 00:27:59.880 --> 00:28:20.560 216-101 Large Conference: Now, with this, we're going to define this coverage of a concept in benchmark B. And so, to do this, I brought some annotations, because I assume I would not be able to draw on the whiteboard, which is a good assumption, because it's over there. So I annotated this on my iPad. And so, so that over there, right, that sum, for summing, this, this activation S, 129 00:28:20.560 --> 00:28:28.199 216-101 Large Conference: for this concept C, over all different data points, right? Because I are the data points of the data set, so we're going to do this over all the data points in the data set. 130 00:28:29.450 --> 00:28:54.290 216-101 Large Conference: here at the bottom, we have the total activation for this concept… sorry, for all concepts in the dataset, right? So we're going to do… we're going to look over all concepts over the entire dataset, and so we're just going to get the entire activation across everything. And then the C, or the, you know, the size of C at the top, that's really us trying to divide the bottom, with the size of the concept dictionary, because if you, you know, if you have, like, a fraction at the bottom, you can 131 00:28:54.290 --> 00:29:17.139 216-101 Large Conference: to flip it. So, like, very basic, just, just algebra there. So this, what this is doing, really, like, intuitively, we're trying to understand, what is the coverage or what is the activation of this concept in the benchmark, if we were to take the entire activation for the whole data set and average it by the number of concepts we have. That's really, it's very simple. Just wanted to be very sort of clear about the notation here. 132 00:29:18.710 --> 00:29:38.290 216-101 Large Conference: Okay, and so now that we have this notion of, you know, benchmark coverage, we can look at different benchmarks and run it, you know, run them through our city, run them through this metric, and then… and then look at different phenomena, different trends. So here, we're showing a bunch of different data sets. There's 10 here, different categories, we have anything, you know, here we did the, sort of. 133 00:29:38.290 --> 00:29:52.819 216-101 Large Conference: prescription, or the categorization of reasoning, and math, and factuality, and ethics and bias. But you have anything from AGI eval, logic bench, math, BPQ, and others. And there's more in the paper if you want to see more. We have, I think, an additional 20 or so. 134 00:29:52.820 --> 00:30:11.489 216-101 Large Conference: And so here, I'm just plotting the benchmark coverage for each of these individual, benchmarks, right? And this is a log scale, so, that's important to note. And we see that there are, you know, there are some concepts that are dominating the benchmarks, and the vast majority are just, like, very, very 135 00:30:11.490 --> 00:30:24.940 216-101 Large Conference: loosely, or, like, not even… almost unrepresented at all. And in fact, if we look at concepts that don't activate at all, like they're below some sort of epsilon with a noise, there's a lot of concepts that just don't get activated at all in these benchmarks. 136 00:30:24.990 --> 00:30:42.439 216-101 Large Conference: Now, two things I want to say here, two caveats. First of all, if we have something like math, again, let's go back to math, it might be okay that it's not testing everything in the model. That might be fine, because math is sort of supposed to test math, and so as long as we get a good coverage for math, we should be fine, presumably. 137 00:30:42.540 --> 00:30:57.580 216-101 Large Conference: And so, you know, you need to be sort of careful, we need to be thinking about which conscious those are, and we can do some filtering there as well, and I'll get into that. And second, there are dead neurons, obviously, in these SAEs, and so we actually accounted for that. So if a neuron was dead, we didn't consider it in these analyses here. 138 00:30:57.580 --> 00:31:05.540 216-101 Large Conference: And so… so that's… so two caveats. But even though we… we sort of aspire for, I guess, like, we hope that these individual benchmarks 139 00:31:05.540 --> 00:31:10.720 216-101 Large Conference: are testing whatever the scope or domain is. If we go back. 140 00:31:10.720 --> 00:31:25.209 216-101 Large Conference: to this, right? Like, this notion of having these big, tables, sorry, with different categories, and just sort of adding to a lot of different benchmarks, the implicit, goal, and sometimes it's even sort of said explicitly, is that if you put together a lot of these different benchmarks. 141 00:31:25.640 --> 00:31:36.450 216-101 Large Conference: presumably, or hopefully, they'll get a pretty good coverage of different, you know, different abilities, different domains, and sort of put together a sense of the model pretty rigorously, sort of, like, you know, run through. 142 00:31:36.550 --> 00:32:01.150 216-101 Large Conference: And so, if we believe that, like, if we, you know, sort of assume that that is the goal here, well, then let's just take all these benchmarks together and consider these suites of benchmarks all together. And if we do that, we still find that there are a handful of concepts that are, like, very heavily overrepresented. Again, this is sort of the same plot, but just putting all the benchmarks together. And there's actually a majority of benchmarks that are just, like, very, very 143 00:32:01.150 --> 00:32:03.160 216-101 Large Conference: Loosely or almost not represented at all. 144 00:32:03.160 --> 00:32:17.239 216-101 Large Conference: And here, we took a suite of benchmarks that, at the time of writing, was very sort of popular or prominent across different model releases, both open source and closed source. So this, at the time, was like a representative sample of, like, this sort of suite of benchmarks. 145 00:32:17.270 --> 00:32:33.160 216-101 Large Conference: And so, so that's one thing I want to say. So, the finding here is that, for these benchmarks, even when you put them together, there's a handful of concepts that are sort of dominating the coverage, and the majority are loosely, or really not represented very much. 146 00:32:33.530 --> 00:32:45.120 216-101 Large Conference: And some specifics here, like, you can look at the specific concepts that are, like, not represented or that are missing from these suites. And here, I bring this up because I think that there are limitations to this, right, when it comes to STEs. Like. 147 00:32:45.120 --> 00:33:07.740 216-101 Large Conference: there are some things, like these concepts, like the word opinion and its derivatives, where it's very hard, in my opinion, to sort of say which benchmarks this should be tested in, like, if that is really meaningful at all. Like, there might be something there, might be a bad label. So, like, there are limitations to this method, for sure, but if you look at the sort of whole, you know, the net and, like, the trends, I think that those are still telling, they are still important. 148 00:33:07.740 --> 00:33:12.309 216-101 Large Conference: And there might be some concepts that are really, sort of, important to you, and that just don't get tested. 149 00:33:13.180 --> 00:33:31.159 216-101 Large Conference: A couple of additional things that I want to… yeah. Try to have some sort of ground truth, like, if a human looked over it, also confirming that there is no trigonometry or no opinion, the word opinion in these things? Yes, yes. So we did have some humans testing this, and for the most part, like, we found that the labels were, were, 150 00:33:31.380 --> 00:33:56.159 216-101 Large Conference: not incorrect, like, we found that in these cases, they were correct, but they weren't specific enough, so, like, I think the word opinion and conservatives was mostly, it was, like, some specific case of it, and then with the mathematical concepts here, it was more specific than that. Like, the labels were not just fine-grained enough, I think was defining there. And we also did some additional, sort of tests where we're trying to use LLMs to sort of say if this concept is relevant to this bench. 151 00:33:56.160 --> 00:33:59.059 216-101 Large Conference: like, we did different things as well in the paper to suggest robustness. 152 00:33:59.060 --> 00:34:03.169 216-101 Large Conference: And overall, we found this to still be robust, but there were some cases where it just didn't… 153 00:34:03.170 --> 00:34:05.409 216-101 Large Conference: It was not a good label, I would say. 154 00:34:06.270 --> 00:34:23.339 216-101 Large Conference: Cool. Okay, so that's, that's on the missing concepts. A couple of additional things I want to sort of highlight here is that once you have this profile, like, you know, one way of thinking about this is, like, it's benchmark coverage, right? Or coverage of different concepts. But really, it's a profile of the benchmark, and you sort of have this density over each concept. 155 00:34:23.340 --> 00:34:27.910 216-101 Large Conference: And so you can then do different things like similarity across different benchmarks. So, like, here we just have this 156 00:34:27.909 --> 00:34:45.360 216-101 Large Conference: this grade where we're sort of comparing similarity, across different benchmarks, and, you know, like, that could do different things. For example, the use case of this could be that if you're selecting benchmarks for your model, you could think about how much new information you're actually getting from any new benchmark. Like, if I already have math. 157 00:34:45.360 --> 00:35:03.050 216-101 Large Conference: and then I want to consider a new benchmark for math. Well, if they're getting a very sort of similar… if they're very similar in what they're testing, in terms of the activations, maybe I'm not going to learn very much from running that benchmark. Maybe it's not, you know, reasonable or justified to run the benchmark. And vice versa, maybe there are areas of my, 158 00:35:03.050 --> 00:35:20.960 216-101 Large Conference: you know, of this sort of activation space, where I'm just, like, not testing at all, and then I can look up, benchmarks with profiles that, like, would be sort of filling that missing space and those gaps. And in fact, we have some people… so actually, there's a, there are some students at Stanford, like, other undergrads that are now sort of doing that with Hugging Face, so they're… 159 00:35:20.960 --> 00:35:23.700 216-101 Large Conference: goes for Hugging Face, and they're using some open source models to 160 00:35:23.810 --> 00:35:43.409 216-101 Large Conference: get these profiles, fragment-based datasets, and building this sort of, like, dataset finder, where, like, you can basically come with your data sets, you can sort of select which ones you have, and then it's going to recommend which ones you could add to really, like, sort of diversify your coverage and test for something that you're not testing for already. So that's another sort of use case of this, potentially. And finally, you could use this to construct 161 00:35:43.560 --> 00:36:01.649 216-101 Large Conference: predict performance in certain contexts or data points, and also construct adversarial data sets. And we actually do this in the paper. We have a bunch of appendix sort of experiments on this. We were able to predict performance pretty well of, you know, different data points, and also construct data sets where, like, the model just, like, failed on the vast majority of 162 00:36:01.650 --> 00:36:04.440 216-101 Large Conference: Of, of, data points. 163 00:36:04.440 --> 00:36:10.199 216-101 Large Conference: So you can use this, you know, information from this method in different ways as well. There are a different set of use cases. 164 00:36:10.740 --> 00:36:15.180 216-101 Large Conference: Now, I want to shift a bit. Oh, yeah, question? Yeah, before you go near the XX. 165 00:36:16.060 --> 00:36:26.809 216-101 Large Conference: I have a broader question, and it's not very well formulated. But I think the point here is you're checking the quality of the benchmark based on a model. Yes. 166 00:36:27.230 --> 00:36:31.079 216-101 Large Conference: So the quality of the model. 167 00:36:31.190 --> 00:36:34.430 216-101 Large Conference: in a sense, decide the quality of the benchmark. 168 00:36:34.770 --> 00:36:51.030 216-101 Large Conference: Do you have, like, a benchmark model that you're using to benchmark your benchmarks? It's excellent. Thank you, guys. This is great. I didn't pay either one of them to help me for this next section. So we're gonna actually… oh, whoops. So this actually should be, 169 00:36:51.080 --> 00:37:03.009 216-101 Large Conference: Yeah, well, the label here should be different, but yes, I'll get into this in the sort of remaining parts of the talk. So, it's a great question. Before you go on, I also have another question, so I'm very curious about… you constructed the adversarial evals, or somebody's working on that. 170 00:37:03.100 --> 00:37:07.199 216-101 Large Conference: So, for that… so I'm just trying to imagine what that process is. 171 00:37:07.300 --> 00:37:10.410 216-101 Large Conference: Would look like, and what kind of things would end up 172 00:37:10.560 --> 00:37:24.660 216-101 Large Conference: an adversarial eval. So, I guess the… the… Physical question is, is… Is this, is this… choosing… 173 00:37:24.800 --> 00:37:29.299 216-101 Large Conference: Specific instances of tasks that have been seen, and 174 00:37:30.180 --> 00:37:39.080 216-101 Large Conference: Different benchmarks and gathering them together, because it's, it's very dense in things that aren't tested by an existing 175 00:37:39.270 --> 00:37:46.020 216-101 Large Conference: E-value have, or is it synthesizing new tasks, new questions that haven't been seen at all? 176 00:37:46.120 --> 00:37:53.829 216-101 Large Conference: to try to, like, which one of those is this activity? Yes, yes. There's a bunch of them. So, so you can go from, like, the very, like. 177 00:37:53.830 --> 00:38:08.880 216-101 Large Conference: data-driven, or just, like, purely just, like, let the model do its thing, where you can just, like, steer with these, and, like, there are… there are clever ways of steering, where it's, like, not producing full-on jargon, but, so you can do that. You can just, like, say, okay, like, during these areas are, like, really poorly represented, and let's just steer towards them, and then you get… 178 00:38:08.880 --> 00:38:15.370 216-101 Large Conference: You know, samples that are, like, that will do the job, but might not be necessarily meaningful to humans, or might not be, like, reasonable tasks. 179 00:38:15.370 --> 00:38:25.669 216-101 Large Conference: So that's, like, one end of the spectrum. So I'm very curious what such a task looks like. Oh, I can… I don't… I can bring that up maybe after the talk. Yeah, so that's one. 180 00:38:25.670 --> 00:38:29.239 216-101 Large Conference: sort of end. And then, I don't have it in the deck, I think. 181 00:38:29.240 --> 00:38:53.590 216-101 Large Conference: I do get into this notion of, like, model performance and, like, model gaps as well, where we're gonna see a bit more of that. And then there's another one where, like, you can do something like, search. Like, if you have just, like, a big pool of tasks, you can just, like, rank them and just, like, choose from them. And then also, one thing that we had is we tested this, sort of, in between where we have, some starting points. Like, you can sort of take math, and then just make math more difficult. So, like, try to, like, use these data points and, like, sort of 182 00:38:53.640 --> 00:39:03.569 216-101 Large Conference: turn them, you know, like, towards concepts, or send them towards concepts that are, like, fully represented. So they were like, cool, I'll just, like, start from scratch, generate fully, fully synthetic data points. 183 00:39:03.660 --> 00:39:16.119 216-101 Large Conference: dataset search and selection, and then also, like, steering. And those… those all work to different degrees. And then there's also this idea where you can predict performance, based on this as well. Great. 184 00:39:16.890 --> 00:39:27.559 216-101 Large Conference: Two more… yeah, I have a more general question. I mean, if you've seen, like, in a lot of, like, NLP literature, there are, like, so many benchmarks on the same topic, like, every other paper would be, like, a benchmark. 185 00:39:27.560 --> 00:39:40.060 216-101 Large Conference: like, I've worked on emotions, and within that emotion topic, they'll be, like… and they'll have very slight differences. Yes. Like, oh, this is… like, the person has started disclosing their emotion, the other person is, like, third person. Yeah. So, is there a way that… 186 00:39:40.200 --> 00:39:48.730 216-101 Large Conference: This… this method can somehow merge them together to identify, like, what does the model really need in terms of 187 00:39:49.380 --> 00:40:05.459 216-101 Large Conference: Interesting. Yeah, yeah, no, I think this is a great question. I think that you could actually take the idea of, like, you know, like, this sort of feature that I mentioned here, like, selecting optimal benchmark suites, like, you could sort of make that more granular and just, like, get a profile for each sort of data point, and then construct something that would be sort of, like. 188 00:40:05.460 --> 00:40:22.729 216-101 Large Conference: you know, given some constraint, like, given that I only want to use 100 data points, what is the best, most diverse, sort of, setup or profile for my data? So I think you could totally do that. I think, and what I'll try to get at with this talk in a more, sort of, general sense is that, these methods are, like, some examples of things where we try to use INTERP 189 00:40:22.730 --> 00:40:46.120 216-101 Large Conference: to design better benchmarks or understand benchmarks better. I'm not saying… I mean, if you want to use a specific method, I think that would be great. I think there might be even many other ways to use different interp methods that we are… you're building, other people are building, to just, like, think about evals in different ways. I think there might be already, like, since this paper, I think there might be newer, even better, more sort of, like, streamlined ways to do what you're saying, but yes, I think you could totally just, like. 190 00:40:46.230 --> 00:41:07.070 216-101 Large Conference: you know, like, sort of refocus the granularity from benchmark as, like, a… as an object to, like, a data point as an object, and then just, like, sort of, like, shifts or remix, benchmarks. Actually, so the thing that, it's something that, I'm sort of helping or advising these, these, some more junior students on, and they're doing this on Hugging phase, and they're sort of… they want to do this report of, like, how much we're actually, like. 191 00:41:07.070 --> 00:41:11.640 216-101 Large Conference: contributing to the diversity. And, like, obviously there's limitations to this, like, there's things that are not captured in this. 192 00:41:11.640 --> 00:41:24.329 216-101 Large Conference: And so there's, like, a bunch of different methods that they're using to do this, but, yeah, like, there's a lot of things that are just, like, kind of redundant in some ways. Like, you're not just adding… or they're not adding, much new, sort of, tested coverage or value in some sense. 193 00:41:26.170 --> 00:41:28.620 216-101 Large Conference: Do models perform measurably worse on 194 00:41:29.230 --> 00:41:46.940 216-101 Large Conference: bench, like, selected benchmarks that are covering concepts that are not usually covered in the benchmark suites? Usually, yes. And we'll actually get into this in just a second. So it's a good question. I think there was one more question over there, or no? Maybe not. Very cool. Alright, okay, so, we covered the, 195 00:41:47.000 --> 00:41:57.889 216-101 Large Conference: the distributions and the evals, themselves with the benchmarks themselves. Now, if we shift focus to, the models themselves and how to use interp directly on the models, 196 00:41:57.890 --> 00:42:14.540 216-101 Large Conference: I want to sort of consider this other question. Can INTERP itself be used as means of evaluation? And there are a couple of examples that I want to offer here, but first I'll start with the models themselves, and sort of using similar ideas from just a second ago to get more granularity within benchmarks. 197 00:42:14.540 --> 00:42:15.919 216-101 Large Conference: So what I mean by this is 198 00:42:15.920 --> 00:42:38.240 216-101 Large Conference: If I'm able to… so, continuing on that, like, transition train, so we still have our, you know, our model here, we still have our SAE and our benchmark B, and concepts and DSE. If I'm able to score each… the performance of a model on each data point, on this range from 0 to 1, so I'm just gonna, like, normalize whatever metric the benchmarks are using to 199 00:42:38.240 --> 00:42:56.940 216-101 Large Conference: to this range from 0 to 1, then we're, we're defining this, sort of, like, model-side performance of a concept in the following way. So, we're saying that the performance of the concept C on benchmark B will basically just be, you know, like, weighted, weighted sort of activation of that concept 200 00:42:56.940 --> 00:43:12.289 216-101 Large Conference: over, weighted by the performance, or by the score on this data point, over the entire activation. So I'm just going to take the activations of this concept across all data points, and just sort of weigh them by, the score of that, of that, on that data point. 201 00:43:12.470 --> 00:43:36.590 216-101 Large Conference: And so if you do that, all of a sudden, for each concept in your dictionary of your SAE, you're going to get a score. And so, you know, we're sort of moving away from these individual scores, right, where we just have one score for the whole model, for the benchmark, to a distribution, where each concept gets its own score for the same benchmark. And so all of a sudden, you have a distribution and not necessarily just a single score. And what happens, or what tends to happen. 202 00:43:36.600 --> 00:43:50.730 216-101 Large Conference: as you notice here, is that for these benchmarks that, for example, AGI eval EN, is ABCD, right? It's a multiple, like, multi-choice question. And so, if the median score, like, the median per concept score is 0.25, 203 00:43:50.730 --> 00:44:00.350 216-101 Large Conference: that's basically guessing, right? Like, that is guessing. And what happens is that the concepts that are here, they're really, like, you know, like, above .75, like, in this, like. 204 00:44:00.600 --> 00:44:12.310 216-101 Large Conference: 0.8.9 range, those are the ones that, on the other plot, were the ones that are, like, sort of, like, maximally represented. And so, the few concepts that are sort of generally the most 205 00:44:12.310 --> 00:44:26.559 216-101 Large Conference: overrepresented are the ones that are, like, really well performing, to your question, over there. Whereas the wider distribution is actually performing in a more, in a poorer or more random, way. What space? The x-axis? 206 00:44:26.580 --> 00:44:46.139 216-101 Large Conference: And the y-axis, sorry, I'm just… Yes, so the y-axis is the number of concepts. So here, you know, like, we are assigning each concept in the dictionary in the SAE its own score, right? So going from one score for each AI value n, where we're going to give each concept its own score, and so here, on the y-axis, we have the number of concepts. 207 00:44:46.140 --> 00:44:58.830 216-101 Large Conference: And here, the x-axis is a normalized score. So, whatever the benchmark, you know, some benchmarks assign, like, you know, points from 1 to 10, and we're just gonna normalize it to from 0 to 1. And so. 208 00:44:59.040 --> 00:45:08.589 216-101 Large Conference: The models were generally, well, so actually, the model we were testing, we generally get performance around here, like, it was at least about .8 on all these benchmarks. 209 00:45:08.590 --> 00:45:22.479 216-101 Large Conference: But then, really, if you get the full distribution, the median score, the median per concept score was actually, like, much, much, much lower. And so what I'm seeing, the claim here is the few concepts that we just saw were overrepresented, or driving up the performance. 210 00:45:22.480 --> 00:45:38.579 216-101 Large Conference: Whereas there are other concepts that are actually performing much worse? There was a question in the back first. I just wanted to make absolutely sure I'm understanding these figures. So, this is a histogram, and if I summed over this, I would get the dictionary size of this Barcelencoder? That's right, yes, the number of elements plotted here are… yes. 211 00:45:39.300 --> 00:45:55.140 216-101 Large Conference: Why do I care if the airplane concept does bad on Mathbench? Exactly, so, great, great question. So again, we can sort of put this together, and, or sort of, like, sort of join this in… in something like this, right? So, like, first of all, you're right to say that, like. 212 00:45:55.350 --> 00:46:11.169 216-101 Large Conference: for some benchmarks, or for some tests, we just don't care about a handful of concepts, and we've done a bunch of experiments where we filtered for those concepts. So, like, you can sort of go through all the auto-interp, labels, and just sort of say, okay, like, for this state of goal, for this, like. 213 00:46:11.220 --> 00:46:19.899 216-101 Large Conference: for this, for this competency, which, like, is this a worthy concept to test? And, so that's one thing. Second thing is that, sometimes. 214 00:46:19.900 --> 00:46:32.109 216-101 Large Conference: You can also think of a competency or a performance within a given scenario, right? Like, you can ask the same question, like, you can ask, like, the same question of, like, you know, if Bob goes to the store and buys 10 watermelons. 215 00:46:32.410 --> 00:46:49.239 216-101 Large Conference: yeah, right? Like, you have a problem, but then you can, like, replace the concepts in that, and we actually see that for… we have this sort of funny, example in the paper, where you can actually look at the concepts, like, really, like, poorly performing, right? So, like, if we find, like, an object, like, say, you know, like, if… 216 00:46:49.240 --> 00:46:52.490 216-101 Large Conference: Say, hypothetically, apples were, like, really, really poorly performing. 217 00:46:52.490 --> 00:47:16.349 216-101 Large Conference: And then, you know, I don't know, like, watermelons were, like, really well performing. Well, if you replace these concepts with these concepts here, even in problems that are, like, overall, you know, like, pretty, like, well, solved by the model, you can actually, like, flip the performance or flip the, the results, which is interesting, to us. It's interesting. And so, we're making a claim here that, even though not all concepts must be critical to, to, like, all 218 00:47:16.350 --> 00:47:23.710 216-101 Large Conference: benchmarks. You do care about your overall performance for the distribution, or you might want to care about it in certain ways. 219 00:47:24.200 --> 00:47:35.470 216-101 Large Conference: So that's one, sort of way of looking at it. Now, to your question here, okay, but what if it comes to… So can you, can you, summarize the… 220 00:47:35.680 --> 00:47:40.850 216-101 Large Conference: So, the physical procedure for getting a score for a concept? Yes. 221 00:47:40.880 --> 00:47:53.159 216-101 Large Conference: So, I go back here, so we're… this is our, our score, for concept C, in, in Benchmark B, or for benchmark B. 222 00:47:53.160 --> 00:48:02.069 216-101 Large Conference: Where, we're basically summing over all the, or, like, the entire activation, all the entire energy of this content and the benchmark, and which has been a way 223 00:48:02.070 --> 00:48:25.580 216-101 Large Conference: that… that activation by the… by the score, or by the performance, which is scored here, right? So, like, if… if it's a multi-choice, then usually you just get, like, a zero, you get one if it's correct, you get 0 if it's incorrect, so you just weigh the… So for every, like, so, like, so for example, every question on the benchmark, do you know which concepts are involved in that? That's right, that's right, yeah, yeah. And then you… you… 224 00:48:25.580 --> 00:48:31.610 216-101 Large Conference: you, you basically… Weight everything by the concepts involved in that 225 00:48:31.900 --> 00:48:37.690 216-101 Large Conference: Yes, so we take the energy of the concept in the benchmark. 226 00:48:37.690 --> 00:48:54.570 216-101 Large Conference: And then at each data point, so, like, the data point here is I, we're gonna take the score, right? So, like, say, like, the data point that I was showing earlier, with the fractions and math, you would either get a zero, if it's incorrect, or 1 if it's correct, and so for that data point, here we just, like, get zero 227 00:48:54.570 --> 00:49:11.449 216-101 Large Conference: times the activation, but here it would still be counted. So that way you sort of get, like, a better sense of… or you get a sense of how the concept is doing, sort of, if it's, like, heavily represented in any given day. Right, but now here you've boosted all the concepts, equally, right? That's right. That's right. It's exactly right, yes. 228 00:49:11.450 --> 00:49:19.400 216-101 Large Conference: But at the same time, like, if your content is not heavily represented in a benchmark, it's, like, pretty low, pretty noise, we do see it generally just, like. 229 00:49:19.400 --> 00:49:25.100 216-101 Large Conference: So average out, and it doesn't really, doesn't really swing either way. But yes, that is true. 230 00:49:25.910 --> 00:49:33.890 216-101 Large Conference: Can you very briefly also explain, why should I care about the distribution? Yes, so we're saying that, 231 00:49:33.890 --> 00:49:51.530 216-101 Large Conference: even if there are specific concepts that are, are, like, critical or, like, sort of hyper-specific to your problem, like, say for math, like, you know, there's going to be variables and, like, addition, like, all these different concepts, and they're sort of very critical to your task, so they're pretty clear. Even if, 232 00:49:51.530 --> 00:50:14.509 216-101 Large Conference: And so I guess one thing you could do, and we do this in the paper, is that we can just filter for those, and just say, like, okay, I'm gonna only consider these concepts, and I care about their performance, their breakdown, and then on its own is informative, right? Like, if you have a math benchmark, you can all of a sudden get a better sense for, like, if your model is doing well on, you know, like, angles or geometry, or if it's doing well on, you know, calc or whatever, like, already that can give you a sort of sense of a breakdown. 233 00:50:14.510 --> 00:50:15.690 216-101 Large Conference: within your domain. 234 00:50:15.690 --> 00:50:34.189 216-101 Large Conference: But we show that in specific cases, the mere presence of a different concept can all of a sudden flip the performance of the model. So, like, the example that I gave earlier was that if you have some basic math word problem, you can just, like, introduce new concepts or new objects to the problem, and they'll actually, like, drive or influence the performance of the model. 235 00:50:34.190 --> 00:50:50.090 216-101 Large Conference: on those tasks, which we do see. And so, like, we can causally show that, at the end as well. So, that still might be the case of, like, maybe not all the concepts are important, because we know that NSA, there are many concepts that are sort of just, like, either weird, or, like, you combine many things, and so, yes, like, ultimately, like. 236 00:50:50.090 --> 00:51:08.429 216-101 Large Conference: probably not all concepts, or definitely not all concepts, are, like, critical to you, but I think it's… it's interesting to look at the full proper distribution and get a sense for… for just how, sort of variable it is. And then, like, the one score… I guess the main thing we're trying to say here, the claim here, is that this one score in the 237 00:51:09.290 --> 00:51:10.040 216-101 Large Conference: Oops. 238 00:51:10.690 --> 00:51:21.359 216-101 Large Conference: here is adding a lot, and you might not need to consider every single concept here, but I think it's noteworthy, or it's worthy of consideration, at least, at certain points. 239 00:51:21.800 --> 00:51:25.130 216-101 Large Conference: Does CHIBC model correlate with CHIBC benchmark? 240 00:51:25.600 --> 00:51:27.320 216-101 Large Conference: Bnc? 241 00:51:27.490 --> 00:51:34.749 216-101 Large Conference: Yes, so they, they do in… in… they… they mostly do, but there are, there are, like, interesting differences as well. 242 00:51:38.760 --> 00:51:39.540 216-101 Large Conference: Cool. 243 00:51:39.830 --> 00:51:41.879 216-101 Large Conference: Cool, awesome. So let's… 244 00:51:42.180 --> 00:51:46.510 216-101 Large Conference: Let's continue. I might actually just, like, stand up this, this is so much better than sitting. 245 00:51:46.540 --> 00:52:02.170 216-101 Large Conference: Okay, so back to your question. So, like, okay, like, this is… this is fun or whatever, like, you know, we can, like, use models, you know, internal activations to understand the distribution of what's being tested, but, like, what if our model is just, like, really, really bad at representing these different things? 246 00:52:02.170 --> 00:52:14.999 216-101 Large Conference: And does this really, like, let us truly consider what is not being, you know, even represented in the model, or, like, being sort of well, worked with in the model? Tlbr, like, what if the model sucks? 247 00:52:16.570 --> 00:52:38.500 216-101 Large Conference: And so, in particular, we asked this question of, are there concepts? The models… so it's a bit of a different question, but I think it sort of gets at this bare question, like, if the model, you know, is not good at, at representing certain concepts, where we previously asked, are there concepts, are there things that models see during training, but are unable to actually represent, or unable to generate at the end? 248 00:52:38.500 --> 00:53:02.179 216-101 Large Conference: And this is where I'm going to flip from language modeling to image modeling, and so to do that, well, actually, two things I want to do. First of all, I want to acknowledge David's work here, because David has this cool paper about what GANs can generate from 2019, and that was very much an inspiration for this paper. And also, thank you for the notes on our paper here. They're very helpful. So, you know, that's already been done before, but we sort of brought this to 249 00:53:02.180 --> 00:53:09.939 216-101 Large Conference: diffusion, and also access to new ways of, sort of, understanding, this, on a, sort of, bigger scale. 250 00:53:09.940 --> 00:53:25.100 216-101 Large Conference: The modification that I'll make here to SAEs, this is where I talk a bit about how SAEs work in vision, so this is the slide that I had on previously about SAEs, right? Like, we're taking a very dense representation, we're trying to get a bunch of sparser concepts from it. 251 00:53:25.340 --> 00:53:43.270 216-101 Large Conference: in images or in vision, one thing people do is they try to, decompose certain representations, like dyno, right? So, like, if we… if we want to use dyno, we could apply SAEs, on Dyno, you know, activates their, I guess, representations, really, and also get some concepts. So it's not… 252 00:53:43.270 --> 00:53:54.460 216-101 Large Conference: so much about getting intermediate layers or intermediate activations, it's really about the final representation from the model, and so that's what we'll be doing here as well. We'll be using Dyno for this work. 253 00:53:54.460 --> 00:54:13.309 216-101 Large Conference: Now, a bit more notation here for those who are interested. I think the TLDR here, the goal that I'm trying to do here, is that we'll try to use SAEs as energy models, because we ultimately want to consider the energy difference between what is in the training data, like what the models see during training, and what the models are able to 254 00:54:13.470 --> 00:54:30.920 216-101 Large Conference: to generate at the end. So, if you're… if you want to see the notation here, I can maybe go over this in more detail later as well, but the goal here is, okay, so we have a distribution, right? Our problem is, are there things that the models see during training but are unable to generate at the end? And so we're going to create these two 255 00:54:31.010 --> 00:54:49.039 216-101 Large Conference: these two energy models, or this sort of energy model, and we'll consider the energy of each concept, in the training distribution versus the one in the, in the, in the actual, like, generated, generated distribution, which, which we'll, get to… so I'll, I'll… 256 00:54:50.500 --> 00:54:58.649 216-101 Large Conference: for people who want to sort of see this derivation, we can get into it later as well, but I think the goal here is to just say that we will define this energy difference. 257 00:54:58.840 --> 00:55:13.450 216-101 Large Conference: here, for concept K, where it'll be a difference between the energy of a particular concept, where concepts correspond to our SAE, between, what I just said, the generated distribution and the distribution and training. Now, 258 00:55:13.970 --> 00:55:19.469 216-101 Large Conference: yeah, if there's questions about this, we can get into that later. I don't want to, yeah, I'll just sort of, 259 00:55:19.540 --> 00:55:34.079 216-101 Large Conference: continue with this. And then once we have this… once we have this energy, right? So, like, once you have this energy difference, we can define the notion of a blind spot, right? So, like, a thing that appears in training, that, like, its energy in training is high, or it exists. 260 00:55:34.080 --> 00:55:47.349 216-101 Large Conference: And then it is not in the, sort of, generated distribution. And so we're going to define this suppressed conceptual blind spot as something where the energy difference is smaller than, than a certain threshold that we're… that we're going to find 261 00:55:47.350 --> 00:56:00.250 216-101 Large Conference: That we're going to define. And then on the other hand, there can also be things that are, sort of exaggerated by the model, right? Like, things that don't appear as often. Its energy in the distribution is not as high as it is in the actual generated distribution. 262 00:56:00.250 --> 00:56:21.769 216-101 Large Conference: Now, to get to the, you know, more practical matter, what we do is we have a data set of many, many, many, many prompts, or I guess, like, captions and images that were used by the model during training, and then to have this matching distribution, or matching condition distribution, we're going to use these same captions to produce images from the model. So that's how we're going to get the training distribution, and also sort of general distribution. 263 00:56:22.020 --> 00:56:29.260 216-101 Large Conference: And so we did this some time ago, so the models here are a bit older, but I'll show you some examples of this. 264 00:56:29.340 --> 00:56:44.569 216-101 Large Conference: We, in particular, used Stability Fusion 2.1, and then a bunch of additional models that you'll see in a minute, like Adinski and Pixar and stuff like that. But so, you know, back to the question, are there things that these models saw during training, but were unable to generate, produce? 265 00:56:44.570 --> 00:56:50.379 216-101 Large Conference: And so we have this whole sort of range from, like, you know, this energy difference from zero, which is suppressed if there's some blind spots. 266 00:56:50.380 --> 00:57:05.470 216-101 Large Conference: to around 1, which is exaggerated, and you do, in fact, see concepts that are just like blind spots. So, this bottle just was unable, for the love of God, to produce a bird feeder. This is the closest thing we ever were able to get to a bird feeder, which was actually described around, right? So, like. 267 00:57:05.510 --> 00:57:13.959 216-101 Large Conference: I'll get to this in just a second, but if you ask this model, give me a bird feeder, it just would not do it. Like, it would give you a tree or something, but, like, not a bird feeder. 268 00:57:13.960 --> 00:57:33.219 216-101 Large Conference: And then if you really, really, really wanted to try to get out of the model, you could say, like, it's a thing that's hanging from the tree, and there are birds, and, like, there's, like, seeds in it, and so you could do that, and it was still, like, this is the closest we got to, and we did over, like, 10,000 generations with this model from, like, a lot of different, sort of conditions and, you know, probes and stuff. 269 00:57:33.220 --> 00:57:39.719 216-101 Large Conference: Similar to orange background. Like, for love of God, it just was not able to produce an orange background. 270 00:57:39.720 --> 00:57:48.309 216-101 Large Conference: On the other hand… Even though these things are in… Exactly, exactly. I mean, we can point to specific data points where we can show that, like, there's orange background, there's bird feeders. 271 00:57:48.310 --> 00:58:04.410 216-101 Large Conference: Stuff like that, yes. So these are things that exist in the data set, during training, but are just, like, unable… the model is unable to produce them, in inference. Did you take into account the frequency? Yes, we did. That's what we sort of do with the energy model there, so that is accounted for. 272 00:58:05.110 --> 00:58:12.430 216-101 Large Conference: That's metaphor. If it didn't really appear that much, then, like, it shouldn't really be penalized. Like, it would sort of be just, like, around, yeah, like, it wouldn't be penalized. 273 00:58:12.850 --> 00:58:22.580 216-101 Large Conference: Does this show up similarly in the inverse generation process, or is it unique to this conceptual decomposition? It's a good point. It does appear, 274 00:58:23.520 --> 00:58:42.929 216-101 Large Conference: to a certain extent and certain distributions. So, like, it depends, is the answer. And so you can then sort of distinguish that, because if you're… your question, which I assume you're sort of asking, like, why is this happening? You can sort of… or maybe not, but, like, that's… you can sort of ask that to… to get to that. And then… then there's, like, different reasons for why this is happening, basically. 275 00:58:42.930 --> 00:58:46.809 216-101 Large Conference: Okay, my question is more like, couldn't I see this by looking at the inverse G? 276 00:58:46.810 --> 00:58:51.749 216-101 Large Conference: But if I can't, then this is very interesting. Yeah, yeah, yeah. For many, for many, you can't. 277 00:58:51.840 --> 00:59:08.290 216-101 Large Conference: Cool. So, and also, I just wanted to sort of highlight that there are concepts that don't appear as much in training, but are just, like, heavily overrepresented. So, like, you might have noticed that if you ask a model, or this is true still of today's models, if you ask the models to, like, you know, produce, like, 278 00:59:08.410 --> 00:59:18.290 216-101 Large Conference: anything, like, you know, like, it could be, like, a phone case, could be, like, some food. They'll put it on, like, a wooden background. And that's the one concept that was, like, heavily over, over-represented and over-exaggerated. 279 00:59:18.290 --> 00:59:31.029 216-101 Large Conference: in the, in the produce distribution. And so all of these were actually, predicted, or sort of, like, found with our method, and then actually panned out, when we tested for them, in, in, you know, in, like, in, in practice at inference. 280 00:59:31.280 --> 00:59:37.850 216-101 Large Conference: And there are these sort of thresholds that we found. It was calculated through the data set, not just, like, arbitrarily. 281 00:59:38.590 --> 00:59:46.630 216-101 Large Conference: some more examples here. So these are actually, so there are certain blind spots, or certain gaps that, exist 282 00:59:46.630 --> 00:59:57.399 216-101 Large Conference: Not just in a single model, but across different models, so that's… this is one of them that we found in multiple models. And I also want to say that, like, we tested for, like, a lot of different 283 00:59:57.400 --> 01:00:09.710 216-101 Large Conference: We were trying to basically, like, induce the model to produce the concept through many different trials and iterations, as well, and we were able to point to, as I said earlier, specific data points from the training set, where they existed, like, where they were clearly present. 284 01:00:10.460 --> 01:00:19.309 216-101 Large Conference: And so there's more, there's also, like, a black computer screen, like, I just want to have, like, a dark computer screen. Like, that was, like, very hard to do for some of the earlier models for some reason. 285 01:00:20.170 --> 01:00:28.659 216-101 Large Conference: I guess this sort of points to the difficulty of just, like, solid color, which was a difficulty for some time. Now it's mostly salt in many more recent models. 286 01:00:29.910 --> 01:00:45.860 216-101 Large Conference: And then here, yeah, this is sort of pointing to the wood in the background, where, like, this is something that appears across the models, where, you know, SD1.5, 2.1, Pixar, Kadinsky, they all have a tendency to just, like, heavily over-represent, over-represent wood or wooden backgrounds, specifically. 287 01:00:46.550 --> 01:01:03.309 216-101 Large Conference: And so you can also look at this from, like, a more distributional perspective, where, again, so we're going to plot the energy differences of these different concepts. So we have this density on the y-axis, the energy difference on the x-axis, so here, 0 would be the blind spots, 1 would be 288 01:01:03.470 --> 01:01:11.919 216-101 Large Conference: sorry, the exaggerate concept, and so in this case, we actually found that there are things, and, you know, I sort of showed you some examples of these, where, in fact. 289 01:01:12.220 --> 01:01:19.489 216-101 Large Conference: The model sees concepts and training stats, sorry, doesn't see them, or it's unable to produce them in first time. 290 01:01:20.960 --> 01:01:38.029 216-101 Large Conference: One thing we also tested at that time was, effects of DPO. So, if you do post-training, you actually get, a much, sort of narrower, citizen median, or you should actually, like, do change the distribution quite significantly. So, DPO seems to be helping in many ways, for image models. 291 01:01:39.510 --> 01:01:47.760 216-101 Large Conference: And then some of the models also, sort of correlate, so you see that SD2.1 and 1.5, correlate, quite, quite well, or quite heavily. 292 01:01:49.440 --> 01:01:57.119 216-101 Large Conference: And so, I think overall, this, you know, back to your question, this goes to show that there are certain gaps in models that exist. 293 01:01:57.120 --> 01:02:13.299 216-101 Large Conference: one way to get around that is to just use, like, a different Activision or a different representation. And so, here we, you know, we were working with these models, like SD or Kadinsky and PixArt, and we specifically chose to use a different distribution that we trusted, to sort of, like, get at the blind spots of the model itself. 294 01:02:13.300 --> 01:02:21.030 216-101 Large Conference: And so that's one way of getting around this issue. Like, if you have a different representation or a different space that you can enter to then search or use for your interp. 295 01:02:21.030 --> 01:02:32.400 216-101 Large Conference: work and evals, that seems to be a pretty good way to go, because here we were able to, in fact, show these blind spots using, like, a different representation. It's not perfect, but it's one way to get around that issue. 296 01:02:33.500 --> 01:02:41.270 216-101 Large Conference: Cool. So, I think on time, because we started 12-10. So… 297 01:02:41.270 --> 01:03:06.130 216-101 Large Conference: To summarize here, or the takeaways here, my goal was to show that you can use specific NSERP methods that we already use. Like, here there's just SAEs, but it could be anything else from NSERP as well, to help or aid evaluations in different ways. First, we looked at gaps in evals and benchmarks, and we were able to actually find some and then use it to predict or build better benchmarks. Second, we were able to actually take existing benchmarks and scores and bring them down into more distributions 298 01:03:06.130 --> 01:03:18.539 216-101 Large Conference: and get a better sense or better feel for different parts of the model, and different sort of competencies, or get a breakdown. And then third, we're also able to find gaps in models, and what they're unable to produce, even though they see it in training. 299 01:03:19.560 --> 01:03:41.110 216-101 Large Conference: And so, you know, to really, like, zoom out fully and go back to these two questions we asked with the plane, when first I, you know, I asked, can we understand what our evals are missing? The answer to that is yes, and one way, I'm sure there's many other ways, but one way to do that could be through these model activations or representations and models themselves. They can tell us in a way where we're not testing for. 300 01:03:41.110 --> 01:03:45.020 216-101 Large Conference: And then, second, hopefully, I mean, my claim here is yes. 301 01:03:45.020 --> 01:04:00.130 216-101 Large Conference: And I guess the real test is whether you're convinced, but my goal here was to convince you that, in fact, we can use INTERP to not just do… I mean, there's many ways to be… to be interested about these models. I think there's just, like, inherent, you know, like. 302 01:04:00.130 --> 01:04:16.019 216-101 Large Conference: beauty, and then just, like, valiant understanding that systems and scientific pursuits. There's also the more, like, you know, like, practical interp, like, you want to improve models, or you can improve evaluations as well. And so there's another, hopefully, reason for you to care about interp, from, like, a different perspective. 303 01:04:16.020 --> 01:04:30.429 216-101 Large Conference: And before I wrap up, I want to just give a big shout out, and thanks to my collaborators and also advisors. Some of this work happened at DeepMind, some of this work happened at Stanford, and also a bit of it with Bill at MIT, and that's really 304 01:04:30.450 --> 01:04:40.959 216-101 Large Conference: it's… and I just… once again, like, if you're into planes, I have a lot of… there's, like, this one amazing documentary about this plane that I'll recommend if you're into planes. But yeah, thanks so much. Thank you, Matty. 305 01:04:45.360 --> 01:04:47.229 216-101 Large Conference: Any questions from Maddie? 306 01:04:50.740 --> 01:04:53.899 216-101 Large Conference: Yeah. In the case of language. 307 01:04:53.920 --> 01:05:08.600 216-101 Large Conference: How many SAEs did you train tonight, and did you compare, like, different concepts between the SAEs? Yes. We tried 5 different models, and different SAEs for them. So we had, we had some llamas, some gemmas, Deep Seek. 308 01:05:08.600 --> 01:05:21.099 216-101 Large Conference: And then some other… two different architectures as well. It's all, like, our appendices from… so this… so the competency gap paper was at ICML, and the, like, you know, at submission, I think we were at, like. 309 01:05:21.150 --> 01:05:23.780 216-101 Large Conference: 40 pages, and after rebuttals, we were at, like. 310 01:05:24.320 --> 01:05:34.759 216-101 Large Conference: almost, like, 100 pages of just, like, appendices and, like, additional experiments, because the reviewers wanted, like, additional models, additional benchmarks, so, like, there's a lot of additional things, and they… for all of them, we found, 311 01:05:34.800 --> 01:05:50.520 216-101 Large Conference: we verified everything, and, like, it always panned out. Obviously, there are things where, like, you know, it's not perfect, like, there are, like, failure cases, of course, but the overall claims that we made in the paper, held up across different models in the States. Not in the sense of perfection, but maybe there are, like, actual interesting 312 01:05:50.530 --> 01:06:00.510 216-101 Large Conference: differences between them that you have? Yes, yes, so the differences, I would say that, like, the… so, you know, these different models usually, like, also have pre-trained assays, and so we use pre-trained assays 313 01:06:00.510 --> 01:06:24.349 216-101 Large Conference: Some of them, and so there are different sizes, right? Like, there's some that are, like, you know, like, on this scale of, like, 80K, 100 plus K, also just, like, 20 or 30K, and so the granularity of the claims you can make changes quite a bit. And so we found that the smaller the SAE, the more, like, I guess it was much easier with the smaller SAEs to then do the adversarial stuff, like to steer, you know, like, to steer the difficult inputs and stuff. 314 01:06:24.350 --> 01:06:48.570 216-101 Large Conference: at the same time, the, like, the insights for, like, a human to, like, look at it and understand what's happening, it was pretty coarse in many ways, so, like, I guess if you really care about just, like, adversarial, like, you know, like, attacks or benchmarks or whatever, like, then you want a smaller essay, I think would be the claim or the finding. If you want more, I guess, like, you know, like, you as a human, as a human researcher want to, like, get a better feel for your model, I would say the, like, 70 to 80K 315 01:06:48.570 --> 01:06:50.720 216-101 Large Conference: concept size was, like, the sweet spot, I would say. 316 01:06:52.860 --> 01:07:06.109 216-101 Large Conference: So, I may have missed some context in between, but, like, from what I understand, you're having a concept being considered as, like, an SE feature, or, like, maybe a set of them. I guess, 317 01:07:06.330 --> 01:07:10.839 216-101 Large Conference: in that case, and, you know, I guess, you know, they asked about, 318 01:07:10.950 --> 01:07:14.629 216-101 Large Conference: choosing, like, from multiple SAEs. 319 01:07:15.130 --> 01:07:34.749 216-101 Large Conference: So, what would you say is, like, in general, like, a definition of a concept, based on, like, what you have, explored, and I guess, how noisy do you feel, like, you know, certain… like, because I imagine not all SAE features are necessarily concept, and so, yeah. 320 01:07:35.330 --> 01:07:42.130 216-101 Large Conference: Yes, yes. No, it's a good point. I mean, here we were kind of… 321 01:07:42.200 --> 01:08:01.170 216-101 Large Conference: I mean, in a way, like, we kind of escaped that question, having to define it, because we're sort of, like, using many of the pre-trained SAEs as well, so we sort of just, like, took them at their word, in a way. In my mind, I don't really… I think that the, the diversity of different concepts, or I guess, like, directions or features in these LLMs. 322 01:08:01.170 --> 01:08:07.030 216-101 Large Conference: Is… is a benefit, because there are some concepts that are very much about, like, an action, or like… like a task, or… or… 323 01:08:07.110 --> 01:08:24.459 216-101 Large Conference: something along those lines, or seemed to correlate with that. Some of them were much more semantic, just like, here's, like, I mean, they're watermelon, or whatever. And I think both were actually very helpful. I think the, so to that end, like, my… I'm using concepts here. I think any sort of units, that, 324 01:08:24.609 --> 01:08:38.900 216-101 Large Conference: are, you know, are able to, sort of discretize, what the LLM is working with in different ways are helpful to me, or, like, seem to be helpful for this. I would say that the, this is, like, the, the, you know, and this is not, like, this is kind of obvious, but, like, I think the… 325 01:08:38.899 --> 01:08:47.259 216-101 Large Conference: the worst ones were the… were the ones that were sort of, like, collapsed, or, like, would sort of combine different things together, where it's neither of those different directions, like, neither, like. 326 01:08:47.260 --> 01:08:54.830 216-101 Large Conference: a task, or, like, an activity, or, like, some, like, you know, some task, or either, like, a semantic concept, like, if it's somewhere in between, like, that's when it gets really… 327 01:08:54.830 --> 01:09:02.219 216-101 Large Conference: messy and complicated. So I think my definition of a concept, is, is, like, very abstract. 328 01:09:02.220 --> 01:09:16.229 216-101 Large Conference: And that's… I just sort of, like, want… Yeah, I think it's very, like, practical, if that makes sense. Like, I, like, as long as it sort of, like, tells me something new, something I can insight, whether it's, like, an activity or a task or, like, a semantic object, I don't really care. 329 01:09:18.490 --> 01:09:25.050 216-101 Large Conference: Assuming labs aren't, like, p-hacking or training on tests, why should I care if an eval isn't comprehensive? 330 01:09:25.750 --> 01:09:40.230 216-101 Large Conference: I think that, for me, the… I think, like, the first assumption is that if you train a model, and you care about its performance, and you care about its robustness, I think, you know, like, even… even if it's purely, like, adversarial, like, like. 331 01:09:40.270 --> 01:09:48.320 216-101 Large Conference: Knowing that your model has these limitations in certain areas, and, like, that, like, me being able to just, like, change a couple of 332 01:09:48.330 --> 01:10:03.369 216-101 Large Conference: you know, like, I guess, like, just a couple of words or a couple of concepts in the prompt or in whatever it is that we're working with, all of a sudden changes your performance, whether that's for cyber or, like, a different capability, I think that is noteworthy. I mean, there's a lot of… so I can't… 333 01:10:03.610 --> 01:10:05.610 216-101 Large Conference: Is this… are we still being recorded? 334 01:10:05.720 --> 01:10:12.989 216-101 Large Conference: Can we stop the recording now? Sure. I'll be able to say more. 335 01:10:14.240 --> 01:10:16.080 216-101 Large Conference: Thank you, Maty. Of course. 336 01:10:17.690 --> 01:10:18.600 216-101 Large Conference: Cool.