WEBVTT 00:05:55.000 --> 00:06:00.000 That way, everybody that reads slightly different things 00:06:00.000 --> 00:06:20.000 Everybody will read stuff relating to the same topic, but people will read slightly different things, and I think that might make discussion a little more interesting as well. 00:06:20.000 --> 00:06:22.000 All right. 00:06:22.000 --> 00:06:32.000 It is 3:05. I'm sending this once again just in case anybody didn't see it. 00:06:32.000 --> 00:06:41.000 Here are the results. Most everybody read some of it thoroughly or skimmed it. 00:06:41.000 --> 00:06:44.000 We will first go over 00:06:44.000 --> 00:06:46.000 A brief 00:06:46.000 --> 00:06:49.000 I'm going to give a very, very brief history 00:06:49.000 --> 00:06:53.000 Which is basically only two bullet points 00:06:53.000 --> 00:06:55.000 And then 00:06:55.000 --> 00:07:01.000 I think the best way to go about this would be for Adam to jump into the demonstration of the workspace tool. 00:07:01.000 --> 00:07:03.000 which is pretty cool. 00:07:03.000 --> 00:07:08.000 And then some wide-ranging discussion on cognition. 00:07:08.000 --> 00:07:11.000 Sounds good. 00:07:11.000 --> 00:07:18.000 All right, first of all, let me get a show of hands reaction wise 00:07:18.000 --> 00:07:24.000 Who is familiar with LogitLens 00:07:24.000 --> 00:07:27.000 At the very least, in the sense of like 00:07:27.000 --> 00:07:32.000 You've seen it around or you've heard of it. 00:07:32.000 --> 00:07:37.000 Okay, this is a very fair amount. 00:07:37.000 --> 00:07:39.000 Wonderful. 00:07:39.000 --> 00:07:48.000 Popular paper. This is a paper by a mysterious author known as Mistalgebraist. It was released in 2020. 00:07:48.000 --> 00:07:52.000 The important thing to know is it says 00:07:52.000 --> 00:07:53.000 Well 00:07:53.000 --> 00:08:11.000 The residual stream is a thing that keeps getting plus equals over and over and over again throughout the model, and then at the very end, we put the residual stream through the unembedding matrix. What if we put an earlier version of the residual stream through the unembedding matrix? And the answer is, you can kind of see what the model's thinking 00:08:11.000 --> 00:08:13.000 JLens 00:08:13.000 --> 00:08:18.000 is basically doing a fancy version of that, if that was not clear to everybody 00:08:18.000 --> 00:08:21.000 Jaylenz says, rather than 00:08:21.000 --> 00:08:25.000 putting everything through the 00:08:25.000 --> 00:08:30.000 rather than putting the residual stream through the unembedding matrix directly 00:08:30.000 --> 00:08:37.000 Which, by the way, just so that everyone's on the same page, when I say putting the residual stream through the unembedding matrix 00:08:37.000 --> 00:08:48.000 We're taking a vector and multiplying it by a matrix. But in reality, what we're doing is decoding something that is the model dimensionality into vocab space 00:08:48.000 --> 00:08:54.000 This is going to be very important for our discussion. The idea of decoding into vocab space. 00:08:54.000 --> 00:09:00.000 So regular logit lens decodes the residual stream directly into vocab space 00:09:00.000 --> 00:09:06.000 JLens, on the other hand, is proposing a small modification 00:09:06.000 --> 00:09:16.000 To the residual stream before putting it through the unembedding matrix to decode it into vocab space. This modification is the multiplication by a square matrix 00:09:16.000 --> 00:09:20.000 That is composed of Jacobians 00:09:20.000 --> 00:09:27.000 Honestly, it's an interesting methodology, but talking about how they put together this Jacobian matrix 00:09:27.000 --> 00:09:33.000 is basically irrelevant to what we want to talk about here today. So if you are very impassioned about 00:09:33.000 --> 00:09:43.000 Talking about Jacobians and calculus, I would suggest you start another NDIF reading group. 00:09:43.000 --> 00:09:58.000 The other important piece of context to know before going into this paper is this was released by Anthropic recently and before the release of this paper, I think on the scale of months before, not within less than a year before 00:09:58.000 --> 00:10:06.000 Anthropic started getting really, really, really into the idea that their models were sentient 00:10:06.000 --> 00:10:09.000 Regardless of whether or not that is true 00:10:09.000 --> 00:10:14.000 Because of this internal belief, it changed the way they talk about and treat 00:10:14.000 --> 00:10:18.000 The models themselves a lot of the time 00:10:18.000 --> 00:10:30.000 Hence, the fact that this paper is published purporting that the models have a global workspace, which is a term from human cognitive science 00:10:30.000 --> 00:10:37.000 comes with a lot of baggage from Anthropic being really excited about the idea that they've made some consciousness. 00:10:37.000 --> 00:10:40.000 Not trying to discredit that 00:10:40.000 --> 00:10:41.000 I'll wait 00:10:41.000 --> 00:10:46.000 at least 10 minutes before trying to do that. 00:10:46.000 --> 00:10:47.000 But 00:10:47.000 --> 00:10:52.000 It is 00:10:52.000 --> 00:11:02.000 This claim is very historically positioned, even though it's something that's happening right now. It is historically positioned in the history that we are currently in. 00:11:02.000 --> 00:11:08.000 Anyways, I think that's all I have to say is Precursor 00:11:08.000 --> 00:11:12.000 Adam, do you feel ready to show off 00:11:12.000 --> 00:11:14.000 cool, shiny stuff 00:11:14.000 --> 00:11:17.000 Yeah, thanks for passing it over to me. 00:11:17.000 --> 00:11:24.000 So hi guys, my name is Adam, I'm one of the NDIF team. We also have Emma and Jaden on the call. 00:11:24.000 --> 00:11:28.000 But sort of the infrastructure work that we've been. 00:11:28.000 --> 00:11:33.000 Though in the past few months was building this, I think we also have Zikai. 00:11:33.000 --> 00:11:35.000 Um, on the call for Godzikai. 00:11:35.000 --> 00:11:42.000 Uh, but we have this infrastructure part of this infrastructure work is, uh, we will bring in our interface. 00:11:42.000 --> 00:11:49.000 Form, it can, uh, quick and easy access to some interpretability methods for exploration. 00:11:49.000 --> 00:11:55.000 Part of the research process, so I'm gonna drop the link in the chat. 00:11:55.000 --> 00:11:58.000 Um… 00:11:58.000 --> 00:12:03.000 And if you want to head over there. 00:12:03.000 --> 00:12:09.000 And check out what we have. So you can access the models that we have on endiv directly through the interface. 00:12:09.000 --> 00:12:17.000 I'm going to share my screen… 00:12:17.000 --> 00:12:20.000 Um, yeah, I'm sort of already into my workspace here, but 00:12:20.000 --> 00:12:26.000 Um, yeah, this is a quick way to do some quick expression. We have logit lens, activation patching. 00:12:26.000 --> 00:12:31.000 Uh, you can even access our big models like Llama 45B. 00:12:31.000 --> 00:12:37.000 Uh, and we've been working on adding support for JLAN, so you can do the basic thing of just like. 00:12:37.000 --> 00:12:41.000 Reading intermediate hidden states using the Jacobian. 00:12:41.000 --> 00:12:45.000 I'm gonna try to show you some. 00:12:45.000 --> 00:12:50.000 what that looks like, and what you can see with it. So. 00:12:50.000 --> 00:13:00.000 Relating to the paper, we read last week, um, the backprop paper had this cool setup where they were training neural networks on 00:13:00.000 --> 00:13:02.000 Predicting relationship between. 00:13:02.000 --> 00:13:06.000 Or was it predicting maybe. 00:13:06.000 --> 00:13:13.000 My family members based on relationships. 00:13:13.000 --> 00:13:14.000 Family members, um, training them on, on trees. 00:13:14.000 --> 00:13:17.000 I think it was the other way around predicting relationships by the family members. 00:13:17.000 --> 00:13:22.000 And these models have gotten so good that you can just do it in context here. 00:13:22.000 --> 00:13:31.000 So I'm using one of the 70B models up right now, and we just passed the entire topology of the family tree in context using like. 00:13:31.000 --> 00:13:33.000 Equal signs and and. 00:13:33.000 --> 00:13:39.000 And arrows and then I just ask Charlotte is Victoria's blank and then answer and then. 00:13:39.000 --> 00:13:44.000 The model does say correctly daughter or predicted. 00:13:44.000 --> 00:13:50.000 You can see here, Charlotte is the product of Victoria and James. 00:13:50.000 --> 00:14:00.000 And to sort of read the results here of JLS on the right-hand side, so we have the layers of the model as the x-axis, and then the tokens from the prompt. 00:14:00.000 --> 00:14:07.000 On the y-axis, and each cell, what you're seeing is the result of applying the Jacobian on the hidden state. 00:14:07.000 --> 00:14:12.000 at that token position, and then the layer norm and then. 00:14:12.000 --> 00:14:14.000 Passing it through the embedded metrics. 00:14:14.000 --> 00:14:16.000 And you can also see what are like the top. 00:14:16.000 --> 00:14:21.000 Predictions from the logic distribution, you can sort of see how that. 00:14:21.000 --> 00:14:25.000 Token progresses throughout the model layers. 00:14:25.000 --> 00:14:37.000 And you can see that the model's thinking a lot about just daughter and think this this task might be too easy that is already locally in the right answer pretty early on. 00:14:37.000 --> 00:14:43.000 Um, but something… so an interesting observation that you can see with logit lens is. 00:14:43.000 --> 00:14:50.000 Even at the very beginning, at the first equal sign, you can sort of see the intermediary representative of married. 00:14:50.000 --> 00:14:54.000 Right there before even sees like. 00:14:54.000 --> 00:14:58.000 The partner, which I think is really cool, and this is just a product of it, seeing family tree. 00:14:58.000 --> 00:15:01.000 In the context. And, um, so. 00:15:01.000 --> 00:15:04.000 You can also try this with LogitLens. 00:15:04.000 --> 00:15:13.000 And so the color gradient represents the probability of the tokens. And so sort of like the confidence of the model in that prediction. 00:15:13.000 --> 00:15:17.000 And you can already see the contrast with J-Lens that. 00:15:17.000 --> 00:15:20.000 The tool logical is a lot more. 00:15:20.000 --> 00:15:26.000 Noisier, and so a lot of low probability tokens and random things would like. 00:15:26.000 --> 00:15:29.000 Come up top and. 00:15:29.000 --> 00:15:33.000 When I wanted to look at the equal sign again, I don't see merit. 00:15:33.000 --> 00:15:36.000 Come up there. Early or at all in this case. 00:15:36.000 --> 00:15:40.000 So just blank. 00:15:40.000 --> 00:15:48.000 So yeah, this is we're rolling out soon other features for our JLN support. So we'd love if you guys want to like. 00:15:48.000 --> 00:15:51.000 Drop any feedback or join the Discord, let us know what you like to see. 00:15:51.000 --> 00:15:54.000 If you come up with any cool. 00:15:54.000 --> 00:16:01.000 Uh, prompts or tests, that would be awesome. 00:16:01.000 --> 00:16:02.000 Thanks for that, Adam. 00:16:02.000 --> 00:16:04.000 Yeah, I hear it. 00:16:04.000 --> 00:16:11.000 Yeah, this is a really cool tool. I can't believe that this is just out there and I can use it 00:16:11.000 --> 00:16:17.000 This is the sort of thing where usually I have to write a few modules in order to 00:16:17.000 --> 00:16:22.000 get an experiment like this going 00:16:22.000 --> 00:16:27.000 I'm going to briefly use — thank you, Emma, Discord link in the chat 00:16:27.000 --> 00:16:31.000 I'm going to briefly use 00:16:31.000 --> 00:16:55.000 This as an example 00:16:55.000 --> 00:17:03.000 Do a brief talk through of LogitLens 00:17:03.000 --> 00:17:11.000 And then we are going to treat JLens as a fancy version of LogitLens. And we're going to spend maybe 5 minutes on that. And then we are going to get into the claims of global workspace. 00:17:11.000 --> 00:17:12.000 So 00:17:12.000 --> 00:17:37.000 I'm going to pull up the paper 00:17:37.000 --> 00:17:51.000 Let's start 00:17:51.000 --> 00:18:21.000 No, we're going to skip over the Jacobians bit and just assume one definition of what the Jacobian matrix is doing. 00:18:23.000 --> 00:18:27.000 Look like a final representation. 00:18:27.000 --> 00:18:33.000 That is a trustworthy claim. We're not going to get into why it's trustworthy, but it is a pretty good claim. 00:18:33.000 --> 00:18:37.000 Then they divide up this matrix into 00:18:37.000 --> 00:18:40.000 Vectors 00:18:40.000 --> 00:18:45.000 So a JLens vector is one row or column of the matrix 00:18:45.000 --> 00:18:52.000 But we should think of this as is a vector in residual space at the layer we're looking at 00:18:52.000 --> 00:18:59.000 Which activates the lens for a specific word in the vocab. 00:18:59.000 --> 00:19:04.000 So, when they say the J lens vector for 00:19:04.000 --> 00:19:11.000 What's one of the examples that they do right down here? 00:19:11.000 --> 00:19:14.000 The J lens vector for soccer 00:19:14.000 --> 00:19:21.000 The soccer lens vector, what that actually is 00:19:21.000 --> 00:19:24.000 A vector at the layer we're looking at 00:19:24.000 --> 00:19:29.000 that after passing through this matrix multiplication, looks like soccer 00:19:29.000 --> 00:19:34.000 at the final layer 00:19:34.000 --> 00:19:36.000 They make 00:19:36.000 --> 00:19:48.000 A very interesting observation, which is typically in the residual stream, only a few JLens vectors are highly present at one time 00:19:48.000 --> 00:19:55.000 Given that this is a high dimensional space with lots of superposition, this is sort of a surprising finding. 00:19:55.000 --> 00:20:00.000 Then they define a thing that they call JSpace. 00:20:00.000 --> 00:20:03.000 Which mathematically is like this 00:20:03.000 --> 00:20:08.000 Pick a number k, take all possible collections of K lens vectors 00:20:08.000 --> 00:20:17.000 Vectors for specific words in the vocab. Take this positive span of those vectors. Each of those spans makes a big k dimensional cone 00:20:17.000 --> 00:20:23.000 And then union all of those. So essentially what we have now is a subset of residual space 00:20:23.000 --> 00:20:26.000 Composed of linear combinations 00:20:26.000 --> 00:20:31.000 of columns of our matrix 00:20:31.000 --> 00:20:35.000 And now that we have a subspace, you can split 00:20:35.000 --> 00:20:40.000 any activation into the part that's in that subspace and the part that's not in that subspace. 00:20:40.000 --> 00:20:52.000 Does this make sense? Because this is an important time to pause 00:20:52.000 --> 00:20:59.000 Or what good definition of space is JSpace a space? It's not a vector space. 00:20:59.000 --> 00:21:04.000 This is a good point 00:21:04.000 --> 00:21:09.000 It is basically just a set of points. It is a 00:21:09.000 --> 00:21:14.000 They're defining space as 00:21:14.000 --> 00:21:16.000 A large set 00:21:16.000 --> 00:21:27.000 That set having a specific dimensionality, and that set being a subset of the vector space that is the residual stream 00:21:27.000 --> 00:21:31.000 But this is a good 00:21:31.000 --> 00:21:34.000 Dimensionality we're defining as the 00:21:34.000 --> 00:21:42.000 number of elements in the vector 00:21:42.000 --> 00:21:47.000 Technically, it's not a vector space because you're only taking the span on the positive sides 00:21:47.000 --> 00:21:52.000 If it was a vector space by formal definition, you'd have to be able to subtract 00:21:52.000 --> 00:21:53.000 Jalen's vectors too, which they're not allowing here 00:21:53.000 --> 00:22:01.000 I missed that. That's interesting. 00:22:01.000 --> 00:22:02.000 Yeah, that's confusing. 00:22:02.000 --> 00:22:08.000 It's not actually that important to their methodology 00:22:08.000 --> 00:22:12.000 The fact that they don't allow subtraction is, I think, a little bit 00:22:12.000 --> 00:22:14.000 weird 00:22:14.000 --> 00:22:16.000 The fact that they are not 00:22:16.000 --> 00:22:22.000 that it is not technically meeting all of the requirements for a vector space is not that important. 00:22:22.000 --> 00:22:24.000 But I think it goes to show 00:22:24.000 --> 00:22:29.000 If it was not clear, I don't really like this paper 00:22:29.000 --> 00:22:32.000 Exactly. 00:22:32.000 --> 00:22:34.000 I don't really like this paper 00:22:34.000 --> 00:22:40.000 And one of the reasons I don't like this paper is because I feel like it's very fudgy with its rigor. 00:22:40.000 --> 00:22:49.000 And this is one unimportant instance among many 00:22:49.000 --> 00:22:51.000 The 00:22:51.000 --> 00:22:54.000 But 00:22:54.000 --> 00:22:57.000 If everyone feels good about JSpace 00:22:57.000 --> 00:23:04.000 as a space, then we can move on. But I think it's very important that we understand 00:23:04.000 --> 00:23:15.000 how it is acting as a space, just insofar as how they have defined a subspace of the residual stream or a subspace of residual space 00:23:15.000 --> 00:23:16.000 Yeah, go ahead, Eric. 00:23:16.000 --> 00:23:25.000 So can you maybe walk us through their construction, I guess? Maybe that would make things clear? 00:23:25.000 --> 00:23:26.000 That's 00:23:26.000 --> 00:23:29.000 clearer and for the people that skimmed, I don't… maybe… 00:23:29.000 --> 00:23:33.000 That might be a good idea. 00:23:33.000 --> 00:23:41.000 The reason this is a little bit funny is because I'm trying to skip over this whole thing 00:23:41.000 --> 00:23:44.000 They use this big equation 00:23:44.000 --> 00:23:49.000 Which is a big expected value of some derivatives 00:23:49.000 --> 00:23:51.000 To say 00:23:51.000 --> 00:23:52.000 When I change 00:23:52.000 --> 00:24:00.000 the residual stream in this direction, it generally has this at the very end effect on the prompt. 00:24:00.000 --> 00:24:06.000 Or more importantly, it has this at the end effect on the residual stream right before we decode it. 00:24:06.000 --> 00:24:09.000 And if you do that over and over and over again. 00:24:09.000 --> 00:24:12.000 You get a 00:24:12.000 --> 00:24:19.000 Matrix that lets you predict if I bump the residual stream at layer 18 in this direction. 00:24:19.000 --> 00:24:21.000 it'll make the 00:24:21.000 --> 00:24:28.000 It'll make the final residual stream decode really heavily to cat. It'll basically inject the cat vector into 00:24:28.000 --> 00:24:31.000 the end of the residual stream 00:24:31.000 --> 00:24:35.000 They then take this logic and put it backwards and say 00:24:35.000 --> 00:24:43.000 Okay, we have found for any layer, this matrix that tells us what's going to make cat 00:24:43.000 --> 00:24:46.000 So let's trace this backwards and say 00:24:46.000 --> 00:24:53.000 I'm going to write this here, actually. We have a matrix that tells us at layer 00:24:53.000 --> 00:24:55.000 18 00:24:55.000 --> 00:24:58.000 Which direction 00:24:58.000 --> 00:25:04.000 Makes the last layer say cat. 00:25:04.000 --> 00:25:09.000 So then they say, let's take 00:25:09.000 --> 00:25:13.000 Let's look at that matrix 00:25:13.000 --> 00:25:18.000 And determine layer 18's most cat direction. 00:25:18.000 --> 00:25:22.000 Call that the cat vector. 00:25:22.000 --> 00:25:24.000 The most cat direction here 00:25:24.000 --> 00:25:26.000 is defined as 00:25:26.000 --> 00:25:28.000 the direction 00:25:28.000 --> 00:25:30.000 of perturbation 00:25:30.000 --> 00:25:35.000 That increases cat the most. 00:25:35.000 --> 00:25:37.000 At the end 00:25:37.000 --> 00:25:39.000 And then we call that the cat vector. 00:25:39.000 --> 00:25:44.000 Do this for every word ever. 00:25:44.000 --> 00:25:49.000 So now we have for every token 00:25:49.000 --> 00:25:51.000 We have a 00:25:51.000 --> 00:25:54.000 vector at our given layer 00:25:54.000 --> 00:25:57.000 that corresponds to 00:25:57.000 --> 00:26:03.000 What we're gonna call right now the concept corresponding to that token 00:26:03.000 --> 00:26:15.000 Because that is what they are heavily implying and trying really, really hard to avoid saying. 00:26:15.000 --> 00:26:17.000 This is how 00:26:17.000 --> 00:26:28.000 Each of these vectors is what they call the J space vector for blank. So anytime you see this pop up, that's what they mean. 00:26:28.000 --> 00:26:32.000 They make a slight jump here 00:26:32.000 --> 00:26:35.000 That jump in logic is 00:26:35.000 --> 00:26:39.000 The direction 00:26:39.000 --> 00:26:46.000 The perturbation at layer 18 that makes the model say cat at the end. 00:26:46.000 --> 00:26:50.000 Is roughly equivalent to 00:26:50.000 --> 00:26:52.000 The direction 00:26:52.000 --> 00:26:55.000 In Layer 18 00:26:55.000 --> 00:27:01.000 That represents cat 00:27:01.000 --> 00:27:03.000 At this point. 00:27:03.000 --> 00:27:08.000 I think it is important to jump into global workspace theory 00:27:08.000 --> 00:27:10.000 But 00:27:10.000 --> 00:27:20.000 I think this explanation is a little bit better than what I did before, but it's still fudging it a little bit. So if this is confusing, someone flag me down now. 00:27:20.000 --> 00:27:21.000 Okay. 00:27:21.000 --> 00:27:25.000 We had a question in the chat. I accidentally answered. I did not mean to answer it. I was like copying it to like think about it and put a quote and it doesn 00:27:25.000 --> 00:27:35.000 Just the cat vector here coincide with the SAE vector feature for cat? No 00:27:35.000 --> 00:27:37.000 It 00:27:37.000 --> 00:27:40.000 Does not necessarily 00:27:40.000 --> 00:27:47.000 They actually take SAE directions and further decompose them into 00:27:47.000 --> 00:27:54.000 combinations of J lens vectors. 00:27:54.000 --> 00:28:00.000 I don't think there is a consistent cosine similarity between J lens vectors and SAE vectors. 00:28:00.000 --> 00:28:08.000 Or if there is, I didn't read far enough for them to mention it. 00:28:08.000 --> 00:28:09.000 Yeah. 00:28:09.000 --> 00:28:10.000 Wait, sorry, a follow up on that? Like, I'm wondering, like, 00:28:10.000 --> 00:28:13.000 If you think about like an SCE vector, it's like a direct. 00:28:13.000 --> 00:28:19.000 Like it's from the normally when people do Syrian stuff, it's from the decoding step, right, so. 00:28:19.000 --> 00:28:21.000 So it's like a direct… 00:28:21.000 --> 00:28:22.000 composition of the residual residual stream. 00:28:22.000 --> 00:28:24.000 Yep. 00:28:24.000 --> 00:28:29.000 But like for this, you are basically like, for example, the cav actor for JSpace. 00:28:29.000 --> 00:28:32.000 It's like you're kind of multiplying like a. 00:28:32.000 --> 00:28:36.000 Like a Jacobian or like a gradient vector with, with the. 00:28:36.000 --> 00:28:38.000 unembedding matrix which. 00:28:38.000 --> 00:28:43.000 It has the same dimensionality with the residual strip space, but like it's. 00:28:43.000 --> 00:28:46.000 Technically, it's like different, right? 00:28:46.000 --> 00:28:49.000 Technically, it is 00:28:49.000 --> 00:28:59.000 Yes, technically it is different because they're decoding residual space directly. But if you consider that residual space is just the composition of a bunch of decodings 00:28:59.000 --> 00:29:00.000 Then 00:29:00.000 --> 00:29:02.000 But it's a gradient, right? 00:29:02.000 --> 00:29:06.000 Like, like the JSpace vector, the J-lens vector. 00:29:06.000 --> 00:29:11.000 The J lens vector 00:29:11.000 --> 00:29:12.000 For CAT, for example. 00:29:12.000 --> 00:29:13.000 The 00:29:13.000 --> 00:29:19.000 I don't believe the J lens vector is itself a gradient. 00:29:19.000 --> 00:29:22.000 Am I wrong about this? 00:29:22.000 --> 00:29:23.000 I think the 00:29:23.000 --> 00:29:24.000 There's an ? 00:29:24.000 --> 00:29:28.000 Sorry, go ahead. Someone else knows the answer better than me. 00:29:28.000 --> 00:29:31.000 Oh, I understand it, it's like, uh… 00:29:31.000 --> 00:29:39.000 So, the Jalens metrics, like the metrics that takes the first, like, the mediocre vector and, like, 00:29:39.000 --> 00:29:43.000 move it through the cat position, it's like the gradient, it's like an… 00:29:43.000 --> 00:29:51.000 Uh, linear approximation of the rest of the… of the layers of the model. So this is, like, the linear observation, then you plug into it the… 00:29:51.000 --> 00:30:00.000 the current point, and then you see, like, how it's linearly approximated in the value. 00:30:00.000 --> 00:30:06.000 Yes, I think that is a very good way of putting it. 00:30:06.000 --> 00:30:07.000 Go ahead. 00:30:07.000 --> 00:30:16.000 So just to hold up. So I just spent like six hours trying to think through this linear approximation perspective. And it like 00:30:16.000 --> 00:30:18.000 kind of doesn't work out, I don't think, because 00:30:18.000 --> 00:30:21.000 Yep. 00:30:21.000 --> 00:30:33.000 You're missing like a constant in that equation, right? So if you think about like the formula for a linear approximation, right? It's like you're at some you're approximating like about some current point. And then you're sort of scaling that gradient by 00:30:33.000 --> 00:30:40.000 Yes. 00:30:40.000 --> 00:30:41.000 Yep. 00:30:41.000 --> 00:30:44.000 the distance from that point of approximation. And so they've just, like, thrown all that way. So, like 00:30:44.000 --> 00:30:46.000 I don't know, the way 00:30:46.000 --> 00:31:00.000 The, like, language in the paper like reads like it's the linear approximation perspective, but, like, I don't know that the math is actually doing that, or if it's, like, equivalent at all. I don't know. I've been trying to figure this out all day 00:31:00.000 --> 00:31:01.000 Yeah. 00:31:01.000 --> 00:31:05.000 I think this is a very important point to bring up. One of the 00:31:05.000 --> 00:31:13.000 assumptions that they make in this paper that is often made nowadays with people that really like the linear representation hypothesis. 00:31:13.000 --> 00:31:17.000 The assumption is that 00:31:17.000 --> 00:31:19.000 features 00:31:19.000 --> 00:31:25.000 in the sense of feature vectors are centered on the origin. 00:31:25.000 --> 00:31:32.000 And so if you accept that they're centered on the origin, then you are only measuring the gradient of these features about the origin. 00:31:32.000 --> 00:31:33.000 The reason 00:31:33.000 --> 00:31:37.000 That this claim is often made is 00:31:37.000 --> 00:31:39.000 Plausible 00:31:39.000 --> 00:31:41.000 That reason being 00:31:41.000 --> 00:31:48.000 Every component that decodes the residual stream is doing so initially via a dot product of some vector onto the residual stream. 00:31:48.000 --> 00:31:51.000 To see if that key vector is present 00:31:51.000 --> 00:31:54.000 And 00:31:54.000 --> 00:32:01.000 In order to do that coherently, you have to be thinking of your vectors as direction centered on the origin 00:32:01.000 --> 00:32:03.000 But 00:32:03.000 --> 00:32:06.000 You get into a sort of circular understanding here. 00:32:06.000 --> 00:32:07.000 Where 00:32:07.000 --> 00:32:15.000 If you assume that everything is centered on the origin because your feature vectors are directions, and then you get a bunch of evidence that says that 00:32:15.000 --> 00:32:25.000 you've got some feature directions that are centered on the origin. You haven't really done your homework on whether or not that's actually going on. There's a significant possibility that 00:32:25.000 --> 00:32:32.000 Much more complicated behavior is occurring, for instance, within the activation functions. But 00:32:32.000 --> 00:32:35.000 That is 00:32:35.000 --> 00:32:40.000 More complex topic. 00:32:40.000 --> 00:32:50.000 The real answer to your complaint is that they are assuming that everything is centered about the origin and the constant being added is 0. 00:32:50.000 --> 00:32:55.000 Under that assumption, their math works out. 00:32:55.000 --> 00:32:58.000 I think also it requires that 00:32:58.000 --> 00:33:00.000 Like, if you if you feed 00:33:00.000 --> 00:33:06.000 the 0 vector into the remaining layers like it's also getting mapped back to back to 0 or something 00:33:06.000 --> 00:33:29.000 Yes. 00:33:29.000 --> 00:33:34.000 And instead you need to do mean ablation if you're ablating something 00:33:34.000 --> 00:33:38.000 I think there is a chance that some of this work falls into the same trap 00:33:38.000 --> 00:33:42.000 Regardless. 00:33:42.000 --> 00:33:50.000 What they have found is 00:33:50.000 --> 00:33:58.000 Here is the, in my opinion, the actual contribution of this paper. Previously, we have been using LogitLens to decode the residual stream into vocab space. 00:33:58.000 --> 00:34:02.000 They have found a methodology that lets you 00:34:02.000 --> 00:34:06.000 at any given layer, find a nice, easy, pretty good transform 00:34:06.000 --> 00:34:11.000 from that layer into something that is better decoded by LaunditLens 00:34:11.000 --> 00:34:13.000 This has been important because 00:34:13.000 --> 00:34:18.000 early layers are poorly decodable by logic lens 00:34:18.000 --> 00:34:24.000 The claims that they make beyond this about the fact that they have discovered a certain space 00:34:24.000 --> 00:34:28.000 I think are 00:34:28.000 --> 00:34:32.000 I personally think these claims are shaky 00:34:32.000 --> 00:34:39.000 Which but I'm willing to be challenged on that and I don't want to be imparting my 00:34:39.000 --> 00:34:45.000 Old man opinions onto the reading group 00:34:45.000 --> 00:34:47.000 So 00:34:47.000 --> 00:34:49.000 How would you feel about moving to 00:34:49.000 --> 00:34:55.000 Some stuff in section one 00:34:55.000 --> 00:34:57.000 Got someone agreeing with me 00:34:57.000 --> 00:35:02.000 That's nice. 00:35:02.000 --> 00:35:05.000 Here is 00:35:05.000 --> 00:35:14.000 A description of the global workspace and conscious access 00:35:14.000 --> 00:35:20.000 There is a claim in cognitive science that when we're thinking about stuff 00:35:20.000 --> 00:35:25.000 We can realize we're thinking about some of this stuff, and we can't realize we're thinking about some of this stuff 00:35:25.000 --> 00:35:28.000 The way that they refer to this is 00:35:28.000 --> 00:35:37.000 Some of our thoughts are accessible, and some of our thoughts are not accessible 00:35:37.000 --> 00:35:45.000 Here are some qualities that have been used in the past to distinguish accessible from not accessible information 00:35:45.000 --> 00:35:49.000 These qualities are reportable 00:35:49.000 --> 00:35:56.000 Controllable in the sense of like you can summon it 00:35:56.000 --> 00:36:01.000 Used in reasoning 00:36:01.000 --> 00:36:04.000 Reusable slash systematic 00:36:04.000 --> 00:36:10.000 and selective 00:36:10.000 --> 00:36:15.000 The 00:36:15.000 --> 00:36:19.000 They spend a lot of the paper making a lot of claims about a lot of these 00:36:19.000 --> 00:36:23.000 different things 00:36:23.000 --> 00:36:26.000 I'm frankly not 00:36:26.000 --> 00:36:33.000 Convinced by a lot of them, but that's okay. They've provided some pretty convincing evidence. 00:36:33.000 --> 00:36:44.000 The more interesting thing is they say this, a representation becomes consciously accessible when it is posted to a shared global workspace from which many downstream processes can read. 00:36:44.000 --> 00:36:47.000 This is a finite workspace 00:36:47.000 --> 00:36:52.000 It is can be thought about as 00:36:52.000 --> 00:36:59.000 Either a subset of thoughts or more accurately, the wherein of a subset of thoughts 00:36:59.000 --> 00:37:03.000 where those thoughts are 00:37:03.000 --> 00:37:07.000 The argument that they are making 00:37:07.000 --> 00:37:15.000 Is that JSpace essentially this union of cones 00:37:15.000 --> 00:37:29.000 A global workspace or is like a global workspace 00:37:29.000 --> 00:37:35.000 Before we actually get into any of their claims, like before I start complaining basically 00:37:35.000 --> 00:37:39.000 I want to hear if anyone has thoughts right off the bat. 00:37:39.000 --> 00:37:55.000 on this claim that they have found a global workspace or of any of these analogs, or even the cognitive science claims of global workspaces 00:37:55.000 --> 00:37:57.000 So, I… 00:37:57.000 --> 00:38:01.000 I don't know much about the cognitive science side of this. 00:38:01.000 --> 00:38:03.000 But in terms of just, like, 00:38:03.000 --> 00:38:08.000 Uh, whether this makes sense as a plausible. 00:38:08.000 --> 00:38:10.000 implementation in a… 00:38:10.000 --> 00:38:12.000 like a language model. 00:38:12.000 --> 00:38:14.000 Like, it feels like 00:38:14.000 --> 00:38:17.000 It's something that could work well. 00:38:17.000 --> 00:38:20.000 whether or not, like, the model, um… 00:38:20.000 --> 00:38:22.000 Like, as a… 00:38:22.000 --> 00:38:25.000 As a, as an abstraction. 00:38:25.000 --> 00:38:28.000 For, you know… 00:38:28.000 --> 00:38:31.000 Um, performing tasks. 00:38:31.000 --> 00:38:36.000 And like, I kind of, I kind of feel like. 00:38:36.000 --> 00:38:40.000 There is some, like… 00:38:40.000 --> 00:38:47.000 Evidence presented towards some semblance of this thing, but I don't… I don't know, like, the 00:38:47.000 --> 00:38:49.000 The depth, uh… 00:38:49.000 --> 00:38:54.000 is… matches the claims, I guess, you know. 00:38:54.000 --> 00:38:59.000 That's sort of my… like, it kind of makes sense to me that, like, 00:38:59.000 --> 00:39:05.000 If you if you're only able to do feed-forward processing. 00:39:05.000 --> 00:39:08.000 Like, and you want to be able to perform 00:39:08.000 --> 00:39:11.000 Multiple, uh, operations. 00:39:11.000 --> 00:39:15.000 On a single input, like, you'd want that input to be. 00:39:15.000 --> 00:39:17.000 uh, accessible by 00:39:17.000 --> 00:39:24.000 You know, multiple layers, multiple functions and downstream layers to be able to like operate on it, right? 00:39:24.000 --> 00:39:32.000 So, I don't know, it seems plausible to me. 00:39:32.000 --> 00:39:35.000 I think that makes sense. 00:39:35.000 --> 00:39:48.000 Anyone else want to contribute? 00:39:48.000 --> 00:39:53.000 I was thinking like, again, I don't really know what I'm talking about, but 00:39:53.000 --> 00:39:56.000 It almost feels like 00:39:56.000 --> 00:40:02.000 Any kind of like sub-quadratic, like linear time attention mechanism would kind of like 00:40:02.000 --> 00:40:07.000 Being inductive bias towards doing this 00:40:07.000 --> 00:40:08.000 Yep 00:40:08.000 --> 00:40:15.000 Since you sort of have just a few channels that kind of are seeing everything. 00:40:15.000 --> 00:40:23.000 Thank you for the link in chat of the nature paper about Global Workspace. 00:40:23.000 --> 00:40:30.000 I would probably agree with you. 00:40:30.000 --> 00:40:38.000 At this point, I think I can reveal my specific qualm with these claims. 00:40:38.000 --> 00:40:42.000 My qualm is that they have divided 00:40:42.000 --> 00:40:48.000 All of activation space or all of residual space into 00:40:48.000 --> 00:40:54.000 Two compliments, it partitioned it into two complements, JSpace and non-JSPACE. 00:40:54.000 --> 00:40:57.000 And they argue JSPACE is where 00:40:57.000 --> 00:40:58.000 The 00:40:58.000 --> 00:41:06.000 Conscious thinking happens and non-JSPACE is where the non-conscious thinking happens. 00:41:06.000 --> 00:41:08.000 And I feel like 00:41:08.000 --> 00:41:14.000 What's actually probably going on is 00:41:14.000 --> 00:41:21.000 The thing that they are referring to, the thing that has the qualities that they're ascribing to JSpace is actually just the entire residual space. 00:41:21.000 --> 00:41:29.000 The residual space is probably acting as some kind of global workspace where information is written in and information is taken out 00:41:29.000 --> 00:41:32.000 Because we know that's how the residual stream works 00:41:32.000 --> 00:41:35.000 And what they have actually found is a subspace of that 00:41:35.000 --> 00:41:40.000 That is decodable into vocab using their method. 00:41:40.000 --> 00:41:57.000 And there are other subspaces of residual space that are probably used in a similar way to a global workspace 00:41:57.000 --> 00:42:00.000 Yeah, I think that's a pretty fair characterization. 00:42:00.000 --> 00:42:02.000 Um… 00:42:02.000 --> 00:42:06.000 Yeah, I like that take. 00:42:06.000 --> 00:42:09.000 Now, I don't know whether this is true. I haven't seen 00:42:09.000 --> 00:42:11.000 Like. 00:42:11.000 --> 00:42:41.000 Obviously, I did not read sections 5 and 6, so there might be something special in there. There is one thing they mentioned at like the beginning of section three, I think that made me feel vindicated 00:42:41.000 --> 00:42:43.000 comparison between 00:42:43.000 --> 00:42:52.000 think about something and talk about something in order to measure introspection 00:42:52.000 --> 00:42:57.000 And what they say is when you do think about something 00:42:57.000 --> 00:42:59.000 And then you 00:42:59.000 --> 00:43:08.000 pull out whatever it's actually thinking about and inject whatever concept you want it to be thinking about instead 00:43:08.000 --> 00:43:11.000 The swap target succeeds in 00:43:11.000 --> 00:43:14.000 With JSpace components 00:43:14.000 --> 00:43:41.000 In 59% of trials 00:43:41.000 --> 00:43:45.000 any conference 00:43:45.000 --> 00:43:51.000 reviews I would get immediately is, what's going on with the other 41%? Why is this 00:43:51.000 --> 00:44:21.000 Like, why is this very intense intervention failing 41% of the time? 00:44:22.000 --> 00:44:42.000 coding verbalizable stuff 00:44:42.000 --> 00:44:48.000 non-word thinking happening. We can get into a whole grounding discussion, but 00:44:48.000 --> 00:44:52.000 I think this is, at the very least a plausible complaint given this evidence. 00:44:52.000 --> 00:44:59.000 What do you feel like is the difference between, like, when you say your critic is, like, that they are 00:44:59.000 --> 00:45:07.000 not made, uh, like, that they are most thought that are un… un… like, not, uh, verbalizable. 00:45:07.000 --> 00:45:10.000 But, like, for me, like, why is it… 00:45:10.000 --> 00:45:18.000 why do you think that they are belonging in the J space? In the global workspace event? 00:45:18.000 --> 00:45:21.000 I think 00:45:21.000 --> 00:45:26.000 So the reason in cognitive science, the reason we talk about the global workspace at all 00:45:26.000 --> 00:45:33.000 is because we can't really… like, I can't tell what you're thinking about except by what you tell me you're thinking about. 00:45:33.000 --> 00:45:35.000 And so 00:45:35.000 --> 00:45:38.000 We have to take a semi-behaviorist approach 00:45:38.000 --> 00:45:42.000 and say, oh, we are just input output machines or 00:45:42.000 --> 00:46:12.000 We have to concede something to the behaviorists and use that and say your reports of what you're thinking about are accurate to what you're thinking about. And so in cognitive science, reportability becomes very, very important 00:46:16.000 --> 00:46:32.000 While you're visualizing that, that image is still in your global workspace by any common sense definition of global workspace. But it is not easily verbalizable, and so it doesn't meet the strict criteria outlined by some cognitive scientists, or outlined by Anthropic here 00:46:32.000 --> 00:46:42.000 More importantly, all of the interventions Anthropic is doing are verbal interventions. So of course, if they found something that works really well with words. 00:46:42.000 --> 00:47:01.000 They're doing a great job. And if there's something, if there's some reasoning that's occurring not in word space where the reasoning gets moved into neural ease and then they're doing some neural ease calculations and then translating it back into words, they're missing all that and they're not testing for it. 00:47:01.000 --> 00:47:05.000 Yeah, it's almost… and so, like, but I feel it's, like, basically… 00:47:05.000 --> 00:47:08.000 Like, I don't… I get, like, the idea of… 00:47:08.000 --> 00:47:14.000 Okay, there are something that are in global worksites, but they are, you know, not verbalizable. 00:47:14.000 --> 00:47:23.000 But, like, I feel that this is, like, as good as we get, uh, in my, like, sense of… 00:47:23.000 --> 00:47:32.000 Yeah, I think that's probably true. I would concede that this is as good as it gets for verbalizable concepts 00:47:32.000 --> 00:47:33.000 This paper is probably the best that we can do with a logit lens-like thing 00:47:33.000 --> 00:47:36.000 of what we can describe. 00:47:36.000 --> 00:47:42.000 But I don't think that means we found the entire place where the conscious thought's happening. 00:47:42.000 --> 00:47:43.000 Oh, I want to see what's in chat 00:47:43.000 --> 00:47:48.000 I was, uh, sorry, another follow-up, like, how would you define a thing that is… 00:47:48.000 --> 00:47:53.000 like, definitely out of the global workspace. 00:47:53.000 --> 00:47:58.000 So this is where it's like, this is something that cognitive scientists have a really hard time 00:47:58.000 --> 00:48:28.000 Defining, I think this nature paper has some good answers. 00:48:28.000 --> 00:48:30.000 your global workspace 00:48:30.000 --> 00:48:35.000 And we can define it as not being in your global workspace by several criteria, but 00:48:35.000 --> 00:48:36.000 That's like a very far 00:48:36.000 --> 00:48:39.000 And can you think… 00:48:39.000 --> 00:48:40.000 Yeah, go ahead. 00:48:40.000 --> 00:48:44.000 of a translation for this, for LLM, like, for something that you say? 00:48:44.000 --> 00:48:53.000 So that's where it's like, I don't really know. I don't buy the idea that global workspace is a useful definition. 00:48:53.000 --> 00:49:05.000 Yeah, that's brainstem activity. I don't buy that global workspace is a useful activity here because there aren't 00:49:05.000 --> 00:49:09.000 I don't feel like there is a good 00:49:09.000 --> 00:49:13.000 Analog, like all of the processing is going 00:49:13.000 --> 00:49:18.000 towards 00:49:18.000 --> 00:49:22.000 No language model ever goes through an optimization 00:49:22.000 --> 00:49:24.000 Procedure 00:49:24.000 --> 00:49:36.000 No standard large language model goes through an optimization procedure that is not optimizing something about its outputs. 00:49:36.000 --> 00:49:44.000 In the sense of like it's actual things that it is saying. 00:49:44.000 --> 00:49:48.000 Whereas 00:49:48.000 --> 00:49:51.000 Like for a person 00:49:51.000 --> 00:50:00.000 There are cognitive processes going on that are unrelated to the things that we declare or the things that we are 00:50:00.000 --> 00:50:11.000 Like the things that we are verbalizing. And so keeping our heart beating is a separate is not in the global workspace because it is not 00:50:11.000 --> 00:50:14.000 Accessible to us in this sense. But 00:50:14.000 --> 00:50:19.000 The model is never going through an optimization procedure 00:50:19.000 --> 00:50:20.000 for anything other than it's talking 00:50:20.000 --> 00:50:23.000 But… 00:50:23.000 --> 00:50:32.000 But there was, like, the question of, like, controllability, like, if I… I don't remember I read the paper a lot of time ago, but there was, like, the… 00:50:32.000 --> 00:50:34.000 They gave the example of, like, uh… 00:50:34.000 --> 00:50:37.000 of, like, the length of the text, right? That it's, like, for them out of the robot space. 00:50:37.000 --> 00:50:39.000 Yeah. 00:50:39.000 --> 00:50:42.000 And, like, I know if this, like, matches that, but… 00:50:42.000 --> 00:50:47.000 I wonder, like, whether, like you said about the output, so even if I try really out, so I can't, like… 00:50:47.000 --> 00:50:52.000 Cause my out a bit strange, so maybe… so maybe, like, going in this direction of, like, 00:50:52.000 --> 00:50:55.000 is there something that they ask the model to do, and it's like… 00:50:55.000 --> 00:50:59.000 even if I ask it, like, specifically, it's do it, like, 00:50:59.000 --> 00:51:01.000 Regardless of my… 00:51:01.000 --> 00:51:04.000 question, because it's out of the… it's, like, out of his… 00:51:04.000 --> 00:51:06.000 Awareness, in a sense. 00:51:06.000 --> 00:51:08.000 Yeah. 00:51:08.000 --> 00:51:15.000 I think that makes sense. I'm looking for the section that you're talking about. I thought it was like 00:51:15.000 --> 00:51:19.000 Does anyone remember who it's before or after adjectives? 00:51:19.000 --> 00:51:25.000 He needs internal reasoning 00:51:25.000 --> 00:51:49.000 Hang on, let me pull up the searchable version. 00:51:49.000 --> 00:52:00.000 No, I'm at the right place 00:52:00.000 --> 00:52:03.000 I want to hear if anyone else has thoughts on 00:52:03.000 --> 00:52:07.000 I have things to say 00:52:07.000 --> 00:52:13.000 About what you've just said, Negev, but I want to hear if anybody else 00:52:13.000 --> 00:52:16.000 If this is resonating with anyone else 00:52:16.000 --> 00:52:24.000 Yeah, I do wanted to say that I also remember something about the fact that Bayer moving, I think, some… 00:52:24.000 --> 00:52:27.000 like the JSpace entirely, they… 00:52:27.000 --> 00:52:38.000 make the model not able to perform some sort of task. But at the same time, I don't really buy the thing that… that means that is related to… 00:52:38.000 --> 00:52:41.000 Like, the… yeah. 00:52:41.000 --> 00:52:48.000 The old point about this, I don't know, it just… okay, you're sort of making the model worse, but that's not really… 00:52:48.000 --> 00:52:50.000 Telling me much about… 00:52:50.000 --> 00:52:54.000 Well, I mean, it's just all about this connection they make. 00:52:54.000 --> 00:52:57.000 At the same time, I'm not ex… I'm not… 00:52:57.000 --> 00:52:59.000 expert at all in this more… 00:52:59.000 --> 00:53:05.000 neuroscience and stuff like that, so I don't really know, and I can't really comment on it. 00:53:05.000 --> 00:53:07.000 But it's just my feeling is… 00:53:07.000 --> 00:53:10.000 And that's it. Yeah. 00:53:10.000 --> 00:53:17.000 Yeah, I think that's a fair complaint. Generally 00:53:17.000 --> 00:53:21.000 Our field has a habit of 00:53:21.000 --> 00:53:22.000 Taking 00:53:22.000 --> 00:53:29.000 a model lobotomizing it in some way, and saying that that's evidence for something, when it 00:53:29.000 --> 00:53:32.000 lobotomized 00:53:32.000 --> 00:53:39.000 Generally, if you are doing a huge intervention and that huge intervention stops 00:53:39.000 --> 00:53:41.000 some capabilities 00:53:41.000 --> 00:53:44.000 That doesn't feel like big evidence or anything 00:53:44.000 --> 00:53:47.000 At the same time 00:53:47.000 --> 00:53:50.000 That section is hinting at 00:53:50.000 --> 00:53:57.000 Something else that is important for us to be thinking about a lot of the time, that thing being the lottery ticket hypothesis 00:53:57.000 --> 00:54:04.000 Which I will link now 00:54:04.000 --> 00:54:08.000 The lottery ticket hypothesis says 00:54:08.000 --> 00:54:11.000 inside of a model 00:54:11.000 --> 00:54:18.000 You 00:54:18.000 --> 00:54:21.000 A smaller model 00:54:21.000 --> 00:54:31.000 That, when trained by itself, can do the same thing. 00:54:31.000 --> 00:54:34.000 And it was unclear to me 00:54:34.000 --> 00:54:39.000 whether or not Anthropic, in writing this paper, was claiming that JSpace 00:54:39.000 --> 00:54:48.000 is a akin to a lottery ticket 00:54:48.000 --> 00:54:53.000 I think it depends on the claims that they actually made 00:54:53.000 --> 00:54:58.000 Regarding non-J space, which was hard for me to find. 00:54:58.000 --> 00:55:00.000 What did Oliver say 00:55:00.000 --> 00:55:07.000 These were causal really hard early on, but didn't or don't have enough fine-grained control to actually do sort of real causal experiments. 00:55:07.000 --> 00:55:17.000 So if you ablate X, which contains feature F, but also sync S, which modulates feature GHI, this is… yep, this is a very good point. So I was doing an experiment yesterday 00:55:17.000 --> 00:55:26.000 Where I found one head that was an attention sink, and then I ablated that head, and the model couldn't perform the task anymore. But if you ablated that head, the model couldn't perform any task anymore. 00:55:26.000 --> 00:55:39.000 So, then what have you found? You found a head that is important to general functionality, which means you've localized exactly nothing. 00:55:39.000 --> 00:55:45.000 We've got five minutes left. I want to hear if anyone has — I've been suggested to cold call 00:55:45.000 --> 00:55:50.000 I am not going to do that because everyone has been nice and vocal 00:55:50.000 --> 00:55:52.000 But 00:55:52.000 --> 00:55:55.000 I want to leave this open right now for 00:55:55.000 --> 00:56:00.000 If anyone who has not spoken has something they would like to contribute 00:56:00.000 --> 00:56:04.000 go right ahead, now is your chance. 00:56:04.000 --> 00:56:11.000 And if really no one has anything to say, we'll do a brief discussion on Section 9. 00:56:11.000 --> 00:56:16.000 I have to get off, but I want to say I really appreciate you all organizing these meetings. That's all. Thank you. 00:56:16.000 --> 00:56:31.000 Yeah, of course. Come back in two weeks. I think we'll read Chris Ola's 2020 circuits from DistillPub. 00:56:31.000 --> 00:56:35.000 I was also looking at 00:56:35.000 --> 00:56:39.000 The Section 5 and 6 of the paper alignment auditing parts 00:56:39.000 --> 00:56:41.000 Yep. 00:56:41.000 --> 00:56:43.000 And if 00:56:43.000 --> 00:56:46.000 The JSpace lets us 00:56:46.000 --> 00:56:51.000 verbalize something that the model isn't going, doesn't say 00:56:51.000 --> 00:56:56.000 during this generation, maybe JSpace could be used as an alignment signal 00:56:56.000 --> 00:56:58.000 To train with 00:56:58.000 --> 00:56:59.000 Yeah. 00:56:59.000 --> 00:57:01.000 That might make the JSPACE less useful 00:57:01.000 --> 00:57:05.000 But maybe it's a good way to align the model 00:57:05.000 --> 00:57:13.000 I think this diagram is very useful for that. This is just some JSPAC decodings at various points in reading a 00:57:13.000 --> 00:57:17.000 An email that's talking about 00:57:17.000 --> 00:57:25.000 Taking down the model 00:57:25.000 --> 00:57:30.000 I think that's certainly true. This sort of thing in general, lenses are 00:57:30.000 --> 00:57:37.000 I would say very useful for auditing 00:57:37.000 --> 00:57:39.000 Given that like 00:57:39.000 --> 00:57:44.000 I mean, look at the layer numbers on these decodings. 00:57:44.000 --> 00:57:50.000 They're pretty late layers. And so this is the sort of thing that we already sort of knew how to do with 00:57:50.000 --> 00:58:16.000 Logic lens, but J lens being just 00:58:16.000 --> 00:58:23.000 I think that's a good point 00:58:23.000 --> 00:58:28.000 Anyone else who hasn't spoken? 00:58:28.000 --> 00:58:33.000 Anybody who actually read the paper, do they compare to like tune lens and stuff, like all these other lenses? I know it's like a million of 00:58:33.000 --> 00:58:45.000 Yeah, they do a brief comparison to tune lens in section two. It's only a few paragraphs. I think they referenced that in one of the appendices they do a stronger comparison to Tunlens 00:58:45.000 --> 00:58:53.000 They argue that JLens is a successor to Logic Lens and that tuned lens is just like going in the wrong direction. 00:58:53.000 --> 00:59:08.000 Interesting. Okay, I'll actually have to read that part 00:59:08.000 --> 00:59:13.000 I guess the thing that's interesting there in the comparison, at least, is like 00:59:13.000 --> 00:59:18.000 These are all just picking some matrix to decode vectors. 00:59:18.000 --> 00:59:20.000 And, like, the way you pick your matrix seems to matter. 00:59:20.000 --> 00:59:22.000 Yep 00:59:22.000 --> 00:59:26.000 Um, and, like, you can… you can, like, verbalize different things. 00:59:26.000 --> 00:59:30.000 I guess if you pick a different matrix, so… 00:59:30.000 --> 00:59:34.000 I'm sure people have already, like 00:59:34.000 --> 00:59:40.000 Tried a bunch of different stuff, but like, for some, yeah, I think it's interesting that the, that the lens… basically, as I understood it. 00:59:40.000 --> 00:59:43.000 Like the lens for this paper is just like 00:59:43.000 --> 00:59:45.000 What's, like, the best… 00:59:45.000 --> 00:59:47.000 Um… 00:59:47.000 --> 00:59:53.000 Mapping between just, like, every possible token, and they just, like, average over all. 00:59:53.000 --> 00:59:59.000 you know, just, like, generic text. And for some reason, that gives, like, good verbalization. 00:59:59.000 --> 01:00:02.000 Um, for some things. 01:00:02.000 --> 01:00:05.000 But I'm curious, like, if you 01:00:05.000 --> 01:00:10.000 Wanted to have, like, a very specific. 01:00:10.000 --> 01:00:18.000 You know, verbalization of some subset of data, like if you pick, if you build a lens on some subset and set it over, like, a big 01:00:18.000 --> 01:00:20.000 Uh… 01:00:20.000 --> 01:00:25.000 corpus, like, if you'd get out other things, you know. 01:00:25.000 --> 01:00:34.000 That's certainly true. You can imagine leaving one token out of the entire set that's training the lens 01:00:34.000 --> 01:00:40.000 It leads me to another qualm, which is 01:00:40.000 --> 01:00:45.000 Potentially, if your training corpus is larger than your lens training corpus 01:00:45.000 --> 01:00:50.000 Then what you've trained the lens on can be missing some important stuff. 01:00:50.000 --> 01:00:58.000 Additionally, because the models are demonstrating such intense generalized generalization abilities 01:00:58.000 --> 01:01:01.000 There is probably also some thought 01:01:01.000 --> 01:01:04.000 some stuff that it can be thinking about 01:01:04.000 --> 01:01:09.000 That we wouldn't even know how to put in a corpus 01:01:09.000 --> 01:01:10.000 But 01:01:10.000 --> 01:01:16.000 Those are large claims 01:01:16.000 --> 01:01:27.000 All right, we got one minute or technically zero minutes. So any last thoughts? 01:01:27.000 --> 01:01:29.000 No. All right 01:01:29.000 --> 01:01:33.000 Thank you all for coming. I think this was a fun session. 01:01:33.000 --> 01:01:41.000 I'm going to declare that two weeks from now, unless you're attached to photo version 01:01:41.000 --> 01:01:45.000 Two weeks from now, we will read 01:01:45.000 --> 01:01:47.000 Let me pull up 01:01:47.000 --> 01:01:51.000 Allow circuit tracing 01:01:51.000 --> 01:01:59.000 We will read the first in David would not appreciate this. 01:01:59.000 --> 01:02:07.000 I would call this the first real Meccenterp paper. 01:02:07.000 --> 01:02:10.000 This will be on the Luma, whatever Luma we make. 01:02:10.000 --> 01:02:15.000 And the assignment I'm going to give to everybody 01:02:15.000 --> 01:02:23.000 Is not to read about any of the specific techniques that they are talking about unless you are interested in them 01:02:23.000 --> 01:02:32.000 Because next week we are not going to talk about the math behind whatever 2020 methodologies they were using for ImageNet stuff 01:02:32.000 --> 01:02:33.000 Instead. 01:02:33.000 --> 01:02:38.000 Look for claims that they make about analogies 01:02:38.000 --> 01:02:49.000 So, right off the bat, there's a big analogy they make about cells, they make a declaration on features and how features work. They do some analogies with neurons, etc. 01:02:49.000 --> 01:02:52.000 Every way that this paper is 01:02:52.000 --> 01:02:55.000 studying how 01:02:55.000 --> 01:02:59.000 Models are analogous to 01:02:59.000 --> 01:03:02.000 Something that is not models 01:03:02.000 --> 01:03:05.000 That's what we're reading about. 01:03:05.000 --> 01:03:10.000 Make sense? 01:03:10.000 --> 01:03:14.000 You email that to everybody? Because I'm not going to remember that prompt. 01:03:14.000 --> 01:03:20.000 Yes, I will make sure to throw that in the next Luma. 01:03:20.000 --> 01:03:23.000 All right, then I will see you all in two weeks 01:03:23.000 --> 01:03:26.000 So, yeah, thanks. 01:03:26.000 --> 01:03:27.000 Happy fall. 01:03:27.000 --> 01:03:28.000 Thanks for organizing. 01:03:28.000 --> 01:03:32.000 Of course, thanks to NDIF. 01:03:32.000 --> 01:04:02.000 Bye-bye.