WEBVTT 1 00:01:37.690 --> 00:01:38.860 Arjun Khurana: Welcome in! 2 00:01:40.020 --> 00:01:44.869 Arjun Khurana: I did not think anyone would be joining right on time, so I was lurking. Pardon. 3 00:01:47.520 --> 00:01:49.580 Arjun Khurana: Let me get into grid mode so I can see everyone. 4 00:02:26.150 --> 00:02:29.040 Arjun Khurana: How is everybody's Wednesday going? 5 00:02:30.710 --> 00:02:32.560 Arjun Khurana: By which I mean anybody's Wednesday. 6 00:02:38.970 --> 00:02:39.839 Yash Mali: It's going good. 7 00:02:41.440 --> 00:02:42.560 Arjun Khurana: Good to hear! 8 00:02:45.030 --> 00:02:46.830 Arjun Khurana: I want to give us at least… 9 00:02:49.120 --> 00:02:53.550 Arjun Khurana: I'm gonna say at least until 3.05 before we actually start talking. 10 00:02:54.370 --> 00:02:58.149 Arjun Khurana: Does anyone have any, banal discoveries that they've had for today? 11 00:02:59.930 --> 00:03:01.310 Arjun Khurana: Like, 12 00:03:01.490 --> 00:03:08.109 Arjun Khurana: The kind of thing that you'd share at the dinner table when you come home from middle school and your mom's like, what did you learn at school today? 13 00:03:19.620 --> 00:03:22.810 owm: Packaging software for Windows is, like, a complete mess. 14 00:03:24.170 --> 00:03:25.150 Arjun Khurana: Oh, yeah? 15 00:03:25.790 --> 00:03:26.460 owm: Oh, yeah. 16 00:03:28.400 --> 00:03:29.260 Arjun Khurana: Why? 17 00:03:30.690 --> 00:03:36.869 owm: It's not Unix, like, I only know Unix… I only know, like, Linux and Unix-based systems, so, like, I think it's kind of weird. 18 00:03:37.950 --> 00:03:39.729 Arjun Khurana: Does it force you to make an MSI? 19 00:03:41.530 --> 00:03:45.740 owm: You don't have to. There's, like… I have to, like, load library differently. 20 00:03:45.920 --> 00:03:48.120 owm: It's fucking weird, and I don't like it. 21 00:03:49.910 --> 00:03:50.910 Arjun Khurana: Disappointed. 22 00:03:52.320 --> 00:03:55.460 Arjun Khurana: Everything should just be Unix. Windows should become Unix. 23 00:03:55.670 --> 00:03:56.460 owm: Agreed. 24 00:04:04.150 --> 00:04:05.820 Arjun Khurana: Wow, we've got a decent population. 25 00:04:07.560 --> 00:04:08.599 Arjun Khurana: It's exciting. 26 00:04:12.600 --> 00:04:16.310 Arjun Khurana: Eric, I said I wanted to give it until 3.05. 27 00:04:17.760 --> 00:04:21.089 Eric Todd: Yeah, I think that sounds great. Let people trickle in. 28 00:04:21.269 --> 00:04:21.889 Arjun Khurana: Yep. 29 00:04:23.079 --> 00:04:27.159 Arjun Khurana: In the meantime, though, I can send my scruple. 30 00:04:28.479 --> 00:04:30.679 Arjun Khurana: And I will resend it in a couple minutes. 31 00:04:35.259 --> 00:04:45.989 Arjun Khurana: Also, sorry, everybody, for… assigning… I forget how massive this paper is, and Anthropic's typical, like. 32 00:04:47.169 --> 00:04:52.699 Arjun Khurana: assigning stuff like this is. I'm gonna be honest, I did not even make it all the way through. I read… 33 00:04:53.199 --> 00:04:58.009 Arjun Khurana: to the middle of section 4, I think, and then I skipped over to discussion. 34 00:04:59.329 --> 00:05:03.649 Arjun Khurana: It's just, like, they're going into so much detail about so much random stuff. 35 00:05:10.029 --> 00:05:11.889 Arjun Khurana: I'm seeing a lot of similar votes. 36 00:05:12.109 --> 00:05:15.279 Arjun Khurana: to what I did of, I read some of it thoroughly. 37 00:05:26.079 --> 00:05:28.979 Arjun Khurana: For the next reading, I'm going to… 38 00:05:31.629 --> 00:05:34.629 Arjun Khurana: Yeah, I had the same suspicion. 39 00:05:35.179 --> 00:05:41.799 Arjun Khurana: For the next reading, I think I'm going, regardless of what it is, though I have something in mind. 40 00:05:42.449 --> 00:05:43.589 Arjun Khurana: I think… 41 00:05:44.459 --> 00:05:59.359 Arjun Khurana: I will assign only part of the reading, not as an explicit, read this part, don't read this part, but as I skip over the sections that are talking about this, read carefully the sections that are talking about this. That way, everybody that reads slightly different things 42 00:05:59.959 --> 00:06:06.279 Arjun Khurana: Everybody will read stuff relating to the same topic, but people will read slightly different things, and I think that might make discussion a little more interesting as well. 43 00:06:20.409 --> 00:06:21.189 Arjun Khurana: Alright. 44 00:06:22.909 --> 00:06:27.769 Arjun Khurana: It is 3.05, I'm sending this once again, just in case anybody didn't see it. 45 00:06:32.639 --> 00:06:37.579 Arjun Khurana: Here are the results. Most everybody read some of it thoroughly. Or skimmed it. 46 00:06:38.549 --> 00:06:45.389 Arjun Khurana: So… We will first go over… a brief… 47 00:06:46.119 --> 00:06:48.859 Arjun Khurana: I'm going to give a very, very brief history. 48 00:06:49.119 --> 00:06:51.619 Arjun Khurana: Which is basically only 2 bullet points. 49 00:06:52.149 --> 00:06:54.559 Arjun Khurana: And then… 50 00:06:54.969 --> 00:07:00.599 Arjun Khurana: I think the best way to go about this would be for Adam to jump into the demonstration of the workspace tool. 51 00:07:00.869 --> 00:07:02.289 Arjun Khurana: Which is pretty cool. 52 00:07:03.199 --> 00:07:06.609 Arjun Khurana: And then, some wide-ranging discussion on cognition. 53 00:07:08.659 --> 00:07:09.779 Arjun Khurana: Sounds good. 54 00:07:11.669 --> 00:07:18.119 Arjun Khurana: Alright, first of all, let me get a show of hands, reaction-wise, 55 00:07:18.599 --> 00:07:21.609 Arjun Khurana: Who is familiar with Logit Lens? 56 00:07:24.219 --> 00:07:26.909 Arjun Khurana: At the very least, in the sense of, like. 57 00:07:27.589 --> 00:07:29.709 Arjun Khurana: You've seen it around, or you've heard of it. 58 00:07:32.689 --> 00:07:35.779 Arjun Khurana: Okay, this is a very fair amount. 59 00:07:37.729 --> 00:07:38.809 Arjun Khurana: Wonderful. 60 00:07:39.269 --> 00:07:48.199 Arjun Khurana: Popular paper. This is a paper by a mysterious author known as Nostalgia Breast. It was released in 2020. 61 00:07:48.889 --> 00:07:52.879 Arjun Khurana: The important thing to know is it says, Well… 62 00:07:53.139 --> 00:08:08.489 Arjun Khurana: The residual stream is a thing that keeps getting plus equals over and over and over again throughout the model, and then at the very end, we put the residual stream through the unembedding matrix. What if we put an earlier version of the residual stream through the unembedding matrix? And the answer is, you can kind of see what the model's thinking. 63 00:08:11.139 --> 00:08:17.939 Arjun Khurana: JLens, Is basically doing a fancy version of that, if that was not clear to everybody. 64 00:08:18.339 --> 00:08:24.599 Arjun Khurana: Jaylenz says, rather than… Putting everything through the… 65 00:08:25.129 --> 00:08:31.609 Arjun Khurana: Rather than putting the residual stream through the unembedding matrix directly, Which, by the way. 66 00:08:31.749 --> 00:08:36.979 Arjun Khurana: Just so that everyone's on the same page, when I say putting the residual stream through the unembedding matrix. 67 00:08:37.139 --> 00:08:47.209 Arjun Khurana: We're taking a vector and multiplying it by a matrix, but in reality, what we're doing is decoding something that is the model dimensionality into vocab space. 68 00:08:48.879 --> 00:08:53.559 Arjun Khurana: This is going to be very important for our discussion, the idea of decoding into vocab space. 69 00:08:53.809 --> 00:08:59.549 Arjun Khurana: So, regular logit lens decodes the residual stream directly into vocab space. 70 00:08:59.749 --> 00:09:05.629 Arjun Khurana: JLens, on the other hand, is proposing A small modification. 71 00:09:06.119 --> 00:09:15.559 Arjun Khurana: to the residual stream before putting it through the unembedding matrix to decode it into vocab space. This modification is the multiplication by a square matrix. 72 00:09:15.889 --> 00:09:19.089 Arjun Khurana: that is composed of Jacobians. 73 00:09:20.669 --> 00:09:26.799 Arjun Khurana: Honestly, It's an interesting methodology, but talking about how they put together this Jacobian matrix. 74 00:09:27.109 --> 00:09:33.199 Arjun Khurana: is basically irrelevant to what we want to talk about here today. So if you are very impassioned about 75 00:09:33.639 --> 00:09:39.079 Arjun Khurana: talking about Jacobians and calculus, I would suggest you start another NDIF reading group. 76 00:09:43.489 --> 00:09:58.639 Arjun Khurana: The other important piece of context to know before going into this paper is this was released by Anthropic recently, and before the release of this paper, I think on the scale of months before, not within a year, like, less than a year before. 77 00:09:58.829 --> 00:10:04.539 Arjun Khurana: Anthropic started getting really, really, really into the idea that their models were sentient. 78 00:10:05.989 --> 00:10:08.879 Arjun Khurana: Regardless of whether or not that is true. 79 00:10:09.759 --> 00:10:14.059 Arjun Khurana: Because of this internal belief, it changed the way they talk about and treat. 80 00:10:14.779 --> 00:10:17.339 Arjun Khurana: The models themselves a lot of the time. 81 00:10:18.349 --> 00:10:28.139 Arjun Khurana: Hence, the fact that this paper is published purporting that the models have a global workspace, which is a term from human cognitive science. 82 00:10:30.119 --> 00:10:32.729 Arjun Khurana: comes with a lot of baggage from Anthropic. 83 00:10:32.849 --> 00:10:36.469 Arjun Khurana: Being really excited about the idea that they've made some consciousness. 84 00:10:37.339 --> 00:10:39.289 Arjun Khurana: Not trying to discredit that. 85 00:10:40.019 --> 00:10:41.089 Arjun Khurana: Oh, wait. 86 00:10:41.529 --> 00:10:44.819 Arjun Khurana: At least 10 minutes before trying to do that. 87 00:10:44.959 --> 00:10:49.149 Arjun Khurana: But… It is… 88 00:10:52.229 --> 00:10:59.219 Arjun Khurana: This claim is very historically positioned, even though it's something that's happening right now, it is historically positioned in the history that we are currently in. 89 00:11:02.459 --> 00:11:04.829 Arjun Khurana: Anyways, I think that's all I have to say as precursor. 90 00:11:06.289 --> 00:11:10.469 Arjun Khurana: Adam, do you feel ready to show off? 91 00:11:10.939 --> 00:11:13.739 Arjun Khurana: your… Cool, shiny stuff. 92 00:11:15.030 --> 00:11:28.860 a.belfki@northeastern.edu: Yeah, thanks for passing it over to me. So, hi guys, my name's Adam, I'm on the NDIF team, we also have Emma and Jaden on the call, but sort of the infrastructure work that we've been, 93 00:11:28.860 --> 00:11:42.720 a.belfki@northeastern.edu: though in the past few months was building this, I think we also have Zikai, on the call. Forgot Zikai. But we have this, infrastructure, part of this infrastructure work is, we'll be building our interface. 94 00:11:42.790 --> 00:11:54.889 a.belfki@northeastern.edu: for making, quick and easy access to some interpretability methods for exploration, part of the research process. So I'm gonna drop the link, in the chat. 95 00:11:55.330 --> 00:12:01.489 a.belfki@northeastern.edu: And if you wanna head over there… 96 00:12:03.340 --> 00:12:09.120 a.belfki@northeastern.edu: And check out what we have. So you can access the models that we have on endiv directly through the interface. 97 00:12:09.450 --> 00:12:11.939 a.belfki@northeastern.edu: I'm going to share my screen. 98 00:12:17.340 --> 00:12:20.479 a.belfki@northeastern.edu: Yeah, I'm sort of already into my workspace here, but… 99 00:12:20.660 --> 00:12:31.400 a.belfki@northeastern.edu: Yeah, this is a quick way to do some quick expression. We have logic lens, activation patching, you can even access our big models, like, LAMA45B. 100 00:12:31.500 --> 00:12:45.260 a.belfki@northeastern.edu: And we've been working on adding support for JLANs, so you can do the basic thing of just, like, reading intermediate hidden states, using the Jacobian. I'm gonna try to show you, some… 101 00:12:45.720 --> 00:12:50.400 a.belfki@northeastern.edu: what that looks like, and what you can see with it. So, 102 00:12:50.830 --> 00:13:06.460 a.belfki@northeastern.edu: Relating to the paper we read last week, the backprop paper had this cool setup where they were trying… training neural networks on predicting relationship between, or was it predicting maybe, 103 00:13:06.670 --> 00:13:10.160 a.belfki@northeastern.edu: family members based on relationships. 104 00:13:10.160 --> 00:13:13.879 Arjun Khurana: I think it was the other way around, predicting relationships by the family members. 105 00:13:13.880 --> 00:13:17.449 a.belfki@northeastern.edu: family members, and training them on, on trees. 106 00:13:17.480 --> 00:13:33.960 a.belfki@northeastern.edu: And these models have gotten so good that you can just do it in context here. So, I'm using one of the 7TB models up right now, and we just passed the entire, like, topology of the family tree in context using, like, equal signs and… 107 00:13:33.960 --> 00:13:41.479 a.belfki@northeastern.edu: and arrows, and then I just ask, Charlotte, is Victoria's blank? And then answer, and then the model does say. 108 00:13:41.570 --> 00:13:49.249 a.belfki@northeastern.edu: correctly, daughter, or predicted, you can see here, Charlotte is the product of Victoria and James. 109 00:13:50.270 --> 00:14:01.690 a.belfki@northeastern.edu: And to sort of read the results here of JLNs on the right-hand side, so we have the layers of the model as the x-axis, and then the tokens from the prompt on the y-axis. 110 00:14:01.710 --> 00:14:17.059 a.belfki@northeastern.edu: And each, cell, what you're seeing, is the result of applying the Jacobian on that hidden state, at that token position, and then, the layer norm, and then passing it through the embedded metrics. And you can also see what are, like, the top. 111 00:14:17.060 --> 00:14:21.829 a.belfki@northeastern.edu: Predictions from the logic distribution, you can sort of see how that 112 00:14:21.830 --> 00:14:35.339 a.belfki@northeastern.edu: Token progresses throughout the model layers, and you can see that the model's thinking a lot about just the daughter, thinks this task might be too easy, that it's already locking in the right answer pretty early on. 113 00:14:37.380 --> 00:14:54.130 a.belfki@northeastern.edu: But something… so an interesting, observation, that you can't see with logic lens is, even at the very beginning, at the first equal sign, you can sort of see the intermediate representative of married, right there, before it even sees, like. 114 00:14:54.370 --> 00:15:05.610 a.belfki@northeastern.edu: the partner, which I think is really cool, and this is just a product of it, seeing Family Tree in the context. And, so you can also try this with LogitLens, and… 115 00:15:05.740 --> 00:15:17.050 a.belfki@northeastern.edu: So the color grading represents the probability of the tokens, and so sort of, like, the confidence of the model in that prediction, and you can already see the contrast with JLens that… 116 00:15:17.100 --> 00:15:36.539 a.belfki@northeastern.edu: The tool, logic lines is a lot more noisier, and so a lot of low probability tokens and random things would, like, come up top, and when I wanted to look at the equal sign again, I don't see merit come up that early or at all in this case. 117 00:15:37.020 --> 00:15:38.280 a.belfki@northeastern.edu: So we're just blank. 118 00:15:40.300 --> 00:15:58.419 a.belfki@northeastern.edu: So yeah, this is… we're rolling out soon other, features for our JLN support, so I'd love if you guys wanna, like, drop any feedback or join the Discord, let us know, we'd like to see if you come up with any cool, prompts or tests. That would be awesome. 119 00:16:01.610 --> 00:16:02.689 Arjun Khurana: Thanks for that, Adam. 120 00:16:02.980 --> 00:16:04.329 a.belfki@northeastern.edu: Yeah. 121 00:16:04.800 --> 00:16:10.640 Arjun Khurana: Yeah, this is a really cool tool, I can't believe that this is just out there and I can use it. 122 00:16:11.310 --> 00:16:16.449 Arjun Khurana: This is the sort of thing where usually I have to write a few modules in order to 123 00:16:16.730 --> 00:16:18.850 Arjun Khurana: Get an experiment like this going. 124 00:16:19.660 --> 00:16:26.460 Arjun Khurana: I'm going to briefly use… thank you, Emma. Discord link in the chat. 125 00:16:27.270 --> 00:16:34.200 Arjun Khurana: I'm going to briefly use… This, as an example. 126 00:16:52.410 --> 00:16:55.270 Arjun Khurana: No, okay, you know what we're gonna do? We're gonna… 127 00:16:55.460 --> 00:16:57.740 Arjun Khurana: do a brief talk-through of Logit Lens. 128 00:17:03.040 --> 00:17:10.549 Arjun Khurana: And then we are going to treat JLens as a fancy version of LogitLens, and we're going to spend maybe 5 minutes on that, and then we are going to get into the claims of global workspace. 129 00:17:11.099 --> 00:17:13.739 Arjun Khurana: So… I'm going to pull up the paper. 130 00:17:16.730 --> 00:17:22.109 Arjun Khurana: This paper printed to 150 pages on PDF. 131 00:17:22.980 --> 00:17:26.500 Arjun Khurana: So, sorry if my computer is running slow. 132 00:17:37.200 --> 00:17:39.000 Arjun Khurana: Let's start… 133 00:17:51.190 --> 00:17:57.789 Arjun Khurana: No, we're gonna skip over the Jacobians bit and just assume one definition of what the Jacobian matrix is doing. 134 00:17:59.040 --> 00:18:01.139 Arjun Khurana: The J lens that they describe. 135 00:18:02.440 --> 00:18:11.429 Arjun Khurana: is a matrix that maps the expected value of the effect of a perturbation at an intermediate layer, perturbation to the residual stream, I should have said. 136 00:18:11.930 --> 00:18:16.420 Arjun Khurana: On the representation in the residual stream right before the unembedding. 137 00:18:16.780 --> 00:18:23.230 Arjun Khurana: So basically, this is one big square matrix that you can use to make an intermediate layer representation 138 00:18:23.600 --> 00:18:26.439 Arjun Khurana: Look like a final representation. 139 00:18:27.080 --> 00:18:31.950 Arjun Khurana: That is a trustworthy claim. We're not going to get into why it's trustworthy, but it is a pretty good claim. 140 00:18:33.220 --> 00:18:38.500 Arjun Khurana: Then, they divide up this matrix into Vectors. 141 00:18:39.940 --> 00:18:44.969 Arjun Khurana: So, a JLens vector is one row or column of the matrix. 142 00:18:45.570 --> 00:18:51.150 Arjun Khurana: What we should think of this as is a vector in residual space at the layer we're looking at. 143 00:18:52.430 --> 00:18:56.369 Arjun Khurana: Which activates the lens for a specific word in the vocab. 144 00:18:58.940 --> 00:19:03.130 Arjun Khurana: So, when they say the JLens vector for… 145 00:19:04.370 --> 00:19:07.050 Arjun Khurana: What's one of the examples that they do right down here? 146 00:19:10.990 --> 00:19:19.340 Arjun Khurana: the JLens vector for soccer, The soccer lens vector, what that actually is, is… 147 00:19:21.240 --> 00:19:23.549 Arjun Khurana: A vector at the layer we're looking at. 148 00:19:23.800 --> 00:19:29.039 Arjun Khurana: That, after passing through this matrix multiplication, looks like soccer. 149 00:19:29.640 --> 00:19:30.869 Arjun Khurana: At the final layer. 150 00:19:34.750 --> 00:19:35.760 Arjun Khurana: they make… 151 00:19:36.240 --> 00:19:43.439 Arjun Khurana: a very interesting observation, which is, typically, in the residual stream, only a few JLens vectors are highly present at one time. 152 00:19:48.940 --> 00:19:54.300 Arjun Khurana: Given that this is a high-dimensional space with lots of superposition, this is sort of a surprising finding. 153 00:19:55.700 --> 00:19:58.519 Arjun Khurana: Then, they define a thing that they call JSpace. 154 00:19:59.920 --> 00:20:02.199 Arjun Khurana: Which, mathematically, is like this. 155 00:20:03.190 --> 00:20:07.640 Arjun Khurana: Pick a number K, take all possible collections of K, JLens vectors. 156 00:20:08.480 --> 00:20:16.799 Arjun Khurana: vectors for specific words in the vocab. Take this positive span of those vectors, each of those spans makes a big k-dimensional cone. 157 00:20:17.160 --> 00:20:18.730 Arjun Khurana: And then union all of those. 158 00:20:18.880 --> 00:20:26.070 Arjun Khurana: So essentially what we have now is a subset of residual space, Composed of linear combinations. 159 00:20:26.590 --> 00:20:28.560 Arjun Khurana: Of columns of our matrix. 160 00:20:31.830 --> 00:20:34.870 Arjun Khurana: And now that we have a subspace, you can split 161 00:20:35.270 --> 00:20:39.629 Arjun Khurana: Any activation into the part that's in that subspace and the part that's not in that subspace. 162 00:20:40.770 --> 00:20:44.379 Arjun Khurana: Does this make sense? Because this is an important time to pause. 163 00:20:52.630 --> 00:20:58.719 Arjun Khurana: Or what good definition of space is JSpace? A space? It's not a vector space. 164 00:20:58.850 --> 00:21:00.460 Arjun Khurana: This is a good point. 165 00:21:04.230 --> 00:21:15.780 Arjun Khurana: it is basically just a set of points. It is a… They're defining space as… a large set. 166 00:21:16.010 --> 00:21:24.149 Arjun Khurana: That set having a specific dimensionality, and that set being a subset of the vector space that is the residual stream. 167 00:21:27.910 --> 00:21:34.119 Arjun Khurana: But this is a good… Dimensionality we're defining as the… 168 00:21:34.720 --> 00:21:36.970 Arjun Khurana: Number of elements in the vector. 169 00:21:41.830 --> 00:21:46.429 Arjun Khurana: Technically, it's not a vector space, because they're only taking the span on the positive sides. 170 00:21:46.890 --> 00:21:51.020 Arjun Khurana: If it was a… if it was a vector space by formal definition, you'd have to be able to subtract 171 00:21:52.630 --> 00:21:55.230 Arjun Khurana: Jalen's vectors, too, which they're not allowing here. 172 00:21:57.150 --> 00:21:59.560 Eric Todd: Hmm, I missed that. That's interesting. 173 00:22:01.700 --> 00:22:02.190 owm: confusing. 174 00:22:02.190 --> 00:22:05.869 Arjun Khurana: Not actually… it's not actually that important to their methodology. 175 00:22:07.370 --> 00:22:13.560 Arjun Khurana: The fact that they don't allow subtraction is, I think, a little bit… Weird. 176 00:22:14.020 --> 00:22:15.810 Arjun Khurana: The fact that they are not… 177 00:22:16.040 --> 00:22:21.809 Arjun Khurana: That it is not technically meeting all of the requirements for a vector space is not that important. 178 00:22:21.970 --> 00:22:23.750 Arjun Khurana: But I think it goes to show… 179 00:22:24.500 --> 00:22:27.340 Arjun Khurana: If it was not clear, I don't really like this paper. 180 00:22:29.900 --> 00:22:31.000 Arjun Khurana: Exactly. 181 00:22:31.950 --> 00:22:33.840 Arjun Khurana: I don't really like this paper. 182 00:22:34.810 --> 00:22:39.089 Arjun Khurana: And one of the reasons I don't like this paper is because I feel like it's very fudgy with its rigor. 183 00:22:40.470 --> 00:22:44.820 Arjun Khurana: And this is… One unimportant instance among many. 184 00:22:49.670 --> 00:22:58.590 Arjun Khurana: the… But… If everyone feels good about JSpace, as a space. 185 00:22:58.770 --> 00:23:02.349 Arjun Khurana: then we can move on. But I think it's very important that we understand 186 00:23:04.130 --> 00:23:13.150 Arjun Khurana: How it is acting as a space, just insofar as, like, how it's… how they have defined a subspace of the residual stream, or a subspace of residual space. 187 00:23:13.770 --> 00:23:16.160 Arjun Khurana: by… Go ahead. 188 00:23:16.160 --> 00:23:25.199 Eric Todd: Can you… can you, can you, like, maybe walk us through, like, their construction, I guess? Maybe that would, like, make things clear… clearer? 189 00:23:25.920 --> 00:23:26.380 Arjun Khurana: That's… 190 00:23:26.380 --> 00:23:29.139 Eric Todd: for… for the people that skimmed, I don't… maybe… 191 00:23:29.770 --> 00:23:31.799 Arjun Khurana: That might be a good idea. 192 00:23:33.310 --> 00:23:37.569 Arjun Khurana: The first… the reason this is a little bit funny is because I'm trying to skip over this whole thing. 193 00:23:41.170 --> 00:23:47.899 Arjun Khurana: They use this big equation, Which is a big expected value of some derivatives. 194 00:23:49.270 --> 00:23:57.540 Arjun Khurana: to say, when I change… the residual stream in this direction, it generally has this 195 00:23:57.720 --> 00:23:59.629 Arjun Khurana: At the very end effect on the prompt. 196 00:24:00.800 --> 00:24:05.690 Arjun Khurana: Or, more importantly, it has this at the end effect on the residual stream right before we decode it. 197 00:24:06.090 --> 00:24:14.610 Arjun Khurana: And if you do that over and over and over again, you get… a… Matrix that lets you predict 198 00:24:14.790 --> 00:24:20.620 Arjun Khurana: If I bump the residual stream at layer 18 in this direction, it'll make the… 199 00:24:21.280 --> 00:24:27.949 Arjun Khurana: it'll make the final residual stream decode really heavily to CAT. It'll basically inject the cat vector into… 200 00:24:28.360 --> 00:24:30.099 Arjun Khurana: The end of the residual stream. 201 00:24:30.980 --> 00:24:34.609 Arjun Khurana: They then take this logic and put it backwards and say. 202 00:24:35.900 --> 00:24:42.359 Arjun Khurana: Okay, we have found, for any layer, this matrix that tells us what's gonna make cat. 203 00:24:43.390 --> 00:24:46.079 Arjun Khurana: So, let's trace this backwards and say… 204 00:24:46.870 --> 00:24:48.320 Arjun Khurana: I'm gonna write this here, actually. 205 00:24:48.490 --> 00:24:54.790 Arjun Khurana: We have a matrix that tells us at layer 18. 206 00:24:55.010 --> 00:24:57.100 Arjun Khurana: Which… direction. 207 00:24:58.500 --> 00:25:01.400 Arjun Khurana: Makes the last layer say cap. 208 00:25:04.180 --> 00:25:11.050 Arjun Khurana: So, then they say, let's take Let's look at that matrix. 209 00:25:13.310 --> 00:25:17.970 Arjun Khurana: And determine layer 18's most cat direction. 210 00:25:18.640 --> 00:25:20.829 Arjun Khurana: Call that the CAT vector. 211 00:25:22.070 --> 00:25:29.840 Arjun Khurana: The most cat direction here, Is defined as… the direction… of perturbation. 212 00:25:30.920 --> 00:25:33.180 Arjun Khurana: That increases CAT the most. 213 00:25:35.110 --> 00:25:36.170 Arjun Khurana: At the end. 214 00:25:36.940 --> 00:25:38.790 Arjun Khurana: And then we call that the CAT vector. 215 00:25:39.640 --> 00:25:41.759 Arjun Khurana: Do this for every word ever. 216 00:25:44.210 --> 00:25:54.250 Arjun Khurana: So, now we have, for every token, We have a… vector at… our given layer. 217 00:25:54.590 --> 00:26:01.120 Arjun Khurana: That corresponds to… What we're gonna call right now the concept corresponding to that token. 218 00:26:03.310 --> 00:26:08.990 Arjun Khurana: Because that is what they are heavily implying and trying really, really hard to avoid saying. 219 00:26:15.480 --> 00:26:16.880 Arjun Khurana: This is how… 220 00:26:17.120 --> 00:26:25.470 Arjun Khurana: Each of these vectors is what they call the J space vector for blank. So anytime you see this pop up, that's what they mean. 221 00:26:28.820 --> 00:26:31.560 Arjun Khurana: They make a slight jump here. 222 00:26:32.600 --> 00:26:36.970 Arjun Khurana: That jump-in logic is… the direction… 223 00:26:39.740 --> 00:26:44.800 Arjun Khurana: a perturbation at layer 18 that makes the model say CAT at the end. 224 00:26:46.460 --> 00:26:57.409 Arjun Khurana: Is roughly equivalent to… the direction… in Layer 18, that represents cat. 225 00:27:01.390 --> 00:27:06.599 Arjun Khurana: At this point, I think it is important to jump into global workspace theory. 226 00:27:08.000 --> 00:27:08.780 Arjun Khurana: But… 227 00:27:10.500 --> 00:27:16.800 Arjun Khurana: I think this explanation is a little bit better than what I did before, but it's still fudging it a little bit, so if this is confusing, someone flag me down now. 228 00:27:18.200 --> 00:27:25.400 owm: We had a question in the chat. I accidentally answered it. I did not mean to answer it. I was, like, copying it to, like, think about it and put a quote in… it doesn't matter. 229 00:27:25.670 --> 00:27:30.539 Arjun Khurana: Does the CAT vector here coincide with the SAE vector feature for CAT? No. 230 00:27:32.680 --> 00:27:36.370 Arjun Khurana: It. 231 00:27:37.550 --> 00:27:39.420 Arjun Khurana: does not necessarily. 232 00:27:40.710 --> 00:27:50.630 Arjun Khurana: they actually take SAE directions and further decompose them into combinations of JLens vectors. 233 00:27:54.020 --> 00:28:00.080 Arjun Khurana: I don't think there's a consistent cosine similarity between JLens vectors and SAE vectors. 234 00:28:00.220 --> 00:28:03.389 Arjun Khurana: Or if there is, I didn't… I didn't read far enough for them to mention it. 235 00:28:06.470 --> 00:28:10.120 Rice Wang: Wait, sorry, a follow-up on that? Like, I'm wondering, like… 236 00:28:10.230 --> 00:28:20.859 Rice Wang: if you think about, like, an SCE vector, it's like a direct… like, it's from the… normally, when people do steering and stuff, it's from the decoding step, right? So… so it's like a direct… 237 00:28:21.010 --> 00:28:32.279 Rice Wang: composition of the residual stream. But, like, for this, you're basically, like, for example, the CAV actor for JSpace is, like, you're kind of multiplying, like, a… 238 00:28:32.500 --> 00:28:45.800 Rice Wang: like, a Jacobian or, like, a gradient vector with an embedding matrix, which… it has the same dimensionality with the residual strip space, but, like, it's… technically, it's, like, different, right? 239 00:28:46.580 --> 00:28:47.970 Arjun Khurana: Technically, it is… 240 00:28:49.720 --> 00:28:58.470 Arjun Khurana: Yes, technically it is different because they're decoding residual space directly, but if you consider that residual space is just the composition of a bunch of decodings. 241 00:28:59.460 --> 00:29:00.430 Arjun Khurana: then… 242 00:29:00.810 --> 00:29:05.709 Rice Wang: But it's a gradient, right? Like… like the J-space vector, the Jalenz vector. 243 00:29:06.850 --> 00:29:08.980 Arjun Khurana: the JLens vector… 244 00:29:11.410 --> 00:29:12.880 Rice Wang: For Cat, for example. 245 00:29:14.010 --> 00:29:18.310 Arjun Khurana: No, I don't believe the JLens vector is itself a gradient. 246 00:29:18.440 --> 00:29:19.849 Arjun Khurana: Am I wrong about this? 247 00:29:22.890 --> 00:29:24.220 Rice Wang: I think the… 248 00:29:24.830 --> 00:29:27.460 Arjun Khurana: Sorry, go ahead, someone else knows the answer better than me. 249 00:29:28.120 --> 00:29:39.420 taglicht.n@northeastern.edu: Oh, I understand it, it's like, so the Jalen's metric, like, the metrics that takes the… the first, like, the… the mediocre vector and, like. 250 00:29:39.420 --> 00:29:56.940 taglicht.n@northeastern.edu: move it through the CAD position, it's like the gradient. It's like a linear approximation of the rest of the… of the layers of the model. So this is, like, the linear approximation, then you plug into it the… the current point, and then you see, like, how it's linearly approximated in the value. 251 00:30:00.170 --> 00:30:04.910 Arjun Khurana: Yes, I think… I think that is a very good way of putting it. The… 252 00:30:04.910 --> 00:30:07.089 Calvin Higgins: So, just to… hold up… 253 00:30:07.090 --> 00:30:07.590 Arjun Khurana: Go ahead. 254 00:30:07.590 --> 00:30:15.769 Calvin Higgins: So I just spent, like, 6 hours trying to think through this linear approximation perspective, and it, like… 255 00:30:16.030 --> 00:30:20.559 Calvin Higgins: kinda doesn't work out, I don't think. Yep. Because, 256 00:30:21.140 --> 00:30:33.299 Calvin Higgins: you're missing, like, a constant in that equation, right? So if you think about, like, the formula for a linear approximation, right? It's like, you're at some… you're approximating, like, about some current point. 257 00:30:33.320 --> 00:30:43.919 Calvin Higgins: And then you're sort of scaling that gradient by, the distance from that point of approximation. And so they're just, like, thrown all that way. So, like. 258 00:30:44.540 --> 00:30:46.030 Calvin Higgins: I don't know, the way… 259 00:30:46.320 --> 00:31:00.169 Calvin Higgins: the, like, language in the paper, like, reads like it's the linear approximation perspective, but, like, I don't know that the math is actually doing that, or if it's, like, equivalent at all. I don't know, I'm gonna try to figure this out. 260 00:31:00.800 --> 00:31:04.450 Arjun Khurana: I think this is a very important point to bring up. One of the… 261 00:31:04.980 --> 00:31:12.830 Arjun Khurana: assumptions that they make in this paper that is often made nowadays with people that really like the linear representation hypothesis. 262 00:31:13.280 --> 00:31:23.640 Arjun Khurana: The assumption is that features, In the sense of feature vectors, Are centered on the origin. 263 00:31:24.960 --> 00:31:31.679 Arjun Khurana: And so, if you accept that they're centered on the origin, then you are only measuring the gradient of these features about the origin. 264 00:31:31.940 --> 00:31:38.170 Arjun Khurana: The reason… That this claim is often made is… plausible. 265 00:31:38.900 --> 00:31:40.510 Arjun Khurana: That reason being… 266 00:31:40.980 --> 00:31:47.930 Arjun Khurana: Every component that decodes the residual stream is doing so initially via a dot product of some vector onto the residual stream. 267 00:31:48.060 --> 00:31:50.859 Arjun Khurana: To see if that key vector is present. 268 00:31:51.810 --> 00:31:53.040 Arjun Khurana: And… 269 00:31:54.590 --> 00:32:00.420 Arjun Khurana: In order to do that coherently, you have to be thinking of your vectors as direction centered on the origin. 270 00:32:01.280 --> 00:32:05.879 Arjun Khurana: But… You get into a sort of circular understanding here. 271 00:32:06.250 --> 00:32:07.330 Arjun Khurana: where… 272 00:32:07.860 --> 00:32:14.770 Arjun Khurana: If you assume that everything is centered on the origin because your feature vectors are directions, and then you get a bunch of evidence that says that 273 00:32:15.080 --> 00:32:19.010 Arjun Khurana: You've got some feature directions that are centered on the origin. You haven't really 274 00:32:19.170 --> 00:32:24.980 Arjun Khurana: done your homework on whether or not that's actually going on. There's a significant possibility that 275 00:32:25.750 --> 00:32:36.560 Arjun Khurana: Much more complicated behavior is occurring, for instance, within the activation functions, but… That is a… more complex topic. 276 00:32:40.330 --> 00:32:49.970 Arjun Khurana: The real… the real answer to your complaint is that they are assuming that everything is centered about the origin and the constant being added is zero. 277 00:32:50.350 --> 00:32:52.409 Arjun Khurana: Under that assumption, their math works out. 278 00:32:55.130 --> 00:32:58.159 Calvin Higgins: I think also it requires that, 279 00:32:58.380 --> 00:33:00.339 Calvin Higgins: Like, if you… if you feed… 280 00:33:00.640 --> 00:33:07.669 Calvin Higgins: the… the zero vector into the remaining layers, like, it's also getting mapped back to… back to zero. Yes. 281 00:33:07.670 --> 00:33:09.939 Arjun Khurana: So this is, this is where, like… 282 00:33:11.460 --> 00:33:15.730 Arjun Khurana: about, I wanna say, a year and a half ago, maybe 2 years ago at this point. 283 00:33:16.060 --> 00:33:22.720 Arjun Khurana: People started really freaking out about the fact that every Mec and Terp ablation experiment was using zero ablation. 284 00:33:23.670 --> 00:33:28.169 Arjun Khurana: Which is not a good way of doing things, because zeros carry signal. 285 00:33:29.480 --> 00:33:33.030 Arjun Khurana: And instead, you need to do mean ablation if you're ablating something. 286 00:33:34.070 --> 00:33:37.979 Arjun Khurana: I think there is a chance that some of this work falls into the same trap. 287 00:33:38.680 --> 00:33:46.410 Arjun Khurana: Regardless, What they have found is… a… 288 00:33:50.540 --> 00:33:58.129 Arjun Khurana: Here is the, in my opinion, the actual contribution of this paper. Previously, we have been using LogitLens to decode the residual stream into vocab space. 289 00:33:58.810 --> 00:34:01.000 Arjun Khurana: They have found a methodology that lets you 290 00:34:01.200 --> 00:34:05.910 Arjun Khurana: At any given layer, find a nice, easy, pretty good transform. 291 00:34:06.370 --> 00:34:10.919 Arjun Khurana: from that layer into something that is better decoded by logic lens. 292 00:34:11.040 --> 00:34:17.220 Arjun Khurana: This has been important because Early layers are poorly decodable by logic lens. 293 00:34:18.170 --> 00:34:22.500 Arjun Khurana: The claims that they make beyond this, about the fact that they have discovered a certain space. 294 00:34:24.110 --> 00:34:30.560 Arjun Khurana: I think are… I personally think these claims are shaky. 295 00:34:32.489 --> 00:34:38.459 Arjun Khurana: Which… but I'm willing to be challenged on that, and I don't want to be imparting my, 296 00:34:39.380 --> 00:34:43.469 Arjun Khurana: Old Man Opinions onto the reading group. 297 00:34:45.679 --> 00:34:51.370 Arjun Khurana: So… How would you feel about moving to… some stuff in Section 1. 298 00:34:55.429 --> 00:34:57.060 Arjun Khurana: Got someone agreeing with me. 299 00:34:57.420 --> 00:34:58.500 Arjun Khurana: That's nice. 300 00:35:02.210 --> 00:35:09.890 Arjun Khurana: Here is… A description of… The global workspace and conscious access. 301 00:35:14.880 --> 00:35:19.879 Arjun Khurana: There is a claim in cognitive science that when we're thinking about stuff. 302 00:35:20.120 --> 00:35:24.940 Arjun Khurana: We can realize we're thinking about some of the stuff, and we can't realize we're thinking about some of the stuff. 303 00:35:25.840 --> 00:35:27.750 Arjun Khurana: The way that they refer to this is… 304 00:35:28.290 --> 00:35:32.179 Arjun Khurana: Some of our thoughts are accessible, and some of our thoughts are not accessible. 305 00:35:37.780 --> 00:35:44.860 Arjun Khurana: Here are some qualities that have been used in the past to distinguish accessible from not accessible information. 306 00:35:45.410 --> 00:35:47.270 Arjun Khurana: These qualities are reportable. 307 00:35:49.120 --> 00:35:52.839 Arjun Khurana: Controllable, in the sense of, like, you can summon it. 308 00:35:56.490 --> 00:35:58.340 Arjun Khurana: used in reasoning. 309 00:36:01.370 --> 00:36:03.890 Arjun Khurana: Reusable slash systematic. 310 00:36:04.140 --> 00:36:05.520 Arjun Khurana: and selective. 311 00:36:10.520 --> 00:36:12.250 Arjun Khurana: the… 312 00:36:15.630 --> 00:36:21.219 Arjun Khurana: They spend a lot of the paper making a lot of claims about a lot of these… different things. 313 00:36:23.640 --> 00:36:31.900 Arjun Khurana: I'm frankly not… convinced by a lot of them, but that's okay, they've provided some pretty convincing evidence. 314 00:36:33.180 --> 00:36:42.460 Arjun Khurana: The more interesting thing is they say this, a representation becomes consciously accessible when it is posted to a shared global workspace from which many downstream processes can read. 315 00:36:44.180 --> 00:36:46.319 Arjun Khurana: This is a finite workspace. 316 00:36:47.780 --> 00:36:50.829 Arjun Khurana: It is… can be thought about as a… 317 00:36:52.340 --> 00:36:59.270 Arjun Khurana: Either a subset of thoughts, or, more accurately, the wherein of a subset of thoughts. 318 00:36:59.460 --> 00:37:01.300 Arjun Khurana: Where those thoughts are. 319 00:37:03.160 --> 00:37:05.179 Arjun Khurana: The argument that they are making 320 00:37:07.620 --> 00:37:14.380 Arjun Khurana: is that JSpace, essentially this union of cones, is… 321 00:37:15.860 --> 00:37:19.100 Arjun Khurana: A global workspace, or is like a global workspace. 322 00:37:29.360 --> 00:37:34.900 Arjun Khurana: Before we actually get into any of their claims, like, before I start complaining, basically. 323 00:37:35.390 --> 00:37:37.620 Arjun Khurana: I want to hear if anyone has thoughts right off the bat. 324 00:37:39.380 --> 00:37:48.330 Arjun Khurana: On this claim that they have found a global workspace, or of any of these analogs, or even the cognitive science claims of global workspaces. 325 00:37:55.490 --> 00:38:01.189 Eric Todd: So, I… I don't know much about the cognitive science side of this. 326 00:38:01.510 --> 00:38:07.979 Eric Todd: But in terms of just, like, whether this makes sense as a plausible 327 00:38:08.220 --> 00:38:12.230 Eric Todd: Implementation in a, like, a language model. 328 00:38:12.480 --> 00:38:17.879 Eric Todd: Like, it feels like it's something that could work well. 329 00:38:18.000 --> 00:38:25.729 Eric Todd: Whether or not, like, the model, like, as a… As a… as an abstraction. 330 00:38:26.010 --> 00:38:30.689 Eric Todd: For, you know, performing tasks. 331 00:38:31.620 --> 00:38:38.660 Eric Todd: And, like, I kind of… I kind of… Feel like… There is some, like… 332 00:38:40.970 --> 00:38:49.719 Eric Todd: evidence presented towards some semblance of this thing, but I don't… I don't know, like, the… the depth, 333 00:38:50.020 --> 00:38:54.019 Eric Todd: is… matches the… the claims, I guess, you know. 334 00:38:54.540 --> 00:39:00.650 Eric Todd: that's sort of my… my… like, it kind of makes sense to me that, like, 335 00:39:01.040 --> 00:39:08.560 Eric Todd: If you… if… if you're only able to do feed-forward processing, like, and you want to be able to perform 336 00:39:08.910 --> 00:39:11.559 Eric Todd: multiple, operations. 337 00:39:11.710 --> 00:39:18.410 Eric Todd: On a single input, like, you'd want that input to be, accessible by… you know. 338 00:39:18.630 --> 00:39:24.290 Eric Todd: multiple layers, multiple functions, and downstream layers to be able to, like, operate on it, right? So… 339 00:39:24.560 --> 00:39:27.529 Eric Todd: So, I don't know, it seems plausible to me. 340 00:39:32.470 --> 00:39:33.659 Arjun Khurana: I think that makes sense. 341 00:39:35.010 --> 00:39:36.730 Arjun Khurana: Anyone else want to contribute? 342 00:39:48.220 --> 00:39:55.330 Calvin Higgins: I was thinking, like… again, I don't really know what I'm talking about, but… It almost feels like, 343 00:39:55.910 --> 00:40:01.000 Calvin Higgins: Any kind of, like, sub-quadratic, like, linear time attention mechanism would kind of, like. 344 00:40:02.190 --> 00:40:05.020 Calvin Higgins: Be an inducted bias towards doing this? 345 00:40:05.440 --> 00:40:10.430 Calvin Higgins: Since you sort of have just a few channels that kind of are seeing everything. 346 00:40:15.430 --> 00:40:19.460 Arjun Khurana: Thank you for the link in chat of the Nature paper about Global Workspace. 347 00:40:23.410 --> 00:40:25.349 Arjun Khurana: I would probably agree with you. 348 00:40:27.080 --> 00:40:34.750 Arjun Khurana: at this point, I think I can reveal my specific qualm with these claims. 349 00:40:36.300 --> 00:40:41.810 Arjun Khurana: My qualm… Is that they have divided 350 00:40:42.630 --> 00:40:49.580 Arjun Khurana: All of activation space, or all of residual space, into two compliments. 351 00:40:49.710 --> 00:40:53.589 Arjun Khurana: They partitioned it into two complements, JSpace and non-J space. 352 00:40:54.270 --> 00:40:57.740 Arjun Khurana: And they argue JSpace is where… the… 353 00:40:58.500 --> 00:41:03.970 Arjun Khurana: conscious thinking happens, and non-J space is where the non-conscious thinking happens. 354 00:41:05.880 --> 00:41:11.330 Arjun Khurana: And I feel like… What's actually probably going on is… 355 00:41:14.010 --> 00:41:21.480 Arjun Khurana: The thing that they are referring to, the thing that has the qualities that they're ascribing to JSpace, is actually just the entire residual space. 356 00:41:21.660 --> 00:41:28.540 Arjun Khurana: The residual space is probably acting as some kind of global workspace where information is written in and information is taken out. 357 00:41:29.040 --> 00:41:31.660 Arjun Khurana: Because we know that's how the residual stream works. 358 00:41:31.960 --> 00:41:35.200 Arjun Khurana: And what they have actually found is a subspace of that 359 00:41:35.390 --> 00:41:39.059 Arjun Khurana: That is decodable into vocab using their method. 360 00:41:40.030 --> 00:41:43.829 Arjun Khurana: And there are other subspaces of residual space. 361 00:41:43.990 --> 00:41:47.249 Arjun Khurana: That are probably used in a similar way to a global workspace. 362 00:41:47.490 --> 00:41:53.159 Arjun Khurana: But these are either not decodable using their method, or they are not vocabulary-oriented at all. 363 00:41:57.400 --> 00:42:04.740 Eric Todd: Yeah, I think that's a pretty fair characterization. Yeah, I like that take. 364 00:42:06.650 --> 00:42:10.420 Arjun Khurana: Now, I don't know whether this is true, I haven't seen… like… 365 00:42:11.440 --> 00:42:21.759 Arjun Khurana: Obviously, I did not read Sections 5 and 6, so there might be something special in there. There was one thing they mentioned at, like, the beginning of Section 3, I think, that made me feel… vindicated. 366 00:42:24.070 --> 00:42:25.710 Arjun Khurana: And it was… 367 00:42:29.190 --> 00:42:31.760 Arjun Khurana: This is them measuring verbalizability. 368 00:42:33.510 --> 00:42:42.179 Arjun Khurana: They say, swapping along the concept vectors… they're doing a, they're doing a comparison between 369 00:42:42.860 --> 00:42:47.079 Arjun Khurana: Think about something and talk about something in order to measure introspection. 370 00:42:49.220 --> 00:42:58.690 Arjun Khurana: And what they say is, when you do a… when you're doing, think about something, And then you… 371 00:42:59.160 --> 00:43:06.829 Arjun Khurana: Pull out whatever it's actually thinking about, and inject whatever is… whatever concept you want it to be thinking about instead. 372 00:43:08.330 --> 00:43:16.519 Arjun Khurana: The swap target succeeds in… with JSpace components, In 59% of trials. 373 00:43:16.830 --> 00:43:21.140 Arjun Khurana: If you use the actual Jalen's vectors, you get an 88% success rate. 374 00:43:21.760 --> 00:43:29.080 Arjun Khurana: But what this means is… So we're doing a targeted intervention here. We're essentially doing steering, using… 375 00:43:30.210 --> 00:43:34.940 Arjun Khurana: Something that's more complicated than diff of means, but is akin to diff of means. 376 00:43:35.260 --> 00:43:40.379 Arjun Khurana: And if I was doing a steering experiment, and I got a 59% result, and I submitted that to… 377 00:43:41.290 --> 00:43:42.790 Arjun Khurana: any conference. 378 00:43:43.400 --> 00:43:50.559 Arjun Khurana: the… reviews I would get immediately is, what's going on with the other 41%? Why is this… 379 00:43:50.670 --> 00:43:55.369 Arjun Khurana: Like, why is this very intense intervention failing 41% of the time? 380 00:43:58.630 --> 00:44:04.109 Arjun Khurana: At the same time, swapping along the non-JSpace component succeeds in only 5% of trials. 381 00:44:06.050 --> 00:44:12.010 Arjun Khurana: This is… somewhat convincing. 382 00:44:12.350 --> 00:44:17.740 Arjun Khurana: But it seems like here… Actually, they have found 383 00:44:19.360 --> 00:44:23.440 Arjun Khurana: The best or one of the best methods for decoding verbalizable stuff. 384 00:44:27.370 --> 00:44:35.229 Arjun Khurana: And so it's not that they're only looking at stuff that's decodable by their methodology, it's that they're looking at stuff that is decodable into words at all. 385 00:44:35.670 --> 00:44:39.510 Arjun Khurana: And so when they do interventions using words, they work great. 386 00:44:40.590 --> 00:44:46.910 Arjun Khurana: But there is probably some also non-word thinking happening. We can get into a whole grounding discussion, but… 387 00:44:48.070 --> 00:44:52.059 Arjun Khurana: I think this is, at the very least, a plausible complaint, given this evidence. 388 00:44:52.830 --> 00:45:07.550 taglicht.n@northeastern.edu: What do you feel like is the difference between, like, like, when you say… your critic is, like, that they have not made, like, that there are more thoughts that are un… like, not, verbalizable? 389 00:45:07.550 --> 00:45:16.119 taglicht.n@northeastern.edu: But, like, for me, like, why is it… why do you think that they are belonging in the JSpace, in the JSpace, in the global workspace event? 390 00:45:19.080 --> 00:45:25.749 Arjun Khurana: I think… So the reason, in cognitive science, the reason we talk about the global workspace at all. 391 00:45:26.180 --> 00:45:32.450 Arjun Khurana: is because we can't really tell… like, I can't tell what you're thinking about except by what you tell me you're thinking about. 392 00:45:33.860 --> 00:45:37.979 Arjun Khurana: And so… We have to take a semi-behaviorist approach. 393 00:45:38.180 --> 00:45:41.900 Arjun Khurana: And say, oh, we are just input-output machines. Or… 394 00:45:42.480 --> 00:45:48.410 Arjun Khurana: We have to concede something to the behaviorists and use that and say, your reports 395 00:45:48.530 --> 00:45:54.479 Arjun Khurana: of what you're thinking about are accurate to what you're thinking about. And so, in cognitive science, reportability becomes very, very important. 396 00:45:55.410 --> 00:46:02.820 Arjun Khurana: Essentially, the global workspace is defined as everything you can tell me that you're thinking about. 397 00:46:04.410 --> 00:46:08.449 Arjun Khurana: And if you're studying yourself, it is everything you can tell yourself you're thinking about. 398 00:46:09.530 --> 00:46:15.620 Arjun Khurana: But, if I ask you to visualize something right now, visualize something that is very difficult to put into words. 399 00:46:16.310 --> 00:46:32.120 Arjun Khurana: while you're visualizing that, that image is still in your global workspace by any common sense definition of global workspace. But it is not easily verbalizable, and so it doesn't meet the strict criteria outlined by some cognitive scientists, or outlined by Anthropic here. 400 00:46:32.460 --> 00:46:35.959 Arjun Khurana: More importantly, all of the interventions Anthropic is doing 401 00:46:36.140 --> 00:46:41.640 Arjun Khurana: are verbal interventions. So, of course, if they found something that works really well with words. 402 00:46:41.780 --> 00:46:55.450 Arjun Khurana: they're doing a great job, and if there's something… if there's some reasoning that's occurring not in word space, where the reasoning gets moved into Neuralese, and then they're doing some Neuralese calculations, and then translating it back into words, they're missing all that, and they're not testing for it. 403 00:46:57.410 --> 00:47:00.979 Arjun Khurana: Does that make sense? Does that answer your question? I feel like I got off track. 404 00:47:01.270 --> 00:47:19.999 taglicht.n@northeastern.edu: Yeah, it's somewhat… and so, like, but I feel that it's, like, basically, like, I don't… I get, like, the idea of, okay, there are some things that are in global voices, but they are, you know, not verbalizable, but, like, I feel that this is, like, as good as we get, in my, like, sense of… 405 00:47:20.240 --> 00:47:21.899 taglicht.n@northeastern.edu: Of what we can describe. 406 00:47:23.450 --> 00:47:31.559 Arjun Khurana: Yeah, I think that's probably true. This is… I would concede that this is as good… as good as it gets for verbalizable concepts, like… 407 00:47:31.790 --> 00:47:35.929 Arjun Khurana: This paper is probably the best that we can do with a logic lens-like thing. 408 00:47:36.800 --> 00:47:40.759 Arjun Khurana: But I don't think that means we've found the entire place where the conscious thought's happening. 409 00:47:42.380 --> 00:47:43.540 Arjun Khurana: Boom. I want to see. 410 00:47:43.540 --> 00:47:48.100 taglicht.n@northeastern.edu: Sorry, another follow-up, like, how would you define a thing that is 411 00:47:48.230 --> 00:47:51.499 taglicht.n@northeastern.edu: Like, definitely out of the global workspace. 412 00:47:53.140 --> 00:47:58.109 Arjun Khurana: So this is where it's, like, this is something that cognitive scientists have a really hard time… 413 00:47:58.770 --> 00:48:02.410 Arjun Khurana: defining, I think this Nature paper has some good answers. 414 00:48:03.230 --> 00:48:15.139 Arjun Khurana: the… not, like… Classic, but, like, a very simple at-hand example is, like, There is a… 415 00:48:15.600 --> 00:48:18.389 Arjun Khurana: Neural process in your head right now? 416 00:48:19.050 --> 00:48:22.910 Arjun Khurana: that… Is keeping your heart beating. 417 00:48:24.230 --> 00:48:29.139 Arjun Khurana: That neural process is occurring in your mind and is not in your global workspace. 418 00:48:30.880 --> 00:48:34.939 Arjun Khurana: And we can define it as not being in your global workspace by several criteria, but… 419 00:48:35.860 --> 00:48:37.829 Arjun Khurana: That's, like, a very far… 420 00:48:38.590 --> 00:48:43.470 taglicht.n@northeastern.edu: Yeah, go ahead. Like, of a translation for this, for LLM? Like, for something that you say… 421 00:48:44.120 --> 00:48:52.039 Arjun Khurana: So that's where it's like, I don't… I don't really know. I don't… I don't buy the idea that, like, global workspace is a useful definition. 422 00:48:53.220 --> 00:48:59.720 Arjun Khurana: Yeah, that's brainstem activity. I don't… I don't buy that brain… that global workspace is a use selectivity here, because there aren't, like. 423 00:49:05.480 --> 00:49:07.779 Arjun Khurana: I don't feel like there is a good… 424 00:49:09.680 --> 00:49:15.020 Arjun Khurana: analog? Like, all of the processing is going… towards… 425 00:49:18.330 --> 00:49:23.120 Arjun Khurana: No language model ever goes through an optimization procedure. 426 00:49:24.720 --> 00:49:31.959 Arjun Khurana: No standard large language model goes through an optimization procedure that is not optimizing something about its outputs. 427 00:49:36.570 --> 00:49:40.160 Arjun Khurana: In the sense of, like, it's actual things that it is saying. 428 00:49:44.520 --> 00:49:45.500 Arjun Khurana: Whereas… 429 00:49:46.970 --> 00:49:47.940 taglicht.n@northeastern.edu: Okay, so… 430 00:49:48.500 --> 00:49:50.450 Arjun Khurana: Like, for a person. 431 00:49:51.500 --> 00:49:58.510 Arjun Khurana: There are cognitive processes going on that are unrelated to the things that we declare, or the things that we are… 432 00:50:00.730 --> 00:50:09.300 Arjun Khurana: Like, the things that we are verbalizing, and so keeping our heart beating is a separate… is not in the global workspace because it is not… 433 00:50:11.040 --> 00:50:13.960 Arjun Khurana: Accessible to us in this sense. But… 434 00:50:14.240 --> 00:50:16.980 Arjun Khurana: The model is never going through an optimization procedure. 435 00:50:19.150 --> 00:50:21.700 Arjun Khurana: For anything other than it's talking. 436 00:50:24.070 --> 00:50:39.280 taglicht.n@northeastern.edu: But there was, like, the question of, like, controllability, like, if I… I don't remember, I read the papers a lot of time ago, but there was, like, the… they gave the example of, like, of, like, the length of the text, right? That it's, like, for them out of the old space. 437 00:50:39.290 --> 00:50:52.310 taglicht.n@northeastern.edu: And, like, I don't know if this, like, matched it, but I wonder, like, whether, like you said about the output, so even if I try really out, so I can't, like, cause my artbit to change, so maybe, like, going in this direction of, like. 438 00:50:52.400 --> 00:51:01.730 taglicht.n@northeastern.edu: is there something that I ask the model to do? And it's like, even if I ask it, like, specifically, it's do it, like, regardless of my… 439 00:51:01.940 --> 00:51:06.440 taglicht.n@northeastern.edu: Question, because it's out of the… it's, like, out of his awareness, in a sense. 440 00:51:06.970 --> 00:51:07.580 Arjun Khurana: Yeah. 441 00:51:08.890 --> 00:51:13.199 Arjun Khurana: I think that makes sense. I'm looking for the section that you're talking about. I thought it was, like. 442 00:51:15.260 --> 00:51:18.909 Arjun Khurana: Does anyone remember who it's before or after adjectives? 443 00:51:19.800 --> 00:51:21.620 Arjun Khurana: Mediates internal reasoning… 444 00:51:25.790 --> 00:51:28.120 Arjun Khurana: Hang on, let me pull up the searchable version. 445 00:51:32.730 --> 00:51:42.689 taglicht.n@northeastern.edu: Yeah, I'm not sure about what I said is I correspond, okay? There was, like, the one thing about the controllability, and I remember this, like, experiment, but I'm not sure exactly what they did, you know. 446 00:51:46.800 --> 00:51:48.460 Arjun Khurana: Counting a line width. 447 00:51:48.850 --> 00:51:50.399 Arjun Khurana: No, I'm at the right place. 448 00:52:00.440 --> 00:52:02.940 Arjun Khurana: I want to hear if anyone else has thoughts on 449 00:52:03.710 --> 00:52:06.820 Arjun Khurana: I, I, I have things to say… 450 00:52:07.520 --> 00:52:11.859 Arjun Khurana: about what you've just said, Nigay, but I want to hear if anybody else 451 00:52:13.520 --> 00:52:15.700 Arjun Khurana: If this is resonating with anyone else. 452 00:52:16.810 --> 00:52:27.720 Jac Zac: Yeah, I do wanted to say that I also remember something about the fact that by removing, I think, some… like, the JSpace entirely, they… 453 00:52:27.720 --> 00:52:39.320 Jac Zac: make the model not able to perform some sort of task. But at the same time, I don't really buy the thing that… that means that is related to… 454 00:52:39.320 --> 00:52:48.379 Jac Zac: like, the… yeah, the old point about this? I don't know, it's just… okay, you're sort of making the model worse, but that's not really… 455 00:52:48.570 --> 00:52:54.659 Jac Zac: Telling me much about… Well, I mean, it's just all about this connection they make. 456 00:52:54.780 --> 00:53:07.469 Jac Zac: At the same time, I'm not no expert at all in this more neuroscience and stuff like that, so I don't really know, and I can't really comment on it, but it's just my feeling is… 457 00:53:07.750 --> 00:53:09.750 Jac Zac: And that's it. Yeah. 458 00:53:10.980 --> 00:53:12.779 Arjun Khurana: Yeah, I think that's a fair complaint. 459 00:53:12.930 --> 00:53:14.300 Arjun Khurana: generally… 460 00:53:17.310 --> 00:53:22.160 Arjun Khurana: Our field has a habit of, taking… 461 00:53:22.460 --> 00:53:30.760 Arjun Khurana: A model lobotomizing it in some way, and saying that that's evidence for something, when it acts lobotomized. 462 00:53:32.000 --> 00:53:40.639 Arjun Khurana: Generally, if you are doing a huge intervention, and that huge intervention stops some capabilities… 463 00:53:40.890 --> 00:53:43.130 Arjun Khurana: That doesn't feel like big evidence for anything. 464 00:53:44.000 --> 00:53:50.200 Arjun Khurana: At the same time, That section is hinting at 465 00:53:50.430 --> 00:53:57.330 Arjun Khurana: Something else that is important for us to be thinking about a lot of the time, that thing being the lottery ticket hypothesis. 466 00:53:57.740 --> 00:53:59.830 Arjun Khurana: Which I will link now. 467 00:54:04.510 --> 00:54:12.350 Arjun Khurana: The lottery ticket hypothesis says… Inside of a model, you… 468 00:54:18.330 --> 00:54:23.649 Arjun Khurana: A smaller model, That, when trained by itself, can do the same thing. 469 00:54:31.090 --> 00:54:39.089 Arjun Khurana: It was unclear to me whether or not Anthropic, in writing this paper, was claiming that JSpace 470 00:54:39.740 --> 00:54:42.990 Arjun Khurana: Is a… akin to a lottery ticket. 471 00:54:48.380 --> 00:54:52.289 Arjun Khurana: I think it depends on the claims that they actually made. 472 00:54:53.460 --> 00:54:57.350 Arjun Khurana: Regarding non-J space, which was hard for me to find. 473 00:54:58.650 --> 00:54:59.990 Arjun Khurana: What did Oliver say? 474 00:55:00.360 --> 00:55:07.200 Arjun Khurana: These were causal really hard early on, but didn't or don't have enough fine-grained control to actually do, sort of, real causal experiments. 475 00:55:07.330 --> 00:55:16.539 Arjun Khurana: So, if you ablate X, which contains feature F, but also sync S, which modulates feature GHI, this is… yep, this is a very good point. So, I was doing an experiment yesterday. 476 00:55:16.700 --> 00:55:25.520 Arjun Khurana: Where I found one head that was an attention sink, and then I ablated that head, and the model couldn't perform the task anymore. But if you ablated that head, the model couldn't perform any task anymore. 477 00:55:25.790 --> 00:55:32.910 Arjun Khurana: So, then what have you found? You found a head that is important to general functionality, which means you've localized exactly nothing. 478 00:55:39.370 --> 00:55:45.249 Arjun Khurana: We've got 5 minutes left. I want to hear if anyone has… I don't… I've been suggested to cold call. 479 00:55:45.810 --> 00:55:50.390 Arjun Khurana: I am… Not going to do that because everyone has been nice and vocal. 480 00:55:50.590 --> 00:55:54.609 Arjun Khurana: But… I want to leave this open right now for… 481 00:55:54.900 --> 00:55:59.390 Arjun Khurana: If anyone who has not spoken has something they would like to contribute. 482 00:55:59.600 --> 00:56:01.560 Arjun Khurana: Go right ahead, now is your chance. 483 00:56:04.040 --> 00:56:07.959 Arjun Khurana: And if really no one has anything to say, we'll do a brief discussion on Section 9. 484 00:56:11.120 --> 00:56:17.080 Calvin Higgins: I have to get off, but I want to say, I really appreciate you all organizing these meetings. That's all. Thank you. 485 00:56:17.080 --> 00:56:22.519 Arjun Khurana: Yeah, of course. Come back in two weeks. I think we'll read Chris Ola's 2020 circuits from DistillPub. 486 00:56:31.670 --> 00:56:34.459 Yash Mali: I was also, looking at 487 00:56:34.900 --> 00:56:39.540 Yash Mali: the Section 5 and 6 of the paper, Alignment Auditing Parts. 488 00:56:39.690 --> 00:56:45.370 Yash Mali: Yup. And if… the JSpace lets us… 489 00:56:46.520 --> 00:56:50.740 Yash Mali: verbalize something that the model isn't going… doesn't say… 490 00:56:51.120 --> 00:56:57.449 Yash Mali: during this generation, maybe JSpace could be used as an alignment signal to train with. 491 00:56:58.160 --> 00:56:58.540 Arjun Khurana: Yeah. 492 00:56:58.540 --> 00:57:00.959 Yash Mali: make the J space less useful. 493 00:57:01.260 --> 00:57:03.990 Yash Mali: But maybe it's a good way to align the model. 494 00:57:06.060 --> 00:57:13.039 Arjun Khurana: I think this diagram is very useful for that. This is just some JSpace decodings at various points in reading a… 495 00:57:13.780 --> 00:57:19.169 Arjun Khurana: An email that's talking about Taking down the model. 496 00:57:21.820 --> 00:57:29.649 Arjun Khurana: I think that's certainly true. This sort of thing. In general, lenses are… 497 00:57:30.250 --> 00:57:32.839 Arjun Khurana: I would say very useful for auditing. 498 00:57:33.770 --> 00:57:42.760 Arjun Khurana: Given that, like, I mean, look at the layer numbers on these decodings. 499 00:57:44.320 --> 00:57:49.800 Arjun Khurana: They're pretty late layers, and so this is the sort of thing that we already sort of knew how to do with… 500 00:57:50.740 --> 00:57:58.530 Arjun Khurana: Logit lens, but JLens being just… strictly… 501 00:57:59.160 --> 00:58:04.270 Arjun Khurana: more powerful. I don't even want to say more accurate, because I don't know how we can measure accuracy, but… 502 00:58:05.100 --> 00:58:07.260 Arjun Khurana: Having a lower garbage rate. 503 00:58:07.590 --> 00:58:13.499 Arjun Khurana: I think is a good sign that we can use methodologies like this, and this specific methodology, like they're saying, for auditing. 504 00:58:16.470 --> 00:58:17.760 Arjun Khurana: I think that's a good point. 505 00:58:23.730 --> 00:58:25.360 Arjun Khurana: Anyone else who hasn't spoken? 506 00:58:28.110 --> 00:58:33.430 owm: Did anybody who actually read the paper, do they compare to, like, Tune Lens and stuff, like, all these other lenses? I know there's, like, a million of them. 507 00:58:33.870 --> 00:58:43.919 Arjun Khurana: Yeah, they do a brief comparison to Toon Lens in Section 2, it's only a few paragraphs. I think in one… I think they reference that in one of the appendices, they do a stronger comparison to Toon Lens. 508 00:58:45.680 --> 00:58:51.459 Arjun Khurana: They… they argue that JLens is a successor to LogicLens, and that Tuned Lens is just, like, going in the wrong direction. 509 00:58:53.750 --> 00:58:55.710 owm: Interesting, okay, I'll actually have to read that part. 510 00:59:08.790 --> 00:59:13.600 Eric Todd: I guess the thing that's interesting there in the comparison, at least, is, like. 511 00:59:13.750 --> 00:59:18.490 Eric Todd: These are all just picking… picking some matrix to decode vectors. 512 00:59:18.780 --> 00:59:22.400 Eric Todd: And, like, the way you pick your matrix seems to matter. 513 00:59:22.680 --> 00:59:29.820 Eric Todd: And, like, you can… you can, like, verbalize different things, I guess, if you, like, pick a different matrix, so… 514 00:59:30.580 --> 00:59:33.580 Eric Todd: I've… I'm sure people have already, like. 515 00:59:34.430 --> 00:59:40.459 Eric Todd: tried a bunch of different stuff, but, like, for some… yeah, I think it's interesting that the… that the lens… basically, as I understood it. 516 00:59:40.590 --> 00:59:46.400 Eric Todd: Like, the lens for this paper is just, like, What's, like, the best… 517 00:59:47.830 --> 00:59:53.289 Eric Todd: Mapping between just, like, every possible token, and they just, like, average over all 518 00:59:53.540 --> 00:59:59.510 Eric Todd: You know, just, like, generic text. And for some reason, that gives, like, good verbalization. 519 01:00:00.210 --> 01:00:02.300 Eric Todd: For some things. 520 01:00:02.470 --> 01:00:04.870 Eric Todd: But I'm curious, like, if you… 521 01:00:05.260 --> 01:00:09.730 Eric Todd: Wanted to have, like, a very specific 522 01:00:10.420 --> 01:00:17.739 Eric Todd: you know, verbalization of some subset of data. Like, if you pick… if you build a lens on some subset instead of over, like, a big 523 01:00:18.240 --> 01:00:23.779 Eric Todd: corpus, like, if you'd get out other things, you know. 524 01:00:24.850 --> 01:00:31.830 Arjun Khurana: I think that's certainly true. You can imagine leaving one token out of the entire set that's training the lens. 525 01:00:32.350 --> 01:00:37.619 Arjun Khurana: It leads me to another qualm, which is… 526 01:00:40.140 --> 01:00:49.230 Arjun Khurana: Potentially, if your training corpus is larger than your lens training corpus, then what you've trained the lens on can be missing some important stuff. 527 01:00:50.230 --> 01:00:56.949 Arjun Khurana: Additionally, because the models are demonstrating such intense generalized… generalization abilities. 528 01:00:57.510 --> 01:01:00.860 Arjun Khurana: There is probably also some thought 529 01:01:01.120 --> 01:01:04.039 Arjun Khurana: Some stuff that it can be thinking about. 530 01:01:04.320 --> 01:01:06.959 Arjun Khurana: That we wouldn't even know how to put in a corpus. 531 01:01:09.200 --> 01:01:11.930 Arjun Khurana: But… Those are large claims. 532 01:01:16.560 --> 01:01:20.759 Arjun Khurana: Alright, we got one minute, or technically 0 minutes. So, any last thoughts? 533 01:01:27.340 --> 01:01:29.360 Arjun Khurana: No? Alright. 534 01:01:29.600 --> 01:01:32.840 Arjun Khurana: Thank you all for coming. I think this was a fun session. 535 01:01:33.920 --> 01:01:40.479 Arjun Khurana: I'm going to declare that two weeks from now, unless you're attached to Footer Buildin. 536 01:01:41.790 --> 01:01:49.769 Arjun Khurana: Two weeks from now, we will read… Let me pull up… A lot of circuit tracing. 537 01:01:51.880 --> 01:01:58.610 Arjun Khurana: We will read the first… In… David would not appreciate this. 538 01:01:59.560 --> 01:02:02.920 Arjun Khurana: I would call this the first real McInterp paper. 539 01:02:05.530 --> 01:02:10.259 Arjun Khurana: This will be on the Luma, whatever Luma we make. 540 01:02:10.890 --> 01:02:14.839 Arjun Khurana: And the assignment I'm going to give to everybody… 541 01:02:15.900 --> 01:02:22.340 Arjun Khurana: Is not to read about any of the specific techniques that they are talking about, unless you are interested in them. 542 01:02:23.920 --> 01:02:31.579 Arjun Khurana: Because next week, we are not going to talk about the math behind whatever 2020 methodologies they were using for ImageNet stuff. 543 01:02:31.860 --> 01:02:37.569 Arjun Khurana: Instead, Look for claims that they make about analogies. 544 01:02:37.840 --> 01:02:47.440 Arjun Khurana: So, right off the bat, there's a big analogy they make about cells, they make a declaration on features and how features work, they do some analogies with neurons, etc. 545 01:02:49.160 --> 01:03:01.239 Arjun Khurana: Every way that this paper is studying how… Models are analogous to Something that is not models. 546 01:03:02.500 --> 01:03:03.969 Arjun Khurana: That's what we're reading about. 547 01:03:05.520 --> 01:03:06.580 Arjun Khurana: Make sense? 548 01:03:10.390 --> 01:03:13.050 owm: Do you email that to everybody? Because I'm not going to remember that prompt. 549 01:03:14.200 --> 01:03:17.009 Arjun Khurana: Yes, I will make sure to throw that in the next Luma. 550 01:03:20.590 --> 01:03:23.200 Arjun Khurana: Alright, then I will see you all in 2 weeks! 551 01:03:24.110 --> 01:03:25.270 Eric Todd: See ya, thanks. 552 01:03:26.100 --> 01:03:26.800 Arjun Khurana: Happy Festival. 553 01:03:26.800 --> 01:03:27.710 Yash Mali: organizing. 554 01:03:28.950 --> 01:03:29.599 Arjun Khurana: Of course. 555 01:03:29.750 --> 01:03:30.899 Arjun Khurana: Thanks to NDIF. 556 01:03:32.230 --> 01:03:33.260 Arjun Khurana: Bye-bye!