WEBVTT 1 00:00:02.020 --> 00:00:06.920 216-101 Large Conference: Yeah Are you okay to be reported? Is that alright? 2 00:00:07.600 --> 00:00:08.830 216-101 Large Conference: Your point is okay? 3 00:00:09.620 --> 00:00:10.680 216-101 Large Conference: All right. 4 00:00:13.640 --> 00:00:15.399 216-101 Large Conference: Nikhil, when did you get back? 5 00:00:16.560 --> 00:00:21.179 216-101 Large Conference: Last week, Thursday night. Okay, welcome back for Amy. Yeah. 6 00:00:22.230 --> 00:00:33.109 216-101 Large Conference: So, so… We're all an internal group here, right? Not everybody. Does anybody not know Nikhil? 7 00:00:33.740 --> 00:00:34.999 216-101 Large Conference: I don't know her. 8 00:00:35.900 --> 00:00:41.769 216-101 Large Conference: Yeah, and I saw a few names on Zoom which I don't recognize. And then we have a couple people online. 9 00:00:42.500 --> 00:00:43.760 216-101 Large Conference: Yeah, it's wonderful. 10 00:00:44.020 --> 00:00:45.770 Alan Eng: Hi, David. How are you? 11 00:00:46.070 --> 00:00:50.599 216-101 Large Conference: So if I'll introduce us, I'll introduce Nikhil for the video. So, 12 00:00:51.040 --> 00:00:54.729 216-101 Large Conference: So, well, we have, speaking today. 13 00:00:55.810 --> 00:00:59.650 216-101 Large Conference: We may know Nikhil, but he may not know it, says Jasmine. 14 00:01:00.220 --> 00:01:07.999 216-101 Large Conference: Nikhil is a, 4th year? Fifth year? Yeah. About to be fifth year. 15 00:01:08.510 --> 00:01:10.490 216-101 Large Conference: PhD student in our lab. 16 00:01:10.750 --> 00:01:15.989 216-101 Large Conference: And, and, you know, the interesting thing about Nikhil is 17 00:01:16.200 --> 00:01:26.899 216-101 Large Conference: When he joined, as a first-year PhD student, he made a list of a few things that he was interested in, and one of the things on the list was he was interested in theory of mind. 18 00:01:27.180 --> 00:01:31.150 216-101 Large Conference: In these AI systems, about 4 years ago. 19 00:01:31.410 --> 00:01:32.720 216-101 Large Conference: And, 20 00:01:33.470 --> 00:01:43.920 216-101 Large Conference: And I thought, well, that's a fantastic thing to look at, but when he tried to look into the open models at the time. 21 00:01:44.210 --> 00:02:02.559 216-101 Large Conference: They couldn't do theory of mind. You could give them little puzzles about, you know, what does this person think, that other person thinks, whatever, and they just completely could not figure it out, so it was really hard to study. And, and so Nikhil started by studying, 22 00:02:02.860 --> 00:02:07.630 216-101 Large Conference: What was a… you know, he thought it was a simpler setting. He started studying binding. 23 00:02:08.080 --> 00:02:16.789 216-101 Large Conference: Which is just, how do AIs keep track of things that you bind to names? Like, variable binding. Like, if you say X equals 5, 24 00:02:17.070 --> 00:02:20.720 216-101 Large Conference: And how does it remember that X is 5? That kind of thing. Or if you say. 25 00:02:20.860 --> 00:02:24.650 216-101 Large Conference: The cat is in this box, and how does that… how does the, 26 00:02:24.770 --> 00:02:27.099 216-101 Large Conference: AI remember what was inside the box. 27 00:02:28.430 --> 00:02:29.849 216-101 Large Conference: And, 28 00:02:29.980 --> 00:02:37.539 216-101 Large Conference: And it's turned out that binding has been a really interesting topic for a lot of people. And so, now. 29 00:02:37.610 --> 00:02:51.709 216-101 Large Conference: Bonnels can, if you're mine too, and he's been studying that as well, but, but he's discovered that binding is a really core aspect of it. And so, so I think that, you just came back from, Good Fire. 30 00:02:52.080 --> 00:02:59.239 216-101 Large Conference: And I think they're interested in theory of mind and binding as well, so you did some more of that for your summer 31 00:02:59.370 --> 00:03:09.450 216-101 Large Conference: summer internship, so I… I don't know all the details of the study, but I'm really looking forward to hearing, what… what's the latest and what Nikhil has found about all these mechanisms. So welcome! 32 00:03:09.560 --> 00:03:11.719 216-101 Large Conference: Welcome, Nikhil. Welcome back. 33 00:03:12.780 --> 00:03:15.509 216-101 Large Conference: That was a great introduction. 34 00:03:16.140 --> 00:03:19.550 216-101 Large Conference: Yeah, so hi everyone, I'm Nikhil, 35 00:03:20.340 --> 00:03:27.229 216-101 Large Conference: And today, I'll be sharing about… sharing details about the project that I did over the summer at Goodwire. I was doing an internship there. 36 00:03:27.940 --> 00:03:34.549 216-101 Large Conference: And the objective for me sharing this work right now is to get, basically, feedback. 37 00:03:38.760 --> 00:03:47.029 216-101 Large Conference: Because this is still a work-in-progress project, and I'm hoping to submit it twice year. So any kind of feedback, questions. 38 00:03:47.250 --> 00:03:52.849 216-101 Large Conference: Minor details or more like an exploratory question would be pretty helpful at this point in time. 39 00:03:53.880 --> 00:03:58.339 216-101 Large Conference: Okay, so yeah, so the title of the project, at least for now, is this one. 40 00:03:58.790 --> 00:04:06.200 216-101 Large Conference: When a pointer isn't enough, how do… how language model Bind entities in reversible sentences. 41 00:04:07.050 --> 00:04:14.340 216-101 Large Conference: Okay, so let's just start with understanding what does binding or variable binding mean. I mean, David, sort of, like. 42 00:04:14.950 --> 00:04:22.220 216-101 Large Conference: touched upon that, from the programming language perspective, where when you say X equals to 5, 43 00:04:22.420 --> 00:04:26.840 216-101 Large Conference: How does it keep track of that piece of information later in the program? 44 00:04:26.960 --> 00:04:31.280 216-101 Large Conference: But, from a more natural setting, what would that mean is, let's say. 45 00:04:32.100 --> 00:04:36.400 216-101 Large Conference: You see this particular picture, and then in your mind, or in your brain, you can… 46 00:04:36.600 --> 00:04:45.100 216-101 Large Conference: keep the representations of horse and fedora to be more associated with each other in comparison to the 47 00:04:45.380 --> 00:04:47.729 216-101 Large Conference: representation of CAP and the CAD. 48 00:04:48.020 --> 00:04:58.629 216-101 Large Conference: And the fact that your mind can do that, basically means that it has the inherent binding capability of associating objects with its, corresponding features. 49 00:05:02.080 --> 00:05:06.170 216-101 Large Conference: And why do we care about binding? 50 00:05:06.900 --> 00:05:11.299 216-101 Large Conference: Yeah, I think two main reasons. One is historical, and one is more contemporary. 51 00:05:11.620 --> 00:05:17.389 216-101 Large Conference: the historical vein, I think in the 1980s or late 1980s, when people were just talking about whether 52 00:05:18.120 --> 00:05:26.660 216-101 Large Conference: they should put in more effort to do research on neural network versus symbolic AI. One of the main arguments, 53 00:05:28.580 --> 00:05:35.429 216-101 Large Conference: Against the neural network was the fact that the models of the neural networks will not be able to do binding, or variable binding. 54 00:05:35.510 --> 00:05:50.409 216-101 Large Conference: For instance, I think their assumption was something like, if you have a sentence like, John loves Mary, there would be some neuron which will activate for John, there would be some neuron which will activate for Mary, and there would probably be some neuron which will activate for the loves relation. 55 00:05:51.170 --> 00:06:03.499 216-101 Large Conference: So when you… I'm moving these chairs because I'm noticing the remoters just see the chairs and they don't see you. So the camera's right here. So if you want… if you'd like, you can sit… you can also sit here, which is more visible to the camera. 56 00:06:03.710 --> 00:06:04.635 216-101 Large Conference: No. 57 00:06:06.240 --> 00:06:24.329 216-101 Large Conference: Yeah, what was I saying? Yeah, so their assumption was something like there would be neuron for each of these three components of the sentence, and if… and they would all fire simultaneously for a sentence like, John loves Mary, and that will create some sort of, like, a combined representation of this particular sentence. But if you go by that particular… 58 00:06:25.970 --> 00:06:36.229 216-101 Large Conference: rational, or that particular form of computation, then the neural network won't be able to distinguish these two different sentences, which is, like, John loves Mary versus Mary loves John, because 59 00:06:36.310 --> 00:06:49.909 216-101 Large Conference: Those are just neurons, and they would get activated, and their combined representation would be similar, and hence, more will not be able to distinguish these two forms of sentences. So that was their argument for saying that neural network is not the correct form of 60 00:06:51.090 --> 00:06:56.719 216-101 Large Conference: Correct framework to study, computational, intelligence. 61 00:06:58.020 --> 00:07:10.369 216-101 Large Conference: That's why I think it becomes important. I mean, now we know that the models can do it. It's a pretty simple task, so I think it becomes important to fundamentally understand how the neural networks are doing it, so that we can better 62 00:07:10.700 --> 00:07:16.770 216-101 Large Conference: We can put the argument against this particular point that they were making. 63 00:07:18.050 --> 00:07:32.010 216-101 Large Conference: And the second point being… second reason for why understanding binding is more important is from the capability perspective. I think, lately there have been some work which has shown that, 64 00:07:32.500 --> 00:07:39.130 216-101 Large Conference: The failure modes… a number of failure modes in various neural networks can be attributed to their binding capabilities. 65 00:07:39.390 --> 00:07:41.420 216-101 Large Conference: That they… 66 00:07:41.720 --> 00:07:51.889 216-101 Large Conference: They're not able to perform a particular task beyond a certain limit, because there is a limitation of the binding capability within their model. 67 00:07:52.680 --> 00:08:04.629 216-101 Large Conference: So if we can fundamentally understand that, then probably we might be able to, overcome those limitations by making fundamental, like, mechanistic updates, or maybe just coming up with… 68 00:08:04.670 --> 00:08:12.780 216-101 Large Conference: Something like, tracer decompile components, which will, overcome those, limitations. 69 00:08:13.340 --> 00:08:17.179 216-101 Large Conference: So those are the two main, I think, reasons for studying binding. 70 00:08:17.650 --> 00:08:23.099 216-101 Large Conference: And one of the other reasons, which is sort of like what David was alluding to, when you have 71 00:08:23.980 --> 00:08:33.780 216-101 Large Conference: I mean… If you need to understand how a neural network is in, like, processing any algorithm. 72 00:08:34.330 --> 00:08:39.850 216-101 Large Conference: You need to be able to understand how it is keeping track of the variables that are 73 00:08:40.299 --> 00:08:41.819 216-101 Large Conference: Present in that algorithm. 74 00:08:42.030 --> 00:08:58.589 216-101 Large Conference: is that, like, variables are part of every algorithm that we study. If we don't understand how does that variable is being tracked within that particular algorithm, it becomes really difficult to understand any kind of algorithm that the model might be implementing. 75 00:08:59.540 --> 00:09:08.560 216-101 Large Conference: One question? Just on the practical side, do you think it's mostly solvent language and mostly a vision problem, or is it also still a problem on language? I think it's both. 76 00:09:08.980 --> 00:09:16.060 216-101 Large Conference: So, the second figure that I have here, I think they showed that as you increase the number of 77 00:09:16.190 --> 00:09:19.389 216-101 Large Conference: Objects in a particular figure. 78 00:09:20.780 --> 00:09:24.470 216-101 Large Conference: The tracking capability decreases. 79 00:09:24.670 --> 00:09:39.809 216-101 Large Conference: So if you ask different kinds of questions, or if you just ask the model to caption a particular image, that decreased, like, that performance degrade, and I think they were attributing that degradation in the performance to the binding limitations of the model. 80 00:09:39.910 --> 00:09:44.669 216-101 Large Conference: So that's more the vision side. Yeah. On the language side, are there still, like, big problems? 81 00:09:44.850 --> 00:09:50.330 216-101 Large Conference: Yeah, for instance, I have here the reversal curse. I mean, theory of mind could also be considered as a… 82 00:09:50.670 --> 00:09:53.899 216-101 Large Conference: limitation because of the binding pun. 83 00:09:54.830 --> 00:09:56.099 216-101 Large Conference: Okay, yeah. 84 00:09:57.400 --> 00:10:00.969 216-101 Large Conference: Okay, and since it is a fundamental… 85 00:10:01.200 --> 00:10:09.139 216-101 Large Conference: capability that we would definitely want to understand. There have been a few existing works in the literature we have which has tried to understand it. 86 00:10:09.150 --> 00:10:21.810 216-101 Large Conference: I'm not gonna go into complete detail on all of them, but I think the main point here is that the model is creating something like a pointer, pointer signal, pointer information to bind 87 00:10:21.820 --> 00:10:27.549 216-101 Large Conference: Values with… to bind variables with their corresponding values. 88 00:10:28.340 --> 00:10:47.789 216-101 Large Conference: What I mean by that, or what I mean by pointer mode mechanistically is, so let's say if you have, two, like, these kinds of… this kind of problem. Like, banana is in box A, the pen is in box B, and blah blah blah blah, and then you finally have this particular sentence, box P contains the… and the answer is pen. 89 00:10:47.820 --> 00:10:49.679 216-101 Large Conference: And the model can solve it. 90 00:10:51.210 --> 00:11:02.790 216-101 Large Conference: When I say the model is sort of solving this particular task using a pointer, what I mean is, it is creating a sort of representation in its internal computation, which is, 91 00:11:03.520 --> 00:11:09.740 216-101 Large Conference: Encoding the… Like, an abstract position of the correct answer. 92 00:11:10.250 --> 00:11:14.249 216-101 Large Conference: So here, the pen is, a… 93 00:11:14.810 --> 00:11:30.459 216-101 Large Conference: The reason Ben is the answer here is because the model is first figuring out that the, sort of, like, second object in the prompt is the correct one, so I need to find the correct, like, the second object of the prompt, and then, sort of, it fetches it. 94 00:11:31.010 --> 00:11:48.080 216-101 Large Conference: And previous works have shown that that actually happens. So there are internal representations in the model, which, when you patch from, let's say, this particular counterfactual sample to this particular clean sample, the final output of the clean changes from apple to book. 95 00:11:48.320 --> 00:11:51.340 216-101 Large Conference: Which shows that the model is actually representing 96 00:11:51.440 --> 00:11:54.979 216-101 Large Conference: A sort of, like, abstract traditional… 97 00:11:55.100 --> 00:11:59.820 216-101 Large Conference: Wider to the second object of a particular prompt, of the counterfactual prompt. 98 00:12:01.220 --> 00:12:02.480 216-101 Large Conference: Wait, is Dr. here? 99 00:12:03.180 --> 00:12:03.910 216-101 Large Conference: Okay. 100 00:12:04.740 --> 00:12:08.840 216-101 Large Conference: Okay, now coming to the… so that was the previous work. 101 00:12:09.050 --> 00:12:15.440 216-101 Large Conference: In this work, what we are trying to do is trying to understand this finding mechanism in the reversible sentences. 102 00:12:15.700 --> 00:12:32.050 216-101 Large Conference: Now, what do I mean by reversible versus irreversible sentences? Is, so this is now… the left-hand side is like a typical example of irreversible sentence. When the nouns of a particular sentences are not reversible, that's just called irreversible sentence. So here, you cannot… we cannot really say, right? 103 00:12:32.260 --> 00:12:34.499 216-101 Large Conference: Paris lives in Alice, that doesn't make sense. 104 00:12:34.750 --> 00:12:39.289 216-101 Large Conference: Whereas on the right-hand side, that's a typical example of reversible sentences. 105 00:12:39.850 --> 00:12:44.239 216-101 Large Conference: So we have, like, multiple, instances of a particular type. 106 00:12:44.350 --> 00:12:52.360 216-101 Large Conference: which is the person type, and those are reversible with each other. Alice introduced Bob to Charlie is still a valid sentence. 107 00:12:52.600 --> 00:13:05.440 216-101 Large Conference: So, in this work, we are primarily focused on understanding the binding in the reversible sentences, and… because most of the previous work… I would say all of the previous works on binding have looked only into the irreversible sentences. 108 00:13:07.740 --> 00:13:14.500 216-101 Large Conference: Yeah, and why do we care about reversible sentences? I think, 109 00:13:14.950 --> 00:13:22.180 216-101 Large Conference: Again, two reasons. I think, historically, yeah, when I was giving you the argument of, historically, people were interested in binding because of 110 00:13:23.110 --> 00:13:33.750 216-101 Large Conference: that neural network was a symbolic AI argument. People primarily use reversible sentences, or reversible sentences examples to argue against neural networks. 111 00:13:33.980 --> 00:13:37.769 216-101 Large Conference: And the second reason being, I think irreversible? 112 00:13:38.210 --> 00:13:45.980 216-101 Large Conference: is that, I would say, an easy case for binding, because all you need is just the pointer. You don't really have to consider the type information. 113 00:13:46.170 --> 00:13:50.989 216-101 Large Conference: But in the reversible case, you do need to consider type information. 114 00:13:51.250 --> 00:13:54.379 216-101 Large Conference: So that's what I think we are trying to… 115 00:13:54.890 --> 00:14:00.810 216-101 Large Conference: Extend our understanding of the binding from irreversible… sorry, from, yeah, from irreversible to the reversible sentences. 116 00:14:02.260 --> 00:14:12.240 216-101 Large Conference: That was a question, I guess. Is it also having a pointer and a data type? Yeah, that's… that's the part of the project. That's the rest of the project. 117 00:14:12.750 --> 00:14:19.510 216-101 Large Conference: So for the irreversible ones, the model use just a pointer, or also the semantic? 118 00:14:19.750 --> 00:14:23.720 216-101 Large Conference: Together, or for the irreversible, it only uses the pointer. 119 00:14:24.430 --> 00:14:33.049 216-101 Large Conference: Yeah, it's the rest of the class. Yeah. I'm… right now, I'm only explaining what I'm… what I'm doing, and why I'm doing it. 120 00:14:33.250 --> 00:14:37.080 216-101 Large Conference: Wait, just to understand the scope of a vertical sentence. 121 00:14:37.300 --> 00:14:40.259 216-101 Large Conference: So, reversible sentences, they… 122 00:14:41.000 --> 00:14:42.800 216-101 Large Conference: don't meet… 123 00:14:42.900 --> 00:15:00.569 216-101 Large Conference: be meaningful. Like, meaningful as in, like, it should not necessarily mean the same thing. Like, you know how certain things, if you were to swap, you might still get the same correct, like… like, if, I can't think of another sentence right now. Yeah, fairly. Yeah. 124 00:15:00.940 --> 00:15:09.970 216-101 Large Conference: then it would mean the same thing. Yeah, like, so we don't… we don't care about them meaning the same thing, ultimately, when they're reversible. It's just only the types you care about, like, because… 125 00:15:09.970 --> 00:15:21.260 216-101 Large Conference: in this… in these two sentences, they don't… like, while the swap is sensible, as you mentioned, it means something else. So, like, we don't… Yeah, the meaning of the sentence would be different, but it is still sensible. 126 00:15:22.010 --> 00:15:25.830 216-101 Large Conference: Okay, alright, so… But in the left-hand side, it's not sensible. 127 00:15:27.570 --> 00:15:33.360 216-101 Large Conference: So we just… we're basically taking meaning out of context, and just looking at, like, type-based sensibility. 128 00:15:38.450 --> 00:15:43.920 216-101 Large Conference: Like, what if… so in this case, it means something else. What if you had, like, 129 00:15:45.040 --> 00:15:50.670 216-101 Large Conference: So I guess, is there anything in between, between irreversible and irreversible? Not that I know of. 130 00:15:51.150 --> 00:16:06.399 216-101 Large Conference: I think you're thinking of, like, Alice met Bob, and Bob met Alice are, like, reversible, but it means the same thing. Would that be in your dataset here? Yeah, that would be considered as reversible. I think the… yeah, if you see the subtitle here, 131 00:16:06.690 --> 00:16:10.289 216-101 Large Conference: If swapping the nouns makes sense, if they are… 132 00:16:10.420 --> 00:16:14.730 216-101 Large Conference: Swappable, then that's sort of… that's a reversible sentence example. 133 00:16:14.830 --> 00:16:19.879 216-101 Large Conference: If the nouns are not swappable, then they are not, they are basically irreversible sentences. 134 00:16:21.700 --> 00:16:22.500 216-101 Large Conference: Sense? 135 00:16:22.990 --> 00:16:32.890 216-101 Large Conference: Do you think it's because of symmetric properties that the model is able to, like, have a reversible sense, or is it something else? 136 00:16:34.720 --> 00:16:38.250 216-101 Large Conference: Right now, I'm just trying to define what is reversible and irreversible. 137 00:16:38.410 --> 00:16:41.259 216-101 Large Conference: I'm still not gotten into how the model works out. 138 00:16:42.660 --> 00:16:48.410 216-101 Large Conference: Yes, my… so far, I've only talked about… tried to convey the information, what each of them mean. 139 00:16:48.540 --> 00:16:52.200 216-101 Large Conference: Sorry, there's just a question in the chat, why not study? 140 00:16:52.400 --> 00:16:57.310 216-101 Large Conference: passive sentences? Is there a reason the meaning is not significant to study? 141 00:16:58.220 --> 00:17:00.890 216-101 Large Conference: So, active to passive, 142 00:17:07.440 --> 00:17:27.150 216-101 Large Conference: I think those are part of the reversible sentences. I would say passive, I would so… I will… I will show in my later slides that there are different types of, we could have, like, different types of reversible sentences. Active and passive are sort of one… I mean, two types of them. So, yeah, we do study them. I think. You're gonna see more details in the later slides. 143 00:17:29.470 --> 00:17:30.300 216-101 Large Conference: Okay. 144 00:17:32.680 --> 00:17:43.159 216-101 Large Conference: One more point, like, a concept that you should know. Thematic role. What does thematic role mean? This is, again, a term coming from linguists… linguistics. 145 00:17:43.920 --> 00:17:55.350 216-101 Large Conference: So, thematic rules is basically associated with a noun, and so, basically, each noun will have a particular thematic rule, depending on its function in a particular sentence. 146 00:17:55.590 --> 00:18:01.450 216-101 Large Conference: So, for instance, here in the active form, we have Anna warned Sam about Grace. 147 00:18:01.770 --> 00:18:06.290 216-101 Large Conference: So Anna is the agent, because Anna is the person who is doing the action. 148 00:18:06.850 --> 00:18:11.789 216-101 Large Conference: That's why it is the agent. And, Sam is sort of the person who is… 149 00:18:11.910 --> 00:18:14.779 216-101 Large Conference: Being action upon, so that's recipient. 150 00:18:15.880 --> 00:18:17.400 216-101 Large Conference: And similarly, the grace. 151 00:18:17.700 --> 00:18:20.529 216-101 Large Conference: And these are position independent. 152 00:18:20.650 --> 00:18:21.640 216-101 Large Conference: So… 153 00:18:21.970 --> 00:18:32.380 216-101 Large Conference: So, these two sentences, this is an active form of a particular sentence, this is a passive form of particular sentences, but the thematic roles of both of these sentences is still the same. 154 00:18:35.200 --> 00:18:37.790 216-101 Large Conference: Anna still is the agent in both of them. 155 00:18:39.690 --> 00:18:40.500 216-101 Large Conference: Okay. 156 00:18:42.190 --> 00:18:52.990 216-101 Large Conference: Yeah, so these are sort of, like, the broader questions that we have answers for in this particular project. Do the models, or at least the language model, understand 157 00:18:53.360 --> 00:18:55.170 216-101 Large Conference: And encode thematic roles. 158 00:18:55.310 --> 00:19:05.539 216-101 Large Conference: And if they are being used for doing binding, and is the mechanism for variable binding generalizable across different languages? 159 00:19:07.520 --> 00:19:13.929 216-101 Large Conference: Okay, so those were the background stuff. Now I'm starting with the actual experimental setup. 160 00:19:15.030 --> 00:19:22.210 216-101 Large Conference: So, I studied… so far, I've only resolved for this particular single model, which is Gemma 20… 7B, instruct. 161 00:19:22.640 --> 00:19:31.720 216-101 Large Conference: And, on the right-hand side, this is the behavioral performance of this particular model on different types of reversible sentences. 162 00:19:32.690 --> 00:19:39.369 216-101 Large Conference: So, I have, like, 5 different types of reversible sentences. So, if you look at this particular task, 163 00:19:39.520 --> 00:19:48.589 216-101 Large Conference: Joey informed Alice about Frank, that's a typical example of active. Kate was reminded about Karen by Philip, that's a typical example of passive. 164 00:19:49.590 --> 00:19:58.390 216-101 Large Conference: And, it was Will who informed James about Susan. That's a typical example of Cliff Agent. And I have two other forms as well. 165 00:19:59.310 --> 00:20:06.200 216-101 Large Conference: And the model can… And this is basically the task. I have… I'm defining what does 166 00:20:06.420 --> 00:20:16.079 216-101 Large Conference: each of the three thematic role means in the instruction, and then I'm asking them all to give me the thematic role of it. Thematic role present in a particular sentence. 167 00:20:16.210 --> 00:20:20.500 216-101 Large Conference: Then I have the sentences, and then finally the model is supposed to give me the answer. 168 00:20:20.760 --> 00:20:28.919 216-101 Large Conference: So, for each of these five types of sentences, the model can answer all the three different thematic roles. 169 00:20:31.120 --> 00:20:32.940 216-101 Large Conference: Pretty… pretty well. 170 00:20:34.510 --> 00:20:35.700 216-101 Large Conference: Is that here? 171 00:20:36.780 --> 00:20:39.090 216-101 Large Conference: It's just behavioral evaluation so far? 172 00:20:43.530 --> 00:20:44.390 216-101 Large Conference: Okay. 173 00:20:45.000 --> 00:20:48.430 216-101 Large Conference: Agent and recipient. Agent, recipient, and topic. 174 00:20:49.160 --> 00:20:52.330 216-101 Large Conference: Those are the three… thematic rules. 175 00:20:52.560 --> 00:20:56.360 216-101 Large Conference: Because the sentences only have 3 nouns, 3 persons. 176 00:21:00.170 --> 00:21:04.100 216-101 Large Conference: Does it matter if you only do sentence 1? 177 00:21:05.530 --> 00:21:15.279 216-101 Large Conference: Does it matter if… what does doing sentence 1 mean? So, if for all questions you ask it to identify who the agent is in sentence 1, 178 00:21:16.350 --> 00:21:26.689 216-101 Large Conference: Does that matter, or does that not matter? Or do you also ask in sentence two and sentence? Yeah, I ask everything. Yeah, yeah, yeah. This behavioral performance is across all the three sentences. 179 00:21:26.930 --> 00:21:28.330 216-101 Large Conference: That is just one example. 180 00:21:29.970 --> 00:21:30.830 216-101 Large Conference: Okay. 181 00:21:32.500 --> 00:21:35.330 216-101 Large Conference: So the first thing that I did was I did probing. 182 00:21:35.910 --> 00:21:39.440 216-101 Large Conference: I… Train two probes. 183 00:21:39.720 --> 00:21:45.529 216-101 Large Conference: on each of the 9% token names that I have in my sentences. 184 00:21:45.800 --> 00:21:56.450 216-101 Large Conference: Will, James, Susan, Joey, Alice, Frank, Kate, Karen, and Philip. So on each of these 9 tokens residual stream, I train 2 probes. 185 00:21:56.700 --> 00:21:59.300 216-101 Large Conference: First proof is to predict 186 00:21:59.450 --> 00:22:13.430 216-101 Large Conference: the syntax index in which this particular token is present. So the input to the probe is the residual vector of each of this, or one of this, person, and the output is 3-class class, like, 3-class vector. 187 00:22:13.640 --> 00:22:16.820 216-101 Large Conference: Which needs to predict which particular 188 00:22:17.960 --> 00:22:20.050 216-101 Large Conference: Sentence this particular person is in. 189 00:22:20.340 --> 00:22:25.299 216-101 Large Conference: So that's the first proof. Second proof is to predict the thematic role of a particular person. 190 00:22:25.500 --> 00:22:36.890 216-101 Large Conference: Input is, again, the residual vector. Output is three-class classifier, predicting, agent, recipient, or the topic, whichever this particular person, is representing in a particular sentence. 191 00:22:37.480 --> 00:22:44.829 216-101 Large Conference: And the rest… yeah, this is basically the result. The top one is for the thematic rule, and the second one is for the sentence index. 192 00:22:45.050 --> 00:22:50.889 216-101 Large Conference: Sentence index is actually… sentence index information is present in all the person tokens. 193 00:22:51.500 --> 00:22:55.149 216-101 Large Conference: Pretty much from the second layer, like, after the first layer. 194 00:22:55.530 --> 00:22:58.410 216-101 Large Conference: That's what this particular… 195 00:22:59.670 --> 00:23:06.320 216-101 Large Conference: Line graph is showing. But the thematic rule results are a little bit more interesting. 196 00:23:06.500 --> 00:23:14.510 216-101 Large Conference: What I found was this thematic rule information was present in each sentence only at the second and the third person. 197 00:23:14.790 --> 00:23:16.730 216-101 Large Conference: It is not present in the first person. 198 00:23:17.710 --> 00:23:23.009 216-101 Large Conference: So what does that mean is, the model does not really encode the thematic rule of will. 199 00:23:23.250 --> 00:23:32.119 216-101 Large Conference: But it does encode the thematic role of, well, sorry, James and Susan. Similarly, here, it won't encode the thematic role of Joey, but it will encode the thematic role of Alice and Frank. 200 00:23:32.280 --> 00:23:33.730 216-101 Large Conference: Let's simulated for the last one. 201 00:23:34.070 --> 00:23:39.580 216-101 Large Conference: Have you tried defining the thematic rules after the sentences? 202 00:23:39.860 --> 00:23:45.449 216-101 Large Conference: And then do the proving, and then what happens? Yeah, that's the second setup, I think. Okay, yeah. 203 00:23:45.900 --> 00:23:50.120 216-101 Large Conference: So, if it's a three-way classification, does that mean that the activation 204 00:23:50.260 --> 00:23:57.499 216-101 Large Conference: space is somehow either similar to the second person and the third person for the first person. That's why it's confused, right? 205 00:24:02.350 --> 00:24:06.320 216-101 Large Conference: Sorry, say a little bit more. Sorry, so if you go to the next slide… 206 00:24:06.610 --> 00:24:16.359 216-101 Large Conference: So, in the top graph, there are the three thematic roles, right? It's only a three-way classification for your activations. 207 00:24:16.590 --> 00:24:28.159 216-101 Large Conference: So does that mean, like, that blue curve basically said whenever, like, your probe sees, like, I think, agent token, either classified as recipient and topic, and it gets confused. 208 00:24:28.460 --> 00:24:42.879 216-101 Large Conference: Okay. Yeah, it's not really whether it sees any particular thematic role. Those three colors are sort of representing whether I'm probing on the first person of a particular sentence, second person of a particular sentence, or third person of a particular sentence. 209 00:24:43.400 --> 00:24:52.269 216-101 Large Conference: So what that blue line is showing is, if you do the probing on the first person of a particular sentence, then that particular first person does not really encode a thematic row. 210 00:24:52.920 --> 00:25:05.390 216-101 Large Conference: Oh, oh, I understand. Oh, okay. So it's not… it's not based on whether they're correctly identifying agent or person, it's just the first or second, third person? Yes. 211 00:25:06.380 --> 00:25:10.459 216-101 Large Conference: So, what does that mean? Let's say if you consider this third sentence. 212 00:25:10.930 --> 00:25:21.859 216-101 Large Conference: If you do the probing on the CAD token residual stream, if you basically ask what is its particular thematic role, you won't be able to decode it linearly. 213 00:25:22.090 --> 00:25:25.719 216-101 Large Conference: But you would be able to decode the thematic role of Karen and Fuller. 214 00:25:25.980 --> 00:25:37.720 216-101 Large Conference: That's what I'm trying to say. I mean, it makes sense, right? Like, you can't know the role, like… it's too early to know the role. You would not be able to decode it even nonlinearly, or any of the… Yeah, yeah. 215 00:25:37.780 --> 00:25:50.230 216-101 Large Conference: And it would write not needed, but yeah, the fact is, the model has not really processed the verb. And since the thematic roles are super dependent on the verb, it cannot really figure out the thematic role of the first person. 216 00:25:52.720 --> 00:26:02.299 216-101 Large Conference: I have a question. Why did you choose to design the task like this? So you only ask about the first sentence? Oh, no, I asked about all the three sentences. This is just an example. 217 00:26:02.900 --> 00:26:19.149 216-101 Large Conference: I ask about all the three thematic rules across all the three sentences. But you're gonna shuffle it later, like, so, like, randomly you'll ask about… Yeah, yeah, this is a normal thing that… Why do you give all three sentences and just ask about one of them? Yeah, why, why, why have three sentences? Because I want one sent, out, like, one token output. 218 00:26:19.890 --> 00:26:27.439 216-101 Large Conference: I mean, what you could have… could you have done your whole study with just one sentence at a time? Yeah, you could do that. 219 00:26:28.150 --> 00:26:29.639 216-101 Large Conference: Okay. 220 00:26:29.790 --> 00:26:36.070 216-101 Large Conference: Because I thought that there was, like, an underlying reason for that. But in that case, you wouldn't really see any pointer information. 221 00:26:36.640 --> 00:26:41.680 216-101 Large Conference: So you want to be able to disambiguate pointers. 222 00:26:42.430 --> 00:26:45.530 216-101 Large Conference: with the thematic rule. I think it will become a little bit clearer in the next slide. 223 00:26:46.440 --> 00:26:55.710 216-101 Large Conference: I don't understand right now. Yeah, let me just go into that. Just to piggyback on that, I know you said you wanted it to be one token, but, like, is there a reason why 224 00:26:55.910 --> 00:27:04.749 216-101 Large Conference: Like, kind of like with precision versus recall, just ask who are all the ones that can access agents, like, so you could give all three of them. 225 00:27:05.400 --> 00:27:14.710 216-101 Large Conference: Icore is just a different task, I just didn't do that. Okay. Yeah, you could… you could do multiple token thing, tasks as well, but I was just… 226 00:27:15.560 --> 00:27:17.660 216-101 Large Conference: Yeah, I wanted to have a single login. 227 00:27:18.790 --> 00:27:28.870 216-101 Large Conference: But, okay, so, what, regarding Ada's point. So, if you do not have a… Multiple sentences. 228 00:27:29.220 --> 00:27:35.669 216-101 Large Conference: So this is sort of like the causal model that I have, the model is using for solving the task at the last token. 229 00:27:35.820 --> 00:27:42.820 216-101 Large Conference: So, if you only have, let's say… 230 00:27:43.300 --> 00:27:48.790 216-101 Large Conference: The task is something like, there is only one sentence, it was Will who informed James about Susan. 231 00:27:49.020 --> 00:27:52.630 216-101 Large Conference: And you only ask determined which of the named person is the agent. 232 00:27:53.180 --> 00:27:58.110 216-101 Large Conference: Right? And the model will not encode any thematic rule for well. 233 00:27:58.360 --> 00:28:05.880 216-101 Large Conference: But it will encode the magic role of James and Susan. And then finally, when you ask the answer, it will… 234 00:28:07.250 --> 00:28:09.369 216-101 Large Conference: It won't use any pointer information. 235 00:28:09.610 --> 00:28:12.409 216-101 Large Conference: It will just use a thematic role information. 236 00:28:12.730 --> 00:28:17.120 216-101 Large Conference: The fact that, 237 00:28:17.670 --> 00:28:30.189 216-101 Large Conference: Yeah, let's say instead of agent, you ask about the recipient, okay? And the James will directly say that I am an agent, because it knows that it is an agent, and model will just use that piece of information to fetch James. 238 00:28:30.370 --> 00:28:33.439 216-101 Large Conference: It doesn't really need to use any pointer information. 239 00:28:33.590 --> 00:28:40.560 216-101 Large Conference: that I need to find the recipient of… What is the information? Yeah, it's more like, sentence index. 240 00:28:41.380 --> 00:28:46.380 216-101 Large Conference: whether I need to find any information, Of a particular sentence. 241 00:28:46.770 --> 00:28:50.250 216-101 Large Conference: Yeah, that's pretty much what I'm calling as pointer. 242 00:28:50.430 --> 00:28:59.199 216-101 Large Conference: But, like, if you ask, you want it from first sentence, it means a pointer of first sentence. Yes, and I wanted that information, so that's why I kept multiple sentences. 243 00:29:00.330 --> 00:29:12.099 216-101 Large Conference: So, the sentences here are, like, have a similar role to the boxes in the other example. Yes. Like, you have a box with 3 things, and a box with 3 things. Like, okay, look at box 2, what is the age? 244 00:29:12.100 --> 00:29:20.819 216-101 Large Conference: or look at the second sentence and that, yeah. Nikhilh, I've been… when I explained your previous work to people, I've started to call it temporal information. 245 00:29:20.830 --> 00:29:39.260 216-101 Large Conference: Because it's about, you know, what time… I think it might be an interesting contrast with your theme, maybe that's something to consider. What do you say? Temporal? I say… Temporal information? Yeah, this is an old-fashioned way of looking at sequences. It's like, oh, like, the old things, the earliest, and the newest things are latest in a sequence. 246 00:29:39.370 --> 00:29:45.439 216-101 Large Conference: And, like, how… where the order things happened is what time order they happened in, in the story. 247 00:29:45.780 --> 00:29:48.719 216-101 Large Conference: And so I've been… I've been explaining it as a temporal… 248 00:29:48.870 --> 00:29:51.179 216-101 Large Conference: Reference, the thing that we've been calling pointers. 249 00:29:51.340 --> 00:30:06.569 216-101 Large Conference: I think that clarifies, because the pointer is very generic, right, but this is the public. I basically took pointer from your slides. Yeah, so now I've changed my slides to… let's call it temporal now, and just… They're temporal pointers, or just temporal? 250 00:30:06.980 --> 00:30:22.510 216-101 Large Conference: Yeah, temporal pointers, a temporal reference, yeah. But, okay, one second. Is your… no, it's not clear. It's not clear to me, like, okay, if you didn't have that, you just had one sentence, what kind of information would you not have in this experiment? So here, if you look at this particular example. 251 00:30:24.050 --> 00:30:33.600 216-101 Large Conference: What we are saying is, at the last token, the model is creating a representation that I need to fetch information from the second sentence for this 252 00:30:34.320 --> 00:30:35.960 216-101 Large Conference: In this counterfactual sample. 253 00:30:36.670 --> 00:30:37.739 216-101 Large Conference: Is that clear? 254 00:30:38.100 --> 00:30:41.109 216-101 Large Conference: Do you agree with that or not? In this example, yeah. 255 00:30:41.740 --> 00:30:47.809 216-101 Large Conference: Okay, and then we are patching it here, that's why, and we are seeing that the output changes from, 256 00:30:48.500 --> 00:30:52.449 216-101 Large Conference: Apple to book. So now switch to the other one, with the three rolls. 257 00:30:53.090 --> 00:31:12.229 216-101 Large Conference: But couldn't that be a long sentence? Like, it could just be, like, you have all the information of the box and be like, so the box… That's right. So, like, one sentence. Yeah, so I think that is, sort of, like, an open question. What actually counts a particular sentence that can happen? You have, like, a… instead of comma there, if you have… 258 00:31:12.490 --> 00:31:25.410 216-101 Large Conference: if you just have, like, the banana is in box A, and the pen is in box B, and the watch is in box H, I would still assume that if you do the patching, that then you'll still get booked there. Right. 259 00:31:26.270 --> 00:31:37.090 216-101 Large Conference: I think, you know what, I bet it would be clearer when you actually describe the experiment. Instead of, like, describing an experiment without your slides, maybe you should go on and just describe what you've done. Okay, 260 00:31:37.690 --> 00:31:44.969 216-101 Large Conference: But this is how, this is the causal model that I have for how the model is solving the task at the last token. 261 00:31:45.550 --> 00:31:57.220 216-101 Large Conference: what it is you are doing is, it is… it still has that syntax index, which is sort of, like, equivalent to the pointer information that I showed in that previous slide. In addition to that. 262 00:31:57.510 --> 00:32:01.810 216-101 Large Conference: It is… it is using, this thematic role variable. 263 00:32:02.640 --> 00:32:06.710 216-101 Large Conference: And an additional variable called FetchFirst. 264 00:32:07.380 --> 00:32:11.920 216-101 Large Conference: So when you ask it to, let's say, give me the agent of a particular sentence. 265 00:32:12.140 --> 00:32:22.849 216-101 Large Conference: It will try to use this role variable. It will use this role variable and the sentence index variable, sort of combine these two pieces of information to actually fetch that particular, 266 00:32:23.180 --> 00:32:27.550 216-101 Large Conference: Pets the person with that particular thematic role in that particular sentence. 267 00:32:28.070 --> 00:32:31.300 216-101 Large Conference: But the problem with that argument is… 268 00:32:31.450 --> 00:32:35.159 216-101 Large Conference: I just showed you that the first person does not really encode a thematic row. 269 00:32:35.620 --> 00:32:47.219 216-101 Large Conference: So, if the answer is the first person, which is Will here, the model will still get the correct answer, but it cannot really use the thematic rule, because Will does not really know what thematic role it encodes. 270 00:32:47.380 --> 00:32:50.319 216-101 Large Conference: So that's why I have this additional variable called fetchWest. 271 00:32:51.180 --> 00:32:58.989 216-101 Large Conference: So the model actually does sort of, like, a conditional, assignment here. If… 272 00:32:59.690 --> 00:33:03.730 216-101 Large Conference: The sentence that you want to fetch the thematic, like. 273 00:33:04.380 --> 00:33:08.890 216-101 Large Conference: That you want to fetch information from is of a particular sentence type. 274 00:33:09.590 --> 00:33:10.470 216-101 Large Conference: Then? 275 00:33:10.940 --> 00:33:14.799 216-101 Large Conference: Either you use the thematic rule or the fetch-first information. 276 00:33:15.930 --> 00:33:19.320 216-101 Large Conference: it will become a little bit more clear when I show the intervention itself. 277 00:33:19.500 --> 00:33:21.500 216-101 Large Conference: So this is the intervention. 278 00:33:21.680 --> 00:33:23.430 216-101 Large Conference: It's not clear at the back, right? 279 00:33:23.670 --> 00:33:25.129 216-101 Large Conference: Can you see the text? 280 00:33:26.040 --> 00:33:27.170 216-101 Large Conference: No. 281 00:33:29.170 --> 00:33:31.309 216-101 Large Conference: How do I share? Speaker funds. 282 00:33:32.670 --> 00:33:34.890 216-101 Large Conference: Fedex. 283 00:33:35.180 --> 00:33:41.409 216-101 Large Conference: So, this is the counterfactual for, aligning the role variable. 284 00:33:41.630 --> 00:33:43.090 216-101 Large Conference: In the Cozuma. 285 00:33:43.380 --> 00:33:44.160 216-101 Large Conference: Okay. 286 00:33:46.170 --> 00:33:51.429 216-101 Large Conference: Now, in the counterfactual sample, I'm asking about the agent of the first sentence, which is well. 287 00:33:51.560 --> 00:33:55.879 216-101 Large Conference: In my clean prompt, I'm asking about the topic of the second sentence, which is Karen. 288 00:33:56.770 --> 00:33:58.759 216-101 Large Conference: Okay? Now… 289 00:33:59.230 --> 00:34:08.990 216-101 Large Conference: I'm doing the patching at the last token, so assume that there is some representation in the residual vector of the last token, which is encoding this role information. 290 00:34:09.320 --> 00:34:12.240 216-101 Large Conference: Now, if you patch that representation. 291 00:34:12.520 --> 00:34:17.589 216-101 Large Conference: From this particular sentence to this particular sentence, what do you think the final output would be? 292 00:34:19.449 --> 00:34:20.380 216-101 Large Conference: Phillip. 293 00:34:21.360 --> 00:34:29.149 216-101 Large Conference: One second, it is… Philip, probably. Yeah, who are… what do you call Philip? 294 00:34:32.600 --> 00:34:36.020 216-101 Large Conference: Okay, let's see, stick with me. 295 00:34:36.480 --> 00:34:39.600 216-101 Large Conference: Both Anna. Who are for Anna? 296 00:34:39.880 --> 00:34:57.750 216-101 Large Conference: Agent of the first. Agent of the first of the 21? I said all the options. Don't consider a residual vector or anything like that. Just assume that there is some representation encoding that scroll. 297 00:34:58.230 --> 00:35:00.479 216-101 Large Conference: And I'm touching in the role representation. 298 00:35:01.780 --> 00:35:04.770 216-101 Large Conference: Topic is grace. Great. 299 00:35:06.700 --> 00:35:17.660 216-101 Large Conference: You're swapping out topic first. That's the second sentence, right? Oh, swapping it? Yes. Okay, okay, let me just go ahead. The answer is for it. 300 00:35:19.150 --> 00:35:23.209 216-101 Large Conference: But there are some there, let me… 301 00:35:24.330 --> 00:35:42.349 216-101 Large Conference: So the rational here is, in this particular counterfactual, the model is encoding this particular information that I need to find the agent of the second sentence. So there is an agent, and there is a second sentence representation. If I… and in the clean, there is this topic representation and the second sentence information, signal. 302 00:35:42.370 --> 00:35:56.349 216-101 Large Conference: So, if I patch in only the role information from the first one to the second one, then in the second one, the model will start thinking that it does not really have to find the topic of the second sentence, but instead it needs to find the agent of the second sentence, which is… 303 00:35:57.200 --> 00:36:00.870 216-101 Large Conference: That is the rational of this intervention. 304 00:36:01.120 --> 00:36:06.220 216-101 Large Conference: Oh, sorry. How do you isolate the role, like, from the activation? 305 00:36:06.830 --> 00:36:10.959 216-101 Large Conference: You do this intervention, And if the intervention works. 306 00:36:11.070 --> 00:36:13.950 216-101 Large Conference: I mean, if you do some kind of patching. 307 00:36:14.440 --> 00:36:19.970 216-101 Large Conference: And if you see that the output of the clean prompt changes from 308 00:36:20.320 --> 00:36:25.310 216-101 Large Conference: Karen to fill up, then the thing that you patch should represent that role information. 309 00:36:26.020 --> 00:36:43.940 216-101 Large Conference: But, the last student should have both the pointer to the first index and the thematic rule, right? Yes. But then only this… but then the second example is only using the thematic rule, and it's not using the positional of the first sentence. Yes. So it's still looking at the second sentence. Yes. 310 00:36:44.020 --> 00:36:46.689 216-101 Large Conference: Why is the position not getting overrated? 311 00:36:47.310 --> 00:36:51.750 216-101 Large Conference: There is two ways to answer that. 312 00:36:52.020 --> 00:36:53.680 216-101 Large Conference: First. 313 00:36:53.880 --> 00:37:02.830 216-101 Large Conference: Let me just… okay, so think of that each of these two pieces of information is present in a specific subspace, and those are independent of each other, and I can patch a specific subspace. 314 00:37:03.320 --> 00:37:06.689 216-101 Large Conference: And that's why you can get Philippa's the output. 315 00:37:07.570 --> 00:37:21.449 216-101 Large Conference: But right now, I'm not even talking about representation at all. I'm just talking about the high-level causal model, which is, like, this is your perception of what is going on. This… the model has this particular particular… this… this… these variables. 316 00:37:21.680 --> 00:37:23.759 216-101 Large Conference: Which are connected in this particular way. 317 00:37:23.910 --> 00:37:40.509 216-101 Large Conference: And, so if you run a particular prompt on your high-level causal model, each of those variables will take this particular value. And now, if you do the intervention on that high-level causal model, this is the expected final output if you do the intervention on that… on a specific variable. 318 00:37:40.770 --> 00:37:43.330 216-101 Large Conference: That is sort of, like, the framework here. 319 00:37:44.310 --> 00:37:46.729 216-101 Large Conference: Okay, that makes sense. Yeah. Okay. 320 00:37:47.130 --> 00:37:50.429 216-101 Large Conference: how he got the question. Nikhil, how do you come up with psychology? 321 00:37:50.630 --> 00:37:51.490 216-101 Large Conference: Good up. 322 00:37:51.830 --> 00:38:02.420 216-101 Large Conference: Jenny's a summer of research. Did you start with that, and then you try to experiment, or you kept experimenting and then changed? 323 00:38:02.600 --> 00:38:19.280 216-101 Large Conference: Yeah, it's a lot of feet to trial. You're probably missing the Feucht first thing at first. Yeah, yeah, yeah. Right. I didn't expect it. I… I don't know, I think Eric has seen those bigger grid-off patch lots. I tried a lot of stuff. Yeah. 324 00:38:19.480 --> 00:38:31.750 216-101 Large Conference: Yeah, this was not the first thing that I came up with. One suggestion, maybe something that would make it clear here that you're talking about just the causal model, is you could have, like, two causal models and, like, show the patch between them. 325 00:38:31.980 --> 00:38:38.060 216-101 Large Conference: Like, you, like, take two sentences and be like, okay, we're populating the variables with these sentences. 326 00:38:38.470 --> 00:38:43.710 216-101 Large Conference: Into our causal model, and then if you, like, swap these, then it might be more clear that, like. 327 00:38:44.070 --> 00:38:48.619 216-101 Large Conference: like, what should the counterfactual output be when I am patching? 328 00:38:48.870 --> 00:38:52.309 216-101 Large Conference: like, because you have, like, I think the text one is nice, but, like. 329 00:38:52.770 --> 00:38:55.910 216-101 Large Conference: You could have the… the parallel 330 00:38:56.030 --> 00:39:00.110 216-101 Large Conference: patch in the cosmology. Okay. Like, that's an option. 331 00:39:00.250 --> 00:39:08.139 216-101 Large Conference: Yeah, like, implementing the causal model on a sentence might clarify what the causal model is doing exactly. Okay. 332 00:39:08.690 --> 00:39:09.400 216-101 Large Conference: Yep. 333 00:39:09.730 --> 00:39:11.789 216-101 Large Conference: One can create such an animation. 334 00:39:12.610 --> 00:39:17.030 216-101 Large Conference: So, what if you patch to a sentence ID that does not have the rule? 335 00:39:17.280 --> 00:39:19.640 216-101 Large Conference: Which you want to answer, like. 336 00:39:21.810 --> 00:39:27.639 216-101 Large Conference: Sentence ID which does not have the rule. Yeah. Let's say, like, there is no topic in a sentence. 337 00:39:28.170 --> 00:39:32.020 216-101 Large Conference: And you patch the sentence ID to a sentence that doesn't have a topic. 338 00:39:32.610 --> 00:39:33.400 216-101 Large Conference: Hmm. 339 00:39:33.710 --> 00:39:36.310 216-101 Large Conference: So, what you're saying is, 340 00:39:36.570 --> 00:39:42.700 216-101 Large Conference: If the second sentence does not have an agent, what would be the final output? 341 00:39:43.440 --> 00:39:45.739 216-101 Large Conference: Ken, maybe I'm… Yeah, for example. 342 00:39:49.530 --> 00:39:53.459 216-101 Large Conference: Doesn't your causal model suggest that it will be fetched first? 343 00:39:55.430 --> 00:39:56.450 216-101 Large Conference: It's not. 344 00:39:57.520 --> 00:40:07.140 216-101 Large Conference: Oh, no, yeah, you're right. Yeah. 345 00:40:08.410 --> 00:40:11.779 216-101 Large Conference: But, but, I mean, there would be the word attaches to that, right? 346 00:40:11.910 --> 00:40:18.110 216-101 Large Conference: I've not tried that, but the causal model that I have would say that fetch the first one or the second sentence. 347 00:40:18.510 --> 00:40:24.489 216-101 Large Conference: Because, you see this, this sort of diamond? This is sort of a condition. 348 00:40:24.830 --> 00:40:28.570 216-101 Large Conference: It's like, if a particular thematic rule is present. 349 00:40:28.910 --> 00:40:35.550 216-101 Large Conference: is encoded in the sentence that I'm interested in, then use the rule. 350 00:40:36.190 --> 00:40:42.270 216-101 Large Conference: But if that particular thematic rule is not present in a particular sentence type, then use the fetch first. 351 00:40:43.500 --> 00:40:52.780 216-101 Large Conference: So you would expect… yeah, so that's a good hypothesis. Yeah, I would expect it to be the first one, whatever is the first one. That's a good extra experiment. I'm not sure how would you create 352 00:40:53.600 --> 00:41:02.450 216-101 Large Conference: Maybe it is possible. I mean, there's also famous linguistic sentences that have no agent, like, it rains today. Agentless. We'll just output none. 353 00:41:03.150 --> 00:41:10.340 216-101 Large Conference: It won't output none. But yeah, I wouldn't… I'm not super sure. Kind of easy to get rid of the topic. 354 00:41:11.160 --> 00:41:15.489 216-101 Large Conference: Yeah, yeah, here you can get rid of the topic. Joey told… 355 00:41:15.760 --> 00:41:19.109 216-101 Large Conference: Joey was told about… Joey was told by Philip, that's it. 356 00:41:20.000 --> 00:41:31.949 216-101 Large Conference: And in that case, I would expect the answer… the causal model that I have would say that the final output should be Joey. Yeah, that is something that is estimate. In the clean prompt itself, would it just say not applicable? 357 00:41:32.150 --> 00:41:44.750 216-101 Large Conference: Like, if you say, what is the topic in the second sentence? With that example, would it just say there is no topic? I don't think so, the model will say… So give… It will… I would still assume that the… 358 00:41:45.710 --> 00:41:54.090 216-101 Large Conference: The token with the highest probability will still be one of the person names, and most likely, it won't say null or none, or something like that. 359 00:41:54.320 --> 00:41:58.680 216-101 Large Conference: Unless I explicitly ask it in the prompt, in the instruction, which I haven't. 360 00:42:01.630 --> 00:42:07.549 216-101 Large Conference: Okay. That was the intervention for role. This is an intervention for Feucht First. 361 00:42:10.440 --> 00:42:11.610 216-101 Large Conference: Don't want to try? 362 00:42:15.460 --> 00:42:16.540 216-101 Large Conference: Oh, sorry. 363 00:42:20.280 --> 00:42:23.810 216-101 Large Conference: That's true. 364 00:42:27.390 --> 00:42:29.799 216-101 Large Conference: We're voting for Joey. 365 00:42:33.470 --> 00:42:34.900 216-101 Large Conference: We're not sure. 366 00:42:37.070 --> 00:42:37.850 216-101 Large Conference: Oh. 367 00:42:38.020 --> 00:42:45.779 216-101 Large Conference: Yeah, it is working. Why are you not sure? I actually never understand patch words. 368 00:42:46.100 --> 00:42:55.050 216-101 Large Conference: Sorry, just the clarification question. So, fetch first just means fetching the first person? Oh, I see. Yeah, so you see this causeomal here? 369 00:42:55.150 --> 00:43:13.239 216-101 Large Conference: the model is sort of combining either the role or the fetch-first information with the sentence index to actually create sort of, like, query vectors for your attention head, which will actually fetch the value. I see. Okay, so, but I'm confused. So roll… so, 370 00:43:14.190 --> 00:43:21.509 216-101 Large Conference: so is… what's the output of Fetch First, or what is the value? Is Fetch First a Boolean? 371 00:43:21.780 --> 00:43:25.569 216-101 Large Conference: Or is that a personal name? Think of it as a Boolean. 372 00:43:25.850 --> 00:43:26.950 216-101 Large Conference: I see. 373 00:43:27.870 --> 00:43:28.800 216-101 Large Conference: I see. 374 00:43:29.570 --> 00:43:31.770 216-101 Large Conference: Like, it, like, it doesn't seem like… 375 00:43:32.100 --> 00:43:39.889 216-101 Large Conference: you have this other side thing, like form, which would have had to be the thing that you patched in order to switch it to Joe. 376 00:43:40.690 --> 00:43:47.610 216-101 Large Conference: So that's the sentence itself in the, like, the second sentence here. Joey was told 377 00:43:47.740 --> 00:43:51.669 216-101 Large Conference: about Karen by Philip. Right. That's sort of like the four. 378 00:43:52.530 --> 00:43:53.530 216-101 Large Conference: Right. 379 00:43:54.400 --> 00:43:55.830 216-101 Large Conference: Right. 380 00:43:58.270 --> 00:44:09.030 216-101 Large Conference: Yeah, I'm a little puzzled. At least to me, the thing that's confusing here is, like, if present, it feeds out role. So, like. 381 00:44:09.600 --> 00:44:14.340 216-101 Large Conference: Like… So the second… the clean prompt has topic as the role. 382 00:44:14.520 --> 00:44:21.379 216-101 Large Conference: But when you patch in Fech first, that, like, ignores or beats out the fact that we had topic. 383 00:44:21.950 --> 00:44:30.100 216-101 Large Conference: So… so Feucht First sort of is, like, another role, but it… but it trumps any existing roles. 384 00:44:30.500 --> 00:44:32.329 216-101 Large Conference: Which is, like, a little confusing. 385 00:44:32.530 --> 00:44:45.759 216-101 Large Conference: Is it the way you think about it? I thought that Fetch First is kind of like a, kind of like a failure mechanism. Like, first, if you can't find any role, then you trigger Fetch First. That's what I thought you meant. 386 00:44:45.870 --> 00:44:48.310 216-101 Large Conference: But, like, you can find a role, it's tough. 387 00:44:48.960 --> 00:44:57.100 216-101 Large Conference: Right? Like, the topic is already there. Oh, you're still talking about that thing, where the topic is gone? No. No, it says topic in it. 388 00:44:57.540 --> 00:45:05.039 216-101 Large Conference: So if there is, let's say, a specific subspace which is encoding FET first, and if I pass that from counterfactual to clean. 389 00:45:05.150 --> 00:45:21.000 216-101 Large Conference: that subspace will say that, okay, Fecht First is true, but there is some other subspace which is still encoding the fact that… The role is also has some value. Yeah, yeah. So my… So there must be, like, some gate or something that's like, if first is true, then ignore the role. 390 00:45:22.050 --> 00:45:30.729 216-101 Large Conference: How are you, like, how do you know that you are getting fetched first as opposed to, like, other heads that maybe are pointing towards another role? 391 00:45:34.630 --> 00:45:36.370 216-101 Large Conference: Because the output is Joey. 392 00:45:36.840 --> 00:45:40.310 216-101 Large Conference: Joey, in this case, is a recipient, right? 393 00:45:40.550 --> 00:45:43.960 216-101 Large Conference: Or… I mean… 394 00:45:44.110 --> 00:45:51.969 216-101 Large Conference: It could be… I mean, I could have other sentences where Joey is not recipient. It could be agent, it could be recipient, or it could be topic. It will still say the first one. 395 00:45:52.710 --> 00:45:56.139 216-101 Large Conference: Irrespective of what the thematic role of the first person is. 396 00:45:56.820 --> 00:45:59.780 216-101 Large Conference: But regarding that, that thing. 397 00:45:59.980 --> 00:46:03.089 216-101 Large Conference: I… in my head, this is… this is what I have. 398 00:46:04.190 --> 00:46:09.769 216-101 Large Conference: So, in… I think in all the cases, there is something… some form of competition going on. 399 00:46:09.940 --> 00:46:12.220 216-101 Large Conference: Competition between Roll and Fetch First. 400 00:46:12.890 --> 00:46:18.380 216-101 Large Conference: And for the cases where 401 00:46:18.740 --> 00:46:31.740 216-101 Large Conference: the thematic role is actually present in the percent token. Let's say here, topic is present in the current token, the role one wins, because the QK of the role is strong there. 402 00:46:33.420 --> 00:46:36.640 216-101 Large Conference: Make sense? So, so, so that's right. So the way I read this… 403 00:46:36.970 --> 00:46:39.350 216-101 Large Conference: So I think that… I don't think I… 404 00:46:39.870 --> 00:46:46.969 216-101 Large Conference: I think I understand the causal model you have in your head. Okay. What I didn't really understand is the diagram you drew. 405 00:46:47.220 --> 00:46:51.970 216-101 Large Conference: Right, and so I… I feel like the diagram you drew doesn't… 406 00:46:52.230 --> 00:46:58.219 216-101 Large Conference: doesn't directly diagram the causal model you're describing in words. Okay. To me. 407 00:46:58.550 --> 00:47:05.870 216-101 Large Conference: And so it, it, it feels like… You know, Rohl is a… 408 00:47:06.820 --> 00:47:08.979 216-101 Large Conference: I feel like there's two things. 409 00:47:09.190 --> 00:47:12.970 216-101 Large Conference: There's role, and then there's the person name who has the role. 410 00:47:13.880 --> 00:47:17.580 216-101 Large Conference: That you'll fetch after resolving the role. 411 00:47:18.000 --> 00:47:23.500 216-101 Large Conference: Right? And… Hmm. 412 00:47:23.970 --> 00:47:25.930 216-101 Large Conference: I see there's this Feucht First thing. 413 00:47:26.760 --> 00:47:33.399 216-101 Large Conference: Oh, let's see… A little confusing. How would you diagram this? 414 00:47:34.820 --> 00:47:43.640 216-101 Large Conference: So, I, like, I almost feel like Feucht First, since it's, like, controlling the if, like, it belongs in the side. Like, when you say there's a form. 415 00:47:44.120 --> 00:47:50.839 216-101 Large Conference: And the form… Basically, the determines whether you should fetch first? Yeah. 416 00:47:51.060 --> 00:47:58.970 216-101 Large Conference: And then, if you fetch first, then you ignore the roll, and then you… 417 00:47:59.460 --> 00:48:08.089 216-101 Large Conference: then you say fetch first instead, and then… or… or if you don't fetch first, then you use the roll, or something like that. Like, I feel like… 418 00:48:08.290 --> 00:48:11.309 216-101 Large Conference: Feucht first should be on the side instead of in the bottom. 419 00:48:11.860 --> 00:48:17.300 216-101 Large Conference: If that makes any sense, but I'm… I don't know. Really? I mean, I can't touch first. 420 00:48:18.060 --> 00:48:25.869 216-101 Large Conference: the value of Fetch first is, like, is decided based on the row. Like, if you can find the row, you just fetch it. 421 00:48:26.880 --> 00:48:36.190 216-101 Large Conference: But, let's say, you can't match with any of the roles. That's what I think. Yeah, so the fetch first, when it… 422 00:48:37.140 --> 00:48:43.789 216-101 Large Conference: Sorry, if you think of it as a Python function, the default value is not true. 423 00:48:44.320 --> 00:49:00.879 216-101 Large Conference: I don't think the default value… like, the Python argument, even a Python function, even takes the fetch first as an argument. Yeah, yeah, yeah. No, I see what you mean. Yeah. But this diagram still seems like it's a little bit off. I'm not sure, like, the right way to diagram what's going on. 424 00:49:01.020 --> 00:49:02.899 216-101 Large Conference: We could just write the Python code. 425 00:49:04.000 --> 00:49:19.770 216-101 Large Conference: Anyways, I think it's worth, you know, brainstorming, like, the right diagram here. But yeah, it's cool. But yeah, what he said is basically what I have in my head. 426 00:49:19.890 --> 00:49:26.099 216-101 Large Conference: But it's surprising to me, like, when you discuss it, right, that there is a materialized… 427 00:49:26.350 --> 00:49:33.530 216-101 Large Conference: variable that you can manipulate Feucht First, because Feucht First in this diagram feels like It's, 428 00:49:36.120 --> 00:49:49.569 216-101 Large Conference: it doesn't seem like the thing that would control whether you're taking the right or the left, right? It feels like the form is the thing that controls whether you're taking the right or the left, and so it's a little weird for you to manipulate one of the leaf variables. 429 00:49:49.790 --> 00:49:57.179 216-101 Large Conference: And then say that manipulating that thing causes the if to go a different way, right? Like, that's surprising for this diagram. 430 00:49:57.630 --> 00:49:58.890 216-101 Large Conference: And, 431 00:49:59.080 --> 00:50:04.820 216-101 Large Conference: Where do you have a right or a left? What right and left are you do not? The roll and force… 432 00:50:05.060 --> 00:50:05.990 216-101 Large Conference: First… 433 00:50:06.230 --> 00:50:20.689 216-101 Large Conference: Maybe it should also be on the left. Yeah. Right? But then maybe also roles should be on the left? I don't know, like, everything's ending up on the left, I'm not really sure. 434 00:50:21.330 --> 00:50:24.550 216-101 Large Conference: Then it can just be, like, the foundation. 435 00:50:24.880 --> 00:50:28.139 216-101 Large Conference: So, let's say this is my… And, 436 00:50:28.370 --> 00:50:32.090 216-101 Large Conference: condition, which only has one where, input, rolled. 437 00:50:33.190 --> 00:50:33.960 216-101 Large Conference: Right? 438 00:50:34.300 --> 00:50:39.230 216-101 Large Conference: And, if it is… Yes. 439 00:50:39.480 --> 00:50:47.730 216-101 Large Conference: I kind of like what Arnab was suggesting. It's almost like roll… Combines with sentence form. 440 00:50:48.340 --> 00:50:51.359 216-101 Large Conference: To output either roll or nothing. 441 00:50:52.060 --> 00:50:57.560 216-101 Large Conference: And then Feucht First combines with roll or nothing to take whatever's the maximum. 442 00:50:57.840 --> 00:51:03.540 216-101 Large Conference: of them. Either the role survives, this… this, filter. 443 00:51:03.690 --> 00:51:12.130 216-101 Large Conference: And it's a stronger thing, or if it didn't survive, then Feucht first is the thing, if that makes any sense. It feels like… it feels like role is combining with sentence form. 444 00:51:14.070 --> 00:51:16.139 216-101 Large Conference: This, this, does this make sense? 445 00:51:17.090 --> 00:51:22.649 216-101 Large Conference: No. No? 446 00:51:23.140 --> 00:51:24.919 216-101 Large Conference: But… but I… 447 00:51:26.030 --> 00:51:36.080 216-101 Large Conference: This one should make sense. What does form do? What does form do? What does form do? Form… are you able to intervene on form, for example? I can, but I haven't. 448 00:51:36.800 --> 00:51:45.520 216-101 Large Conference: Okay. So form is… is like the sentence form that you have here. So, let's say we are talking about the second sentence here. Yep. 449 00:51:46.100 --> 00:51:57.670 216-101 Large Conference: the structure of that sentence will determine which of the three thematic rows is actually encoded. And is it probably… and is that represent as the last token, or do you have to go back to the sentence period to manipulate that thing? 450 00:51:58.510 --> 00:52:04.819 216-101 Large Conference: No, it's at the sentence. It's at the percent tokens which are present in that particular sentence. 451 00:52:05.350 --> 00:52:10.410 216-101 Large Conference: Okay, so what I'm saying is, let's say here the agent value is… Topic. 452 00:52:10.930 --> 00:52:12.180 216-101 Large Conference: Yep, okay. 453 00:52:12.590 --> 00:52:24.739 216-101 Large Conference: then this particular if statement will look at the three persons in the second sentence. It will check if any of those three persons is saying I'm in topic or not. 454 00:52:24.920 --> 00:52:32.229 216-101 Large Conference: Right, and that's actually a function of the form. Yes. Right? That's why I have this form as the input. Okay, yeah. 455 00:52:32.390 --> 00:52:42.240 216-101 Large Conference: It will look for those 3 persons, it will look at those three persons and will check if any of those three persons says, I am a topic or not. 456 00:52:42.340 --> 00:52:46.289 216-101 Large Conference: Yep. If any of this 3 persons says that I'm a topic. 457 00:52:46.560 --> 00:52:49.000 216-101 Large Conference: Then, this top thing goes… this… 458 00:52:49.280 --> 00:53:00.200 216-101 Large Conference: then that role directly just goes to the next… Then it becomes… says topic. Yes. Find the topic of… Yes, topic, yeah. If none of those three persons says, I'm topic. 459 00:53:00.430 --> 00:53:11.859 216-101 Large Conference: then this Feuch first comes into the picture, which is like, okay, nobody's saying that they are the topic. Then, the first person… then just find the, like, the fetch first. Yeah. 460 00:53:11.980 --> 00:53:24.169 216-101 Large Conference: Does that make sense? I see, so it's really role and sentence form, which is combining and then turning into a variable which is either role or fetch first. 461 00:53:24.720 --> 00:53:32.290 216-101 Large Conference: Maybe that's a better… The sentence should… the sentence index shouldn't connect to a form somehow? Yeah, yeah, you can connect sentence index with the form. 462 00:53:32.420 --> 00:53:51.279 216-101 Large Conference: Or that this particular condition. Oh, yeah. Another art. So, what if multiple, names have the same topic in the same sentence? What if there are multiple names that have the same topic in the same sentence? 463 00:53:51.800 --> 00:53:59.749 216-101 Large Conference: Like, something that breaks this, diagram. I'm not sure if we could create such… 464 00:54:01.640 --> 00:54:06.360 216-101 Large Conference: I can select one by Karen and Philip. 465 00:54:06.500 --> 00:54:10.520 216-101 Large Conference: Sorry. Was told about by… 466 00:54:11.210 --> 00:54:22.569 216-101 Large Conference: Yeah, so in that case, the clause would be different. The sentence index that I'm defining is sort of in a single clause. When you include an AND, I think that becomes sort of a different clause. 467 00:54:22.870 --> 00:54:27.499 216-101 Large Conference: So in that case, the sentence index will be different, but the thematic rule will still remain the same. 468 00:54:27.930 --> 00:54:28.720 216-101 Large Conference: Okay. 469 00:54:29.640 --> 00:54:36.689 216-101 Large Conference: Third question? In the prompt, original prompt, you did say what the three rows are, right? 470 00:54:37.400 --> 00:54:45.569 216-101 Large Conference: probably didn't get to that yet, but I'm curious if you remove that, if you would find the same causal… Yeah, that's the next, next setup. But, yeah. 471 00:54:45.690 --> 00:54:50.120 216-101 Large Conference: Because right now, it knows that it has only three options, right? You tell it. 472 00:54:50.280 --> 00:54:57.629 216-101 Large Conference: Yes. Yeah, so I think in the next setup, I don't really talk about thematic groups at all, and… 473 00:54:57.840 --> 00:55:00.779 216-101 Large Conference: The results there are pretty much the same. 474 00:55:02.000 --> 00:55:07.880 216-101 Large Conference: Yeah, but yeah, your current further graph, you're saying 475 00:55:08.560 --> 00:55:11.539 216-101 Large Conference: Bill and Agent are not submitted models. 476 00:55:11.890 --> 00:55:14.789 216-101 Large Conference: can speak together, it's just, like, rule of elimination. 477 00:55:15.370 --> 00:55:20.490 216-101 Large Conference: Go ahead. Bill and AJ… Thank you. 478 00:55:20.640 --> 00:55:25.199 216-101 Large Conference: The causal graph is suggesting that the model doesn't know that relate the agent. 479 00:55:25.470 --> 00:55:28.730 216-101 Large Conference: Even, like, at the last token, like, at the answer colon. 480 00:55:30.900 --> 00:55:40.519 216-101 Large Conference: Like, there's no theme for that very token, so it's like, sure, I'll just fetch the first token. Yeah, something like, yeah. 481 00:55:40.770 --> 00:55:47.620 216-101 Large Conference: Well, like, isn't there… Taking autoregressive models should somehow figure out by 482 00:55:47.840 --> 00:55:52.810 216-101 Large Conference: Like, it was Will who informed. By the token informed, it should understand that Will is the agent. 483 00:55:54.290 --> 00:55:57.859 216-101 Large Conference: But the thing is, the model is fetching 484 00:55:58.010 --> 00:56:04.970 216-101 Large Conference: So, at the last token… sorry, yeah, at the last token, the later layers are sort of fetching the will information directly from the will token. 485 00:56:05.420 --> 00:56:10.460 216-101 Large Conference: It's not fetching the information from, let's say, Informed or the PA2. 486 00:56:10.840 --> 00:56:14.800 216-101 Large Conference: So that means those set of heads which are actually fetching in the will information. 487 00:56:15.010 --> 00:56:21.310 216-101 Large Conference: So somehow know that… no… I mean, somebody should tell it that, okay, you need to fetch theken, right? 488 00:56:21.480 --> 00:56:27.430 216-101 Large Conference: Yeah. And that piece of information is coming from something like the elimination 489 00:56:27.530 --> 00:56:37.630 216-101 Large Conference: sort of, like, analogy, like, if you don't find, in the previous set of computation, if you don't find any particular person which says, I'm an agent, then fetch the first one. 490 00:56:38.850 --> 00:56:49.489 216-101 Large Conference: So, do you think if I just change the setup to it was William for Henry and David? No, I can't do that. And then I just add their prenames? 491 00:56:49.610 --> 00:56:57.449 216-101 Large Conference: Okay, in this next step, there's will enable it. By the time, the model should know that bill is Agent… 492 00:56:57.960 --> 00:57:09.140 216-101 Large Conference: So you're saying you have the same sentence, and then you start a new sentence where you have, well… Yeah, all the three names. All the prenames. The second, if you look… if you patch a, 493 00:57:09.490 --> 00:57:17.380 216-101 Large Conference: Most probably, if you do the probing of the second, occurrence of any of those three names, the… that particular residual vector will have its corresponding time. 494 00:57:18.900 --> 00:57:21.379 216-101 Large Conference: Yeah, actually, I've done that. It does show up. 495 00:57:22.980 --> 00:57:31.210 216-101 Large Conference: Yeah, I had the same question as him. Like, I was also… I would have imagined if the verb would at some verb, like, informed would at some point encode the agent, but you're saying, like, if you probe the verb. 496 00:57:31.490 --> 00:57:38.019 216-101 Large Conference: No, Work will have it. I'm not saying the Wrb… I mean, if you do the probing on, I think, any token after… 497 00:57:38.200 --> 00:57:48.979 216-101 Large Conference: the work, it will, I think, figure out which of the three persons is the agent, recipient of the topic. Okay. But the thing is, those won't be causal in nature. 498 00:57:49.340 --> 00:57:57.729 216-101 Large Conference: Because in the causal path, the model is directly fetching in the values from that particular percent drop. So it never attends the verb, basically, never pays attention to the verb. 499 00:57:58.230 --> 00:58:02.540 216-101 Large Conference: Or, like, in order to figure out the… Edmund. 500 00:58:07.360 --> 00:58:20.419 216-101 Large Conference: Because at least in linguistics, there's also so many theories that the verb is usually the one that kind of, like, is the central one that connects everything, but I guess here, the model… Okay, I don't think so. I would say it doesn't… it never attends to, 501 00:58:21.410 --> 00:58:29.740 216-101 Large Conference: Yeah, Bob. Yeah, so in this project, I've not really looked too much into the circuit stuff. I was doing a little bit of that. 502 00:58:30.210 --> 00:58:42.480 216-101 Large Conference: Though this is something that I'm not super sure about. So my hypothesis is, if the early layers, the model sort of figures out the role in the sentence, and then sort of in the middle layers, it starts looking for that particular role. 503 00:58:43.010 --> 00:58:45.420 216-101 Large Conference: In the essence of that particular sentence. 504 00:58:45.870 --> 00:59:00.389 216-101 Large Conference: If it finds it, then it's good, it goes up, but if it doesn't find it, I think there are specific heads which attend to the words, which actually sort of, I think, writes this information, or Feucht First Boolean information at the last token. Okay. 505 00:59:00.580 --> 00:59:13.260 216-101 Large Conference: But I'm not super confident about that yet. Is there a fetch first that is, like, literally fetch first from, like, the first sentence itself, like a failure case, like an all-capped failure case, where… 506 00:59:13.380 --> 00:59:16.469 216-101 Large Conference: Maybe it doesn't even know… I've not tried that. 507 00:59:16.650 --> 00:59:17.710 216-101 Large Conference: It might be. 508 00:59:18.840 --> 00:59:24.360 216-101 Large Conference: Could you quickly explain what, Feucht First in this example is, and why the answer is Jay? 509 00:59:25.530 --> 00:59:34.000 216-101 Large Conference: In this one? Yeah, so here, if you look at this new model, new causal model. 510 00:59:34.220 --> 00:59:37.559 216-101 Large Conference: In this particular clean prompt, the role is topic. 511 00:59:37.780 --> 00:59:39.810 216-101 Large Conference: Syntax is the second one. 512 00:59:40.160 --> 00:59:53.490 216-101 Large Conference: So both of this piece of information goes to this condition, in addition to this form information. So this condition basically checks if, any of these three persons 513 00:59:53.910 --> 00:59:59.230 216-101 Large Conference: which is, Joey, Karen, or Topic, if any of them says they are Topic or not. 514 01:00:00.630 --> 01:00:08.719 216-101 Large Conference: and, in this example, in the… without the intervention, there is one person. 515 01:00:09.100 --> 01:00:12.359 216-101 Large Conference: Karen, who says I'm a… I'm an agent, sorry, I'm a topic. 516 01:00:12.740 --> 01:00:16.020 216-101 Large Conference: So, that topic information directly goes here. 517 01:00:16.600 --> 01:00:17.420 216-101 Large Conference: Okay. 518 01:00:23.740 --> 01:00:30.120 216-101 Large Conference: So that's the normal causal model. But here, what I'm doing is, I am explicitly patching this 519 01:00:31.460 --> 01:00:36.830 216-101 Large Conference: Or in the counterfactual, this particular path will be active instead of the top one. 520 01:00:37.100 --> 01:00:39.840 216-101 Large Conference: Okay. Does that make sense? Okay. 521 01:00:39.950 --> 01:00:48.900 216-101 Large Conference: So, what I'm doing is, I'm basically patching in this fetch first from the top one, where that particular path is active. 522 01:00:49.150 --> 01:00:53.950 216-101 Large Conference: To explicitly make this particular path active in the clean one. 523 01:00:55.410 --> 01:00:58.689 216-101 Large Conference: Okay? And if you make this one active. 524 01:00:59.690 --> 01:01:04.819 216-101 Large Conference: The moral is sort of, so, in that case, both 525 01:01:05.020 --> 01:01:09.649 216-101 Large Conference: The topic and the Feet first will be active, because the topic was active by default. 526 01:01:09.840 --> 01:01:12.410 216-101 Large Conference: And we are explicitly making the Feucht 527 01:01:12.640 --> 01:01:14.840 216-101 Large Conference: To be active after the intervention. 528 01:01:15.130 --> 01:01:24.070 216-101 Large Conference: But in that case, if both of them are active, then the model is giving preference to the fetch person. And that's why, after the intervention, the model 529 01:01:25.560 --> 01:01:31.570 216-101 Large Conference: Attends to and feds the… the first person of the second sentence. 530 01:01:33.570 --> 01:01:42.460 216-101 Large Conference: Yeah, that was the whole story. So, patch first is always the first token of the sentence. Yes, first person of the… first person of the… okay, yeah. 531 01:01:43.430 --> 01:01:47.330 216-101 Large Conference: So, you got output as a Jew. Yeah. 532 01:01:47.640 --> 01:01:52.320 216-101 Large Conference: Okay, and what if there is a sentence which does not have a rule? So it's… 533 01:01:52.610 --> 01:02:01.480 216-101 Large Conference: like, by default, that Feucht First is the, like, true Boolean there, but what if you ablate fetch first out of that sentence? 534 01:02:02.570 --> 01:02:04.629 216-101 Large Conference: Like, make both of them false. 535 01:02:05.230 --> 01:02:06.920 216-101 Large Conference: role in Feuchverse. 536 01:02:10.040 --> 01:02:11.950 216-101 Large Conference: Reducing, 537 01:02:13.550 --> 01:02:20.880 216-101 Large Conference: In the counterfactual, where the Feucht first is actually active, if I somehow ablate out the fetch first, what would happen? Yeah. 538 01:02:22.060 --> 01:02:29.550 216-101 Large Conference: Yeah, I'm not sure what will happen. I think the model might just be confused. 539 01:02:30.060 --> 01:02:36.909 216-101 Large Conference: Probably generate fetch first again, something like that. Or it might just give equal probability to all three. 540 01:02:37.750 --> 01:02:48.270 216-101 Large Conference: Do you want to actually move on to the, yeah, actual meaty stuff, the interventions? Yeah, but yeah, that one I'm not sure. Yeah, we're already over time. 541 01:02:48.700 --> 01:02:49.870 216-101 Large Conference: Okay, Chris. 542 01:02:50.210 --> 01:02:52.090 216-101 Large Conference: How do I quote it? 543 01:02:58.400 --> 01:03:02.999 216-101 Large Conference: How many verses? It's okay, the next ones, I can go pretty quickly. 544 01:03:04.150 --> 01:03:06.250 216-101 Large Conference: But how do I clear this thing? 545 01:03:07.070 --> 01:03:18.109 216-101 Large Conference: Okay, so these are the interventions for each of the variables. Oh, there's one more important thing. 546 01:03:18.790 --> 01:03:21.699 216-101 Large Conference: So, I had it… okay, there's one more thing. 547 01:03:22.660 --> 01:03:28.700 216-101 Large Conference: So, previously I showed the intervention for each of the, like, individual variables. 548 01:03:28.900 --> 01:03:47.869 216-101 Large Conference: But, you can also have intervention for this compound variable, which has both role and sentence index. So if you do the batching for that one, for that particular variable, the output would change from, in this particular example, topic of the second sentence, to the agent of the first sentence, which is, grace. 549 01:03:48.230 --> 01:03:48.930 216-101 Large Conference: Okay. 550 01:03:51.150 --> 01:03:55.519 216-101 Large Conference: Okay, so then what I did was I basically used this intervention to train DAS. 551 01:03:55.900 --> 01:04:06.020 216-101 Large Conference: And what I found was I could train the DAS for each of the three independent variables, which is role, sentence, index, and fetch first. 552 01:04:06.400 --> 01:04:10.690 216-101 Large Conference: But when I combined those trained subspaces. 553 01:04:11.070 --> 01:04:14.530 216-101 Large Conference: to do the compound intervention, that did not work. 554 01:04:15.400 --> 01:04:16.949 216-101 Large Conference: So what does that mean is. 555 01:04:17.100 --> 01:04:25.480 216-101 Large Conference: I trained the role subspace using a DAS. I trained the sentence index subspace with the DAS, both of… 556 01:04:25.600 --> 01:04:27.890 216-101 Large Conference: For both of them have individual interventions. 557 01:04:28.570 --> 01:04:34.089 216-101 Large Conference: And then I sort of combined both the subspaces, so I just concatenated and did the… 558 01:04:34.290 --> 01:04:38.640 216-101 Large Conference: And just orthonormalize the combined, subspace. 559 01:04:38.950 --> 01:04:44.559 216-101 Large Conference: And now I have this new subspace, the combined one, and I use that on the compound intervention. 560 01:04:45.440 --> 01:04:50.550 216-101 Large Conference: And, the red line is… its corresponding IRA. 561 01:04:51.310 --> 01:04:56.110 216-101 Large Conference: And that does not work. That way. Sorry, you get the compound intervention. 562 01:04:56.360 --> 01:05:06.339 216-101 Large Conference: like, by training a task from scratch. No. You take the two things, and then you also make sure that before you combine it. 563 01:05:06.500 --> 01:05:13.729 216-101 Large Conference: these two things, role and sent and index subspaces, they are autonomorable with each other as well? No, no, no. 564 01:05:14.050 --> 01:05:29.080 216-101 Large Conference: No, you didn't. That's not the objective test. First, you found the two subspaces separately. Yes. They're not orthogonal. Yeah, they are not orthogonal. So then, if you just add them, it doesn't make sense. Now, so what I did was, at these two subspaces, I concatenated it. 565 01:05:29.720 --> 01:05:35.739 216-101 Large Conference: So if you… right, exactly. So you have a A-dimensional subspace, you have a B-dimensional subspace. 566 01:05:35.970 --> 01:05:42.500 216-101 Large Conference: So now you take the… Join subspace, and then now it's A plus B dimensional. That is not orthonormal. 567 01:05:42.650 --> 01:05:44.980 216-101 Large Conference: Right, but then he just… I auto-normalize it. 568 01:05:45.370 --> 01:05:48.200 216-101 Large Conference: QR? Just to make a basis, yeah, QR, yeah. Just Q. 569 01:05:48.820 --> 01:05:55.329 216-101 Large Conference: So did you concatenate, or I added in line control? I concatenated, and then… then you just apply QR on that. 570 01:05:55.630 --> 01:05:58.430 216-101 Large Conference: Just to make sure that the columns are authentic. 571 01:05:58.600 --> 01:06:00.890 216-101 Large Conference: Just, like, apply Gram Smith on it. 572 01:06:01.990 --> 01:06:08.189 216-101 Large Conference: What? Benjamin, here's what that, what's that mean? 573 01:06:08.960 --> 01:06:19.940 216-101 Large Conference: So… And the idea… Huh? What's IIA? Intervention Accuracy. Okay. Interchange intervention accuracy. Now, if the subspaces have some… 574 01:06:20.490 --> 01:06:24.389 216-101 Large Conference: You know, geometry where they're like this, very close to each other. 575 01:06:25.090 --> 01:06:31.379 216-101 Large Conference: Right, then those… that… that dimension where they're just slightly different in that dimension will come up 576 01:06:31.660 --> 01:06:33.110 216-101 Large Conference: in your subspace. 577 01:06:33.600 --> 01:06:41.250 216-101 Large Conference: But it's really just noise. Like, you could be, like, A could be a subspace, B could be a subspace. 578 01:06:41.540 --> 01:06:46.019 216-101 Large Conference: But the… but, like, the real subspace 579 01:06:46.210 --> 01:07:00.459 216-101 Large Conference: that joins them could be in a lower dimension than A and B, because there could be some noise. I'm not trying to find the intersection, I'm trying to find the union. No, I know, but, like, the real union space. So, like, the intersect… like, the assumption would be that in the generic case. 580 01:07:00.810 --> 01:07:04.550 216-101 Large Conference: the intersection subspace is the null subspace, right? There's… 581 01:07:04.840 --> 01:07:09.490 216-101 Large Conference: Because you've got two generic subspaces you find empirically, they're not going to intersect. 582 01:07:09.710 --> 01:07:15.780 216-101 Large Conference: Yeah, but I think generally they are not, but let's assume that. Right, but then… but then… but they could actually… 583 01:07:16.910 --> 01:07:22.130 216-101 Large Conference: be very close to intersecting in subspace, so you could find, like, an approximate 584 01:07:22.250 --> 01:07:29.479 216-101 Large Conference: intersection subspace, and then subtract that from your union. So I guess that's… that's a question, like, is this… is this failing because 585 01:07:29.770 --> 01:07:31.549 216-101 Large Conference: The joint subspace is too vague. 586 01:07:31.670 --> 01:07:45.490 216-101 Large Conference: He has a solution for it. Oh, you have a solution? Yeah. Yeah. Not just, not just jointly optimized. We already said it, oh, there's joint optimization, yeah. Oh, you did the joint optimization, but it's just… so does the joint optimization tell you that this subspace was too big? 587 01:07:50.520 --> 01:07:53.270 216-101 Large Conference: I'm still not sure about the join… the… 588 01:07:53.570 --> 01:08:06.379 216-101 Large Conference: Like, is this… is the joint optimization first space that you're going to find on the next slide? So you go to the next slide, you can show the joint optimization. So what I did was, instead of training one DAS, 589 01:08:06.540 --> 01:08:12.640 216-101 Large Conference: I… sorry, one DAS for each variable. I trained a single shared one for all of them. 590 01:08:14.250 --> 01:08:21.749 216-101 Large Conference: I have this, matrix, where I have, like, let's say first R columns for one particular variable, next. 591 01:08:21.750 --> 01:08:35.299 216-101 Large Conference: N minus R column for the another variable. I use the first R columns for doing the intervention for one variable, next N minus R column for doing the intervention for the next variable, and I use all of the columns to do the intervention for… 592 01:08:35.380 --> 01:08:37.160 216-101 Large Conference: The combined variable. 593 01:08:38.600 --> 01:08:40.390 216-101 Large Conference: That's the objective here. 594 01:08:40.439 --> 01:08:58.780 216-101 Large Conference: And then for… yeah, for each of the interventions, we will find one particular loss, and I do combine all the losses, and do one gradient descent update on that combined subspace. Is that the boundless task? How do you define R? Yeah, no, I just do… that's test hyperparametric, yeah. 595 01:09:00.290 --> 01:09:08.639 216-101 Large Conference: Absolutely. In the previous slide, why should union, why should the role and sentence ID be orthogonal? 596 01:09:10.310 --> 01:09:11.180 216-101 Large Conference: No, we… 597 01:09:11.819 --> 01:09:23.840 216-101 Large Conference: It might be, it might not be. So the general idea is, since these two variables are sort of completely different, they're encoding different semantic information, and since the residual vector is super large, we'd, sort of expect that 598 01:09:23.979 --> 01:09:27.640 216-101 Large Conference: This, this subspaces might be, almost near Otago. 599 01:09:28.090 --> 01:09:30.389 216-101 Large Conference: So on your joint optimization here. 600 01:09:30.600 --> 01:09:39.149 216-101 Large Conference: Are you still doing just role and index, or are you doing role and index and combined as three objectives? 601 01:09:39.460 --> 01:09:43.689 216-101 Large Conference: Oh, I see. Does it work if you just do roll and index without the third one? 602 01:09:44.420 --> 01:09:45.379 216-101 Large Conference: It doesn't. 603 01:09:45.510 --> 01:09:50.279 216-101 Large Conference: Because, like, here you're enforcing orthogonality. 604 01:09:50.870 --> 01:09:55.909 216-101 Large Conference: So if you… so it's still different if you just do roll and index jointly. 605 01:09:56.330 --> 01:10:00.750 216-101 Large Conference: But you tried that, and it doesn't work. Yeah, it doesn't work. You need to have the joint training. 606 01:10:01.640 --> 01:10:08.480 216-101 Large Conference: combined… compounding. And my intuition is something like this. I think the subspaces is not a unique thing. 607 01:10:08.600 --> 01:10:20.169 216-101 Large Conference: There is a family of subspaces which would satisfy, let's say, variable A intervention. There is a family of subspaces which would satisfy the variable B intervention. 608 01:10:20.620 --> 01:10:23.259 216-101 Large Conference: So when I'm only optimizing for one of them. 609 01:10:24.250 --> 01:10:34.879 216-101 Large Conference: I'm finding one subspace from that family for each of the variables, but their compound… there's no reason for the… their joint or the concatenated one being… being a good one. 610 01:10:35.060 --> 01:10:42.440 216-101 Large Conference: But when I'm explicitly adding this particular training loss in the task, I'm finding, 611 01:10:43.090 --> 01:10:52.470 216-101 Large Conference: a subspace in both those families, which are sort of at the intersection. And that's why that particular one works, and the previous one's just not. I think that's my high-level intuition. 612 01:10:52.950 --> 01:10:54.019 216-101 Large Conference: Does that make sense? 613 01:10:54.290 --> 01:10:55.050 216-101 Large Conference: Nope. 614 01:10:55.850 --> 01:10:57.970 216-101 Large Conference: I mean, it makes sense to see your intuition. 615 01:10:58.270 --> 01:11:00.120 216-101 Large Conference: Oh, okay. But it's very unintuitive. 616 01:11:02.870 --> 01:11:14.970 216-101 Large Conference: But it's okay, I mean, it's great. I mean, if that's what the experiment is showing, then that's what it shows. Yeah, that is what it's showing. Yes, it's interesting. 617 01:11:15.320 --> 01:11:16.000 216-101 Large Conference: -Oh. 618 01:11:16.310 --> 01:11:17.080 216-101 Large Conference: Okay. 619 01:11:18.220 --> 01:11:23.800 216-101 Large Conference: Did you try training multiple subspaces in each category with different initial scenes? 620 01:11:24.300 --> 01:11:28.489 216-101 Large Conference: And did they end up having, sort of, different properties in this way, or did they all end up 621 01:11:28.870 --> 01:11:34.580 216-101 Large Conference: Like, every time you did a role swap-only DAS optimization, did it not end up generalizing? 622 01:11:36.050 --> 01:11:39.950 216-101 Large Conference: When you say properties, which properties are you? Like, flip back. 623 01:11:40.600 --> 01:11:48.040 216-101 Large Conference: This one, or this one? No, the one with the stripes. The one with the stripes? Yes, this one. So, if you… 624 01:11:48.660 --> 01:12:01.679 216-101 Large Conference: train on just objectives 1 and 2, but not 3, and try that on 100 different seeds. Do all 100 seeds, not generalize? I did not try 100, but I tried, like, 3 or 4. Sure. That did not work. 625 01:12:02.250 --> 01:12:03.450 216-101 Large Conference: Okay, yeah. 626 01:12:05.270 --> 01:12:11.320 216-101 Large Conference: then… Let's generalize it… let's generalize your 3 or 4 to 100 real quick. 627 01:12:12.230 --> 01:12:18.359 216-101 Large Conference: If we were to do this 100 times and none of the times it worked, then flip to the next slide. 628 01:12:18.720 --> 01:12:22.549 216-101 Large Conference: Wouldn't that mean that middle section is either really small or non-existent? 629 01:12:22.700 --> 01:12:28.499 216-101 Large Conference: Yeah, it could be very small. Yeah, I'm not really saying about the size of the family here. I'm just saying there is a family. 630 01:12:29.190 --> 01:12:31.400 216-101 Large Conference: I see. Okay, that makes sense. 631 01:12:34.330 --> 01:12:35.920 216-101 Large Conference: But it's still not intuitive. 632 01:12:37.000 --> 01:12:38.940 216-101 Large Conference: Can you say your intuition again? 633 01:12:39.470 --> 01:12:42.370 216-101 Large Conference: So… this orange circle. 634 01:12:42.700 --> 01:12:49.369 216-101 Large Conference: think of it as a family of subspaces which will satisfy the role intervention, okay? 635 01:12:49.680 --> 01:12:58.489 216-101 Large Conference: this… Blue one is representing the family of interventions, which will satisfy the sentence index intervention. 636 01:12:59.780 --> 01:13:07.590 216-101 Large Conference: So let's say if I only do their separate independent DAS training, so for the sentence index, I find this particular 637 01:13:08.200 --> 01:13:09.190 216-101 Large Conference: subscript? 638 01:13:09.470 --> 01:13:13.930 216-101 Large Conference: And for, role I find this particular service. 639 01:13:14.050 --> 01:13:20.329 216-101 Large Conference: Right? Then when I try to concatenate it, and then try to apply the compound intervention. 640 01:13:20.560 --> 01:13:25.100 216-101 Large Conference: There is no reason for it to… be in a… 641 01:13:25.910 --> 01:13:31.630 216-101 Large Conference: There's no reason for both the subspaces to be aligned in a way that both of them just will work for the compound variable. 642 01:13:32.360 --> 01:13:35.230 216-101 Large Conference: Because they were not really trained. 643 01:13:35.680 --> 01:13:42.280 216-101 Large Conference: Okay? But when I explicitly enforce that training, like, enforce that laws, that 644 01:13:42.500 --> 01:13:47.730 216-101 Large Conference: You don't let only have… Just to… to modify the compound? Yeah. Okay. 645 01:13:47.900 --> 01:13:50.440 216-101 Large Conference: when I explicitly ask it to find 646 01:13:50.830 --> 01:14:00.109 216-101 Large Conference: a subspace in this particular family and this particular family, which should work for the compound one. Then, it will find me one which 647 01:14:00.390 --> 01:14:02.580 216-101 Large Conference: Yeah, which basically does work. 648 01:14:02.940 --> 01:14:13.699 216-101 Large Conference: What is this one? The intersection of these two? The intersection of these two is sort of, like, representing the fact that I could use the subspaces 649 01:14:14.010 --> 01:14:19.000 216-101 Large Conference: In that intersection family to do the compound intersection, with compound intervention. 650 01:14:19.590 --> 01:14:23.440 216-101 Large Conference: So these are cases that did work for combining, or… 651 01:14:24.210 --> 01:14:33.009 216-101 Large Conference: This is… What does the intersection of the two circles represent? Certain family, certain… or some number of subspaces. 652 01:14:33.330 --> 01:14:35.530 216-101 Large Conference: That were found by the joint. 653 01:14:36.490 --> 01:14:38.490 216-101 Large Conference: Optimization? 654 01:14:38.730 --> 01:14:47.390 216-101 Large Conference: Yeah. To satisfy. Maybe even if they weren't found, you're, like, saying, these ones satisfy the role. Exactly. The intersection is where both of them work together. 655 01:14:47.750 --> 01:14:52.070 216-101 Large Conference: So the joint one… Did both of them work, or is this another subspace that works? 656 01:14:53.340 --> 01:15:09.080 216-101 Large Conference: Is this the two subspaces combined that work together, or is this a new subspace that works? It's a new subspace, I believe. Every point is a new subspace. Each point is… each point is a choice of subscrees, subspaces, right? 657 01:15:09.250 --> 01:15:12.120 216-101 Large Conference: Is… each point is, 658 01:15:12.630 --> 01:15:23.130 216-101 Large Conference: Yeah, maybe this is not a very good diagram. First, I thought it's, like, intervention, like, intersection is where they both conflict each other, so they cannot be combined. 659 01:15:23.450 --> 01:15:28.960 216-101 Large Conference: That's what I was thinking earlier. That's exactly a positive. Maybe, like, a better… 660 01:15:29.060 --> 01:15:42.980 216-101 Large Conference: So your intuition is that there is a subspace for X, subspace for Y, and another subspace for X and Y? No. No, no, no. There's many, many different possible ways of solving it. So you're arguing 661 01:15:43.210 --> 01:15:52.710 216-101 Large Conference: Or, like, non-determinism. So you're saying… you're saying, hey, you know, that, like, Atticus has been going around saying, you find the subspace for my causal variable. 662 01:15:52.940 --> 01:16:04.010 216-101 Large Conference: And you're like, oh, that's… that's an ill-posed question. There are many different subspaces that all can achieve 100% on your causal mediation metric. 663 01:16:04.140 --> 01:16:12.140 216-101 Large Conference: And a lot of the subspaces are very different from each other. And so, you know, just because your optimizer goes to 100% doesn't mean you found the unique 664 01:16:12.520 --> 01:16:21.080 216-101 Large Conference: Subspace. There's many places that it could have ended up. And if you add more constraints to it, then you might discover one of the other subspaces. 665 01:16:21.250 --> 01:16:26.740 216-101 Large Conference: it could also get to 100%. Is this what you found? Like, did different seeds produce 666 01:16:26.920 --> 01:16:29.710 216-101 Large Conference: Perfectly optimized to different stuff pieces. 667 01:16:29.810 --> 01:16:44.930 216-101 Large Conference: That was intuitive. Compare the similarity between different seed, who's four different seed? Like, I like… the similarity would be, like, 0.6 is the old story. The old story was intuitive. Is there anything meaningful? Right. 668 01:16:44.930 --> 01:16:52.459 216-101 Large Conference: I could teach it. I don't think it's intuitive. Always comment subspace or whatever. For something like that. 669 01:16:52.460 --> 01:17:03.850 216-101 Large Conference: Not super sure about that, rather. Intuitive. I'm basically making it clear that, like, this, if you, 670 01:17:04.350 --> 01:17:20.310 216-101 Large Conference: There's several papers that show that, like, mechanisms don't have to be unique. 671 01:17:20.310 --> 01:17:30.139 216-101 Large Conference: I was gonna say, I thought this is a common thing… Like, this is dissatisfying in the same way that the backup heads are, like, you know, dissatisfying. Who likes backup heads? 672 01:17:30.140 --> 01:17:45.390 216-101 Large Conference: Do I… any fans of backup yet? Yeah, so, like, backup… They make the model more robust. Okay. And so, do you think this is just a robustness mechanism? No, no, no, this is unsatisfying for you. 673 01:17:45.710 --> 01:17:48.260 216-101 Large Conference: This is related to backup. 674 01:17:48.540 --> 01:17:52.420 216-101 Large Conference: What? I don't think that this is related to backup. You don't think that this is backup? This is just… 675 01:17:52.560 --> 01:18:04.409 216-101 Large Conference: this geometry is, like, space is super large, and there is not a unique subspace which is satisfying the condition that you care about. This is, like, big family of… there is this, like, family of… 676 01:18:04.530 --> 01:18:08.050 216-101 Large Conference: satisfy the condition they look. That's it. 677 01:18:08.620 --> 01:18:25.660 216-101 Large Conference: So, in the, the ones that, that encomp… like, the role, role and, sent, sentence ID, you can just, like, test, like, put all the families, like, all the subspaces and check which are all programar. You'll get the union. 678 01:18:26.210 --> 01:18:28.730 216-101 Large Conference: Like, it could be… 679 01:18:29.230 --> 01:18:39.119 216-101 Large Conference: If you are able to organize one of the independent rule subspace with another sentence ID, It means, like. 680 01:18:39.530 --> 01:18:41.320 216-101 Large Conference: It's in the union. 681 01:18:42.550 --> 01:18:50.879 216-101 Large Conference: No, that doesn't mean that… But you can just, orthogonalize two subspaces, right? And create a… You already tried that. 682 01:18:51.280 --> 01:19:03.600 216-101 Large Conference: So what you're saying is I can find one roll subspace, and then I find a specific sentence index subspace which is orthogonal to the roll one? Yeah. Yeah, I can do that. Yeah, yeah, but that does not work. That does not work the combined one. 683 01:19:04.600 --> 01:19:08.149 216-101 Large Conference: But it might not work for one subspace, but we don't know, right? 684 01:19:08.480 --> 01:19:11.740 216-101 Large Conference: What do you mean? Like, there are multiple subspaces, no? 685 01:19:12.080 --> 01:19:21.209 216-101 Large Conference: Well, he found one that works in the end by adding a third constraint, and then he, of course, he can find it, you can make it work in the end by adding a third constraint, but without the third constraint, then… Yeah. 686 01:19:21.490 --> 01:19:24.210 216-101 Large Conference: Exactly. And there's many that don't work. 687 01:19:24.720 --> 01:19:29.319 216-101 Large Conference: Are these all on the same layer, by the way? Did you search… All in December. 688 01:19:29.570 --> 01:19:31.669 216-101 Large Conference: One second, huh? 689 01:19:33.180 --> 01:19:37.150 216-101 Large Conference: This is across layers. This is basically the results. 690 01:19:38.000 --> 01:19:50.199 216-101 Large Conference: I can… I can do it across layers. These pieces of information, this, like, these subspaces of information is present across layers at the last two. So if you enter in one layer, it just works? It works. 691 01:19:50.570 --> 01:19:55.040 216-101 Large Conference: One Layer 1 token. I mean, if, yeah, if, yeah, if, I mean… 692 01:19:55.610 --> 01:20:01.939 216-101 Large Conference: So think of this, like, information is present at start… start to be present and causally relevant at layer 20. 693 01:20:02.050 --> 01:20:10.860 216-101 Large Conference: And it is being relevant up to layer 13. If you intervene on any… anything in between, everything after that will be your intervene information, right? 694 01:20:12.000 --> 01:20:30.660 216-101 Large Conference: So it leads to this question, well, so now you have this third constraint that you've, you know, you've got this extra constraint, which means that, you know, the work that you did in the previous work to say, this is where the index is, this is where the pointer index is, is actually maybe not close to the truth, because 695 01:20:30.830 --> 01:20:37.299 216-101 Large Conference: You didn't have this role interaction that you should have used as a supplementary constraint to clean up 696 01:20:37.480 --> 01:20:50.369 216-101 Large Conference: your understanding of where the index subspace was. But that now leaves open the question, oh, other than role, then there may be, you know, 10 other factors that you also have to worry about the interaction with. 697 01:20:50.640 --> 01:20:53.109 216-101 Large Conference: In order to find, like, the queen. 698 01:20:53.340 --> 01:20:59.660 216-101 Large Conference: And what are those 10 other interactions, 10 other variables that you haven't thought of? 699 01:20:59.920 --> 01:21:18.180 216-101 Large Conference: Including, right? Is that… is that… I think that is a good… yeah. There's one caveat in the last project, I did not use DAS. I used a different thing. Yeah, yeah, but you could have, might as well use it, yes. That's correct. So you have to find all the… all the ones, then find the intersection of that, and that's probably the… 700 01:21:18.240 --> 01:21:35.409 216-101 Large Conference: the closest thing that we care about. To the truth. Is there a truth? Or maybe there's no truth? Or maybe, depending on the subset of variables we have… I mean, we are basically labeling, putting names on numbers at the end of the day. Right, of course, but, like, but once… but maybe if you have enough interactions. 701 01:21:35.410 --> 01:21:38.310 216-101 Large Conference: You try to do this, and eventually the optimization starts failing. 702 01:21:38.620 --> 01:21:44.050 216-101 Large Conference: Like, you, like, maybe you don't have a subspace that satisfies all the interactions at once. I mean, it's possible, right? 703 01:21:45.420 --> 01:21:47.140 216-101 Large Conference: Yep. Yeah. 704 01:21:47.430 --> 01:22:06.150 216-101 Large Conference: But how many intersection… I mean, how many, interactions would you consider, and which ones and which ones? Yeah, I don't know. You just came up with another one. Huh? Sorry, what do you mean? I know, like, this is, like, it's, like, one per year? So in 10 years, not 10? Like, you know, I don't know, right? Does that make sense? Yeah, that's the problem. Right. 705 01:22:06.210 --> 01:22:17.689 216-101 Large Conference: I have another question. If you would have completely changed the prompt, and the names, and the… maybe even the types of the sentences, would the intervention work, role to role? 706 01:22:17.920 --> 01:22:29.790 216-101 Large Conference: I think I have an answer for that, I think yes. I have, I've done a similar kind of experiment. Okay. It's one of the later slides, and I think it works. Okay. 707 01:22:29.790 --> 01:22:38.979 216-101 Large Conference: we can… depends on how different you want to make the sentence, it… it will depend on that, but I have a few other types of sentences where the interventions works. 708 01:22:39.680 --> 01:22:43.429 216-101 Large Conference: So I think when we go to that slide, maybe we… you can… 709 01:22:44.270 --> 01:22:47.809 216-101 Large Conference: Yeah, let me know if that's what you were thinking or something else. 710 01:22:48.650 --> 01:22:56.469 216-101 Large Conference: But yeah, I couldn't find those, variables, so that means that the variable, the causal model that I was thinking about was more or less… 711 01:22:56.680 --> 01:22:57.750 216-101 Large Conference: Verified. 712 01:22:58.040 --> 01:23:06.870 216-101 Large Conference: I'm gonna skip this part. But then I actually started looking into a different setup. This is where I do not ask about thematic goals at all. 713 01:23:07.960 --> 01:23:18.549 216-101 Large Conference: This is just normal Q&A question. I skip the sentence, and then I ask who did what to whom kind of question, and the model still answers. The model is still pretty good at it. 714 01:23:19.570 --> 01:23:24.829 216-101 Large Conference: thing still remains the same. The probing result still holds the same, that the first 715 01:23:25.140 --> 01:23:29.190 216-101 Large Conference: Person does not encode the thematic rule, where the second and third one does. 716 01:23:29.850 --> 01:23:45.730 216-101 Large Conference: Did you train these probes in this new setting, or did you… okay, yeah, it's a new training. Can you just take those probes and then apply it, and then generalize to this OT setting? Slightly OD setting. I did not transfer the probes, but the interventions… 717 01:23:45.890 --> 01:23:50.659 216-101 Large Conference: does transfer. But those are last tokens. Interventions transferred. Those are last tokens, yeah. 718 01:23:51.510 --> 01:23:59.249 216-101 Large Conference: Those are the what? Those are the last what? Those are at the last tokens, not really at the percent tokens, where I do the probing. 719 01:23:59.710 --> 01:24:11.489 216-101 Large Conference: So he's asking about if the probe stronger. Oh, I see, I see. Something that is not, right? Okay. Yeah, yeah. It might. Yeah, it might. Yeah, it'd be stronger if it does. I can, I can do that. 720 01:24:12.180 --> 01:24:31.779 216-101 Large Conference: But yeah, similar kinds of intervention, similar tasks, experiment, and it does work there as well. Same layer, same layer, I think. Yeah. So, the task now is, like, asking who did something, and then you take maybe the role from the previous task, and you use it to steer? 721 01:24:33.400 --> 01:24:43.749 216-101 Large Conference: Yeah, that's probably the next, next slide. Isn't what you just showed? This is still training on the… this particular setup. Okay. 722 01:24:44.460 --> 01:24:48.439 216-101 Large Conference: But I can actually train a subspace. 723 01:24:48.890 --> 01:24:55.919 216-101 Large Conference: Where I train it using the interventions in the previous setup, and the interventions in the new setup, but not cross… 724 01:24:56.050 --> 01:24:57.590 216-101 Large Conference: Across different setup? 725 01:24:57.700 --> 01:25:02.640 216-101 Large Conference: If I use those two interventions for each of the variables, then I can come up with a subspace. 726 01:25:03.260 --> 01:25:12.800 216-101 Large Conference: Which, which I can do the patching across different… between these two setups, for each of the variables that I have in the Cosmo. What's the difference in both of those recordings? 727 01:25:12.990 --> 01:25:14.240 216-101 Large Conference: It's the same thing. 728 01:25:15.680 --> 01:25:34.329 216-101 Large Conference: That's the whole point. What I'm saying is, the way a model solves, when you explicitly ask about a specific thematic role, and the way the model solves it, when you just ask about who did what to whom kind of question, the model is still thinking about those thematic roles to solve the task. And that's the main result. 729 01:25:35.650 --> 01:25:38.020 216-101 Large Conference: What are the blue and the gray ones? 730 01:25:39.580 --> 01:25:45.509 216-101 Large Conference: Blue and the gray ones are if you just train the task on one specific setup. 731 01:25:46.150 --> 01:25:50.640 216-101 Large Conference: And the orange one is when you train the DAS. 732 01:25:50.870 --> 01:25:56.400 216-101 Large Conference: On both the setup intervention, but not across interventions. 733 01:25:56.800 --> 01:26:00.199 216-101 Large Conference: Yeah, but what's the difference between the gray and the blue? 734 01:26:00.500 --> 01:26:16.399 216-101 Large Conference: So, blue is if you train each of those, each of this, let's say this role variable on this particular setup, and apply it on this setup and this setup, and that it averages this one. The average, okay. And the… 735 01:26:16.880 --> 01:26:19.320 216-101 Large Conference: Gray is the opposite? Exactly. 736 01:26:19.840 --> 01:26:28.490 216-101 Large Conference: And orange one is, you train a DAS for this particular role and this particular role simultaneously, but without doing the intervention cross-setup. 737 01:26:29.360 --> 01:26:34.339 216-101 Large Conference: Yeah, so you train it, and then you do the intervention cross-setup, then it works. 738 01:26:34.660 --> 01:26:35.750 216-101 Large Conference: Thank you. 739 01:26:36.080 --> 01:26:41.090 216-101 Large Conference: Sorry, go ahead. So if it's the same, causal… 740 01:26:41.690 --> 01:26:44.960 216-101 Large Conference: Shouldn't the blue and gray already be high? 741 01:26:45.700 --> 01:26:51.140 216-101 Large Conference: I do think that… Yeah, I think that again comes back to that family of subspaces. 742 01:26:51.490 --> 01:27:02.030 216-101 Large Conference: It's not a unique subspace… So this is that experiment. So you're just showing that single variable training experiment again, with the blue and the gray, is that right? Am I misunderstanding? 743 01:27:02.430 --> 01:27:21.900 216-101 Large Conference: No, it's something else? No, it's something else? Single variable, but applied to the other… Applied to the other. So it doesn't generalize. So, like, the single variable thing that didn't generalize to the joint thing in one setting also doesn't transfer between settings. Yeah. 744 01:27:21.900 --> 01:27:26.590 216-101 Large Conference: Yeah, I see. So this single variable thing is problematic. 745 01:27:27.330 --> 01:27:30.840 216-101 Large Conference: I see. But the joint… but the joint thing… 746 01:27:31.230 --> 01:27:34.420 216-101 Large Conference: also transfer is better between settings. No. 747 01:27:34.720 --> 01:27:38.460 216-101 Large Conference: No, no, no, no, no. Actually, forget about the single one. 748 01:27:38.570 --> 01:27:45.480 216-101 Large Conference: So, joint on the one setup, versus joint on both the setup? 749 01:27:46.320 --> 01:27:51.060 216-101 Large Conference: Oh. Which one is… which color is which? Oranges, both the setup. 750 01:27:51.190 --> 01:27:57.330 216-101 Large Conference: Oh, so you… you have to train across both setups? It's actually not across. 751 01:27:57.800 --> 01:28:01.769 216-101 Large Conference: It's like, I do the intervention on this particular variable in this setup. 752 01:28:01.970 --> 01:28:17.799 216-101 Large Conference: that is one loss for DAS, and I do the same intervention for this, similar intervention for the same variable on this other setup, and get one loss, and I use both the losses simultaneously to train the DAS. But you're training a common subspace? 753 01:28:18.190 --> 01:28:26.559 216-101 Large Conference: Would you… yeah, I see. But then after you use both of these losses to train the subspace, then you can transfer between them. 754 01:28:26.840 --> 01:28:35.609 216-101 Large Conference: Oh, I see, but it's a little bit cheating, because you've used both of them in the training. Yeah. Also, it's much lower, I think, than… like, you can get, like, 100% if you just do any one in… 755 01:28:35.800 --> 01:28:49.589 216-101 Large Conference: Like, separately, right? Yeah, it's a little bit lower. It's not exactly as high as the individual. My follow-up question would be, if you take a third setting now, would that generalize? When you say third setting, you mean still… 756 01:28:50.030 --> 01:28:59.700 216-101 Large Conference: like, similar kinds of sentences, or… Like, the second one was a completely different task, but requires the same skills, right? So a third one. 757 01:29:00.880 --> 01:29:02.569 216-101 Large Conference: Different from the first two. 758 01:29:02.890 --> 01:29:04.909 216-101 Large Conference: Okay, maybe this will answer the question. 759 01:29:05.650 --> 01:29:19.450 216-101 Large Conference: I also like David's question, like, you train the common subspace in a single setup for all the three thematic… all the three causal variables. Would that transfer across both setups? Because it's already, like, kind of… 760 01:29:19.580 --> 01:29:21.010 216-101 Large Conference: Slightly passing. 761 01:29:21.920 --> 01:29:30.380 216-101 Large Conference: space. So is that one of the bars? So if you go back to the previous thing. Yeah, those are the left bars. So is that… is that what the blue bar is? What is the blue bar? 762 01:29:30.550 --> 01:29:33.569 216-101 Large Conference: Blue bar is if I train a subspace. 763 01:29:33.870 --> 01:29:51.319 216-101 Large Conference: On one building, on one… And using all three constraints from previous experiments. So, like, one variable, another variable, joint variable, and then now you try to use those subspaces to transfer roles over to the other? Yes. And it only works half the time. Yes. I see, and what is the gray bar? 764 01:29:52.150 --> 01:29:56.739 216-101 Large Conference: It's just doing on the other side. Oh, just from the other direction, that's all. Yeah. 765 01:29:57.080 --> 01:30:01.570 216-101 Large Conference: I see. So, and which one is from the more structured to the less structured? Is it… 766 01:30:01.770 --> 01:30:06.489 216-101 Large Conference: I think blue is the first one to the second one. 767 01:30:06.770 --> 01:30:17.029 216-101 Large Conference: The more structured, the least structured. Left is less structured. Left is more structured. Wait, implicit is the first one? 768 01:30:17.240 --> 01:30:32.429 216-101 Large Conference: I'm not sure… what do you mean by more structure? The one with the longer prompt where you… Yeah, that is the first one, which is the blue one, which is what I'm calling as implicit cyclic. That is explicit once looked. So it is, gray one, gray one. 769 01:30:32.530 --> 01:30:43.360 216-101 Large Conference: Yeah, that is gray one, sorry, go ahead. Okay, okay. So, what's the difference between implicit and explicit? Those are just different setups. One of them has a long prompt, and the other one has a short prompt. It explains what the roles are to the module. 770 01:30:43.640 --> 01:30:49.590 216-101 Large Conference: Oh, that's the only difference? Yeah. Go ahead, you asked about a name instead of a role. 771 01:30:49.970 --> 01:30:58.369 216-101 Large Conference: In general, like, this experiment aside, this is a lack of my knowledge on DAS, can you make one of those orange bars out of any two unrelated concepts? 772 01:30:58.490 --> 01:31:03.380 216-101 Large Conference: Like, if I have 10 completely unrelated tasks, and I do your methodology on them. 773 01:31:03.710 --> 01:31:09.759 216-101 Large Conference: Can I get a coherent… are there any cases of there's two concepts where they won't form a coherent… 774 01:31:10.100 --> 01:31:14.479 216-101 Large Conference: Cross-patchable subspace, in whatever sense of cross-patchable you want to define there. 775 01:31:16.110 --> 01:31:18.020 216-101 Large Conference: My intuition is that, like. 776 01:31:18.390 --> 01:31:22.659 216-101 Large Conference: If both of them are in the… if both losses are in the training objective, then you're just gonna find 777 01:31:23.630 --> 01:31:25.659 216-101 Large Conference: The unity of the two subspaces. 778 01:31:27.610 --> 01:31:30.419 216-101 Large Conference: Yeah, it's like a control baseline. 779 01:31:33.320 --> 01:31:39.389 216-101 Large Conference: I don't know if Daz has some incentive for, like, making the subspaces, but I'm keeping the size of the subspaces the same. 780 01:31:41.450 --> 01:31:43.139 216-101 Large Conference: The sizes is not expanding. 781 01:31:45.520 --> 01:31:48.689 216-101 Large Conference: But I get your point. It's a valid point. 782 01:31:50.490 --> 01:31:58.800 216-101 Large Conference: I'm not… I'm not… let's say if I train a subspace for role length from one setup to the sentence index of the other one. 783 01:31:58.930 --> 01:31:59.810 216-101 Large Conference: Yeah. 784 01:32:02.220 --> 01:32:09.229 216-101 Large Conference: I'm still keeping the sizes, let's say the original size of the rolls, subspace to still remain the same. 785 01:32:10.480 --> 01:32:15.989 216-101 Large Conference: But if you… but if you can… if you can successfully patch between those, then that's an indicator that 786 01:32:16.550 --> 01:32:20.350 216-101 Large Conference: The size of the orange bar does not indicate something about the concepts. 787 01:32:23.940 --> 01:32:29.689 216-101 Large Conference: Like, if you could, if you could use this method to patch between role and sentence index, whatever that might mean. 788 01:32:30.150 --> 01:32:33.709 216-101 Large Conference: Then, you can patch between, like, anything and anything, right? 789 01:32:34.430 --> 01:32:39.439 216-101 Large Conference: Yes, yeah, I think I caught that point, but what I'm saying is, 790 01:32:41.300 --> 01:32:51.009 216-101 Large Conference: For that to work, you may have to use a bigger subspace. You may have to train a bigger subspace with your DAS. Oh, oh, I see. To accommodate both of them. 791 01:32:51.370 --> 01:32:53.630 216-101 Large Conference: So that you can catch both of them. 792 01:32:53.930 --> 01:32:57.159 216-101 Large Conference: But I did not really increase the size. 793 01:32:57.650 --> 01:32:59.629 216-101 Large Conference: So this is, this is evidence. 794 01:32:59.790 --> 01:33:07.709 216-101 Large Conference: because it is the same size subspace. It's not, as clean as it should be, but kind of. 795 01:33:08.900 --> 01:33:18.760 216-101 Large Conference: What is the value of R for role, for fetch first, for position? Our role is 2, Fetch first is, I think, 2 as well, sentence index is 16. 796 01:33:20.150 --> 01:33:20.860 216-101 Large Conference: Okay. 797 01:33:21.430 --> 01:33:22.510 216-101 Large Conference: Let me go back. 798 01:33:23.360 --> 01:33:34.079 216-101 Large Conference: So, should… in explicit, it has that definition of agent, topic, and role, right? Yes. So, shouldn't the scores be more on the gray bar? 799 01:33:34.580 --> 01:33:39.719 216-101 Large Conference: I think this is either the… yeah, this is averaged over both the directions. 800 01:33:39.840 --> 01:33:45.020 216-101 Large Conference: It's not just trained and tested on the same setup, it is trained on one setup and tested on both the setup. 801 01:33:46.140 --> 01:34:03.000 216-101 Large Conference: Okay, I'm gonna move forward. Yeah. What's the next big step? Yeah, but the main point here is the model is using thematic rules in both the setup. That means when you just ask who did what to whom, the model is still using thematic rule in addition to the point of information to solve a task. 802 01:34:05.790 --> 01:34:12.290 216-101 Large Conference: It's okay, cool. And the same thing, everything that I've told, generalizes across languages. 803 01:34:12.450 --> 01:34:28.790 216-101 Large Conference: Or at least left to light language. Oh, wait, can you train it on one language without training it on another language, and then it works on that language? I still… no, that is not what that one. Yeah, that's what it means for me to generalize. Unseen language. That's the problem of DAS. I think DAS is a little too powerful. 804 01:34:29.760 --> 01:34:33.099 216-101 Large Conference: If you paid it only… That's overfits. 805 01:34:35.010 --> 01:34:39.959 216-101 Large Conference: So that's why I have to train it on the 7 sentence, all the seven languages. 806 01:34:40.130 --> 01:34:49.500 216-101 Large Conference: But then once you train it, you make the DAS constrained enough, then you can do… you can use that constrained subspace to be patching across different languages. 807 01:34:53.390 --> 01:34:58.549 216-101 Large Conference: Yeah, but if you don't really constrain it, then the transfer does not work. 808 01:34:59.370 --> 01:35:06.829 216-101 Large Conference: So the subspace is theoretically then segmentable to sub-subspaces corresponding to each language? 809 01:35:08.540 --> 01:35:16.379 216-101 Large Conference: some subspace… So if we're… if we've got… let's say this is the whole subspace of… 810 01:35:17.710 --> 01:35:21.229 216-101 Large Conference: But the actual, like, material task based on the concepts. 811 01:35:22.750 --> 01:35:30.839 216-101 Large Conference: If you're fitting your DAS just on English, you're gonna get this little zone. If you're fitting your DAS just on Chinese, you're gonna get this little zone. Okay. 812 01:35:31.770 --> 01:35:34.250 216-101 Large Conference: The fact that you can distinguish these little zones. 813 01:35:34.770 --> 01:35:37.770 216-101 Large Conference: And that the zones don't generalize must mean that, like. 814 01:35:39.080 --> 01:35:44.460 216-101 Large Conference: There's a part of your subspace of… Freezing this task. 815 01:35:44.610 --> 01:35:48.899 216-101 Large Conference: that is reasoning about that in English versus another part that's reasoning about it in Chinese. 816 01:35:49.610 --> 01:35:55.109 216-101 Large Conference: Is that right? Yeah, I think that's… that's what I think the kind of intuition that I was trying to give with this paragraph. Oh, yes, yeah, okay. 817 01:35:55.270 --> 01:36:01.299 216-101 Large Conference: So, there is sort of a family of subspaces which works for, let's say, English. There is a family that works with 818 01:36:02.250 --> 01:36:03.230 216-101 Large Conference: Mandarin. 819 01:36:03.950 --> 01:36:08.230 216-101 Large Conference: It's a little dissatisfying, right? Yeah, so if you… 820 01:36:08.460 --> 01:36:26.509 216-101 Large Conference: Yeah, what's your question, Sheng Yu? Oh, yeah. You're targeting a concept. Arunam, you've got to ask enough questions. Sheng Yu! Okay, yeah. Sorry, my question is pretty dumb. So, if you fit on one language, why not fit on, like, five languages at the same time, trying to just a single desk? 821 01:36:26.890 --> 01:36:28.130 216-101 Large Conference: Better work. 822 01:36:28.340 --> 01:36:30.130 216-101 Large Conference: Like, it's not… doesn't work. 823 01:36:30.440 --> 01:36:42.690 216-101 Large Conference: Yeah, that works. It works. It works. It also works. Yeah, but what does it teach us about the model? Oh, but, like, the thing is, like, if you train on a lot of languages, it fits. If you train on even more languages, we're generalized to unseen languages. 824 01:36:42.800 --> 01:36:59.540 216-101 Large Conference: Oh, so if you train on 10 languages, does it go to the 11th on? I think if you… if I train on 4 or 5, then it will generalize to… Have you tried? Yeah, I've tried that. Oh, okay, good. That's good. Okay, yeah. Okay, well then that taught us something. It depends on how much… 825 01:37:00.050 --> 01:37:09.600 216-101 Large Conference: then some of the constraints do you put on the DAS? If you don't put the constraint, then it can learn some… it will learn something that probably won't generalize, but if you put enough constraint on the DAS, 826 01:37:09.840 --> 01:37:14.519 216-101 Large Conference: then I think you will find… What is the effect? So, like, if you train it only on French. 827 01:37:15.460 --> 01:37:20.830 216-101 Large Conference: And then you try to apply… to do transformations on English. 828 01:37:21.480 --> 01:37:29.239 216-101 Large Conference: the accuracy goes down, but does it go down in some interesting way? Like, does it… does it do everything but in French or something like that? 829 01:37:29.480 --> 01:37:31.100 216-101 Large Conference: You know what I mean? 830 01:37:31.300 --> 01:37:36.399 216-101 Large Conference: I did not look into that. Yeah, we should look at the instances to build some intuition. 831 01:37:37.770 --> 01:38:00.030 216-101 Large Conference: I did not look into that qualitatively. How can you do that in French, though? It's just names, people's names. That's the first token. That's the objective here. Yeah, I don't know. Maybe instead of Steven, it's like Stefano. Like, what does it mean for the… Stefano? I don't know, but for the accuracy to go down, it's just some other token. I wonder what it is. 832 01:38:00.480 --> 01:38:07.379 216-101 Large Conference: It would be more interesting to try it with non-subject-verb object languages, like languages that are still left to right, but that flip around your order. 833 01:38:08.370 --> 01:38:14.630 216-101 Large Conference: like, oh, I know that Chinese is one of them, but that's, like, subject verb, object is weird there. I'm thinking, like, 834 01:38:16.010 --> 01:38:23.099 216-101 Large Conference: like, Latinizations of… Swahili. They do subject, object, verb, I think. 835 01:38:24.010 --> 01:38:28.539 216-101 Large Conference: Subject, object, work? If I'm remembering right. Okay. It's okay. 836 01:38:31.130 --> 01:38:33.649 216-101 Large Conference: Yeah, different language forms, that's kind of neat. 837 01:38:34.010 --> 01:38:35.210 216-101 Large Conference: It's a good idea. 838 01:38:36.930 --> 01:38:37.870 216-101 Large Conference: Yeah. 839 01:38:39.220 --> 01:38:41.989 216-101 Large Conference: No, that's it, I think that's, that's it. 840 01:38:42.540 --> 01:38:50.960 216-101 Large Conference: Yeah, it generalizes, across languages. What's your summary? Conclusion? Oh, yeah, read your conclusion slide. Yeah, maybe you can read it. 841 01:38:53.070 --> 01:38:54.809 216-101 Large Conference: Can you advise what you think. 842 01:38:55.650 --> 01:38:58.669 216-101 Large Conference: Yeah, so I think what I think is, 843 01:38:58.790 --> 01:39:09.450 216-101 Large Conference: The model is sort of combining or composing this pointer information with thematic role information, and it actually links to fetch information. 844 01:39:09.630 --> 01:39:16.190 216-101 Large Conference: not only pointer, which is relevant for binding. There is at least one more piece of information, which 845 01:39:17.360 --> 01:39:20.459 216-101 Large Conference: I'm describing using this framework of thematic rules. 846 01:39:20.760 --> 01:39:22.879 216-101 Large Conference: I think that's the main takeaway. 847 01:39:23.230 --> 01:39:29.240 216-101 Large Conference: Yeah, and then this fact is generalizable across languages. 848 01:39:29.900 --> 01:39:36.379 216-101 Large Conference: So, I would say you have a very interesting meta result, which is a… it's a limitation. You got everything to work. 849 01:39:36.560 --> 01:39:41.440 216-101 Large Conference: But, like, you can see, like, the people around the table are dissatisfied 850 01:39:41.590 --> 01:39:44.340 216-101 Large Conference: Like, now we're not happy with subspaces anymore. 851 01:39:44.590 --> 01:39:53.100 216-101 Large Conference: Right? You've made us all unhappy with subspaces, which is… right? Which I think is an interesting result. I mean, like, you've gotten the subspaces to work. 852 01:39:54.100 --> 01:40:01.629 216-101 Large Conference: Right? But they're not acting, like, the way that you expect subsurfaces to work. Like, you'd expect… like, that experiment that you showed. 853 01:40:02.270 --> 01:40:08.960 216-101 Large Conference: Where you should join the subspaces and it should just work naturally. That's what you would expect. If… 854 01:40:09.260 --> 01:40:13.250 216-101 Large Conference: If, you know, your linear algebra class is working the way that 855 01:40:13.640 --> 01:40:23.070 216-101 Large Conference: You know, if it was like a Fourier decomposition, and you said, I've got the low frequency things, and I've got the high frequency things, if I join the subspaces together, now I've got 856 01:40:23.330 --> 01:40:29.430 216-101 Large Conference: You know, the combination of all the frequencies, and it just works, and it's beautiful, it's what they teach you. 857 01:40:29.590 --> 01:40:33.290 216-101 Large Conference: You know… An engineering class, or whatever, right? 858 01:40:33.540 --> 01:40:36.760 216-101 Large Conference: And, and somehow, It's not doing that. 859 01:40:36.930 --> 01:40:40.359 216-101 Large Conference: Right? So, now, it's not to say the subspaces are… 860 01:40:41.120 --> 01:40:43.299 216-101 Large Conference: Irrelevant, like, you got it to work. 861 01:40:43.550 --> 01:40:46.920 216-101 Large Conference: But somehow, it's missing something. I think that that might be… 862 01:40:47.390 --> 01:40:53.410 216-101 Large Conference: I mean, so, like, I feel like there's two levels of the thing you found. Like, one thing you found 863 01:40:53.760 --> 01:40:59.150 216-101 Large Conference: This really interesting substructure for… binding. 864 01:40:59.370 --> 01:41:02.829 216-101 Large Conference: Which is… A nice clarification. 865 01:41:03.020 --> 01:41:12.819 216-101 Large Conference: I think that it could… might have been anticipated, like, some people could have… have, like, hallucinated, oh yeah, there's probably what is under there. Like, I sort of would have imagined that there's some substructure. 866 01:41:12.930 --> 01:41:14.380 216-101 Large Conference: Underneath those pointers. 867 01:41:14.510 --> 01:41:25.059 216-101 Large Conference: But then you also found this limitation, which, I don't… I think it's kind of surprising to see this limitation, and it asks this question for the next researchers. 868 01:41:25.310 --> 01:41:33.250 216-101 Large Conference: Well, if not subspaces, then what is the right abstraction for dividing up a big representation? We have these giant rep… you know, we've got these giant 869 01:41:34.010 --> 01:41:35.959 216-101 Large Conference: 8,000 dimensional vectors. 870 01:41:36.370 --> 01:41:50.979 216-101 Large Conference: And Atticus and, you know, we have been going around, you know, looking at subspaces of these 8,000-dimensional spaces to say, information's in this subspace, but you're saying, I don't know. Once you find a subspace. 871 01:41:51.920 --> 01:41:54.529 216-101 Large Conference: You might not be as smart as you think you are. 872 01:41:54.980 --> 01:41:57.750 216-101 Large Conference: Right? There's some… there's some other stuff going on. 873 01:41:58.740 --> 01:42:08.830 216-101 Large Conference: You know, I think that's kind of this interesting question. Well, if not subspaces, then what? Should we be looking at covariance matrices or something? Should be, like, weighted subspaces? 874 01:42:08.900 --> 01:42:24.589 216-101 Large Conference: Where… what? Weight space, I don't think… I think weight space is even worse. Now, instead of starting with 8,000 dimensions, now you start with, you know, 150,000 dimensional space, or something like that, you know, weights in a layer. Like, even… 875 01:42:25.450 --> 01:42:27.519 216-101 Large Conference: You're in a worse situation. 876 01:42:27.840 --> 01:42:32.009 216-101 Large Conference: I don't know. I'm explaining it, but maybe the explanation is more… Yeah, I don't know, I don't… 877 01:42:32.300 --> 01:42:33.500 216-101 Large Conference: Maybe. 878 01:42:36.240 --> 01:42:43.570 216-101 Large Conference: But, like, one thing you haven't looked at, and I don't really… it's probably not the right thing to cover in this paper, because you're covering so much already, but it's kind of an interesting question, right, is… 879 01:42:43.910 --> 01:42:47.080 216-101 Large Conference: is… is this… this question of angles. It's like, so… 880 01:42:47.500 --> 01:42:51.239 216-101 Large Conference: You know, is our… is there a thing about… 881 01:42:51.820 --> 01:42:58.579 216-101 Large Conference: subspaces being closer to each other and farther. I think Sheridan had this interesting perspective, also. 882 01:42:58.680 --> 01:43:05.319 216-101 Large Conference: Where, you know, so Sheridan was looking at larger variables, rather than, like, what a role is, which has, like, 3 883 01:43:05.380 --> 01:43:24.429 216-101 Large Conference: settings or something like that, so you can kind of fit it into a 10-dimensional subspace or something like this. Like, what dimension… what is the dimensionality of the spaces that you end up using? Two dimensions for the roll? Yeah, two for the roll, two for edge first, and 16 for… 16 for the number, for the count, for the… yeah. 884 01:43:24.630 --> 01:43:32.329 216-101 Large Conference: Yeah, that makes sense. Sorry, I just wanted to say, it's kind of, like, interesting, right? There are three rows, two bits of information. 885 01:43:32.630 --> 01:43:34.400 216-101 Large Conference: Like, locked to off… 886 01:43:34.640 --> 01:43:42.159 216-101 Large Conference: 3 is… the seal of it is just 2, so it kind of makes sense that you'll need at least 2… At least 2 bits? Yeah. 887 01:43:42.410 --> 01:43:47.590 216-101 Large Conference: Yeah. And that's the same thing why you get two, for the… 888 01:43:48.460 --> 01:43:54.110 216-101 Large Conference: For the fetch first. Why do you need two for the fetch first? Yeah, you could just use one. Yeah. 889 01:43:55.280 --> 01:43:58.040 216-101 Large Conference: I think you can get, like, depends, 890 01:43:58.390 --> 01:44:06.969 216-101 Large Conference: How much IA are you expecting? Here, I think I was going beyond 95, but I think you can still get, like, 80%-ish with a single dimension. 891 01:44:09.940 --> 01:44:25.029 216-101 Large Conference: So one thing I was, like, when you show the DAG, right, the causal DAG, you have all these very nicely named… that you think are going to generalize, like, okay, it detects it on, you know, the roles and the topics and things like that. 892 01:44:25.260 --> 01:44:38.159 216-101 Large Conference: And when you train on… without the prompts, like, you still find them. The fetch first seems, like, very specific to your sentence structure. That, like, just works because you know 893 01:44:38.270 --> 01:44:48.679 216-101 Large Conference: you always have 3 roles, and the first one is when it's not. So, like, I would expect a model, like, when I read the question, like, I'm thinking, okay, which role happens to each one? 894 01:44:49.090 --> 01:44:51.580 216-101 Large Conference: But the Fech first wouldn't be, like. 895 01:44:52.160 --> 01:44:59.389 216-101 Large Conference: I wouldn't expect it to be a mechanism that works all, like, in most sentence structures. Like, it… 896 01:45:00.430 --> 01:45:03.600 216-101 Large Conference: from your variables, I think the Fed first is kind of… 897 01:45:04.610 --> 01:45:12.020 216-101 Large Conference: only useful as an algorithm if you know, I'm going to have the three roles and only one instance of the three roles. I think 898 01:45:12.410 --> 01:45:23.880 216-101 Large Conference: if you have one missing, your DAG would imply, okay, I'll just say whatever is first, if one of… if the topic is missing. So I think, like, the DAG actually might… 899 01:45:24.590 --> 01:45:30.530 216-101 Large Conference: Like, maybe one reason this is confusing is that the Fech first isn't a single variable. 900 01:45:30.640 --> 01:45:36.360 216-101 Large Conference: So when you train a subspace for it, you're, like, learning a bunch of different things that you don't know how to make. 901 01:45:38.580 --> 01:45:44.459 216-101 Large Conference: Well, I think… Regarding a sort of, like, first point, if I got it correctly, that 902 01:45:45.200 --> 01:45:49.090 216-101 Large Conference: Wait, is your first point was, like, it won't generalize to other setups? 903 01:45:49.220 --> 01:45:58.969 216-101 Large Conference: like, the fetch first makes sense, because you know the sentence has 3 roles. So, oh, if I don't see my role, I'll just look at the first one. 904 01:45:59.470 --> 01:46:10.290 216-101 Large Conference: But the role thing would be something that's relevant for every sentence to keep in track forever. The thing, it doesn't really matter whether I have, like, 3 rows or 10 rows. 905 01:46:10.540 --> 01:46:22.910 216-101 Large Conference: the fact that the first thing, the better, huh? If you get 10 rules, then you have more bits of information, then your subspace will not cover that. I'm talking about the causal model, not really the representation. 906 01:46:23.770 --> 01:46:27.900 216-101 Large Conference: It… but your interventions will not work if you have more than 3 roles. 907 01:46:28.240 --> 01:46:39.200 216-101 Large Conference: Why? Five flows? Because the tools… Yeah, every change, every access of change you make, you'll need to change. 908 01:46:39.500 --> 01:46:42.779 216-101 Large Conference: Train subspaces, and to make it work, right? 909 01:46:43.240 --> 01:46:52.739 216-101 Large Conference: Yeah, you… yeah, that will… that is true, but I think the here point here is about the Cozumaro, the fact that the new information is being used 910 01:46:53.090 --> 01:47:04.619 216-101 Large Conference: It may be represented in, like, a two-dimensional, three-dimensional, 4-dimensional… yeah, that is… that is, I think, setup-dependent, but the fact that it is using a different type of information for binding is, I think, the main point. 911 01:47:06.110 --> 01:47:14.390 216-101 Large Conference: Nikhil, do you think you could have come up with an alternative causal model and also prove it with a subspaces? Yeah, I think I was… 912 01:47:15.110 --> 01:47:21.859 216-101 Large Conference: I might do that for the paper. Something, like, instead of using thematic roll, just Do something like… 913 01:47:22.030 --> 01:47:28.469 216-101 Large Conference: fetch the first one, or the second one, or the third one. But maybe, maybe you're asking the question in a different way. So, like. 914 01:47:28.740 --> 01:47:37.289 216-101 Large Conference: So, you have a causal model, and you said that this was not intuitive for you. You, like, you discovered this Feet first. 915 01:47:37.600 --> 01:47:40.639 216-101 Large Conference: Mechanism, you know, partway through your research. 916 01:47:40.850 --> 01:47:45.180 216-101 Large Conference: So you started with simpler causal models that you disproved. 917 01:47:45.480 --> 01:47:55.359 216-101 Large Conference: Do you have, like, a good way… like, so in this slide deck, you didn't… you try not to confuse us, which is good, because it takes up time just to explain what you have. 918 01:47:55.640 --> 01:48:05.930 216-101 Large Conference: But do you have, like, a good way of presenting, or proving, or walking a reader through in the paper that you've… you've disproven alternate hypotheses? 919 01:48:06.440 --> 01:48:07.200 216-101 Large Conference: Yeah. 920 01:48:07.860 --> 01:48:16.300 216-101 Large Conference: In a sense, you can say that if your causal model does not have the fetch-first variable… Yes. …then… 921 01:48:16.580 --> 01:48:25.410 216-101 Large Conference: it won't be complete. Right, so I think that would be really nice. Like, I don't know if, like, how complete is your write-up? Do you talk about that? 922 01:48:25.720 --> 01:48:33.180 216-101 Large Conference: I mean, I'm still writing. You're still writing it, yeah, I think that would be really interesting, even if it's just… if it's, like, in the appendix or something, but just to say, look. 923 01:48:33.480 --> 01:48:44.829 216-101 Large Conference: you know, here's a simpler causal model, and the numbers are really bad for it. Let me show you, like, why you couldn't simplify it in this natural way. 924 01:48:45.200 --> 01:48:47.750 216-101 Large Conference: Does that make sense? 925 01:48:48.030 --> 01:48:55.889 216-101 Large Conference: Yeah, I can do that. Sorry, one more thing, like, I think… I'll ask if this is, like, a counterexample to your bank. Like, if you say. 926 01:48:56.160 --> 01:49:00.230 216-101 Large Conference: Anna and Bob do whatever. You have two agents. 927 01:49:00.710 --> 01:49:08.799 216-101 Large Conference: You would predict that it's going to check, is there an agent here? Yes, Bob. Anna doesn't have any encoding, so Bob is an agent. 928 01:49:09.190 --> 01:49:14.099 216-101 Large Conference: Sorry, say that sentence again? Anne and Bau do. Yeah. 929 01:49:14.290 --> 01:49:16.630 216-101 Large Conference: So maybe Anna doesn't have any… 930 01:49:16.940 --> 01:49:25.260 216-101 Large Conference: represent, you know, it doesn't have a role. Bau would. No, because Bau would. Oh, because you need the verb. Yes. Yeah. 931 01:49:25.480 --> 01:49:29.969 216-101 Large Conference: So you would say, like, the one who reminded was Bob to Anna. 932 01:49:30.230 --> 01:49:32.899 216-101 Large Conference: So then, like, maybe your first one will already have a command 933 01:49:33.460 --> 01:49:48.739 216-101 Large Conference: The one who reminded. The one who reminded was Bob. Was Bob, comma, it was about Anna, too. There is no such thing. 934 01:49:50.860 --> 01:50:03.739 216-101 Large Conference: I guess the simplest one… Sorry, simplest one is when one is missing, then you look for it, you don't see it, so you give whatever is first, which is, like, the wrong role. That's what you would predict from this data. 935 01:50:03.850 --> 01:50:05.840 216-101 Large Conference: That, that is the wrong rule. 936 01:50:06.550 --> 01:50:19.070 216-101 Large Conference: You're just trying to construct a sentence form which would cause the model to make a mistake based on this, is that right? Yeah, you can get a much simpler sentence form to contradict it, just do passive voice of the same fact. 937 01:50:19.640 --> 01:50:24.950 216-101 Large Conference: like, Alice called Bob becomes… Bob was called by Alice. 938 01:50:25.130 --> 01:50:25.890 216-101 Large Conference: Right? 939 01:50:26.450 --> 01:50:32.579 216-101 Large Conference: Because that would… that would eliminate the… that would screw up the fetch-first mechanism. I think it would be fine, though. 940 01:50:32.740 --> 01:50:39.409 216-101 Large Conference: Like, it… it's… the fetch first work always works, like, doesn't have to be, like, passable, active. Like, the… 941 01:50:40.150 --> 01:50:44.770 216-101 Large Conference: It's just, like, whenever the first one, like, you can't access, then… Oh, oh, oh, I see. 942 01:50:45.050 --> 01:50:53.449 216-101 Large Conference: Nikhil, most of your experiments are looking at the mechanism from the last token before the answer. 943 01:50:54.120 --> 01:50:57.850 216-101 Large Conference: But you also have probing experiments that show that 944 01:50:58.220 --> 01:51:02.190 216-101 Large Conference: This thematic indexing happens at the word token? Yes. 945 01:51:02.900 --> 01:51:11.429 216-101 Large Conference: Do you think you can also train subspaces to change roles of each shared inside, so that you don't do anything in the final token, it's… 946 01:51:12.110 --> 01:51:14.540 216-101 Large Conference: It's just you doing the other end of… 947 01:51:14.870 --> 01:51:22.979 216-101 Large Conference: the intervention. I think it is doable, but I've just not done it. But yeah. You, like, do you not think it's relevant to the work, or what it is? 948 01:51:23.450 --> 01:51:35.559 216-101 Large Conference: It is, I think, doable. It is a very… yeah, I think it will work if I do it as well, but I'm just not doing it. I think for this project, I was just primarily focusing on the last token. What happens at the last token. 949 01:51:35.760 --> 01:51:46.750 216-101 Large Conference: So, I want to add something to Hadas and David's point. Like, during… I don't know, like, David Goodfair has more say in what to write in the paper, but when you're presenting, I would… 950 01:51:46.940 --> 01:51:53.029 216-101 Large Conference: But most of the things… that was my first question, right? Like, how did you come up with this graph for the Cosmic model? 951 01:51:53.890 --> 01:52:01.110 216-101 Large Conference: When you're explaining things, most of these are really surprising, from your point of view, I'm sure. 952 01:52:01.460 --> 01:52:06.790 216-101 Large Conference: When I'm listening for the first time and I'm not super aware of DAS or any of these techniques. 953 01:52:06.960 --> 01:52:10.219 216-101 Large Conference: The thing that would really help me was if I see a control. 954 01:52:10.690 --> 01:52:20.630 216-101 Large Conference: Where you say, oh, this 100% is not something that happens for anything. Like, for example, like, Hadas asked the question, oh, like, can you do the same thing by just 955 01:52:20.770 --> 01:52:26.050 216-101 Large Conference: changing the entire causal graph, coming up with a new graph, and I can still do all the DAS variables. 956 01:52:26.620 --> 01:52:35.020 216-101 Large Conference: Yeah. Right? Like, Fet second may be a variable that you can try to train DAS on, and maybe DAS would give you, like, 50% or 20%. 957 01:52:35.490 --> 01:52:41.800 216-101 Large Conference: So, like, if you can show such control examples for every experiment we have, I think that would, like… 958 01:52:42.150 --> 01:52:45.960 216-101 Large Conference: Enhance my understanding, too, that, oh, yes, this is, in fact, surprising. 959 01:52:46.990 --> 01:52:59.899 216-101 Large Conference: And maybe that's partly the reason why there's, like, some dissatisfaction around, like… Interpretability. Yeah. Right. Like, we tend to not control our experiments well enough, because… and partly it's because 960 01:53:00.150 --> 01:53:07.060 216-101 Large Conference: The results can be complicated, like, you've got, like, multiple things going on, and you feel like, oh, now I've filled out my 8 pages. 961 01:53:07.150 --> 01:53:25.209 216-101 Large Conference: But I think that… I think it's good. I think that even if it means that the controls are in the appendix or something like that, we should… we should make Nikhil's run a bunch of these, they're just not in the presentation. They're just not there. They didn't… because they weren't relevant to what actually worked out. Yeah, I… I totally… I know, and that's why he… that's why you work. What? I mean, that might be just too much. 962 01:53:25.410 --> 01:53:32.389 216-101 Large Conference: if I go through the rationale of each of these different… Yeah, I know, you always do this, you always have, like, 150 pages of results. 963 01:53:32.670 --> 01:53:40.589 216-101 Large Conference: So… so, but I… but I do… but I… but I agree with you, so I think that, you know, it… it probably… there's probably some editorial judgment. 964 01:53:40.770 --> 01:53:43.519 216-101 Large Conference: About, like, what would be reasonable controls? 965 01:53:43.650 --> 01:53:47.430 216-101 Large Conference: to include, but I do think that it would be helpful. 966 01:53:47.720 --> 01:53:57.940 216-101 Large Conference: to basically show people, the alternate… some alternate hypotheses that don't work. Like, ones that you think might be reasonable that people would suggest, or something like that, like… 967 01:53:58.070 --> 01:54:05.429 216-101 Large Conference: You know, even if that's explained in the appendix. Yeah, so the question I was trying to ask me for is, were there alternate hypotheses 968 01:54:05.830 --> 01:54:09.550 216-101 Large Conference: really don't work, because from what you showed, it seems like you can train for anything. 969 01:54:11.050 --> 01:54:11.820 216-101 Large Conference: Great. 970 01:54:12.830 --> 01:54:16.660 216-101 Large Conference: Like, it could be just predicate logic plus events. 971 01:54:16.970 --> 01:54:18.589 216-101 Large Conference: Instead of thematic rules. 972 01:54:20.320 --> 01:54:27.239 216-101 Large Conference: Or fetch 2nd, fetch third. Would it not work, or did you know whether it would work if you called with a completely different one? 973 01:54:28.330 --> 01:54:29.280 216-101 Large Conference: Which one? 974 01:54:29.550 --> 01:54:35.189 216-101 Large Conference: A different causal graph, and do… and train doubts on it, on, like, the causal variables there. 975 01:54:35.960 --> 01:54:39.670 216-101 Large Conference: Again, it will depend on the cost of model that you come up with. 976 01:54:39.910 --> 01:54:40.870 216-101 Large Conference: So… 977 01:54:41.880 --> 01:54:50.380 216-101 Large Conference: It might be the case that you can come up with another causal model where the variables that I have in my causal model is sort of, like, split into more fine-grained things. 978 01:54:50.660 --> 01:54:51.820 216-101 Large Conference: And that's my board. 979 01:54:52.230 --> 01:54:57.640 216-101 Large Conference: But, if you come up with a causal model where you come up with a variable which is actually not 980 01:54:58.130 --> 01:55:01.040 216-101 Large Conference: It is actually not what is happening in the model, then it won't work. 981 01:55:01.730 --> 01:55:02.660 216-101 Large Conference: Great. 982 01:55:03.320 --> 01:55:05.489 216-101 Large Conference: That will convince me if I see that. 983 01:55:05.840 --> 01:55:10.230 216-101 Large Conference: Okay. Because of all the limitations you showed. 984 01:55:10.740 --> 01:55:17.769 216-101 Large Conference: So, it isn't in the… In the original DAS paper, they come… they… 985 01:55:18.000 --> 01:55:26.980 216-101 Large Conference: They work with a simple, summation setup in plus B plus C, and there are different ways you can combine them. Summation, or was it range? 986 01:55:27.360 --> 01:55:45.210 216-101 Large Conference: There was a summation paper, yeah. I think the one with a different concept set out. Okay, never mind. Let's see, then Tyke balancing, like, sorry, parenthesis balancing, stuff like that. Okay. And they actually came up with, different causal 987 01:55:46.910 --> 01:56:04.630 216-101 Large Conference: causal graphs, and they were showing that one of them works, or two of them works, or multiple of them. I'm not really sure. But I think a lot of these questions are already answered in the original DAS paper. Well, for the… for that toy example. For the toy example. For us, the example is more important than the DAS example. 988 01:56:04.770 --> 01:56:05.610 216-101 Large Conference: Right? 989 01:56:06.330 --> 01:56:07.700 216-101 Large Conference: Yeah… 990 01:56:07.970 --> 01:56:16.360 216-101 Large Conference: But also, like, like, in this specific example, can you think of another causal graph? Yeah. Fletch second, fetch third, like, how you can say. 991 01:56:17.440 --> 01:56:21.980 216-101 Large Conference: How would that work? Maybe, I don't know, like, maybe Dad can find it. 992 01:56:22.150 --> 01:56:25.020 216-101 Large Conference: Like, find the second person. 993 01:56:25.200 --> 01:56:38.289 216-101 Large Conference: Well, if you, if you just did, like, like, the initial thing, well, like, so, you know, Beno or whoever would say, oh, it's obvious that you can't do that because it doesn't know what the role is, but the obvious thing to do would be just to drop the Feucht first. 994 01:56:38.770 --> 01:56:43.689 216-101 Large Conference: Variable, right, and try… just try to say, hey, can you get a causal graph 995 01:56:43.790 --> 01:56:49.029 216-101 Large Conference: to work with just the role. And, you know, it'll fail, because there's no way that it knows the role. 996 01:56:49.180 --> 01:56:50.910 216-101 Large Conference: Right? Okay. 997 01:56:51.240 --> 01:56:56.100 216-101 Large Conference: That's, like, one example of that, that the cause of all that I come up with is not really good. 998 01:56:56.510 --> 01:57:06.269 216-101 Large Conference: I thought the fetch was very good. That would have been a nice resource to see that there's, like, some gap when you don't bring the fetch first, and once you add it, it pops up. That'd be nice. 999 01:57:07.790 --> 01:57:09.990 216-101 Large Conference: Also, like, 1000 01:57:10.630 --> 01:57:22.450 216-101 Large Conference: If you just use your pointer method from previous works, does this task… That could be also another… Another baseline. Another baseline, another cause of all, that I forget about anything. 1001 01:57:24.250 --> 01:57:34.720 216-101 Large Conference: forget about anything on the left-hand side of the circuitry. Just include the third one, then also it will be pretty bad. Yeah, I think that's nothing better. It'll be bad? Yeah, it will be bad. I mean… 1002 01:57:34.860 --> 01:57:40.250 216-101 Large Conference: If you just tell it to find some person of a particular sentence, then there are 3 options. 1003 01:57:40.940 --> 01:57:47.600 216-101 Large Conference: it won't be good, right? You could fetch… you could fetch any of this 3-1. Yeah. So, like, do you think that… 1004 01:57:47.730 --> 01:57:53.359 216-101 Large Conference: In the next coming 10 years, you can come up with more setups that would, like, start extracting more and more. 1005 01:57:54.870 --> 01:58:01.239 216-101 Large Conference: causal inputs to your… Yeah, so I think the hope here is that if we have enough 1006 01:58:02.080 --> 01:58:11.490 216-101 Large Conference: Variables that go into form… that goes into binding different types of features to a particular object. 1007 01:58:12.080 --> 01:58:20.049 216-101 Large Conference: then… I think we can… or the way I think about it is that the North Pole here is… 1008 01:58:20.610 --> 01:58:25.770 216-101 Large Conference: If we can, come up with, let's say, tracer programs. 1009 01:58:26.260 --> 01:58:35.630 216-101 Large Conference: Using the variables that we have in our causal model, and replace the specific model components which are responsible for generating those variables in the model. 1010 01:58:35.850 --> 01:58:38.640 216-101 Large Conference: With the tracer program that we have come up with. 1011 01:58:39.170 --> 01:58:43.199 216-101 Large Conference: And do it at a sufficiently good performance level. 1012 01:58:43.430 --> 01:58:48.870 216-101 Large Conference: then I think I would say that, like, if we can achieve that, then I would say that, okay, binding is a problem. 1013 01:58:50.330 --> 01:58:53.609 216-101 Large Conference: That's what I would call it, as, like, a solid problem. 1014 01:58:54.050 --> 01:58:55.869 216-101 Large Conference: If we can actually program 1015 01:58:56.080 --> 01:59:08.200 216-101 Large Conference: all the different variables which are important for doing the banding with something like a tracer thing. Isn't this what the symbolic AI people have… was trying to do for, like, 50 years and could not do it? 1016 01:59:08.730 --> 01:59:14.999 216-101 Large Conference: they were trying to do it for the symbolic, you mean neurosymbolic people, or… Neurosymbolic people, like… Yeah, but maybe they were not… 1017 01:59:15.270 --> 01:59:32.509 216-101 Large Conference: Creative enough. This is what I mean by gradient descent is probably more creative. They didn't have gradient descent. Gradient Descent has learned something that they were not able to come up with, and since we are reverse engineering that… 1018 01:59:32.700 --> 01:59:35.210 216-101 Large Conference: Then we have a good chance of getting there. 1019 01:59:36.200 --> 01:59:44.539 216-101 Large Conference: I'll ask you for a prediction. If you train it on, like, Fez 2nd or 1st, 3rd, do you think that would work similarly well? 1020 01:59:47.220 --> 01:59:56.520 216-101 Large Conference: I'm not sure about that. I think that's one of the another baselines that I need to try. Also, like, it could be just model-specific or parameter-specific graph you made. 1021 01:59:57.000 --> 02:00:09.949 216-101 Large Conference: Yeah, so this one has… this one has only been tested on Gemma 27B, but I think I'm gonna try it on other models. Like, maybe Kimi K3 is a really large… it might have… 1022 02:00:10.280 --> 02:00:11.500 216-101 Large Conference: Yeah, I mean… 1023 02:00:12.200 --> 02:00:28.929 216-101 Large Conference: Yeah, I can try QiniK3 with good fire cluster. Actually, I'm gonna say who's gonna give me the compute, but I think they are the one. But yeah, I'm gonna… and I'm hopeful that the causal model will generally generalize to the other model. 1024 02:00:31.480 --> 02:00:37.090 216-101 Large Conference: Okay, I think we can call it off, we are going way over time. Great, thank you, Nikhil, very much. That's great. 1025 02:00:38.830 --> 02:00:40.070 216-101 Large Conference: Maybe divestations.