WEBVTT

00:04:27.000 --> 00:04:32.000
But, from a more natural setting, what would that mean is, let's say

00:04:32.000 --> 00:04:37.000
you see this particular picture, and then in your mind, in your brain, you can

00:04:37.000 --> 00:04:46.000
Keep the representations of horse and fedora to be more associated with each other in comparison to the

00:04:46.000 --> 00:04:48.000
representation of CAP and the cap.

00:04:48.000 --> 00:05:03.000
And the fact that your mind can do that basically means that it has the inherent binding capability of associating objects with its corresponding features

00:05:03.000 --> 00:05:07.000
And why do we care about binding

00:05:07.000 --> 00:05:12.000
Yeah, I think two main reasons. One is historical, and one is more contemporary

00:05:12.000 --> 00:05:18.000
The historical being, I think in the 1980s or late 1980s, when people were just talking about whether

00:05:18.000 --> 00:05:29.000
They should put in more effort to do research on neural network versus symbolic AI. One of the main arguments

00:05:29.000 --> 00:05:46.000
Against the neural network was the fact that the models, or the neural networks will not be able to do binding or variable binding. For instance, I think their assumption was something like if you have a sentence like John loves Mary, there would be some neuron which will activate for John, there will be some neuron which will activate for Mary

00:05:46.000 --> 00:05:51.000
And there would probably be some neuron which will activate for the loves relation

00:05:51.000 --> 00:06:04.000
So, when you… I'm moving these pairs because I'm noticing that the motors just see the chairs, and they don't see you. So the camera's right here. So if you'd like, you can also sit here, which is more visible to the camera.

00:06:04.000 --> 00:06:06.000
Yeah.

00:06:06.000 --> 00:06:22.000
Okay. Yeah, what was I saying? Yeah, so their assumption was something, like, there would be neuron for each of these three components of the sentence, and if… and they would all fire simultaneously for a sentence like John loves Mary, and that will create some sort of, like, a combined representation of this particular sentence

00:06:22.000 --> 00:06:26.000
But if we go by that particular

00:06:26.000 --> 00:06:37.000
cash flow, or that particular form of computation, then the neural network won't be able to distinguish these two different sentences, which is, like, John loves Mary versus Mary loves John, because

00:06:37.000 --> 00:06:51.000
Those are just neurons, and they would get activated, and their combined representation would be similar, and hence, Maul will not be able to distinguish these two forms of sentences. So that was their argument for saying that neural network is not the correct form of

00:06:51.000 --> 00:06:58.000
Correct framework to study, computational intelligence

00:06:58.000 --> 00:07:11.000
That's why I think it becomes important. I mean, now we know that models can do it. It's a pretty simple task. So I think it becomes important to fundamentally understand how the neural networks are doing it so that we can better

00:07:11.000 --> 00:07:18.000
You can put the argument against this particular point that they were making.

00:07:18.000 --> 00:07:33.000
And the second point being… second reason for why understanding binding is more important is from the capability perspective, I think lately there have been some work which has shown that

00:07:33.000 --> 00:07:40.000
The failure modes, a number of failure modes in various neural networks can be attributed to their binding capabilities

00:07:40.000 --> 00:07:42.000
That they

00:07:42.000 --> 00:07:53.000
They are not able to perform a particular task beyond a certain limit, because there is a limitation of the binding capability within their models.

00:07:53.000 --> 00:08:05.000
So, if we can fundamentally understand that, then probably we might be able to overcome those limitations by making fundamental, like, mechanistic updates, or maybe just coming up with

00:08:05.000 --> 00:08:15.000
Something like, tracer decompiled components, which will overcome those limitations

00:08:15.000 --> 00:08:18.000
So those are the two main, main, I think, reasons for studying binding

00:08:18.000 --> 00:08:26.000
And… and one of the other reasons, which is sort of, like, what David was alluding to, when you have

00:08:26.000 --> 00:08:32.000
If you need to understand how a neural network is in, like

00:08:32.000 --> 00:08:35.000
Processing any algorithm

00:08:35.000 --> 00:08:41.000
You need to be able to understand how it is keeping track of the bind, the variables that are

00:08:41.000 --> 00:08:43.000
present in that algorithm.

00:08:43.000 --> 00:09:00.000
It's just, like, variables are part of every algorithm that we study. If we don't understand how does that variable is being tracked within a particular algorithm, it becomes really difficult to understand any kind of algorithm that the model might be

00:09:00.000 --> 00:09:09.000
One question. Just on the practical side, do you think it's mostly software language and mostly a vision problem, or is it also still a problem on language? I think it's both

00:09:09.000 --> 00:09:17.000
So, the second figure that I have here, I think they showed that as you increase the number of

00:09:17.000 --> 00:09:21.000
objects in a particular figure

00:09:21.000 --> 00:09:37.000
The tracking capability decreases. So if you ask different kinds of questions, or if you just ask the model to caption a particular image that decrease, like, that performance degrade, and I think they were attributing that degradation in the performance

00:09:37.000 --> 00:09:45.000
to the binding limitations of the model. So that's mother vision side. On the language side, are there still like big problems

00:09:45.000 --> 00:09:51.000
Yeah, for instance, I have here the reversal cars. I mean, theory of mind could also be considered as

00:09:51.000 --> 00:09:56.000
limitation because of the binding one

00:09:56.000 --> 00:09:58.000
Okay. Yeah

00:09:58.000 --> 00:10:02.000
Okay, and since it is a fundamental

00:10:02.000 --> 00:10:19.000
capability that we would definitely want to understand. There have been a few existing works in the literature we have, which has tried to understand it. I'm not going to go into complete detail on all of them, but I think the main point here is that the model is creating something like a pointer

00:10:19.000 --> 00:10:29.000
Pointer signal, pointer information to bind values with… to bind variables with their corresponding values.

00:10:29.000 --> 00:10:44.000
What I mean by that, or what I mean by pointer mode mechanistically is, so let's say if you have two like this kinds of prompt, like banana is in box A, the pen is in box B, and blah blah blah, and then you finally

00:10:44.000 --> 00:10:51.000
have this particular sentence, box P contains the, and the answer is 10. And the model can solve it

00:10:51.000 --> 00:11:04.000
When I say the model is sort of solving this particular task using a pointer, what I mean is, it is creating a sort of representation in its internal computation, which is

00:11:04.000 --> 00:11:06.000
Encoding the

00:11:06.000 --> 00:11:10.000
like an abstract position of the correct answer

00:11:10.000 --> 00:11:15.000
So here, the pen is

00:11:15.000 --> 00:11:31.000
The reason, Pen, is the answer here is because the model is first figuring out that the second object in the prompt is the correct one. So I need to find the correct object of the prompt, and then sort of it fetches it

00:11:31.000 --> 00:11:48.000
In previous works have shown that that actually happens. So there are internal representations in the model, which when you patch from let's say, this particular counterfactual sample to this particular clean sample, the final output of the clean changes from apple to book

00:11:48.000 --> 00:11:56.000
Which shows that the model is actually representing a sort of like abstract positional

00:11:56.000 --> 00:12:01.000
Why not get a second object of a particular of the counterfactual prompt

00:12:01.000 --> 00:12:03.000
Is that clear

00:12:03.000 --> 00:12:05.000
Okay.

00:12:05.000 --> 00:12:10.000
Okay, now coming to the… so that was the previous work.

00:12:10.000 --> 00:12:16.000
In this work, what we are trying to do is trying to understand this binding mechanism in the reversible sentences.

00:12:16.000 --> 00:12:32.000
Now, what do I mean by reversible versus irreversible sentences is so this is on the left-hand side is like a typical example of irreversible sentence. When the nouns of a particular sentences are not reversible, that's just called irreversible sentence. So here, you cannot… we cannot really say, right?

00:12:32.000 --> 00:12:35.000
Paris lives in Alice, that doesn't make sense.

00:12:35.000 --> 00:12:40.000
Whereas on the right-hand side, that's a typical example of reversible sentences

00:12:40.000 --> 00:12:53.000
So we have, like, multiple, instances of a particular type, which is the person type, and those are reversible with each other. Alice introduced Bob to Charlie is still a valid sentence.

00:12:53.000 --> 00:13:10.000
So, in this work, we are primarily focused on understanding the binding in the reversible sentences, and because most of the previous work… I would say all of the previous works on binding have looked only into the irreversible sentences

00:13:10.000 --> 00:13:15.000
Yeah, and why do we care about reversible sentences? I think

00:13:15.000 --> 00:13:23.000
Again, two reasons, I think, historically, yeah, when I was giving you the argument of, historically, people were interested in binding because of

00:13:23.000 --> 00:13:34.000
That neural network was a symbolic AI argument. People primarily use reversible sentences or reversible sentences, examples to argue against neural networks

00:13:34.000 --> 00:13:39.000
And the second reason being, I think irreversible

00:13:39.000 --> 00:13:48.000
He said, I would say an easy case for mining, because all you need is just a pointer. You don't really have to consider the type information

00:13:48.000 --> 00:13:52.000
But in the reversible case, you do need to consider type information

00:13:52.000 --> 00:13:55.000
So that's what I think we are trying to

00:13:55.000 --> 00:14:02.000
extend our understanding of the binding from, yeah, from irreversible to the reversible sentences

00:14:02.000 --> 00:14:13.000
That was a question, I get. Is it also having pointer and a data type? Yeah, that's the part of the project. That's the rest of the project.

00:14:13.000 --> 00:14:20.000
So for the irreversible ones, the model use just a pointer, also the semantic

00:14:20.000 --> 00:14:25.000
Together, or for the irreversible, it only uses the appointment

00:14:25.000 --> 00:14:34.000
Yeah, it's the rest of the process. Right now, I'm only explaining what I'm… what I'm doing, and why I'm doing it.

00:14:34.000 --> 00:14:38.000
Wait, just to understand the scope of reversible sentences

00:14:38.000 --> 00:14:41.000
So reversible sentences, they

00:14:41.000 --> 00:14:56.000
Don't need to be meaningful. Like, meaningful as in, like, it should not necessarily mean the same thing. Like, you know how certain things, if you were to swap, you might still get the same correct like like if

00:14:56.000 --> 00:15:01.000
I can't think of another sentence right now. Yeah, fairly. Yeah

00:15:01.000 --> 00:15:20.000
Then it would mean the same thing. Yeah, like, so we don't… we don't care about them meaning the same thing, ultimately, when they're reversible. It's just only the types you care about, like, because in this case, in these two sentences, they don't… like, while the swap is sensible, as you mentioned, it means something else. So, like, we don't… Yeah, the meaning of the sentence would be different

00:15:20.000 --> 00:15:22.000
But it is still sensible

00:15:22.000 --> 00:15:28.000
Okay, all right. But in the left-hand side, it's not sensible

00:15:28.000 --> 00:15:39.000
So we just, we're basically taking meaning out of context and just looking at like type based sensibility

00:15:39.000 --> 00:15:45.000
Like, what if… so in this case, it means something else. What if you had like

00:15:45.000 --> 00:16:03.000
So, I guess, is there anything in between between irreversible and reversible? Not that I know of. I think you're thinking of like Alice met Bau and Bob met Alice are like reversal, but it means the same thing. Would that be in your dataset here? Yeah, that would be considered as reversible. I think the

00:16:03.000 --> 00:16:07.000
Yeah, if you see the subtitle here,

00:16:07.000 --> 00:16:23.000
If swapping the nouns make sense, if they are swappable, then that's sort of… that's a reversible sentence example. If the nouns are not swappable, then they are not… they are basically reversible sentences

00:16:23.000 --> 00:16:35.000
Do you think it's because of symmetric properties that the model is able to, like, have a reversible sense, or is it something else

00:16:35.000 --> 00:16:43.000
Right now, I'm just trying to define what is reversible and irreversible. I'm still not gotten into how the model is

00:16:43.000 --> 00:16:53.000
Yes, my… so far, I've only talked about… tried to convey the information, what each of them mean. Oh, sorry, that's just a question in the chat. Why not study

00:16:53.000 --> 00:16:58.000
passive sentences? Is there a reason the meaning is not significant to study?

00:16:58.000 --> 00:17:08.000
So, active to passive,

00:17:08.000 --> 00:17:23.000
I think those are part of the reversible sentences. I would say passive, I would say, I will… I will show in my later slides that there are different types of, we could have, like, different types of reversible sentences, active and passive are sort of one… I mean, two types of them

00:17:23.000 --> 00:17:30.000
So, yeah, we do study them, I think you're gonna see more details in the later slides

00:17:30.000 --> 00:17:33.000
Okay.

00:17:33.000 --> 00:17:45.000
One more point, like a concept that you should know, thematic role. What does thematic role mean? This is again a term coming from linguists linguistics.

00:17:45.000 --> 00:17:56.000
So, thematic rules is basically associated with a noun, and so basically each noun will have a particular thematic rule depending on its function in a particular sentence.

00:17:56.000 --> 00:18:02.000
So, for instance, here in the active form, we have Anna warned Sam about Grace

00:18:02.000 --> 00:18:07.000
So, Anna is the agent, because Anna is the person who is doing the action

00:18:07.000 --> 00:18:16.000
That's why it is the agent. And, Sam is sort of the person who is being actioned upon, so that's recipient.

00:18:16.000 --> 00:18:18.000
And similarly, the grace.

00:18:18.000 --> 00:18:22.000
And these are position independent. So

00:18:22.000 --> 00:18:35.000
So, these two sentences, this is an active form of particular sentence, this is a passive form of particular sentences, but the thematic roles of both of these sentences is still the same

00:18:35.000 --> 00:18:40.000
And I still is the agent in both of them

00:18:40.000 --> 00:18:42.000
Okay.

00:18:42.000 --> 00:18:54.000
Yeah, so these are sort of, like, the broader questions that we have answers for in this particular project. Do the models, or at least the language model, understand

00:18:54.000 --> 00:19:08.000
And encode thematic roles. And if they are being used for doing binding and is the mechanism for variable binding generalizable across different languages

00:19:08.000 --> 00:19:15.000
Okay, so those were the background stuff. Now I'm starting with the actual experimental setup

00:19:15.000 --> 00:19:21.000
So, I studied… so far, I have only reserved for this particular single model, which is Gemma 20

00:19:21.000 --> 00:19:23.000
7B instruct

00:19:23.000 --> 00:19:33.000
And on the right-hand side, this is the behavioral performance of this particular model on different types of reversible sentences

00:19:33.000 --> 00:19:40.000
So, I have, like, 5 different types of reversible sentences. So, if you look at this particular task.

00:19:40.000 --> 00:19:50.000
Joey informed Alice about Frank, that's a typical example of active. Kate was reminded about Karen by Philip, that's a typical example of passive

00:19:50.000 --> 00:19:59.000
And it was well who informed James about Susan. That's a typical example of cleft agent. And I have two other forms as well.

00:19:59.000 --> 00:20:03.000
And the model can

00:20:03.000 --> 00:20:07.000
And this is basically the task. I have… I'm defining what does

00:20:07.000 --> 00:20:17.000
Each of the three thematic role means in the instruction, and then I'm asking the model to give me the thematic role of it. Thematic role present in a particular sentence

00:20:17.000 --> 00:20:21.000
Then I have the sentences, and then finally the model is supposed to give me the answer.

00:20:21.000 --> 00:20:31.000
So for each of these five types of sentences, the model can answer all the three different thematic rules

00:20:31.000 --> 00:20:35.000
pretty… pretty well

00:20:35.000 --> 00:20:37.000
Is that year

00:20:37.000 --> 00:20:44.000
It's just behavioral evaluation so far

00:20:44.000 --> 00:20:45.000
Okay.

00:20:45.000 --> 00:20:49.000
Agent and recipient. Agent, recipient, and topic.

00:20:49.000 --> 00:20:52.000
Those are the three

00:20:52.000 --> 00:20:53.000
Thematic rules

00:20:53.000 --> 00:21:00.000
Because the sentences only have 3 nouns, 3 persons.

00:21:00.000 --> 00:21:06.000
Does it matter if you only do entrance 1,

00:21:06.000 --> 00:21:17.000
Does it matter if… what is doing sentence one mean? So it's for all questions you asked it to identify who the agent is in sentence one

00:21:17.000 --> 00:21:27.000
Does that matter? What does that… does that not matter? What do you also ask in sentence to in sentence? Yeah, I ask everything. Yeah, yeah, yeah. This behavioral performance is across all the three sentences

00:21:27.000 --> 00:21:30.000
That is just one example

00:21:30.000 --> 00:21:33.000
Okay.

00:21:33.000 --> 00:21:36.000
So the first thing that I did was I did probing

00:21:36.000 --> 00:21:38.000
I…

00:21:38.000 --> 00:21:40.000
train two probes

00:21:40.000 --> 00:21:46.000
on each of the 9% token names that I have in my sentences

00:21:46.000 --> 00:21:57.000
Well, James, Susan, Joey, Alice Frank, Kate, Karen, and Philip. So, on each of these 9 tokens residual stream, I trained 2 probes

00:21:57.000 --> 00:22:10.000
First proof is to predict, the sentence index in which this particular token is present. So the input to the probe is the residual vector of each of these or one of this person

00:22:10.000 --> 00:22:14.000
And the output is 3 class clash, like, 3 class vector

00:22:14.000 --> 00:22:18.000
Which needs to predict which particular

00:22:18.000 --> 00:22:21.000
sentence, this particular person is in.

00:22:21.000 --> 00:22:26.000
So that's the first pro. Second proof is to predict the thematic role of a particular person.

00:22:26.000 --> 00:22:38.000
Input is again the residual vector, output is 3 class classifier predicting agent, recipient of the topic, whichever this particular person, is representing in a particular sentence

00:22:38.000 --> 00:22:46.000
And there was… yeah, this is basically the result. The top one is for the thematic rule, and the second one is for the sentence index.

00:22:46.000 --> 00:22:52.000
Content Index is actually… sentence index information is present in all the percent token

00:22:52.000 --> 00:22:56.000
pretty much from the second year, like, after the first layer.

00:22:56.000 --> 00:23:00.000
That's what this particular…

00:23:00.000 --> 00:23:07.000
Line graph is showing. But the thematic rule results are a little bit more interesting.

00:23:07.000 --> 00:23:15.000
What I found was this thematic rule information was present in each sentence only at the second and the third person

00:23:15.000 --> 00:23:18.000
It is not present in the first person

00:23:18.000 --> 00:23:24.000
So what does that mean is the model does not really encode the thematic rule of will

00:23:24.000 --> 00:23:33.000
But it does include the thematic role of, well, sorry, James and Susan. Similarly, here it won't encode the thematic role of Joey, but it will encode the thematic role of Alice and Frank.

00:23:33.000 --> 00:23:34.000
I simulated for the last one

00:23:34.000 --> 00:23:40.000
Have you tried defining the thematic rules after the sentences?

00:23:40.000 --> 00:23:46.000
And then do the proving, and then what happens? Yeah, that's the second setup, I think also yeah

00:23:46.000 --> 00:24:03.000
So if it's a three-way classification, does that mean that the activation space is somehow either similar to the second person and the third person for the first person? That's why it's confused, right?

00:24:03.000 --> 00:24:07.000
Sorry, say a little bit more. Sorry, so if you go to the next slide

00:24:07.000 --> 00:24:17.000
So in the top graph there are the three semantic rules, right? It's only a three-way classification for your activations

00:24:17.000 --> 00:24:29.000
So does that mean, like, that blue curve basically said whenever like your probe sees like, I think, agent token either classified as recipient and topic and gets confused

00:24:29.000 --> 00:24:44.000
Okay, yeah, it's not really whether it sees any particular thematic role. Those three colors are sort of representing whether I'm probing on the first person of a particular sentence, second person of a particular sentence, or third person of a particular sentence.

00:24:44.000 --> 00:24:53.000
So what that blue line is showing is, if you do the probing on the first person of a particular sentence, then that particular first person does not really encode a thematic row

00:24:53.000 --> 00:25:07.000
Oh, oh, I understand. Oh, okay. So it's not, it's not based on whether they're correctly identifying agent or it's just a first or second, third person. Yes

00:25:07.000 --> 00:25:11.000
So, what does that mean? Let's say if you consider this third sentence

00:25:11.000 --> 00:25:22.000
If you do the probing on the CAI token residual stream, if you basically ask what is its particular thematic role, you won't be able to decode it linearly

00:25:22.000 --> 00:25:26.000
But you would be able to decode the thematic role of carry and fella

00:25:26.000 --> 00:25:38.000
That's what I'm trying to say. I mean, it makes sense, right? Like, you can't know the role, like… It's too early to know the role. You would not be able to decode it even non-linearly, or any… Yeah, yeah.

00:25:38.000 --> 00:25:53.000
I never tried non-ED, but yeah, the fact is, the model has not really processed the work. And since the thematic rules are super dependent on the verb, it cannot really figure out the thematic role of the first person

00:25:53.000 --> 00:26:03.000
I have a question. Why did you choose to design the task like this? So you only ask about the first sentence? Oh, no, I asked about all the three sentences. This is just an example

00:26:03.000 --> 00:26:20.000
I ask about all the three thematic rules across all the three sentences. But you're going to shuffle it later, like, so, like, randomly you'll ask about it. Yeah, this is a normal thing that you… Why did you go three sentences and just ask about one of them? Yeah, why have three sentences? Because I want token output.

00:26:20.000 --> 00:26:29.000
I mean, what you could have… could you have done your whole study with just one sentence at a time? Yeah.

00:26:29.000 --> 00:26:37.000
Okay. I thought that there was like an underlying reason for that. But in that case, you wouldn't really see any pointer information.

00:26:37.000 --> 00:26:40.000
So you want to be able to disambiguate

00:26:40.000 --> 00:26:43.000
Pointers

00:26:43.000 --> 00:26:47.000
With the thematic group. I think it will become a little bit clearer in the next slide.

00:26:47.000 --> 00:26:56.000
I don't understand. Yeah, let me just to piggyback on that, I know you said you wanted it to be one token, but like, is there a reason why

00:26:56.000 --> 00:27:06.000
Like, kind of like with precision versus recall, just ask who are all the ones that can act as agents? Like, so we could give all three of them

00:27:06.000 --> 00:27:16.000
Core is just a different task, I just didn't do that. Yeah, you could do multiple token thing task as well, but I was just

00:27:16.000 --> 00:27:19.000
Yeah, I wanted to have a single token.

00:27:19.000 --> 00:27:28.000
But, okay, so what… regarding Ada's point. So, if you do not have

00:27:28.000 --> 00:27:30.000
Multiple sentences

00:27:30.000 --> 00:27:36.000
So this is sort of like the causal model that I have. The model is using for solving the task as the last token

00:27:36.000 --> 00:27:42.000
So, if you only have

00:27:42.000 --> 00:27:44.000
Let's say

00:27:44.000 --> 00:27:50.000
The task is something like, there is only one sentence, it was Will who informed James about Susan.

00:27:50.000 --> 00:27:53.000
And you only has determined which of the named person is the agent.

00:27:53.000 --> 00:27:59.000
Right? And the model will not encode any thematic rule for well

00:27:59.000 --> 00:28:07.000
But it will encode the medical of James and Susan. And then finally, when you ask the answer, it will

00:28:07.000 --> 00:28:10.000
It won't use any pointer information.

00:28:10.000 --> 00:28:13.000
It will just use a thematic role information

00:28:13.000 --> 00:28:18.000
The fact that,

00:28:18.000 --> 00:28:31.000
Yeah, let's say instead of agent, you ask about the recipient, okay? And the James will directly say that I am an agent, because it knows that it is an agent, and model will just use that piece of information to fetch James.

00:28:31.000 --> 00:28:42.000
It doesn't really need to use any partner information that I need to find the recipient of information? It's more like sentence index.

00:28:42.000 --> 00:28:47.000
Whether I need to find an information of a particular sentence

00:28:47.000 --> 00:29:00.000
Yeah, that's pretty much what I'm calling as pointer. But if you ask what is from the first sentence, it means a pointer of first sentence. Yes, and I wanted that information. So that's why I kept multiple sentences

00:29:00.000 --> 00:29:16.000
So, the synthesis here are, like, have a similar role to the boxes in the other example. Yes. Like, you have a box with three, and a box with three things. It's like, okay, look at box two, what is the age? Look at the second sentence in that, yeah

00:29:16.000 --> 00:29:37.000
Nikhil, I've been… when I explained your previous work to people, I've started to call it temporal information because it's about what time… I think it might be an interesting contrast with your theme, maybe that's something to consider. Temporal? I say temporal information? Yeah, this is an old-fashioned way of looking at sequences. It's like, oh, like the old things earliest

00:29:37.000 --> 00:29:46.000
The newest things are latest in a sequence and with the order of things happen is what time order they happened in in the story

00:29:46.000 --> 00:29:52.000
And so I've been… I've been explaining it as a temporal reference. The thing that we've been calling pointers

00:29:52.000 --> 00:30:07.000
I think that clarifies, because the pointer is very generic, right? Basically to pointer from your slides. I've changed my slides. Let's call it temporal now. The temporal pointers are just temporal

00:30:07.000 --> 00:30:23.000
Yeah, temporal pointer is a temporal reference, yeah. But okay, one second. But okay, if you didn't have that, you just had one sentence, what kind of information would you not have in this experiment? So here, if you look at this particular example

00:30:23.000 --> 00:30:35.000
What we are saying is, at the last token, the model is creating a representation that I need to fetch information from the second sentence for this

00:30:35.000 --> 00:30:37.000
In this counterfactual sample

00:30:37.000 --> 00:30:38.000
Is that clear?

00:30:38.000 --> 00:30:42.000
Do you agree with that or not? In this example, yeah.

00:30:42.000 --> 00:30:49.000
Okay, and then we are patching it here, that's why… and we are seeing that the output changes from

00:30:49.000 --> 00:30:53.000
Apple to book. So now switch to the other one, with the three rolls.

00:30:53.000 --> 00:31:10.000
But couldn't that be a long sentence? It could just be like, you have all the information of the box and be like, so the box so like one sentence. Yeah, so I think that is sort of like an open question. What actually counts a particular sentence? You have like a

00:31:10.000 --> 00:31:13.000
Instead of comma there, if you have

00:31:13.000 --> 00:31:26.000
If you just have, like, the banana is in box A and the pen is in box B, and the watch is in box H, I would still assume that if you do the batching that then you still get booked there. Right. So what

00:31:26.000 --> 00:31:38.000
I think, you know what, I bet it would be clear when you actually described the experiment. Instead of describing an experiment without your slides, maybe you should go on and just describe what you've done.

00:31:38.000 --> 00:31:46.000
But this is how, this is the cosmo model that I have for how the model is solving the task at the last token.

00:31:46.000 --> 00:31:56.000
What it is you are doing is, it still has that syntax index, which is sort of like equivalent to the pointer information that I showed in that previous slide.

00:31:56.000 --> 00:31:58.000
In addition to that

00:31:58.000 --> 00:32:03.000
It is… it is using this thematic role variable

00:32:03.000 --> 00:32:08.000
And an additional variable called fetch first

00:32:08.000 --> 00:32:13.000
So when you ask it to, let's say, give me the agent of a particular sentence.

00:32:13.000 --> 00:32:23.000
It will try to use this role variable. It will use this role variable and the sentence index variable, sort of combine these two pieces of information to actually fetch that particular

00:32:23.000 --> 00:32:28.000
It's the person with that particular thematic role in that particular sentence.

00:32:28.000 --> 00:32:36.000
But the problem with that argument is, I just showed you that the first person does not really encode the thematic row

00:32:36.000 --> 00:32:48.000
So, if the answer is the first person, which is Will here, the model will still get the correct answer, but it cannot really use the thematic rule, because Will does not really know what thematic role it encodes.

00:32:48.000 --> 00:32:51.000
So that's why I have this additional variable called fetchFirst.

00:32:51.000 --> 00:33:00.000
So the model actually does sort of, like, a conditional assignment here. If

00:33:00.000 --> 00:33:05.000
The sentence that you want to fetch the thematic, like.

00:33:05.000 --> 00:33:10.000
That you want to fetch information from is of a particular sentence type

00:33:10.000 --> 00:33:11.000
Then

00:33:11.000 --> 00:33:16.000
either you use the thematic rule or the fetch-first information

00:33:16.000 --> 00:33:22.000
It will become a little bit more clear when I show the intervention itself. So this is the intervention

00:33:22.000 --> 00:33:24.000
It's not clear at the back, right?

00:33:24.000 --> 00:33:26.000
Can you see the text

00:33:26.000 --> 00:33:29.000
No.

00:33:29.000 --> 00:33:33.000
How do I speak your funds?

00:33:33.000 --> 00:33:36.000
Oh, feathers.

00:33:36.000 --> 00:33:42.000
So, this is the counterfactual for, aligning the role variable

00:33:42.000 --> 00:33:44.000
In the Cozuma

00:33:44.000 --> 00:33:46.000
Okay.

00:33:46.000 --> 00:33:52.000
Now, in the counterfactual sample, I'm asking about the agent of the first sentence, which is well

00:33:52.000 --> 00:33:57.000
In my clean prompt, I'm asking about the topic of the second sentence, which is Karen.

00:33:57.000 --> 00:34:00.000
Okay? Now…

00:34:00.000 --> 00:34:10.000
I'm doing the patching at the last token, so assume that there is some representation in the residual vector of the last token, which is encoding this role information

00:34:10.000 --> 00:34:13.000
Now, if you pass that representation

00:34:13.000 --> 00:34:20.000
from this particular sentence to this particular sentence, what do you think the final output would be?

00:34:20.000 --> 00:34:22.000
Philip.

00:34:22.000 --> 00:34:33.000
Yeah, who are voting for Philip

00:34:33.000 --> 00:34:37.000
Okay, let's see, stick with me.

00:34:37.000 --> 00:34:40.000
2 are for Anna

00:34:40.000 --> 00:34:59.000
Agent of the first of the federal residual vector or anything like that. Just assume that there is some representation encoding the scroll

00:34:59.000 --> 00:35:02.000
And I'm touching in the role representation.

00:35:02.000 --> 00:35:06.000
Topic is Greece, right?

00:35:06.000 --> 00:35:07.000
Okay

00:35:07.000 --> 00:35:18.000
We're swapping out topic for the second sentence, right? Oh, swapping with… I wrote press. Okay, let me just go ahead. The answer is valid.

00:35:18.000 --> 00:35:24.000
Yeah. But the rational there, let me

00:35:24.000 --> 00:35:42.000
The rationale here is, in this particular counterfactual, the model is encoding this particular information that I need to find the agent of the second sentence. So there is an agent, and there is a second sentence representation. And in the clean, there is this topic representation and the second sentence information

00:35:42.000 --> 00:35:57.000
Signal. So, if I patch in only the rule information from the first one to the second one, then in the second one, the model will start thinking that it does not really have to find the topic of the second sentence, and instead it needs to find the agent of the second sentence, which is

00:35:57.000 --> 00:36:03.000
That is the rationale of this intervention.

00:36:03.000 --> 00:36:07.000
How do you isolate the role? Like, from the activation?

00:36:07.000 --> 00:36:09.000
You do this intervention

00:36:09.000 --> 00:36:15.000
And if the intervention work… I mean, if you do some kind of matching

00:36:15.000 --> 00:36:21.000
And if you see that the output of the clean prompt changes from

00:36:21.000 --> 00:36:26.000
Karen to Philip, then the thing that you batch should represent that role information.

00:36:26.000 --> 00:36:43.000
But, the last student should have both the pointer to the first index and the thematic rule, right? Yes. But then only the second example is only using the thematic rule, and it's not using the positional of the first sentence, because it's still looking at the second

00:36:43.000 --> 00:36:50.000
Yes. Why is the physician not getting overrated?

00:36:50.000 --> 00:36:52.000
There is two way to answer that

00:36:52.000 --> 00:36:54.000
First

00:36:54.000 --> 00:37:04.000
Okay, so think of that each of these two pieces of information is present in a specific subspace, and those are independent of each other, and I can patch a specific subspace

00:37:04.000 --> 00:37:08.000
And that's why you can get Philippa's logo.

00:37:08.000 --> 00:37:22.000
But right now, I'm not even talking about representation at all. I'm just talking about the high-level causal model, which is like, this is your perception of what is going on. The model has this particular variables

00:37:22.000 --> 00:37:39.000
interconnected in this particular way and so if you run a particular prompt on your high-level causal model, each of those variables will take this particular value. And now, if you do the intervention on that high level causal model, this is the expected final output. If you do the intervention on

00:37:39.000 --> 00:37:41.000
on a specific variable

00:37:41.000 --> 00:37:44.000
That is sort of like the framework here.

00:37:44.000 --> 00:37:47.000
Okay, that makes sense. Yeah. Okay.

00:37:47.000 --> 00:37:52.000
Oh, he got the question. Nikhil, how do you come up with psychology

00:37:52.000 --> 00:38:03.000
A summer of research. Did you start with that? You kept experimenting and then

00:38:03.000 --> 00:38:18.000
Yeah, it's a lot of you to try. You're probably missing the pitch first thing at first. Yeah, yeah, yeah. Right. I didn't expect it. I don't. I think Eric has seen those bigger grid of fashion plots

00:38:18.000 --> 00:38:20.000
Tried a lot of stuff. Yeah.

00:38:20.000 --> 00:38:32.000
So, yeah, this was not the first thing that I came over. One suggestion, maybe something that would make it clear here that you're talking about just the causal model, is you could have, like, two causal models and, like, show the patch between them

00:38:32.000 --> 00:38:39.000
Like, you, like, take two sentences and be like, okay, we're populating the variables with these sentences

00:38:39.000 --> 00:38:44.000
into our causal model, and then if you, like, swap these, then it might be more clear that like

00:38:44.000 --> 00:38:49.000
like, what should a counterfactual alpha be when I am patching

00:38:49.000 --> 00:38:53.000
Like, because you have, like, I think the text one is nice, but like

00:38:53.000 --> 00:39:01.000
You could have the parallel patch in the like that's an option.

00:39:01.000 --> 00:39:10.000
Yeah, like, implementing the causal model on the sentence might clarify what the causal model is doing exactly. Okay. Yeah

00:39:10.000 --> 00:39:13.000
One can create such an animation

00:39:13.000 --> 00:39:18.000
So, what if you patch to a sentence ID that does not have the rule

00:39:18.000 --> 00:39:22.000
which you won't answer. Like

00:39:22.000 --> 00:39:28.000
Sentence ID which does not have the rule. Yeah. Let's say, like, there is no topic in a sentence

00:39:28.000 --> 00:39:34.000
And you patch the sentence ID to a sentence that doesn't have a topic

00:39:34.000 --> 00:39:37.000
So what you're saying is

00:39:37.000 --> 00:39:44.000
If the second sentence does not have an agent, what would be the final output?

00:39:44.000 --> 00:39:50.000
And maybe I'm… Yeah, for example.

00:39:50.000 --> 00:39:56.000
Doesn't your causal model suggest that it will be fetch first?

00:39:56.000 --> 00:40:01.000
It's not

00:40:01.000 --> 00:40:09.000
Customers may suggest. Oh, no, yeah, you're right. Yeah

00:40:09.000 --> 00:40:19.000
But I've not tried that. But the causal model that I have would say that fetch the first one or the second sentence

00:40:19.000 --> 00:40:25.000
Because, you see this… this sort of diamond? This is sort of a condition.

00:40:25.000 --> 00:40:29.000
It's like, if a particular thematic rule is present

00:40:29.000 --> 00:40:36.000
is encoded in the sentence that I'm interested in, then use the rule

00:40:36.000 --> 00:40:44.000
But if that particular thematic rule is not present in a particular sentence type, then use the fetch first.

00:40:44.000 --> 00:40:54.000
Actually, you would expect, yeah, so that's a good one, whatever. That's a good actually experiment. I'm not sure how would you create

00:40:54.000 --> 00:41:03.000
Maybe it is possible. I mean, there's also famous linguistic sentences that have no agent like it rains today. We'll just output none

00:41:03.000 --> 00:41:11.000
It won't output. But yeah, I wouldn't… I'm not sure. It's kind of easy to get rid of the topic

00:41:11.000 --> 00:41:16.000
Yeah, yeah, here you can get rid of the topic. Joey told

00:41:16.000 --> 00:41:20.000
Joey was told about… Joey was told by Philip, that's it.

00:41:20.000 --> 00:41:33.000
And in that case, I would expect the answer causal model that I have would say that the final output should be Joey. Yeah, that is something that is sensible. In the clean prompt itself, would it just say not applicable?

00:41:33.000 --> 00:41:46.000
Like, if you say, what is the topic in the second sentence with that example, would it just say there is no topic? I don't think so. The model will change. So give it well. I would still assume that the

00:41:46.000 --> 00:41:55.000
The token with the highest probability will still be one of the person names, and most likely it won't say null or none, or something like that

00:41:55.000 --> 00:42:02.000
Unless I explicitly ask it in the prompt, in the instruction, which I haven't.

00:42:02.000 --> 00:42:11.000
Okay. That was the intervention for all. This is an intervention for fetchFirst

00:42:11.000 --> 00:42:16.000
One wants to try

00:42:16.000 --> 00:42:28.000
Oh, sorry.

00:42:28.000 --> 00:42:34.000
While voting for Joey

00:42:34.000 --> 00:42:37.000
We're not sure.

00:42:37.000 --> 00:42:47.000
Oh. Yeah, it is why. Why are you not sure?

00:42:47.000 --> 00:43:03.000
Sorry, just the clarification question. So batch first just mean fetching the first person. Oh, I see. Yeah, so you see this console model here. The model is sort of combining either the role or the fetch first information with the sentence index

00:43:03.000 --> 00:43:16.000
To actually create, sort of, like, query vectors for the attention head, which will actually fetch the value. I see. Okay, so I'm confused. So

00:43:16.000 --> 00:43:22.000
So what's the output of fetch first? So what is the value is fetchFirst a boolean

00:43:22.000 --> 00:43:26.000
Or is that a person's name? Think of it as a Boolean.

00:43:26.000 --> 00:43:28.000
I see.

00:43:28.000 --> 00:43:30.000
I see.

00:43:30.000 --> 00:43:33.000
Like, it doesn't it seem like

00:43:33.000 --> 00:43:42.000
You have this other side thing, like form, which would have had to be the thing that you patched in order to switch it to Joe

00:43:42.000 --> 00:43:53.000
So that's the sentence itself in the second sentence here, Joey was told about Karen by Philip, right? That's sort of like the form

00:43:53.000 --> 00:43:55.000
Right?

00:43:55.000 --> 00:43:58.000
Right

00:43:58.000 --> 00:44:10.000
Yeah, I'm a little puzzled. At least to me, the thing that's confusing here is if fetch versus present, it beats out role.

00:44:10.000 --> 00:44:11.000
Like

00:44:11.000 --> 00:44:22.000
So the second… the clean prompt has topic as the role, but when you patch in fetch first, that like ignores or beats out the fact that we had topic

00:44:22.000 --> 00:44:33.000
So Feet First sort of is like another role but it trumps any existing roles, which is like a little confusing

00:44:33.000 --> 00:44:44.000
Is it the way you think about it? I thought that fetch first is kind of like a kind of like a failure mechanism. Like first, if you can't find any role

00:44:44.000 --> 00:44:49.000
Then you trigger Fetchworth. But like you can't find a role, it's topic

00:44:49.000 --> 00:44:58.000
Great. Are we still talking about that thing where the topic is gone. No, this has topic in it.

00:44:58.000 --> 00:45:06.000
So if there is, let's say, specific subspace which is encoding FED first, and if I pass that from counterfactual to clean.

00:45:06.000 --> 00:45:22.000
That subspace will say that, okay, fetch first is true, but there is some other subspace which is still encoding the fact that the role has some value. Yeah, yeah. So my there must be like some gate or something that's like if fetch first is true, then ignore the role.

00:45:22.000 --> 00:45:35.000
How are you, like, how do you know that you're getting fetch first as opposed to, like, other heads that maybe are pointing towards another role?

00:45:35.000 --> 00:45:37.000
Because the output is chewy

00:45:37.000 --> 00:45:41.000
Joey, in this case, is a recipient, right?

00:45:41.000 --> 00:45:43.000
Or

00:45:43.000 --> 00:45:53.000
I mean, it could be… I mean, I could have other sentences where Joey is not recipient, could be agent, it could be recipient, or it could be topic. It will still say the first one.

00:45:53.000 --> 00:45:57.000
Irrespective of what the thematic role of the first person is.

00:45:57.000 --> 00:46:01.000
But regarding that thing

00:46:01.000 --> 00:46:04.000
I… in my head, this is… this is what I have.

00:46:04.000 --> 00:46:10.000
So, in… I think in all the case, there is something… some form of competition going on

00:46:10.000 --> 00:46:15.000
Competition between Role and FeudgeFirst

00:46:15.000 --> 00:46:19.000
And for the cases where

00:46:19.000 --> 00:46:34.000
The thematic rule is actually present in the percent token. Let's say here topic is present on the current token, the role one wins, because the QK of the role is strong there.

00:46:34.000 --> 00:46:37.000
Make sense? So that's right. So the way I read this

00:46:37.000 --> 00:46:40.000
So I think that… I don't think I

00:46:40.000 --> 00:46:48.000
I think I understand the causal model you have in your head. Okay. What I didn't really understand is the diagram you drew

00:46:48.000 --> 00:46:53.000
Right, and so I… I feel like the diagram you drew doesn

00:46:53.000 --> 00:46:59.000
Who doesn't directly diagram the causal model you're describing in words. Okay. To me

00:46:59.000 --> 00:47:03.000
And so it feels like

00:47:03.000 --> 00:47:07.000
Your role is a

00:47:07.000 --> 00:47:10.000
I feel like there's two things

00:47:10.000 --> 00:47:14.000
There's role and then there's the person name who has the role

00:47:14.000 --> 00:47:18.000
That you'll fetch after resolving the roll

00:47:18.000 --> 00:47:22.000
Right? And

00:47:22.000 --> 00:47:24.000
Hmm.

00:47:24.000 --> 00:47:27.000
I see there's this fetch first thing

00:47:27.000 --> 00:47:30.000
Well, let's see

00:47:30.000 --> 00:47:35.000
A little confusing. How would you diagram this?

00:47:35.000 --> 00:47:44.000
So I almost feel like Feucht, since it's like controlling the if, like it belongs in the side. Like when you say there's a form

00:47:44.000 --> 00:47:47.000
And the

00:47:47.000 --> 00:47:52.000
Basically, the condition determines whether you should fetch first

00:47:52.000 --> 00:48:00.000
And then if you fed first, then you ignore the role and then you

00:48:00.000 --> 00:48:09.000
Then you say fetch first instead, and then… or… or if you don't fetch first, then you use the roll or something like that. Like, I feel like

00:48:09.000 --> 00:48:18.000
Feucht first should be on the side instead of the bottom, if that makes any sense. But I don't know. Let's first

00:48:18.000 --> 00:48:27.000
The value of Feet First is decided based on the row. Like, if you can find the row, you just fetch it

00:48:27.000 --> 00:48:37.000
But let's say you can't match with any of the roles. That's what I think. Yeah, so the fetch first when it

00:48:37.000 --> 00:48:40.000
Sorry,

00:48:40.000 --> 00:48:45.000
If you think of it as a Python function, the default value is not true

00:48:45.000 --> 00:48:55.000
I don't think the default value… like, the Python argument, even Python function even takes the fetch first as an argument. Yeah, yeah, yeah. No, I see what you mean. Yeah

00:48:55.000 --> 00:49:02.000
But this diagram still seems like it's a little bit off. I'm not sure, like, the right way to diagram what's going on

00:49:02.000 --> 00:49:04.000
Maybe just write the Python code

00:49:04.000 --> 00:49:15.000
Anyways, I think it's worth, you know, verifying the right diagram here

00:49:15.000 --> 00:49:20.000
But… but yeah, it's cool. But yeah, what he said is basically what I have in my head.

00:49:20.000 --> 00:49:27.000
But it's surprising to me, like, when you discuss it, that there is a materialized

00:49:27.000 --> 00:49:33.000
variable that you can manipulate catch first, because fetch first in this diagram feels like

00:49:33.000 --> 00:49:36.000
It's

00:49:36.000 --> 00:49:50.000
It doesn't seem like the thing that would control whether you're taking the right or the left, right? It feels like the form is the thing that controls whether you're taking the right or the left. And so it's a little weird for you to manipulate one of the leaf variables

00:49:50.000 --> 00:49:58.000
And then say that manipulating that thing causes the if to go a different way, right? Like, that's surprising for this diagram.

00:49:58.000 --> 00:50:05.000
And where do you have a right or left? What right and left are you talking about? The role and

00:50:05.000 --> 00:50:07.000
First

00:50:07.000 --> 00:50:22.000
Maybe it should also be on the left. Yeah. Right. But then maybe also role should be on the left. I don't know, like everything's ending up on the left. I'm not really sure.

00:50:22.000 --> 00:50:25.000
then it can just be, like, the peak condition

00:50:25.000 --> 00:50:27.000
So, let's say this is my

00:50:27.000 --> 00:50:29.000
And

00:50:29.000 --> 00:50:33.000
condition. It's working as one where the input role

00:50:33.000 --> 00:50:35.000
Right?

00:50:35.000 --> 00:50:37.000
And,

00:50:37.000 --> 00:50:38.000
If it is

00:50:38.000 --> 00:50:40.000
Yes

00:50:40.000 --> 00:50:45.000
I kind of like what Arnab was suggesting. It's almost like role

00:50:45.000 --> 00:50:49.000
Combines with sentence form

00:50:49.000 --> 00:50:52.000
To output either roll or nothing

00:50:52.000 --> 00:50:58.000
And then fetch first combines with roller nothing to take whatever is the maximum

00:50:58.000 --> 00:51:02.000
of them either the role survived this

00:51:02.000 --> 00:51:14.000
This filter, and it's a stronger thing, or if it didn't survive, then Feucht Burst is the thing. If that makes any sense. It feels like roll is combining with sentence form

00:51:14.000 --> 00:51:17.000
Does this make sense?

00:51:17.000 --> 00:51:23.000
No. No?

00:51:23.000 --> 00:51:26.000
But

00:51:26.000 --> 00:51:37.000
This one should make sense. What does form do? What does form do? Are you able to intervene on form, for example? I can, but I haven't.

00:51:37.000 --> 00:51:46.000
Okay, so form is… is like the sentence form that you have here. So let's say we are talking about the second sentence here. Yep.

00:51:46.000 --> 00:51:59.000
The structure of that sentence will determine which of the three thematic roles is actually encoded. And it's probably… and is that represented the last token, or do you have to go back to the sentence period to manipulate that thing?

00:51:59.000 --> 00:52:06.000
No, it's at the… it's at the percent tokens which are present in that particular sentence.

00:52:06.000 --> 00:52:10.000
Okay, so what I'm saying is, let's say here the agent value is

00:52:10.000 --> 00:52:11.000
Topic

00:52:11.000 --> 00:52:13.000
Yep, okay.

00:52:13.000 --> 00:52:25.000
Then this particular if statement will look at the three persons in the second sentence. It will check if any of those three person is saying I'm a topic or not

00:52:25.000 --> 00:52:33.000
Right, and that's actually a function of the form. Yes. Right. That's why I have this form as length. Okay, yeah

00:52:33.000 --> 00:52:43.000
It will look for those 3 persons, it will look at those 3 persons, and it will check if any of those 3 person says, I am a topic or not.

00:52:43.000 --> 00:52:47.000
Yep. If any of this 3% says that I'm a topic

00:52:47.000 --> 00:52:51.000
Then, this top thing goes this

00:52:51.000 --> 00:53:01.000
Then that role directly just goes to the next topic. Yes. Find the topic of if none of those three person says I'm topic.

00:53:01.000 --> 00:53:12.000
Then, this fetch first comes into the picture, which is like, okay, nobody's saying that they are the topic. Then, the first… then just find the, like, yeah.

00:53:12.000 --> 00:53:25.000
Does that make sense? I see. So it's really role and sentence form, which is combining and then turning into a variable which is either role or fetchford.

00:53:25.000 --> 00:53:34.000
Maybe that's actually the sentence index shouldn't connect to form somehow? Yeah, yeah, you can connect sentence index with the form.

00:53:34.000 --> 00:53:50.000
Or that this particular condition. Oh, yeah. Yeah, another arc. So what if multiple names have the same topic in the same sentence? What if there are multiple names that have the same

00:53:50.000 --> 00:53:55.000
topic in the same sentence.

00:53:55.000 --> 00:54:02.000
Like, something that breaks this diagram. I'm not sure if you could create such

00:54:02.000 --> 00:54:07.000
I can select one by Karen and Philip.

00:54:07.000 --> 00:54:11.000
Sorry, was told about

00:54:11.000 --> 00:54:23.000
Yeah, so in that case, the clause would be different. The sentence index that I'm defining is sort of in a single clause. When you include an AND, I think that becomes sort of a different clause

00:54:23.000 --> 00:54:28.000
So in that case, the sentence index will be different, but the thematic rule will still remain the same

00:54:28.000 --> 00:54:30.000
Okay

00:54:30.000 --> 00:54:38.000
Third question, in the prompt, original prompt, you did say what's the three rows are, right?

00:54:38.000 --> 00:54:46.000
Probably I didn't get to that yet, but I'm curious if you remove that, if you would find the same causal… Yeah, that's the next setup. But yeah.

00:54:46.000 --> 00:54:58.000
Right now, it knows that it has only three options, right? You tell it has three options. Yeah, so I think in the next setup, I don't really talk about thematic rules at all

00:54:58.000 --> 00:55:02.000
The results there are pretty much the same.

00:55:02.000 --> 00:55:09.000
Yeah, but yeah, can you share me with your current photograph, you're saying

00:55:09.000 --> 00:55:12.000
Bill and agent are not coming to audits

00:55:12.000 --> 00:55:20.000
can speak together, discuss, like, rule of elimination. Will and age.

00:55:20.000 --> 00:55:26.000
The color graph is suggesting that the model doesn't know that will is the agent.

00:55:26.000 --> 00:55:31.000
Even, like, at the last token, like, at the answer call and

00:55:31.000 --> 00:55:41.000
There's no team for that very token. So it's like, sure, I'll just fetch the first token. Yeah, something like, yeah

00:55:41.000 --> 00:55:44.000
Well, like, isn't that

00:55:44.000 --> 00:55:48.000
Taking autoregressive model, it should somehow figure out by

00:55:48.000 --> 00:55:54.000
Like, it was well who informed by the token informed, it should understand that Will is the agent.

00:55:54.000 --> 00:56:06.000
But the thing is, the… the model is fetching… so at the last token… sorry, yeah, at the last token, the later layers are sort of fetching the valid information directly from the will token.

00:56:06.000 --> 00:56:11.000
It's not fetching the will information from, let's say, Informed or the PA2.

00:56:11.000 --> 00:56:16.000
That means those set of heads which are actually fetching in the well information

00:56:16.000 --> 00:56:23.000
So somehow know that no… I mean, somebody should tell it that, okay, you need to fetch the will token, right?

00:56:23.000 --> 00:56:39.000
Yeah. And that piece of information is coming from something like the elimination, sort of, like, analogy. Like if you don't find in the previous set of computation, if you don't find any particular person which says I'm a agent, then fetch the first

00:56:39.000 --> 00:56:50.000
So, do you think if I just change the setup to it, it was built in table? No, I can't pay it. And then I just add the three names

00:56:50.000 --> 00:56:58.000
Okay, by the time the model should know that Bill is agent

00:56:58.000 --> 00:57:10.000
So you're saying you have the same sentence, and then you start a new sentence where you have, well… Yeah, all the three names. All the three names. Yeah, in the second, if you look… if you patch,

00:57:10.000 --> 00:57:19.000
Most probably, if you do the probing of the second, occurrence of any of those three names, that particular residual vector will have its corresponding time.

00:57:19.000 --> 00:57:23.000
Yeah, actually, I've done that. It does, sure.

00:57:23.000 --> 00:57:32.000
Yeah, I had the same question. I would have imagined that the verb with some verb, like informed would at some point encode the agent, but you're saying if you probe the verb

00:57:32.000 --> 00:57:39.000
No, Work will have it. I'm not saying the verb… I mean, if you do the probing on, I think, any token after

00:57:39.000 --> 00:57:50.000
The work, it will, I think, figure out which of the three persons is the agent recipient of the topic. Okay. But the thing is, those won't be causal in nature

00:57:50.000 --> 00:57:58.000
Because in the causal path, the model is directly fetching in the values from that particular percent. So it never tells the verb, basically, never pays attention to the verb

00:57:58.000 --> 00:58:02.000
In order to figure out the

00:58:02.000 --> 00:58:08.000
Agent

00:58:08.000 --> 00:58:22.000
So the least in linguistics, there's also so many theories that the verb is usually the one that kind of, like, is the central one that connects everything. But I guess here the model… Okay, I don't think so. I would say it doesn't it never attends to

00:58:22.000 --> 00:58:30.000
Yeah, so in this project, I've not really looked too much into the circuit stuff. I was doing a little bit of that

00:58:30.000 --> 00:58:43.000
So, this is something that I'm not super sure about. So, my hypothesis is, if the early layers, the model sort of figures out the role in the sentence, and then sort of in the middle layers, it starts looking for that particular role

00:58:43.000 --> 00:58:46.000
In the presence of that particular sentence

00:58:46.000 --> 00:59:01.000
If it finds it, then it's good, it goes up, but if it doesn't find it, I think there are specific heads which attain to the words which actually sort of, I think, writes this fetch first information or fetch first boolean information in the last token. Okay.

00:59:01.000 --> 00:59:14.000
But I'm not super confident about that yet. Is there a fetch first that is literally fetch first from the first sentence itself, like a failure case, like an all cap failure case where

00:59:14.000 --> 00:59:19.000
Maybe it doesn't even know that. It might be.

00:59:19.000 --> 00:59:28.000
Could you quickly explain what, Feuch first in this example is, and why the answer is joined?

00:59:28.000 --> 00:59:35.000
In this one? Yeah, so here, if you look at this new model, new causal model.

00:59:35.000 --> 00:59:40.000
In this particular clean prompt, the role is topic, sentence index is the second one.

00:59:40.000 --> 00:59:54.000
So, both of this piece of information goes to this condition. In addition to this form information. So this condition basically checks if any of these 3 persons

00:59:54.000 --> 01:00:03.000
Which is, joy, Karen, or topic. If any of them says they are topic or not.

01:00:03.000 --> 01:00:09.000
And in this example, without the intervention, there is one person

01:00:09.000 --> 01:00:13.000
Karen, who says I'm an agent? Sorry, I'm a topic

01:00:13.000 --> 01:00:17.000
So, that topic information directly goes here

01:00:17.000 --> 01:00:24.000
Okay.

01:00:24.000 --> 01:00:32.000
So that's the normal causal model. But, here what I'm doing is, I am explicitly patching this

01:00:32.000 --> 01:00:37.000
In the counterfactual, this particular path will be active instead of the top one.

01:00:37.000 --> 01:00:49.000
Okay? Does that make sense? Okay. So what I'm doing is I'm basically patching in this fetch first from the the top one where that particular path is active

01:00:49.000 --> 01:00:56.000
To explicitly make this particular path active in the clean one.

01:00:56.000 --> 01:01:00.000
Okay? And if you make this one active

01:01:00.000 --> 01:01:03.000
The moral is sort of

01:01:03.000 --> 01:01:10.000
So in that case, both the topic and this fetch first will be active, because the topic was active by default.

01:01:10.000 --> 01:01:13.000
And we are explicitly making the fetch voice

01:01:13.000 --> 01:01:16.000
To be active after the intervention

01:01:16.000 --> 01:01:26.000
But in that case, if both of them are active, then the model is giving preference to the fetch person. And that's why, after the intervention, the model

01:01:26.000 --> 01:01:34.000
Attends to and fetch the… the first person of the second sentence.

01:01:34.000 --> 01:01:44.000
Yeah, that was the whole story. So fetch first is always the first token of the sentence. Yes, first person of the first person of the sentence

01:01:44.000 --> 01:01:48.000
So, you got output as a Jew. Yeah.

01:01:48.000 --> 01:01:53.000
Okay, and what if there is a sentence which does not have a rule? So, it's

01:01:53.000 --> 01:02:03.000
Like, by default, that fetchFirst is the, like, true boolean there, but what if you ablate fetchFirst out of that sentence?

01:02:03.000 --> 01:02:05.000
Like, make both of them false.

01:02:05.000 --> 01:02:10.000
roll and February.

01:02:10.000 --> 01:02:14.000
You're saying,

01:02:14.000 --> 01:02:26.000
In the count of actual where the fetch first is actually active, if I somehow ablate out the fetch first, what would happen? Yeah.

01:02:26.000 --> 01:02:30.000
Yeah, I'm not sure what will happen. I think the model might just be confused

01:02:30.000 --> 01:02:38.000
Probably generate fetch fast again. Or it might just give equal probability to all three.

01:02:38.000 --> 01:02:49.000
Do you want to actually move on to the actual meaty stuff, the interventions? Yeah, but yeah, that one I'm not sure. Yeah, we're already over time.

01:02:49.000 --> 01:02:51.000
Okay.

01:02:51.000 --> 01:02:59.000
how do I quote it

01:02:59.000 --> 01:03:04.000
How many more? It's okay that the next ones, I can go pretty quickly

01:03:04.000 --> 01:03:07.000
How do I clear this thing?

01:03:07.000 --> 01:03:19.000
Okay, so these are the interventions for each of the variables I can skip this one. Oh, there is one more important thing.

01:03:19.000 --> 01:03:23.000
So, I had… okay, there's one more thing.

01:03:23.000 --> 01:03:29.000
So, previously I showed the intervention for each of the individual variables

01:03:29.000 --> 01:03:42.000
But, you can also have intervention for this compound variable, which has both the role and sentence index. So if you do the badging for that one, for that particular variable, the output would change from

01:03:42.000 --> 01:03:49.000
In this particular example, topic of the second sentence to the agent of the first sentence, which is Grace.

01:03:49.000 --> 01:03:51.000
Okay.

01:03:51.000 --> 01:03:56.000
Okay, so then what I did was I basically used this intervention to train DAS

01:03:56.000 --> 01:04:07.000
And what I found was I could train the DAS for each of the 3 independent variables, which is role, sentence, index and fetch first

01:04:07.000 --> 01:04:11.000
But when I combined those trained subspaces

01:04:11.000 --> 01:04:16.000
To do the compound intervention, that did not work.

01:04:16.000 --> 01:04:26.000
So what does that mean is, I train the role subspace using DAS. I train the sentence index subspace with the DAS, both of

01:04:26.000 --> 01:04:29.000
For both of them, I have individual interventions.

01:04:29.000 --> 01:04:35.000
And then I sort of combined both the subspaces. So I just concatenated and did the

01:04:35.000 --> 01:04:38.000
And just autonormalize the combined

01:04:38.000 --> 01:04:39.000
subspace

01:04:39.000 --> 01:04:46.000
And now I have this new subspace, the combined one, and I use that on the compound intervention.

01:04:46.000 --> 01:04:49.000
And the red line is

01:04:49.000 --> 01:04:51.000
It's corresponding II

01:04:51.000 --> 01:04:57.000
And that does not work. That way. Sorry, you get the compound intervention

01:04:57.000 --> 01:05:07.000
Like, by training a task from scratch, you take the two things, and then you also make sure that before you combine it

01:05:07.000 --> 01:05:14.000
These two things, Roland sent an index, subspaces, they are orthonormal with each other as well. No, no, no.

01:05:14.000 --> 01:05:30.000
No, you didn't. That's not the objective of that. First, you found the two subspaces separately. Yes, they're not orthogonal. Yeah, they are not orthogonal. So then, if you just add them, it doesn't make sense. Now, so what I did was add these 2 subspaces. I concatenated it

01:05:30.000 --> 01:05:36.000
So if you break exactly so you have a dimensional subspace, you have a B dimensional subspace

01:05:36.000 --> 01:05:38.000
So now you take

01:05:38.000 --> 01:05:46.000
And then now it's A plus B dimensional. That is not orthonormal. Right, but then he just — I orthonormalize it.

01:05:46.000 --> 01:05:49.000
QR? Just to make the basis, yeah, QR, yeah.

01:05:49.000 --> 01:05:56.000
So did you come to me, or I had to download? I concatenated, then you just applied QR on that

01:05:56.000 --> 01:06:02.000
Just to make sure that the columns are orthonormic. Just like apply Gram Smith on it

01:06:02.000 --> 01:06:09.000
What?

01:06:09.000 --> 01:06:11.000
So

01:06:11.000 --> 01:06:21.000
And the idea what's IIA? Intervention accuracy. Interchange Intervention. Now, if the subspace is have some

01:06:21.000 --> 01:06:25.000
You know, geometry where they're like this very close to each other

01:06:25.000 --> 01:06:32.000
Then that dimension where they're just slightly different in that dimension will come up

01:06:32.000 --> 01:06:42.000
in your subspace. But it's really just noise like you could be like like a could be a subspace B could be a subspace.

01:06:42.000 --> 01:06:58.000
But the… but, like, the real subspace that joins them could be in a lower dimension than A and B, because there could be some noise. I'm not trying to find the intersection. I'm trying to find the union. No, I know. But like the the real union space. So like the intersect

01:06:58.000 --> 01:07:01.000
The assumption would be that in the generic case

01:07:01.000 --> 01:07:05.000
the intersection subspace is the null subspace, right?

01:07:05.000 --> 01:07:10.000
Because you've got two generic subspaces you find empirically, they're not going to intersect.

01:07:10.000 --> 01:07:17.000
Yeah, but I think generally they are not, but let's assume that. Right. But then… but then… but they could actually

01:07:17.000 --> 01:07:23.000
Be very close to intersecting in subspace. So you could find like an approximate

01:07:23.000 --> 01:07:30.000
Intersection subspace, and then subtract that from your union. So I guess that's a question, like, is this… is this failing because

01:07:30.000 --> 01:07:51.000
The joint subspace is too vague. He has a solution for it. Oh, you have a solution. Yeah, yeah, not just joint optimization. Oh, you do the joint optimization. But it's just so does the joint optimization tell you that the subspace was too big

01:07:51.000 --> 01:07:54.000
I'm still not sure about the

01:07:54.000 --> 01:08:07.000
The big banana. Like, is this… is the joint optimization for space that you're going to find on the next slide? So you go to the next slide, you can show the joint optimization. What I did was, instead of training one task.

01:08:07.000 --> 01:08:14.000
I… sorry, one dash for each variable. I trained a single shared one for all of them

01:08:14.000 --> 01:08:31.000
I have this matrix where I have, like, let's say, first R columns for one particular variable, next N minus R column for another variable. I use the first R columns for doing the intervention for one variable, next N minus R column for doing the intervention for the next variable.

01:08:31.000 --> 01:08:39.000
And I use all of the columns to do the intervention for the combined variable

01:08:39.000 --> 01:08:55.000
That's the objective here. And then for… yeah, for each of the intervention, we will find one particular loss, and I do combine all the losses and do one gradient descent update on that combined subspace. Is that the boundless task? How do you define R

01:08:55.000 --> 01:09:00.000
Yeah, no, I just do… that's hyperparameter. Yeah

01:09:00.000 --> 01:09:11.000
In the previous slide, why should union, why should the rule and sentence ID be orthogonal

01:09:11.000 --> 01:09:12.000
No, we

01:09:12.000 --> 01:09:24.000
It might be, it might not be. So the general idea is, since these two variables are sort of completely different, they are including different semantic information, and since the residual vector is super large, we'd sort of expect that

01:09:24.000 --> 01:09:28.000
This subspaces might be almost near Othaga

01:09:28.000 --> 01:09:31.000
So on your joint optimization here

01:09:31.000 --> 01:09:40.000
Are you still doing just roll and index, or are you doing roll and index and combined S3 objectives?

01:09:40.000 --> 01:09:45.000
Oh, I see. Does it work if you just do roll and index without the third one?

01:09:45.000 --> 01:09:49.000
It doesn't. Because the queue you're enforcing

01:09:49.000 --> 01:09:51.000
orthogonality

01:09:51.000 --> 01:09:57.000
So if you… so it's still different if you just do roll and index jointly

01:09:57.000 --> 01:10:02.000
But you tried that and it doesn't work. Yeah, it doesn't work. You need to have the joint training

01:10:02.000 --> 01:10:09.000
And my intuition is something like this. I think the subspaces is not a unique thing.

01:10:09.000 --> 01:10:21.000
There is a family of subspaces which would satisfy, let's say, variable A intervention. There is a family of subspaces which will satisfy the variable B intervention.

01:10:21.000 --> 01:10:24.000
When I'm only optimizing for one of them.

01:10:24.000 --> 01:10:35.000
I'm finding one subspace from that family for each of the variable, but their compound… there's no reason for the… their joint or the concatenated one being… being a good one

01:10:35.000 --> 01:10:43.000
But when I'm explicitly adding this particular training loss in the task, I'm finding

01:10:43.000 --> 01:10:53.000
A subspace in both those families, which are sort of at the intersection. And that's why that particular one works, and the previous ones does not. I think that's my high-level intuition.

01:10:53.000 --> 01:10:55.000
Does that make sense?

01:10:55.000 --> 01:10:56.000
No

01:10:56.000 --> 01:10:59.000
I mean, it makes sense to see your intuition.

01:10:59.000 --> 01:11:03.000
Okay. But it's very unintuitive.

01:11:03.000 --> 01:11:17.000
But it's okay, I mean, it's great. I mean, if that's what the experiment is showing, then that's what it shows. Yeah, that is what it's showing, yes, it's interesting.

01:11:17.000 --> 01:11:18.000
Okay

01:11:18.000 --> 01:11:25.000
Did you try training multiple subspaces in each category with different initial seeds?

01:11:25.000 --> 01:11:29.000
And did they end up having sort of different properties in this way, or did they all end up

01:11:29.000 --> 01:11:36.000
like, every time you did a role swap only Das optimization, did it not end up generalizing?

01:11:36.000 --> 01:11:41.000
When you say properties, which properties are you? Like, flip back?

01:11:41.000 --> 01:11:49.000
This one, not this one. The one with the stripes. The one with the stripes? Yes, this one. So if you

01:11:49.000 --> 01:12:02.000
train on just objectives 1 and 2, but not 3, and try that on 100 different seeds. Do all 100 seeds not generalize? I did not try 100, but I tried like three or four sure that did not work

01:12:02.000 --> 01:12:05.000
Okay, yeah.

01:12:05.000 --> 01:12:08.000
Then

01:12:08.000 --> 01:12:12.000
Let's generalize it, let's generalize your 3 or 4 to 100 real quick

01:12:12.000 --> 01:12:19.000
If we were to do this 100 times and none of the times it worked, then flip to the next slide

01:12:19.000 --> 01:12:29.000
Wouldn't that mean that middle section is either really small or non-existent? Yeah, it could be really small. Yeah, I'm not really saying about the size of the family here, I'm just saying there is a family

01:12:29.000 --> 01:12:35.000
I see, okay, that makes sense.

01:12:35.000 --> 01:12:37.000
But it's still unintuitive.

01:12:37.000 --> 01:12:40.000
Can you say your intuition again?

01:12:40.000 --> 01:12:43.000
So… this orange circle

01:12:43.000 --> 01:12:51.000
Think of it as a family of subspaces which will satisfy the rural intervention.

01:12:51.000 --> 01:13:00.000
Blue one is representing the family of interventions, which will satisfy the sentence index intervention.

01:13:00.000 --> 01:13:08.000
So, let's say if I only do their separate independent DAS training, so for the sentence index, I find this particular

01:13:08.000 --> 01:13:10.000
Subscript

01:13:10.000 --> 01:13:12.000
And for,

01:13:12.000 --> 01:13:21.000
role I find this particular substrate, right? Then when I tried to concatenate it, and then try to apply the compound intervention

01:13:21.000 --> 01:13:24.000
There is no reason for it to

01:13:24.000 --> 01:13:26.000
be in a

01:13:26.000 --> 01:13:33.000
There's no reason for both the subspaces to be aligned in a way that both of them just will work for the compound variable.

01:13:33.000 --> 01:13:36.000
Because they were not really trained

01:13:36.000 --> 01:13:43.000
Okay, but when I explicitly enforce that training, like, enforce that laws, that

01:13:43.000 --> 01:13:48.000
You don't let only have to modify the compound

01:13:48.000 --> 01:13:51.000
When I explicitly ask it to find

01:13:51.000 --> 01:14:01.000
a subspace in this particular family and this particular family, which should work for the compound 1, then it will find me one which

01:14:01.000 --> 01:14:03.000
Yeah, which basically does work

01:14:03.000 --> 01:14:14.000
What is this one, the intersection of these two? The intersection of these 2 is sort of like representing the fact that I can use the subspaces

01:14:14.000 --> 01:14:20.000
In that intersection family to do the compound intersection, with compound intervention.

01:14:20.000 --> 01:14:25.000
So these are cases that did work for combining or

01:14:25.000 --> 01:14:34.000
This is… What does the intersection of the two circles represent? Certain family, certain or some number of subspaces.

01:14:34.000 --> 01:14:37.000
that were found by the joint

01:14:37.000 --> 01:14:39.000
Optimization

01:14:39.000 --> 01:14:48.000
Maybe even if they weren't found, you're, like, saying, these ones satisfy the role. Exactly. The intersection is where both of them work together.

01:14:48.000 --> 01:14:54.000
So the joint one… Both of them work, or is this another subspace that works?

01:14:54.000 --> 01:15:09.000
Is this the two subspaces combined that work together or is this a new subspace that works? It's a new subspace, I believe. Every point is a new subspace. Each point is a choice of subspaces, right

01:15:09.000 --> 01:15:13.000
Is… each point is,

01:15:13.000 --> 01:15:24.000
Yeah, maybe this is not very good diagram. First, I thought it's like intervention mean, like, intersection is where they both conflict each other, so they cannot become

01:15:24.000 --> 01:15:43.000
That's what I was thinking earlier, and exactly apost. Maybe like a better So your intuition is that there is a subspace for the X subspace for y and another subspace for X and Y. No, no, no. There's many, many different possible ways of solving it. So you're arguing

01:15:43.000 --> 01:15:53.000
Or like non-determinism. So you're saying you're saying, hey, you know that like Atticus has been going around saying, you find the subspace for my causal variable

01:15:53.000 --> 01:16:05.000
And you're like, oh, that's… that's an ill-posed question. There are many different subspaces that all can achieve 100% on your causal mediation metric

01:16:05.000 --> 01:16:13.000
And a lot of the subspaces are very different from each other. And so, you know, just because your optimizer goes to 100% doesn't mean you find the unique

01:16:13.000 --> 01:16:16.000
Subspace. There's many places that it could have ended up.

01:16:16.000 --> 01:16:27.000
And if you add more constraints to it, then you might discover one of the other subspaces. It could also get to 100%. Is this what you found? Like, did different seeds produce

01:16:27.000 --> 01:16:45.000
Perfectly optimized similarity between different seeded. It's like if it's like I like the similarity would be like 0.6 is old story. Is there anything meaningful, right

01:16:45.000 --> 01:17:03.000
I could teach it. I don't think it's intuitive. Always comment subspace or whatever. Not super sure about that, rather. Intuitive. It is basically making that, like.

01:17:03.000 --> 01:17:05.000
If you

01:17:05.000 --> 01:17:21.000
There's several papers that show that, like, mechanisms don't have to be unique

01:17:21.000 --> 01:17:36.000
This is dissatisfying in the same way that the backup heads are, like, you know, dissatisfying. Who likes backup heads? Any fans of backup heads? Yeah, so like backup heads?

01:17:36.000 --> 01:17:49.000
They make the model more robust. Okay. And so, do you think this is just a robustness mechanism, or no, this is unsatisfying for you.

01:17:49.000 --> 01:18:05.000
You don't think of this as backup. This is just geometry is like space is super large and there is not a unique subspace which is satisfying the condition that you care about. This is like big family of there is this like family of

01:18:05.000 --> 01:18:09.000
Some spaces will satisfy the condition itself. That's it.

01:18:09.000 --> 01:18:25.000
So, in the ones that encompass, like, the rule and sentence ID, you can just, like, test, like, put all the families, like, all the subspaces, and check which are orthogonal

01:18:25.000 --> 01:18:26.000
You'll get the union

01:18:26.000 --> 01:18:29.000
It could be

01:18:29.000 --> 01:18:38.000
If you are able to organize one of the independent subspace with another sentence ID

01:18:38.000 --> 01:18:40.000
It means, like

01:18:40.000 --> 01:18:43.000
It's in the union

01:18:43.000 --> 01:18:52.000
No, that doesn't mean that… but you can just, orthogonalize two subspaces, right? And create a… I already tried that

01:18:52.000 --> 01:19:05.000
So what you're saying is I can find one rolls of space, and then I find a specific sentence index of space which is orthogonal to the role one? Yeah. Yeah, I can do that. Yeah, yeah, but that does not work. That does not work the combined one.

01:19:05.000 --> 01:19:09.000
But it might not work for one subspace, but you don't know, right?

01:19:09.000 --> 01:19:13.000
What do you mean? Like, there are multiple subspecies, no?

01:19:13.000 --> 01:19:22.000
Well, he found one that works by adding a third constraint, and then, of course, you can find it, you can make it work in the end by adding a third constraint, but without the third constraint, then

01:19:22.000 --> 01:19:29.000
Exactly. And there's many that don't work. Are these all all on the same layer, by the way, did you search

01:19:29.000 --> 01:19:30.000
On December

01:19:30.000 --> 01:19:33.000
One second

01:19:33.000 --> 01:19:38.000
This is across layers. This is basically the results.

01:19:38.000 --> 01:19:51.000
I can… I can do it across layers. These pieces of information, this, like, this subspaces of information is present across layers at the last one. So if you interlude in one layer, it just works. It works

01:19:51.000 --> 01:19:56.000
One layer, one token. I mean, if… I mean

01:19:56.000 --> 01:20:12.000
Think of this, like, information is present at start… start to be present and causally relevant at layer 20, and it is being relevant up to layer 30. If you intervene on any… anything in between, everything after that will be your intervening information.

01:20:12.000 --> 01:20:29.000
So this is question. Well, so now you have this third constraint that you've got this extra constraint, which means that the work that you did and the previous work to say this is where the index is, this is where the pointer index is actually maybe

01:20:29.000 --> 01:20:38.000
close to the truth because you didn't have this role interaction that you should have used as a supplementary constraint to clean up

01:20:38.000 --> 01:20:51.000
Your understanding of where the index subspace was. But that now leaves open the question, oh, other than roll, then there may be 10 other factors that you also have to worry about the interaction with

01:20:51.000 --> 01:20:54.000
in order to find, like, the queen

01:20:54.000 --> 01:21:00.000
And what are those and other interactions and other variables that you haven't thought of

01:21:00.000 --> 01:21:15.000
including, right? Is that… is that… I think that is a good end, yeah, there is one caveat in the last project, I did not use that. I used a different thing. Yeah, yeah, but you could have might as well use it. Yes, that's correct. So you have to find all the… all the ones

01:21:15.000 --> 01:21:30.000
Then find the intersection of that, and that's probably the the closest thing that we care about to the truth. Or maybe there's no truth, or maybe, depending on the subset of verification… I mean, we are basically labeling, putting names on numbers at the end of the day

01:21:30.000 --> 01:21:39.000
Right, of course, but, like, but once… but maybe if you have enough interactions, you try to do this, and eventually the optimization starts failing

01:21:39.000 --> 01:21:46.000
like you, like, maybe you don't have a subspace that satisfies all the interactions at once. I mean, it's possible, right?

01:21:46.000 --> 01:21:48.000
Yep. Yeah

01:21:48.000 --> 01:22:06.000
But how many intersections… I mean, how many interactions would you consider, and which ones and which ones? Yeah, I don't know. You just came up with another one. Sorry, what you said? I know, like this is like, it's like one per year. I don't know, right? Does that make sense? Yeah, that's the problem

01:22:06.000 --> 01:22:18.000
I have another question. If you would have completely changed the prompt and the names and the maybe even the types of the sentences, would the intervention work role to role

01:22:18.000 --> 01:22:34.000
I think I have an answer for that, I think, yes, go. I have done a similar kind of experiment it's one of the later slides, and I think it works. Okay. We can… depends on how different you want to make the sentence

01:22:34.000 --> 01:22:44.000
It will depend on that, but I have a few other types of sentences where the intervention works. So, I think when we go to that slide, maybe we… you can

01:22:44.000 --> 01:22:49.000
Yeah, let me know if that's what you were thinking or something else.

01:22:49.000 --> 01:22:57.000
But yeah, I couldn't find those, variables, so that means that the variable, the causal model that I was thinking about was more or less

01:22:57.000 --> 01:22:59.000
Verified.

01:22:59.000 --> 01:23:08.000
I'm gonna skip this part. But then I actually started looking into a different setup. This is where I do not ask about thematic rows at all

01:23:08.000 --> 01:23:20.000
This is just normal Q&A question. I skip the sentence, and then I ask who did what to whom kind of question, and the monster answer is the model is still pretty good at it

01:23:20.000 --> 01:23:25.000
Things still remains the same. The probing result still holds same, that the first

01:23:25.000 --> 01:23:31.000
Poison does not include the thematic rule, where the second and third one does

01:23:31.000 --> 01:23:46.000
Did you train these folks in this new setting, or did you… Okay. Yeah, it's a new training. Can you just take those probes and then apply it and generalize to this OTC? Slightly OTC. I did not transfer the probes, but the interventions

01:23:46.000 --> 01:23:52.000
dust transfer. But those are last intervention transfer. Those are last tokens, yeah.

01:23:52.000 --> 01:24:00.000
Those are the what? Those are the last what? Those are the last tokens, not really at the percent tokens where I do the proving

01:24:00.000 --> 01:24:12.000
But if the probes transfer. Oh, I see, I see. Something that I've not tried. Okay. Yeah, yeah. It might. Yeah, it'd be stronger if it does. I can do that

01:24:12.000 --> 01:24:31.000
But yeah, similar kinds of interventions, similar tasks, experiment, and it does work there as well. Yeah. So the task now is like asking who did something and then you take maybe the role from the previous step. Can you use it

01:24:31.000 --> 01:24:34.000
Two steers

01:24:34.000 --> 01:24:45.000
Yeah, that's probably the next slide. What you just showed. This is still training on this particular setup. Okay.

01:24:45.000 --> 01:24:49.000
But I can actually train a subspace

01:24:49.000 --> 01:24:57.000
Where I train it using the interventions in the previous setup and the interventions in the new setup, but not cross

01:24:57.000 --> 01:25:03.000
Across different setup. If I use those two interventions for each of the variable, then I can come up with a subspace

01:25:03.000 --> 01:25:13.000
Which, which I can do the patching across different… between these two setups for each of the variables that I have in the Cosmo. What's the difference in the both of the models?

01:25:13.000 --> 01:25:16.000
It's the same thing

01:25:16.000 --> 01:25:32.000
That's the whole point. What I'm saying is, the way a model solves when you explicitly ask about a specific thematic rule, and the way the model solves it when you just ask about who did what to whom kind of question, the model is still thinking about those thematic rules

01:25:32.000 --> 01:25:40.000
to solve the task. And that's the main result. What are the blue and the gray ones?

01:25:40.000 --> 01:25:46.000
Blue and the gray ones are if you just train the task on one specific setup.

01:25:46.000 --> 01:25:51.000
And the orange one is when you train the DAS

01:25:51.000 --> 01:25:57.000
on both the setup intervention, but not across interventions

01:25:57.000 --> 01:26:01.000
Yeah, but what's the difference between the gray and the blue?

01:26:01.000 --> 01:26:16.000
So blue is if you train each of those… each this, let's say this rule variable on this particular setup, and apply it on this setup, and this setup, and that its average is this one. The average, okay

01:26:16.000 --> 01:26:17.000
And

01:26:17.000 --> 01:26:20.000
grade the opposite? Exactly.

01:26:20.000 --> 01:26:30.000
And orange one is, you train a DAS for this particular role and this particular role simultaneously, but without doing the intervention cross-setup

01:26:30.000 --> 01:26:37.000
Yes, so you train it, and then you do the intervention cross-setup, then it works.

01:26:37.000 --> 01:26:46.000
Sorry, go ahead. So if it's the same causal, shouldn't the blue and gray already be high?

01:26:46.000 --> 01:26:52.000
I do think that the combined training… Yeah, I think that again comes back to that family of subspaces.

01:26:52.000 --> 01:27:03.000
It's not a unique subspace. So this is that experiment, so you're just showing that single variable training experiment again with the blue and the gray, is that right? Am I misunderstanding?

01:27:03.000 --> 01:27:18.000
Single variable, but applied to the other… so it doesn't generalize. So, like, the single variable thing, they didn't generalize to the

01:27:18.000 --> 01:27:28.000
joint thing in one setting also doesn't transfer between settings. Yeah, yeah. So this single variable thing is problematic

01:27:28.000 --> 01:27:32.000
I see. But the joint thing

01:27:32.000 --> 01:27:35.000
Also, transfer is better between settings.

01:27:35.000 --> 01:27:47.000
No, no, no, no, no. Actually, forget about the single one. Joint on the one setup, versus joint on both the setup

01:27:47.000 --> 01:27:52.000
Oh, which one is… which color is which? Orange is both the center.

01:27:52.000 --> 01:27:58.000
Oh, so you… you have to train across both setups? It's actually not across

01:27:58.000 --> 01:28:03.000
It's like, I do the intervention on this particular variable in this setup

01:28:03.000 --> 01:28:13.000
That is one loss for DAS, and I do the same intervention for this or similar intervention for the same variable on this other setup, get one loss

01:28:13.000 --> 01:28:27.000
And that I use both the losses simultaneously to train the DAS. But you're training a common subspace? Yes. I see. But then after you use both of these losses to train the set space, then you can transfer between them

01:28:27.000 --> 01:28:46.000
Oh, I see, but it's a little bit cheating, because you've used both of them in the training. Also, it's much lower, I think, than, like, you can get, like, 100% if you just do anyone, like, separately, right? Yeah, it's a little bit lower. It's not exactly as high as the individual. So my follow-up question would be if you take a third setting now, would that generalize?

01:28:46.000 --> 01:28:50.000
When you say third setting, you mean still

01:28:50.000 --> 01:29:01.000
Like, similar kinds of sentences, or… Like, the second one was completely different task, but requires the same skills, right? So a third one

01:29:01.000 --> 01:29:03.000
It's different from the first 2.

01:29:03.000 --> 01:29:06.000
Okay, maybe this will answer the question

01:29:06.000 --> 01:29:22.000
I also like David's question. You train the common subspace in a single setup for all the three causal variables. Would that transfer across both setups? Because it's already like kind of

01:29:22.000 --> 01:29:31.000
Yes. So is that one of the bars? So if you go back to the previous thing. Yeah, those are the left bars. So is that what the blue bar is? What is the blue bar

01:29:31.000 --> 01:29:34.000
Blue bar is if I train a subspace

01:29:34.000 --> 01:29:50.000
On one thing. On one using all three constraints from previous experiments. So, like, one variable, other variable, joint variable, and then now you try to use those subspaces to transfer roles over to the other cell, and it only works half the time. Yes

01:29:50.000 --> 01:29:52.000
I see, and what is the gray bar?

01:29:52.000 --> 01:29:57.000
It's just doing on the other side. Oh, just from the other direction, that's all.

01:29:57.000 --> 01:30:02.000
I see. So, and which one is from the more structured to the less structured? Is it

01:30:02.000 --> 01:30:07.000
I think blue is the first one to the second one

01:30:07.000 --> 01:30:18.000
The more structure to the least structure. Left is less structured. Implicit is the first one.

01:30:18.000 --> 01:30:33.000
I'm not sure what you mean by more structure. The one with the longer prompt for you. Yeah, that is the first one, which is the blue one, which is what I'm calling as implicit. That is explicit one.

01:30:33.000 --> 01:30:44.000
Yeah, that is great one, sorry, okay. So, what's the difference between implicit and explicit? Those are just different setups. One of them has a long prompt, and the other one has a short prompt. What explains what the roles are to the model?

01:30:44.000 --> 01:30:50.000
Oh, that's the only difference? Yeah, go ahead. You asked about a name instead of a role.

01:30:50.000 --> 01:30:59.000
in general, like, this experiment aside, this is a lack of my knowledge on DAS. Can you make one of those orange bars out of any two unrelated concepts?

01:30:59.000 --> 01:31:04.000
Like, if I have 10 completely unrelated tasks and I do your methodology on them

01:31:04.000 --> 01:31:11.000
Can I get a coherent… are there any cases of there's two concepts where they won't form a coherent

01:31:11.000 --> 01:31:16.000
cross-catchable subspace, in whatever sense of cross-patchable you want to define there.

01:31:16.000 --> 01:31:19.000
My intuition is that

01:31:19.000 --> 01:31:24.000
If both of them are in the… if both losses are in the training objective, then you're just going to find

01:31:24.000 --> 01:31:28.000
So the union of the two subspaces.

01:31:28.000 --> 01:31:34.000
Yeah, it's like a control baseline.

01:31:34.000 --> 01:31:42.000
I don't know if Dazz has some incentive for, like, making the subspaces, but I'm keeping the size of the subspace the same.

01:31:42.000 --> 01:31:46.000
The size is not expanding

01:31:46.000 --> 01:31:51.000
But I get your point. Yes, it's a valid point.

01:31:51.000 --> 01:31:59.000
I'm not… let's say if I train a subspace for rolling from one setup to the sentence index of the other one

01:31:59.000 --> 01:32:02.000
Yeah

01:32:02.000 --> 01:32:11.000
I'm still keeping the sizes, let's the original size of the rolls, subspace to still remain the same

01:32:11.000 --> 01:32:17.000
But if you can successfully patch between those, then that's an indicator that

01:32:17.000 --> 01:32:24.000
the size of the orange bar does not indicate something about the concepts

01:32:24.000 --> 01:32:30.000
Like, if you could use this method to patch between role and sentence index, whatever that might mean

01:32:30.000 --> 01:32:35.000
then you can pass between, like, anything and anything, right?

01:32:35.000 --> 01:32:41.000
Yes, yeah, I think I got that point, but what I'm saying is,

01:32:41.000 --> 01:32:52.000
For that to work, you may have to use a bigger subspace. You may have to train a bigger subspace with your DAS. Oh, oh, I see. To accommodate both of them

01:32:52.000 --> 01:32:55.000
So that you can catch both of them.

01:32:55.000 --> 01:32:58.000
But I did not really increase the size

01:32:58.000 --> 01:33:09.000
So this is… this is evidence because it is the same size subspace. As clean as it should be, but kind of.

01:33:09.000 --> 01:33:20.000
What is the value of R for roll, or fetch first, for position? Our role is to… fetch first is, I think, 2 as well, sentence index is 16.

01:33:20.000 --> 01:33:22.000
Okay.

01:33:22.000 --> 01:33:24.000
We go back

01:33:24.000 --> 01:33:35.000
So, sure, in explicit, it has the definition of agent, topic, and rule, right? Yes. So, shouldn't the scores be more on the gray bar?

01:33:35.000 --> 01:33:46.000
I think this is either the… yeah, this is averaged over both the direction. It's not just trained and tested on the same setup, it is trained on one setup and tested on both the setup.

01:33:46.000 --> 01:34:06.000
Okay, I'm gonna move forward. Yeah. What's the next big spot? Yeah, but the main point here is the model is using thematic rules in both the setup. That means when you just ask who did what to whom, the model is still using thematic role in addition to the pointer information to solve a task

01:34:06.000 --> 01:34:13.000
It's okay, I've got… and the same thing, everything that I've told, generalizes across languages

01:34:13.000 --> 01:34:30.000
At least left to light. Oh, wait. Can you train it on one language without training it on another language, and then it works in that language? No, that does not work that way. Yeah, that's what it means for me, to generalize. Unseen language. I think Dash is a little too powerful

01:34:30.000 --> 01:34:35.000
That's overfits.

01:34:35.000 --> 01:34:41.000
So that's why I have to train it on the 7 sentences, the 7 languages

01:34:41.000 --> 01:34:54.000
But then, once you train it, you make the DAS constrained enough, then you can do… you can use that constraint subspace to be patching across different languages

01:34:54.000 --> 01:35:00.000
Yeah, but if you don't really constrain it, then the transfer does not have worked

01:35:00.000 --> 01:35:09.000
So the subspace is theoretically then segmentable to sub-subspaces corresponding to each language.

01:35:09.000 --> 01:35:18.000
subspace… So if we're… if we've got, let's say this is the whole subspace of

01:35:18.000 --> 01:35:23.000
But the actual, like, material tasks based on the concepts.

01:35:23.000 --> 01:35:24.000
Hmm?

01:35:24.000 --> 01:35:32.000
If you're fitting your gas just on English, you're gonna get this little zone. If you're fitting your DAS just on Chinese, you're gonna get this little zone.

01:35:32.000 --> 01:35:35.000
The fact that you can distinguish these little zones

01:35:35.000 --> 01:35:39.000
And that the zones don't generalize must mean that like

01:35:39.000 --> 01:35:43.000
There's a part of your subspace of

01:35:43.000 --> 01:35:50.000
Freezing in this task that is reasoning about that in English versus another part that's reasoning about it in Chinese.

01:35:50.000 --> 01:35:56.000
Is that right? Yeah, I think that's what I think the kind of intuition that I was trying to give with this diagram.

01:35:56.000 --> 01:36:02.000
So, there is sort of a family of subspaces which works for, let's say, English. There is a family that works with

01:36:02.000 --> 01:36:04.000
Mandarin

01:36:04.000 --> 01:36:20.000
It's a little dissatisfying, right? Yeah. So what's your question, Shengyu? No, I think you are targeting a concept. Arnab you got to ask enough questions. My question

01:36:20.000 --> 01:36:29.000
Pretty dumb. So, if you fit on one language, why not fit on, like, five languages at the same time during just a single death?

01:36:29.000 --> 01:36:31.000
I guess now doesn't work.

01:36:31.000 --> 01:36:43.000
Yeah, that works. It works. It also works. Yeah, but what does it teach us about the model? Oh, but, like, the thing is, like, if you train on a lot of languages and fits, if you train on even more languages, we're generalized to unseen languages

01:36:43.000 --> 01:37:00.000
Right, so if you train on 10 languages, does it go to the 11th on C? I think if you… if I train on 4 or 5, then it will generalize to… Have you tried? Yeah, I've tried that. Oh, okay, good. That's good. Okay, yeah. Okay, well then that taught us something. Yeah, it depends on how much attention

01:37:00.000 --> 01:37:10.000
Then some constraint do you put on the DAS? If you don't put the constraint, then it can learn something that probably won't generalize. But if you put enough constraint on the DAS

01:37:10.000 --> 01:37:16.000
Then I think you will find… What is the effect? So, like, if you train it only on French

01:37:16.000 --> 01:37:18.000
And then you try to apply

01:37:18.000 --> 01:37:22.000
To do transformations on English

01:37:22.000 --> 01:37:30.000
The accuracy goes down, but does it go down in some interesting way? Like, does it does it do everything but in French or something like that

01:37:30.000 --> 01:37:32.000
You know what I mean?

01:37:32.000 --> 01:37:38.000
I did not look into that. Yeah, we should look at the instances to build some intuition.

01:37:38.000 --> 01:38:01.000
Yeah, I will not look into that qualitatively. How can you do that in French, though? It's just names, people's names. That's the first token. That's the objective here. Yeah, I don't know. Maybe instead of Steven, it's like Stefano. Like, what does it mean for the next… Stefano? I don't know, but for the accuracy to go down, it's just some other token. I wonder what it is

01:38:01.000 --> 01:38:09.000
It would be more interesting to try it with non-subject verb objects, like languages that are still left to right, but then flip around your order

01:38:09.000 --> 01:38:16.000
Like, oh, I know the Chinese is one of them, but that's like subject verb object is weird there. I'm thinking like

01:38:16.000 --> 01:38:19.000
Like Latinizations of

01:38:19.000 --> 01:38:24.000
Swahili. They do subject, object, verb, I think.

01:38:24.000 --> 01:38:31.000
Subject-object work. If I'm remembering right okay it's okay.

01:38:31.000 --> 01:38:34.000
Yeah, different language forms, that's kind of neat.

01:38:34.000 --> 01:38:37.000
It's a good idea.

01:38:37.000 --> 01:38:39.000
Yes.

01:38:39.000 --> 01:38:43.000
No, that's it, I think that's it.

01:38:43.000 --> 01:38:53.000
Yeah, it generalizes across languages. What's your summary conclusion? Oh yeah, I read your conclusion slide. Yeah, maybe you can read it.

01:38:53.000 --> 01:38:56.000
Sell your wife what you think

01:38:56.000 --> 01:39:10.000
Yeah, so I think what I think is, the model is sort of combining or composing this pointer information with thematic role information when it actually needs to fetch information.

01:39:10.000 --> 01:39:18.000
Not only pointer, which is relevant for binding. There is at least one more piece of information, which

01:39:18.000 --> 01:39:21.000
I'm describing, using this framework of thematic rules

01:39:21.000 --> 01:39:26.000
I think that's the main takeaway

01:39:26.000 --> 01:39:30.000
Yeah, and then this fact is generalizable across languages.

01:39:30.000 --> 01:39:37.000
So I would say you have a very interesting meta result, which is a limitation. You got everything to work

01:39:37.000 --> 01:39:42.000
But, like, you can see, like, the people around the table are dissatisfied

01:39:42.000 --> 01:39:45.000
Like, now we're not happy with subspaces anymore

01:39:45.000 --> 01:39:54.000
But you've made us all unhappy with subspaces, which is which I think is an interesting result. I mean, you've gotten the subspaces to work

01:39:54.000 --> 01:40:02.000
But they're not acting like the way that you expect subspaces to work. That experiment that you showed

01:40:02.000 --> 01:40:10.000
Where you should join the subspaces and it should just work naturally. That's what you would expect if

01:40:10.000 --> 01:40:14.000
If, you know, you're a linear algebra class is working the way that

01:40:14.000 --> 01:40:24.000
You know, if it was like a Fourier decomposition, you said, I've got the low frequency things, and I've got the high frequency things. If I join the subspaces together, now I've got

01:40:24.000 --> 01:40:30.000
You know, the combination of all the frequencies, and it just works, and it's beautiful. It's what they teach you

01:40:30.000 --> 01:40:31.000
You know

01:40:31.000 --> 01:40:34.000
In engineering class or whatever right

01:40:34.000 --> 01:40:36.000
And somehow

01:40:36.000 --> 01:40:41.000
It's not doing that right? So now it's not to say the subspaces are

01:40:41.000 --> 01:40:44.000
irrelevant, like, you got it to work

01:40:44.000 --> 01:40:48.000
But somehow it's missing something. I think that that might be

01:40:48.000 --> 01:40:54.000
I mean, so, like, I feel like there's two levels of the thing you found. Like, one thing you found

01:40:54.000 --> 01:40:57.000
This really interesting substructure

01:40:57.000 --> 01:41:00.000
for binding

01:41:00.000 --> 01:41:01.000
Which is

01:41:01.000 --> 01:41:13.000
A nice clarification. I think that it might have been anticipated, like some people could have like hallucinated, oh yeah, there's probably what is under there that like I sort of would have imagined that there's some substructure

01:41:13.000 --> 01:41:26.000
I don't know if there's pointers. But then you also found this limitation, which I think it's kind of surprising to see this limitation and it asks this question for the next researchers

01:41:26.000 --> 01:41:34.000
Well, if not subspaces, then what is the right abstraction for dividing up a big representation? We have got these giant

01:41:34.000 --> 01:41:37.000
8,000 dimensional vectors

01:41:37.000 --> 01:41:52.000
And Atticus, and, you know, we have been going around, you know, looking at subspaces of these 8,000-dimensional spaces to say, information's in this subspace, but you're saying, hmm, I don't know. Once you find a subspace.

01:41:52.000 --> 01:41:55.000
You might not be as smart as you think you are

01:41:55.000 --> 01:41:59.000
Right? There's some other stuff going on.

01:41:59.000 --> 01:42:09.000
You know, I think that's kind of this interesting question. Well, if not subspaces, then what should we be looking at covariance matrices or something? Should we be like weighted subspaces

01:42:09.000 --> 01:42:26.000
Where? What? Wait space, I don't think… I think WaitSpace is even worse. Instead of starting with 8,000 dimensions, now you start with, you know, 150,000 dimensional space, or something like that, you know, weights in a layer like

01:42:26.000 --> 01:42:28.000
You're in a worse situation.

01:42:28.000 --> 01:42:33.000
I don't know. Explaining it, but maybe the explanation is more… Yeah, I don't know, I don't

01:42:33.000 --> 01:42:37.000
Maybe.

01:42:37.000 --> 01:42:44.000
But, like, one thing you haven't looked at, and I don't… it's probably not the right thing to cover in this paper, because you're covering so much already, but it's kind of interesting question, right, is

01:42:44.000 --> 01:42:48.000
is this question of angles. It's like, so

01:42:48.000 --> 01:42:52.000
You know, is… is there a thing about

01:42:52.000 --> 01:42:59.000
Subspaces being closer to each other and farther. I think Sheridan had this interesting perspective also

01:42:59.000 --> 01:43:14.000
Where, you know, so Sheridan was looking at larger variables rather than, like, what a role is, which has, like, three settings or something like that. So you can kind of fit it into a 10 dimensional subspace or something like this, like, what dimension what is the dimensionality of the spaces that you end up using

01:43:14.000 --> 01:43:25.000
Two dimensions for the role. Yeah, two for the role, two for face first, and 16 first. 16 for the number for the count, for the… yeah

01:43:25.000 --> 01:43:33.000
Yeah, that makes sense. Sorry, I just want to say it's kind of like interesting, right? There are 3 roles, 2 bits of information.

01:43:33.000 --> 01:43:35.000
Like, log two of

01:43:35.000 --> 01:43:43.000
3 is the seal of it is just 2, so it kind of makes sense that you'll need at least two beds. Yeah

01:43:43.000 --> 01:43:49.000
Yeah, and that's the same thing where you get two, for the

01:43:49.000 --> 01:43:56.000
Why do you need two for the Fezfas? Yeah, you can just use one.

01:43:56.000 --> 01:43:59.000
I think you can get, like, depends,

01:43:59.000 --> 01:44:10.000
How much IAA are you expecting? Here, I think I was going beyond 95, but I think you can still get, like, 80%-ish with a single dimension.

01:44:10.000 --> 01:44:25.000
So one thing I was like when you fill the bag, the causal bag, you have all these very nicely named that you think are going to generalize. Okay, it detects it on the roles and the topics and things like that

01:44:25.000 --> 01:44:39.000
And when you train without the prompts, you still find them. The fetch first seems like very specific to your sentence structure that's just works because you

01:44:39.000 --> 01:44:49.000
You always have 3 roles, and the first one is when it's not. So, like, I would expect a model, like, when I read the question, like, I'm thinking, okay, which role happens to each one

01:44:49.000 --> 01:44:52.000
But the fetch first wouldn't be

01:44:52.000 --> 01:45:01.000
I wouldn't expect it to be a mechanism that works all like in most sentence structures.

01:45:01.000 --> 01:45:05.000
From your variables, I think the Fed first is kind of

01:45:05.000 --> 01:45:13.000
Only useful as an algorithm if you know, I'm going to have the three roles and only one instance of the three roles. I think

01:45:13.000 --> 01:45:25.000
If you have one missing, that would imply, okay, I'll just say whatever is first if the topic is missing. So I think like the DAG actually might

01:45:25.000 --> 01:45:31.000
Like maybe one reason this is confusing is that the Feuch first isn't a single variable

01:45:31.000 --> 01:45:38.000
So when you train a substrate for it, you're like learning a bunch of different things that you don't know how to make.

01:45:38.000 --> 01:45:41.000
Well, I think

01:45:41.000 --> 01:45:45.000
Regarding a sort of, like, first point, if I got it correctly, that

01:45:45.000 --> 01:45:56.000
We use your first point was, like, it won't generalize to other setups? Like, the first makes sense, because you know the sentence has three roles

01:45:56.000 --> 01:46:01.000
So, oh, if I don't see my row, I'll just look at the first one

01:46:01.000 --> 01:46:11.000
But the role thing would be something that's relevant for every sentence of keeping track forever. The thing… it doesn't really matter whether I have, like, 3 rows or 10 rows

01:46:11.000 --> 01:46:24.000
The fact that the first thing… huh? If you get 10 rows, then you have more bits of information than your subspace will not cover that. Talking about the causal model, not really the representation

01:46:24.000 --> 01:46:29.000
But your interventions will not work if you have more than three rows

01:46:29.000 --> 01:46:40.000
Why? Because the two… I will have… So, yeah, every change, every axis of change you make, you'll need to change

01:46:40.000 --> 01:46:44.000
trains of spaces, and to make it work, right?

01:46:44.000 --> 01:46:53.000
Yeah, you… yeah, you… yeah, that will… that is true, but I think the point here is about the causal model. The fact that the new information is being used

01:46:53.000 --> 01:47:06.000
It may be represented in, like, a two-dimensional, three-dimensional, four-dimensional… yeah, that is… that is, I think, setup dependent, but the fact that it is using a different type of information for binding is, I think, the main point.

01:47:06.000 --> 01:47:15.000
Nikhil, do you think you could have come up with an alternative causal model and also prove it with a subspace? Yeah, I think I

01:47:15.000 --> 01:47:21.000
I might do that for the paper. Something like instead of using thematic roll, just

01:47:21.000 --> 01:47:29.000
You something like fetch the first one or the second one, or the third one. But maybe you're asking the question in a different way. So like

01:47:29.000 --> 01:47:38.000
So, you have a causal model, and you said that this was not intuitive for you. You discovered this fetch first

01:47:38.000 --> 01:47:41.000
Mechanism, you know, partway through your research.

01:47:41.000 --> 01:47:56.000
So you started with simpler causal models that you disproved. Do you have like a good way? So in this slide deck, you try not to confuse us, which is good because it takes up time just to explain what you have

01:47:56.000 --> 01:48:07.000
But do you have, like, a good way of presenting or proving, or walking a reader through in the paper that you've disproven alternate hypotheses?

01:48:07.000 --> 01:48:08.000
Yeah.

01:48:08.000 --> 01:48:17.000
In a sense, you can say that if your causal model does not have the fetch-first variable, then

01:48:17.000 --> 01:48:26.000
it won't be complete. Right, so I think that would be really nice. Like, I don't know if, like, like, how complete is your write-up? Do you talk about that?

01:48:26.000 --> 01:48:34.000
I mean, I'm still writing. You're still writing it. Yeah, I think that would be really interesting, even if it's just… if it's, like, in the appendix or something, but just to say, look.

01:48:34.000 --> 01:48:47.000
You know, here's a simpler causal model, and the numbers are really bad for it. Let me show you, like, why you couldn't simplify it in this natural way.

01:48:47.000 --> 01:48:48.000
That makes sense.

01:48:48.000 --> 01:48:57.000
Yeah, I can do that. Sorry, one more thing. I think I'll ask if this is like a counterexample to your bank. Like, if you say

01:48:57.000 --> 01:49:01.000
Anna and Bob do whatever. You have two agents

01:49:01.000 --> 01:49:09.000
You would predict that it's going to check, is there an agent here? Yes, Bob, Anna doesn't have any encoding, so Bob is an agent.

01:49:09.000 --> 01:49:15.000
Sorry, say that sentence again? Anna and Bob do.

01:49:15.000 --> 01:49:17.000
So maybe Anna doesn't have any

01:49:17.000 --> 01:49:26.000
You know, it doesn't have a role all wood. No, even Bob would add. Oh, because you need the verb. Yes, yeah.

01:49:26.000 --> 01:49:31.000
So you said, like, the one who reminded was Bob to Anna.

01:49:31.000 --> 01:49:34.000
So then, like, maybe your first one will already have a thematic

01:49:34.000 --> 01:49:51.000
The one who reminded. The one who reminded was Bob was Bob, it was about Anna to…

01:49:51.000 --> 01:50:04.000
I guess the simplest one, sir. Sorry, simplest one is when one is missing, then you look for it, you don't see it. So you give whatever is first, which is like the wrong role. That's what you would predict from this tag

01:50:04.000 --> 01:50:07.000
That is the wrong rule

01:50:07.000 --> 01:50:22.000
You're trying to construct a sentence form which would cause the model to make a mistake based on this. You get a much simpler sentence for contradict it. Just do passive voice of the same fact.

01:50:22.000 --> 01:50:26.000
Alice called Bob becomes… Bob was called by Alice.

01:50:26.000 --> 01:50:27.000
Right

01:50:27.000 --> 01:50:33.000
Because that would… that would eliminate the… that would screw up the fetch first mechanism. Nothing would be final

01:50:33.000 --> 01:50:40.000
The fetch first always, like, doesn't have to be, like, passed by active. Like, the

01:50:40.000 --> 01:50:45.000
It's just like whenever the first one, like, you can't access them. Oh, oh, oh, I see

01:50:45.000 --> 01:50:54.000
are looking at the mechanism from the last token before the answer.

01:50:54.000 --> 01:50:59.000
But you also have probing experiments that show that

01:50:59.000 --> 01:51:03.000
This thematic indexing happens at the word token. Yes

01:51:03.000 --> 01:51:12.000
Do you think you can also train subspaces to change roles of each bird inside, so that you don't do anything in the final token?

01:51:12.000 --> 01:51:15.000
It's just you doing the other end of

01:51:15.000 --> 01:51:24.000
I think it is doable, but I'm just not done it. But yeah, like, do you not think it's relevant to the work, or what it is?

01:51:24.000 --> 01:51:36.000
It is, I think, doable. It is a very, yeah, I think it will work if I do it as well, but I'm just not doing it. I think for this project, I was just primarily focusing on the last token, what happens at the last token

01:51:36.000 --> 01:51:47.000
So, I want to add something to Adas and David's point. Like, during, I don't know, like, every good fire has more seen what to write in the paper, but when you're presenting, I would

01:51:47.000 --> 01:51:54.000
But most of the things… that was my first question, right? Like, how did you come up with this graph?

01:51:54.000 --> 01:52:02.000
When you're explaining things, most of these are really surprising from your point of view, I'm sure.

01:52:02.000 --> 01:52:07.000
When I'm listening for the first time and I'm not super aware of DAS or any of these techniques

01:52:07.000 --> 01:52:11.000
The thing that would really help me was if I see a control

01:52:11.000 --> 01:52:21.000
Where you say, oh, this 100% is not something that happens for anything. Like, for example, like, Hadith asked the question, oh, like, can you do the same thing by just

01:52:21.000 --> 01:52:27.000
Changing the entire causal graph, coming up with a new graph, and I can still do all the DAS variables

01:52:27.000 --> 01:52:36.000
Yeah. Right, like, 5 seconds, maybe a variable that you can try to train DAS on, and maybe DAS would give you, like, 50% or 20%.

01:52:36.000 --> 01:52:41.000
So, like, if you can show such control examples for every experiment you have

01:52:41.000 --> 01:52:43.000
I think that would, like…

01:52:43.000 --> 01:52:47.000
enhance my understanding, too, that, oh, yes, this is, in fact, surprising

01:52:47.000 --> 01:53:00.000
And maybe that's partly the reason why there's like some dissatisfaction around like interpretability. Yeah. Right. Like we tend to not control our experiments well enough because partly it's because

01:53:00.000 --> 01:53:14.000
The results can be complicated, like you've got, like, multiple things going on, and you feel like, oh, now I've filled out my 8 pages. But I think that I think it's good. I think that even if it means that the controls are in the appendix or something like that

01:53:14.000 --> 01:53:26.000
Which means Nikhil's running a bunch of these, they're just not in the presentation. They're just not there. They didn't… because they weren't relevant to what actually worked out. Yeah, I totally… I know, and that's why you were. What? I mean, that might be just too much

01:53:26.000 --> 01:53:33.000
If I go through the rationale of each of these different… Yeah, I know, like, you always do this, you always have, like, 150 pages of results.

01:53:33.000 --> 01:53:41.000
But I agree with you. So I think that, you know, it probably… there's probably some editorial judgment

01:53:41.000 --> 01:53:48.000
About, like, what would be reasonable controls to include. But I do think that it would be helpful

01:53:48.000 --> 01:53:59.000
To basically show people the alternate some alternate hypotheses that don't work like ones that you think might be reasonable that people would suggest, or something like that. Like

01:53:59.000 --> 01:54:06.000
you know, and even if that's explained in the appendix. Yeah, so the question was for you to ask before is were there alternate hypotheses

01:54:06.000 --> 01:54:11.000
really don't work, because from what you showed, it seems like you can train for anything.

01:54:11.000 --> 01:54:13.000
Right.

01:54:13.000 --> 01:54:17.000
Like, it could be just predicated logic plus events

01:54:17.000 --> 01:54:21.000
Instead of thematic rooms

01:54:21.000 --> 01:54:29.000
Perfect, 2nd, Fearch, 3rd. Would it not work, or did you know whether it would work if you come up with a completely different one?

01:54:29.000 --> 01:54:30.000
Which one?

01:54:30.000 --> 01:54:36.000
A different causal graph, and train DAS on it, on, like, the causal variables there.

01:54:36.000 --> 01:54:40.000
Again, it will depend on the cost of all that you come up with

01:54:40.000 --> 01:54:42.000
So

01:54:42.000 --> 01:54:51.000
It might be the case that you can come up with another causal model where the variables that I have in my causal model is sort of like split into more fine-grained things.

01:54:51.000 --> 01:54:53.000
And that might work

01:54:53.000 --> 01:54:58.000
But, if you come up with a causal model where you come up with a variable which is actually not

01:54:58.000 --> 01:55:02.000
It is actually not what is happening in the model, then it won't work.

01:55:02.000 --> 01:55:03.000
Great.

01:55:03.000 --> 01:55:06.000
That will confuse me if I see that.

01:55:06.000 --> 01:55:11.000
Okay, I think I have all the limitations issue.

01:55:11.000 --> 01:55:14.000
It isn't in the

01:55:14.000 --> 01:55:19.000
In the original DAS paper, they come… they…

01:55:19.000 --> 01:55:28.000
They work with a simple summation setup, plus B2C, and there are different ways you can combine them. Summation, or was it range?

01:55:28.000 --> 01:55:47.000
Those are the summation paper, yeah. I think they work with a different concept analysis. Okay, never mind. Then take balancing, like, sorry, parenthesis balancing, stuff like that. Okay. And they actually came up with, different causal

01:55:47.000 --> 01:56:00.000
Causal graph, and they were showing that one of them works or two of them works or multiple of them. I'm not really sure. But I think a lot of these questions are already answered in the original last paper.

01:56:00.000 --> 01:56:05.000
Well, for the… for that toy example. For the toy example. For us, the example is more important than

01:56:05.000 --> 01:56:06.000
Right

01:56:06.000 --> 01:56:08.000
Yeah…

01:56:08.000 --> 01:56:18.000
But also, like, like in this specific example, can you think of another causal graph that will fet second, fetch third, like how you can find

01:56:18.000 --> 01:56:23.000
How will that work? Maybe, I don't know, like, maybe Dash can find it.

01:56:23.000 --> 01:56:39.000
Like, find the second person off. Well, if you just the initial thing, well, like, so, you know, Ben or whoever would say, oh, obviously you can't do that because it doesn't know what the role is, but the obvious thing to do would be just to drop the fetch first

01:56:39.000 --> 01:56:50.000
Variable, right, and just try to say, hey, can you get a causal graph to work with just the role? And it'll fail because there's no way that it knows the role

01:56:50.000 --> 01:56:52.000
Right?

01:56:52.000 --> 01:56:57.000
That's like one example of that, that the cause of all that I come up with is not really good

01:56:57.000 --> 01:57:08.000
Without the fetch was better. That there's, like, some gap when you don't bring the fetch first, and once you add it, it comes up.

01:57:08.000 --> 01:57:11.000
Also, like.

01:57:11.000 --> 01:57:24.000
If you just use your pointer method from previous works, does this task? That could be also another cause and all that I forget about anything.

01:57:24.000 --> 01:57:35.000
Forget about anything on the left-hand side of the circuitry. Yeah, just include the third one, then also it will be pretty bad. Yeah, in fact, that's nothing better. It'll be bad? Yeah, it will be bad. I mean

01:57:35.000 --> 01:57:41.000
If you just tell it to find some person of a particular sentence, and there are 3 options

01:57:41.000 --> 01:57:48.000
It won't be correct. It could fetch… it could fetch any of these 3 ones. Yeah, so, like, do you think then

01:57:48.000 --> 01:57:55.000
In the next coming 10 years, you can come up with more setups that would, like, start extracting more and more

01:57:55.000 --> 01:58:02.000
Covering inputs to your… Yeah, so I think the hope here is that if we have enough

01:58:02.000 --> 01:58:14.000
Variables that go into binding different types of features to a particular object.

01:58:14.000 --> 01:58:21.000
I think we can… or the way I think about it is that the North Pole here is

01:58:21.000 --> 01:58:26.000
If we can, come up with, let's say, tracer programs

01:58:26.000 --> 01:58:36.000
Using the variables that we have in our causal model, and replace the specific model components which are responsible for generating those variables in the model

01:58:36.000 --> 01:58:39.000
With the treasure program that we have come up with

01:58:39.000 --> 01:58:44.000
And do it at a sufficiently good performance level.

01:58:44.000 --> 01:58:51.000
Then I think I would say that if we can achieve that, then I would say that, okay, binding is a hard time problem

01:58:51.000 --> 01:58:54.000
That's what I would call it, as, like, a salt problem

01:58:54.000 --> 01:58:57.000
If we can actually program

01:58:57.000 --> 01:59:09.000
All the different variables which are important for doing the banding with something like a tracer thing. Isn't this good? The symbolic AI people have was trying to do for like 50 years and could not do it.

01:59:09.000 --> 01:59:16.000
They were trying to do it for the symbolic, I mean, neurosymbolic people, or… Neurosymbolic people. Yeah, but maybe they were not

01:59:16.000 --> 01:59:31.000
Creative enough. This is what I mean by creative decision is probably more creative. It is happening here. They didn't have creative. Gradient descent has learned something that they were not able to come up with, and

01:59:31.000 --> 01:59:33.000
Since we are reverse engineering that

01:59:33.000 --> 01:59:36.000
then we have a good chance of getting there.

01:59:36.000 --> 01:59:47.000
I'd ask you for a prediction. If you train it on, like, FedStack and on first third, you think that would work similarly well?

01:59:47.000 --> 01:59:57.000
I'm not sure about that. I think that's one of the another reason that I need to try. Also, like, it could be just moral specific or parameter-specific graph you made

01:59:57.000 --> 02:00:11.000
Yeah, so this one is… this one has only been tested on Chema 27B, but I think I'm gonna try it on other models. Like, maybe Kimi K3 is a really large… it might have

02:00:11.000 --> 02:00:12.000
Yeah, I

02:00:12.000 --> 02:00:32.000
Yeah, I can try Kimitia 3 with good fire cluster. Who's going to give me the compute, but I think the other one will. But yeah, and I'm hopeful that the causal model will generally generalize

02:00:32.000 --> 02:00:43.000
Okay, I think we can call it out, we are going way over time. Great, thank you, Nikhil, very much. That's great.