Nikhil Prakash: So… Today we have… Not visiting speaker.
Nikhil Prakash: Nikhil Prakash.
Nikhil Prakash: And so Nikhil is actually one of… It's a very unusual case.
Nikhil Prakash: For a PhD student.
Nikhil Prakash: Go ahead. Because… Because, you know, when he arrived, sort of his first month here.
Nikhil Prakash: We sat down, we chatted about what kinds of things he was interested in studying.
Nikhil Prakash: And we looked at a whole variety of different things, and
Nikhil Prakash: And he pretty quickly says, you know that, right there, this topic is interesting to me. What is that? Theory of mind.
Nikhil Prakash: Right? Do these AI systems that we're building
Nikhil Prakash: Do they think like humans? Do they look at other agents and think about what they're thinking? Do they have a theory of mind? Can they… can they… can they put themselves
Nikhil Prakash: In the… in the shoes of others?
Nikhil Prakash: That's what I'd like to study.
Nikhil Prakash: And… and and so he… he started working on it right away.
Nikhil Prakash: But from the beginning, it was, like, challenges. Like, the models that were available his first year.
Nikhil Prakash: They had no theory of mind. We've had some papers about this.
Nikhil Prakash: Right? Like, they just… they couldn't do it at all. So it's taken a few years to get to the point when models maybe credibly have some… some evidence of even having this.
Nikhil Prakash: But… but the interesting thing, and the thing that makes them unusual, is that
Nikhil Prakash: Yeah, if I look at him now, you know, a few years later, oh, he's, like, working on the thing that he said that he was gonna work on.
Nikhil Prakash: And so…
Nikhil Prakash: So today, he's not going to talk about a theory of mind paper, but he's… But it's very closely related. It's very closely related. It's binding, right? Which is, which is… which is a, which is a large aspect of the theory of mind task.
Nikhil Prakash: And he's looking at it in an interesting setting.
Nikhil Prakash: There's not language models, and so… So, so…
Nikhil Prakash: Now, I think that he also is going to do the talk in an unusual way, because… are you? No. No, you're not. So, I've been asking him to, practice his, his, his, his sort of talk technique, and trim down his talks, sort of really short, really polished.
Nikhil Prakash: You know, to, like, you know, the kind of, like, more formal presentations. And I thought, well, maybe he'll practice that today, but it sounds like he's not going to practice that today, so we'll see. But welcome, welcome, Mikael.
Nikhil Prakash: But, take it.
Nikhil Prakash: Yeah, thank you for that great introduction. Yeah, unfortunately, yeah, I… I… so, I had prepared the slides by yesterday, but then I… but, yeah, and I had a chat with David about the slides.
Nikhil Prakash: And he gave me a lot of feedback.
Nikhil Prakash: And I just couldn't incorporate All of the feedback in, like, 3, 4, 5 hours.
Nikhil Prakash: That's the… effectively, that's the time that I had. So, I started making the changes in the slide, but in the middle, I figured out I won't be able to complete it. I mean, the kind of quality that I think you were trying to push me towards, that was too much for me to get in 3 or 4 to 5 hours. Okay.
Nikhil Prakash: So, I did a few changes. In fact, you might see that there might be significant difference between the quality of slides.
Nikhil Prakash: like, the earliest sites versus later sites.
Nikhil Prakash: But hopefully, I think I should be able to explain the ideas and the main results from the paper still clearly. I think that's the main goal here. So I think, yeah, so this is… this is gonna… still gonna be our normal conversation, kind of.
Nikhil Prakash: presentation, not like a formal, polished presentation. So feel free, feel free to interrupt me and ask as many questions as you want.
Nikhil Prakash: Okay.
Nikhil Prakash: Yeah, so this is basically the title of the paper, The Dual Mechanisms of Spatial Reasoning in VLMs. This is under, review right now at ICML.
Nikhil Prakash: Let's see what happens.
Nikhil Prakash: Okay, so…
Nikhil Prakash: Okay, I'll start the presentation by actually describing this idea or concept of variable binding. Yeah, I think they would…
Nikhil Prakash: briefly mentioned about it, that this paper is about binding as well. It's, I think, a very old idea. I think it's an idea which is older than most of us.
Nikhil Prakash: People have been talking about in neuroscience and Cox science for the last… at least 3 decades that… that… that I've known of.
Nikhil Prakash: And the idea is actually pretty simple and fundamental.
Nikhil Prakash: It's just the ability to associate features of an object. So now, right now, you're seeing a lot of people, and most of us are wearing different color
Nikhil Prakash: t-shirts or sweaters.
Nikhil Prakash: And you can ascribe
Nikhil Prakash: you can figure out that, okay, the color of the t-shirt that this person is wearing is this color. And you're not confusing between the colors of different people. And the reason that you are able to do that, that's because your mind is able to do, or is able to solve the binding problem.
Nikhil Prakash: Yeah, so essentially the binding problem is basically to…
Nikhil Prakash: associate features of an object, and keep it separate between different objects. So, for instance, in this particular slide, this is a nice photo of a horse wearing a fedora and a cat wearing a…
Nikhil Prakash: Cap.
Nikhil Prakash: Now… if…
Nikhil Prakash: any intelligent system, be it human brain or a neural model, if we say that that system has variable binding capability, what I mean is, it can sort of take that image and then create this list of
Nikhil Prakash: A set of tuples, where each tuple is actually representing an object and its corresponding feature.
Nikhil Prakash: Yeah, if a system can do that, then we say that the system has a variable binding capability.
Nikhil Prakash: Why do we care about variable binding in quant spatial reasoning? Because I would argue that this is one of the fundamental properties that any system needs to have to be able to do spatial reasoning. So, for instance, if you give this particular image to a neural model to generate a caption, and let's say it creates this caption, which describes the image.
Nikhil Prakash: I would argue that to be able to represent or generate that caption coherently, it needs to do variable binding in its internal thinking process. If it cannot do variable binding properly, then the caption that it's gonna generate is more likely be incorrect, and even if it's correct, it's…
Nikhil Prakash: It's something that is not really trustworthy.
Nikhil Prakash: So that's why variable binding is an essential skill or essential task for any system to do spatial reasoning.
Nikhil Prakash: In fact, a number of previous works have shown that the… the limited…
Nikhil Prakash: facial capability of VLMs can be attributed to their restricted variable binding capabilities.
Nikhil Prakash: Okay.
Nikhil Prakash: Okay, so variable mining seems very important for spatial reasoning. Then the question…
Nikhil Prakash: Comes up. The main question that comes up, at least for us.
Nikhil Prakash: the Macintah people is, can we better understand it in the visual space? In the hope that maybe we can probably improve the model performance as well, with the better insights of how the model
Nikhil Prakash: models are actually… Forming those tuples in its internal activations.
Nikhil Prakash: Before going into… the VLM space.
Nikhil Prakash: Actually, people have looked into this problem in the language model space, and I'm first gonna describe what we already… You're so humble. People have looked into it. I wonder who people…
Nikhil Prakash: Okay, yeah, so some of those contributions have been from our lab as well.
Nikhil Prakash: Okay, okay, okay, from me.
Nikhil Prakash: Yeah, okay, so…
Nikhil Prakash: Yeah. Yeah, so let me first describe what we already know about this variable binding in the language model space, and then maybe we can start thinking about and talking about the same problem in the vision space.
Nikhil Prakash: So assume this is, the task, the task that you see on the slide. You have a context, something like, apple is in box A, banana is in box B, cheese in box C, and then you ask, or you just have a query sentence, box A contains the… and the answer should be apple.
Nikhil Prakash: Pretty simple, right? And what previous, or some of my works have shown is the way models, or especially language models, solve this task is by first creating some kind of, like, abstract, ordering representation for each kind of important token that you have in the prompt.
Nikhil Prakash: Okay.
Nikhil Prakash: abstract ordering representation, and each kind of important token in the prompt. Those are the two main key… key terms in what I said.
Nikhil Prakash: So, essentially, what that means is, here in this particular prompt, I would say that there are two main types of important tokens.
Nikhil Prakash: First is the object, and the second one is the box label.
Nikhil Prakash: Now, what the model does, it basically creates a label.
Nikhil Prakash: for each of these two type of, information. So for Apple, since it's the first one in the prompt, it says that, okay, this is the first
Nikhil Prakash: object in the prompt. And A is the first label in the prompt.
Nikhil Prakash: Similarly, for banana, it says that, okay, this is the second object, and B is the second label.
Nikhil Prakash: So it creates these abstract representations, to encode important information which is present in the prompt.
Nikhil Prakash: And then, finally, when you ask a question, or some kind of…
Nikhil Prakash: features about a particular label or a particular box, what it does is basically uses that ordering information to fetch its corresponding features that it will predict as the next token, where, in this case, the feature would be actual object which is present in that box.
Aruna Sankaranarayanan: Nikhil, I have a question. I'm online, sorry, sorry. Can I ask now, or should I wait till the end?
Nikhil Prakash: Yeah, yeah, go ahead, Matt. Yeah, yeah.
Aruna Sankaranarayanan: Okay, so my question was, so in this case, like, the structure of the sentence is such that the…
Aruna Sankaranarayanan: It has, like, this very constrained structure, right, where you have, you know, word followed by is in box, and is in box is identical, you know, in all the three segments, and then you have, kind of these labels that come up.
Aruna Sankaranarayanan: So,
Aruna Sankaranarayanan: Do you think that even if we didn't have this constraint structure, and, you know, if the sentence was pretty free-flowing, that semantically similar elements would still share
Aruna Sankaranarayanan: this ordering ID, like, the model would… like, is the model doing it… doing this based on, like, some kind of semantic similarity between these components, or is there, like, some other…
Aruna Sankaranarayanan: reason for… for why these, you know, for why these IDs exist.
Nikhil Prakash: So I think for the first question, if… so the question was, for a given, like, free-form text, would we expect to see similar kind of…
Nikhil Prakash: ordering IDs.
Nikhil Prakash: Or not. I think my intuition is, I think we would still see these kinds of ordering ID. But the problem there is, since those are, like, general text, it will be more difficult to actually prove
Nikhil Prakash: that…
Nikhil Prakash: it… they are indeed there. I mean, we can still do, like, causal experiments to say that, okay, they are ex… they are present there, but in that case, the
Nikhil Prakash: The… the process of coming up with the causal experiments would be way more difficult, because there would be way more confounds.
Nikhil Prakash: Yeah, I think… but I… but my general intuition is that even in the freeform text, the model does create this kind of, ordering ID representation.
Nikhil Prakash: Because I think we have seen this kind of representation across a bunch of…
Nikhil Prakash: settings now. Even though most of those settings are still synthetic, kind of… still, in a sense, synthetic, but I think we have seen this… this kind of representation now across a bunch of tasks, bunch of settings.
Nikhil Prakash: Which kind of give me the confidence that, okay, this is a generic thing, it's not just…
Nikhil Prakash: constrained to this kind of synthetic setting.
Aruna Sankaranarayanan: Yeah, okay, awesome. Thank you so much, yeah. I think Yuav's work also is probably, like, relevant here, maybe. Like, he also had these similar findings across many, many tasks, but they were all, again, synthetic, so…
Nikhil Prakash: Yeah, even, yeah, the tasks that he used, I think he… those were more or less synthetic as well. I don't think so they… they had general… general text.
Aruna Sankaranarayanan: Yeah, yeah, yeah, yeah, yeah, cool, thank you.
Nikhil Prakash: We did see generalization to the mental deficit, which is much less.
Nikhil Prakash: So, like, even if you can't find them in the playbook setting, you can show generalization or not. Yeah, that's right. I mean, yeah, it depends what you call synthetic funnel. The belief tracking was also not super synthetic.
Nikhil Prakash: Yeah, but this mechanism is clear.
Nikhil Prakash: So, okay, so that was the language model space, now let's talk about the… Huh.
Nikhil Prakash: What if you have more than one object?
Nikhil Prakash: Yeah, good question. I'm not super sure what will that have… what will that do.
Nikhil Prakash: What do you think?
Nikhil Prakash: Yeah, I'm not 100% sure.
Nikhil Prakash: Yeah.
Nikhil Prakash: them to have the same article.
Nikhil Prakash: But then how… So it's not an order, something else.
Nikhil Prakash: So it's gonna still be, like, the first object, first object? Yeah, I think both are gonna… it's gonna be counted as one object, that's what I… First…
Nikhil Prakash: The phrase will be… what is the object inside box A that starts with A, or something like that? Like, if you target a specific one.
Nikhil Prakash: Yes, it should be… Yeah, it shouldn't be exactly the same. There should be some difference, but…
Nikhil Prakash: No, but for Boxide contains B.
Nikhil Prakash: It will be the same, more or less the same marker for F1. No, it decides about the marker, where it sees the query.
Nikhil Prakash: It decides about the markers before it sees the query.
Nikhil Prakash: Yeah, but I would guess it creates different markers for usage.
Nikhil Prakash: Yeah, I think so.
Nikhil Prakash: Andrew.
Nikhil Prakash: Martinez Lab.
Nikhil Prakash: How to work… pull off words for music.
Nikhil Prakash: yours that shows that there are different markers… Oh, he has a paper on this? I've not seen that paper.
Nikhil Prakash: Is it outbound? Yeah, I think so.
Nikhil Prakash: Oh, I've not seen that paper.
Nikhil Prakash: Okay, Adish, yeah, if you can send that, it would be really helpful.
Nikhil Prakash: Do you have an idea?
Nikhil Prakash: What is getting binded to what?
Nikhil Prakash: Anyone?
Nikhil Prakash: I can't be buying books because… Really?
Nikhil Prakash: Oh, oh, you asked about… But there is no happiness.
Nikhil Prakash: No, no, you can get some templates.
Nikhil Prakash: You asked me about Philadelph Edwards.
Nikhil Prakash: How do we send them back.
Nikhil Prakash: up above every little house.
Nikhil Prakash: And… with the… I don't think that happened in the morning.
Nikhil Prakash: Nico, wouldn't you say that your diagram
Nikhil Prakash: Suggest, because you call it IDs.
Nikhil Prakash: you're showing here is that, no, neither one. They're both being bound.
Nikhil Prakash: To number one.
Nikhil Prakash: They're not… it's not apples being bound to A, or A is being bound to apple.
Nikhil Prakash: They've been this ID, right?
Nikhil Prakash: there's an indirect… binding happening through the ID.
Nikhil Prakash: Interesting.
Nikhil Prakash: Basically.
Nikhil Prakash: The finding that happens.
Nikhil Prakash: So is it more like a 4.9? Yes, exactly, exactly.
Nikhil Prakash: Yeah, but the… yeah, it's more like a pointer, but there's still the question of many-to-one binding, where, let's say, we have multiple objects, like, multiple… yeah, multiple objects in a single box.
Nikhil Prakash: How… So that would be binded is something that I'm still not very sure.
Nikhil Prakash: Okay, we can move forward. So, there was the existing results in the language model space. Now, let's talk… start talk… talking about the VLM results. So, before we get into the results, I just wanted to brush up on the architecture of a VLM. It's actually simple.
Nikhil Prakash: Yes.
Nikhil Prakash: So, yes, so there are primarily two components, or actually three components. There is a vision encoder and a projector on top of the vision encoder.
Nikhil Prakash: And then finally, we have a language model backbone. So the image is first, broken down into smaller patches. Each of the patches becomes a specific token, which is fed into the vision encoder.
Nikhil Prakash: Which is just VIT.
Nikhil Prakash: It processes those… token information.
Nikhil Prakash: And, yeah, then output of each of the token is basically given to the projector, which is supposed to trans… transpose the…
Nikhil Prakash: Those vision encoder into language… language model space.
Nikhil Prakash: And then those… Transformed vision encoder, vision embedding, are actually passed on to the language model, just as
Nikhil Prakash: Normal token embedding, in addition with the…
Nikhil Prakash: the token embeddings from the text. That's it. And then, finally, the language model predicts the next token in the text domain.
Nikhil Prakash: Yeah, so there was this paper last year,
Nikhil Prakash: Which show that the kind of mechanism that I showed for the language model in the last-to-last slide actually generalizes to VLMs as well.
Nikhil Prakash: Though they only studied the language model backbone, but they show that this… the same kind of mechanism actually generalizes,
Nikhil Prakash: that setting as well. So, just to it.
Nikhil Prakash: They only show, sorry, but they only show, like, what's happening in the text tokens. They didn't say anything about the visual text.
Nikhil Prakash: So they… they have just one… one experiment. They say that the information comes from the visual. But they do one experiment on the key vectors of visual tokens.
Nikhil Prakash: Yeah, it's… so they don't say it as clearly as I'm gonna explain it to you.
Nikhil Prakash: But I think, based on what I've explained to you in the language setting, and based on their result, I think we can infer that.
Nikhil Prakash: Okay, so essentially, this is what happens, so…
Nikhil Prakash: Yeah, the image gets passed on from Vision Encoder and the projector, and then in the language model.
Nikhil Prakash: in the language model backbone, we… or they have shown that the model creates this ordering ID for, let's say, both horse
Nikhil Prakash: and CAD, as well as Fedora and, CAP.
Nikhil Prakash: In the visual token residual string.
Nikhil Prakash: And then when we ask about cat is wearing a, and the answer should be cat, the model basically… the language model basically fetches the ordering ID of the cat, which is 2,
Nikhil Prakash: from the vision token to the CAT token residual stream, and then it passes on that piece of information to the last token, the A token.
Nikhil Prakash: Which is used to, again, fetch in the corresponding, object ordering ID.
Nikhil Prakash: And then finally, the model uses that object ordering ID to actually fetch its value, the cap.
Nikhil Prakash: Which is actually the final prediction as the answer.
Nikhil Prakash: Good.
Nikhil Prakash: How does we know that if this rectangle represents a cat?
Nikhil Prakash: How… I'm not saying that if it… it could be representing anything.
Nikhil Prakash: But… Let's say… so when we…
Nikhil Prakash: Break down the image into smaller batches.
Nikhil Prakash: Let's say the last patch.
Nikhil Prakash: Which, the last patch, which has some… region of the cat.
Nikhil Prakash: will have this ordering ID.
Nikhil Prakash: Okay. Even that's not… I don't think so they explained that in that paper. I think… I think in that paper… in their paper, they just had…
Nikhil Prakash: like… Shapes, which can be covered with a single token.
Nikhil Prakash: Single batch. My single batch.
Nikhil Prakash: So if you have an object which spread across multiple patches.
Nikhil Prakash: then I think…
Nikhil Prakash: things become a little bit more complicated, but I think we can say that at least the last token of that
Nikhil Prakash: the last batch of that particular object should encode this ordering ID in the language model space.
Nikhil Prakash: It's the same as Rome, basically. It's kind of like the last entity of the, like, the last token of the entity includes…
Nikhil Prakash: Kind of like that. Interesting, yeah, it's the same in… But here, the passions are not, Sequential.
Nikhil Prakash: Like, a cat can be some, like, tentacles, and then… Yeah, yeah. That's right, that's right.
Nikhil Prakash: There could be a lot of space in between. Yeah.
Nikhil Prakash: Though they don't show it in the paper, I… yeah, this is my understanding and a little bit of our experiment.
Nikhil Prakash: Okay, but so this is…
Aruna Sankaranarayanan: I have another… I have another question. So, you're saying that each patch then is associated directly with an object, and the object is not spread across multiple patches?
Aruna Sankaranarayanan: In the datasets that you're using in this experiment?
Nikhil Prakash: In the paper that I'm talking about.
Aruna Sankaranarayanan: In the paper that you're talking about. Okay, okay, go ahead.
Nikhil Prakash: In this particular paper.
Nikhil Prakash: Okay, so, yeah, basically the same story in the… VLM setting.
Nikhil Prakash: But I think there's still a lot… there are still many questions that remains unanswered. So I think this is one of the first questions that we study in this paper, which is, where do
Nikhil Prakash: Where does… where does the ordering ID get formed?
Nikhil Prakash: Is it in the language model backbone itself, where we have already some evidence that it is present there?
Nikhil Prakash: Or maybe it is not getting generated in the language model, it is actually getting generated in the vision encoder, and is being passed on to the…
Nikhil Prakash: language model backbone.
Nikhil Prakash: Okay, so that's, like, one of the questions that we study. Another question that we study is how are they represented?
Nikhil Prakash: Is it actually localized?
Nikhil Prakash: Or, is it diffused across tokens?
Nikhil Prakash: When I say it, I mean…
Nikhil Prakash: Yeah, okay, I mean the ordering ID.
Nikhil Prakash: And again, we know that ordering IDs are present in the language model backbone.
Nikhil Prakash: So can we better characterize them?
Nikhil Prakash: I mean, what kinds of…
Nikhil Prakash: what kinds of representation are they? Are they encoding, like, a relative… sort of like a relative position, or are they encoding more like an abstract position, like…
Nikhil Prakash: This is the two coordinates in the image, or is it, like, the first object in the image?
Nikhil Prakash: So those are the three main questions that we study, and I think it will become more clear, I think, as I move… as I show the result and the experimental setups.
Nikhil Prakash: Okay. Yeah, so we studied these four settings,
Nikhil Prakash: 3 of them are synthetic, and one is actually coming from a real benchmark.
Nikhil Prakash: The first one is actually the simplest one, where we generate these squares of equal sizes, but different colors. They are either spread horizontally or vertically. Here, I'm only showing the horizontal one, but we have settings where
Nikhil Prakash: The squares and the shapes and the objects are spread vertically.
Nikhil Prakash: And the question that we ask is.
Nikhil Prakash: So let's say… let's assume the square… this particular square example… square image example.
Nikhil Prakash: So, for this particular square, we could ask, what is the color of the square to the left of the green square? Or the color of the
Nikhil Prakash: Squared to the… Left of green square is
Nikhil Prakash: And the answer should be red.
Nikhil Prakash: That's basically the task.
Nikhil Prakash: And we study two models, Quen and Gemma models.
Nikhil Prakash: And both the models can do this task perfectly.
Nikhil Prakash: Okay, so that was…
Nikhil Prakash: That's the setup. Now, we are starting to get into the experimental setups, or the experiments that we did in the paper.
Nikhil Prakash: Now, the first thing that we did was to actually cross-check if the…
Nikhil Prakash: VLM is, again, using this ordering ID information to solve this task or not. Even though the previous work has already shown that, but that was for a different task. So we wanted to confirm that this particular result actually generalizes to our setting or not.
Nikhil Prakash: So for that, we did, a patching experiment.
Nikhil Prakash: Which is pretty simple, actually. So we have this…
Nikhil Prakash: two samples. The first one is the clean sample.
Nikhil Prakash: Where the answer to the question is red, and for the sample on the right, the answer to the question is black.
Nikhil Prakash: Okay?
Nikhil Prakash: The other major difference between these two samples are…
Nikhil Prakash: In the first sample, the first square is the correct answer.
Nikhil Prakash: And in the second sample, the third square is the correct sample.
Nikhil Prakash: Okay? And we are doing the intervention at the last text token, which is in this particular setting.
Nikhil Prakash: So, we are taking the residual vector at the ACE token from the counterfactual run, and pasting it onto its corresponding location and layer.
Nikhil Prakash: In the clean run.
Nikhil Prakash: And then checking how does the final output change.
Nikhil Prakash: And we are expecting that if a particular layer is encoding the ordering ID information.
Nikhil Prakash: When that layer is patched from the counterfactual run to the clean run, the final output of the clean run should change from red
Nikhil Prakash: to blue Is that… does that make sense? I… I have just spoken a lot of words.
Nikhil Prakash: Okay.
Nikhil Prakash: So, we did that experiment, and this is the result. We see that in the later-ish layer, something after layer 20, we do see that
Nikhil Prakash: That the last,
Nikhil Prakash: token is encoding this ordering ID representation, and if we patch that from the counterfactual to the clear run, we ex… we get blue color as our… as the final output.
Aruna Sankaranarayanan: Nikhil, can you explain this again? Like, why would you not expect the color to be black? Like, why would you expect it to be blue, like, when you're doing this patching?
Nikhil Prakash: Okay, so idea here is, we are… Wait, wait, wait, would you expect it to be black?
Aruna Sankaranarayanan: I don't know, like, I feel like it should be black, yeah, I do expect it to be black.
Nikhil Prakash: That's reasonable. It should be black.
Nikhil Prakash: Didn't… didn't your figure in the next slide show that at the later layers, it does start getting black? Yeah, so you guys… yeah, you already have partial answer. I mean, you, yeah, almost have the answer. So, essentially, what the model does at the last token, it first forms this ordering ID information.
Nikhil Prakash: Of the correct square.
Nikhil Prakash: Which it needs to predict as the next token.
Nikhil Prakash: And once it has formed that ordering ID information, then it uses that piece of information to actually fetch the feature associated with that square that it needs to answer. So in this case, in this task, the feature is the color of that square.
Nikhil Prakash: So, in the later-ish layer, after it has formed the ordering ID, it actually uses it to fetch the color of that square, and hence.
Nikhil Prakash: In further later layers, we see black color as the final output.
Aruna Sankaranarayanan: Oh, so wait, sorry, so if I understand correctly, both the feature information as well as the ordering information is encoded in that representation you're using to patch, but the ordering information from the representation is recovered earlier, like, before the feature information.
Nikhil Prakash: Yes, you can say that. So, model is first forming the ordering information, and then using it to fetch the feature information at further layers, or later layers.
Nikhil Prakash: Could… would this be, like, a correct, very high-level understanding of, like, what this represents? Like, around layer 20, when that blue peak is happening, it's forming, like, the index to look up, but it's not actually doing the lookup yet, so when you patch right at those layers, you intervene on, like.
Nikhil Prakash: the location, that it's looking up before it actually retrieves the value there. Okay. That's exactly what… that's 100% correct.
Nikhil Prakash: I even had a question. Yeah, so, what exactly did you catch? Is this the M of the alpha, or something? This is just residual vector.
Nikhil Prakash: Was it true?
Nikhil Prakash: Okay, so from this experiment, we can say that even for this particular task, the VLM.
Nikhil Prakash: is using this ordering ID information to solve the task.
Nikhil Prakash: Okay, so now, coming to our… first… I…
Nikhil Prakash: Like, one of the main questions that we study in this paper.
Nikhil Prakash: Which is to understand where exactly this ordering ID information is getting generated in the VLM.
Nikhil Prakash: Yeah. Double question, you look at, the varying colors themselves.
Nikhil Prakash: Right.
Nikhil Prakash: just seeing, you know, this should be an invariant, so I'd be changing some, like, orange.
Nikhil Prakash: Brown, right?
Nikhil Prakash: Yes, I think I should have mentioned it. So this… this graph that you see is… is actually averaged over 50 samples. Oh.
Nikhil Prakash: Yeah, I think we should mention that. It's… yeah, and those 50 samples have various different combinations of colors.
Nikhil Prakash: So, it's not really specific to this particular sample. And yeah, so results are actually…
Nikhil Prakash: Averaged over a bunch of samples with different colors.
Nikhil Prakash: Okay, So you did residual…
Nikhil Prakash: Do you also look at… Any specific mechanism, like… Does this,
Nikhil Prakash: Like, I see that skins.
Nikhil Prakash: Like, 80%.
Nikhil Prakash: This is not an intervention like this, right? But even, even, even that, I would say, maybe, maybe there is some information in a…
Nikhil Prakash: in previous token, one or two previous tokens, like, square token might have some information about the ordering ID, which we are not really patching upon.
Nikhil Prakash: maybe that still holds the ordering ID of… Of the left square.
Nikhil Prakash: And that's why we're not seeing super high intervention. Well, you know more than this. I mean, when you get to the end of the paper, you're gonna say that…
Nikhil Prakash: Instead of saying maybe, you're going to say, oh, there is more than one.
Nikhil Prakash: Nope.
Nikhil Prakash: Oh, sorry, we are not seeing… 100%. Yeah, 100% here. Because there's one mechanism? Yeah. Because there's more than one mechanism? No, but this is… this is the end of both the mechanism. Both the mechanisms have already come by. They both come together. Yeah, yeah, yeah. This is almost the end of the process. Okay.
Nikhil Prakash: So end is… the end is merged. The end is merged. Yeah, end is almost merged. This is almost the end of the competition.
Nikhil Prakash: Yeah, so it's like, if the thing that you're patching in is indeed this, like, the right side index, then what? Cleanly patch it, then you should probably get it
Nikhil Prakash: Yeah, so my guess is maybe we are…
Nikhil Prakash: There are… there is some information maybe diffused over some other…
Nikhil Prakash: Tokens that we are not patching upon, and that's why we're not seeing a completely high… Intervention… effect.
Nikhil Prakash: Still my TV clone.
Nikhil Prakash: The gap between saying grid and saying that is so small.
Nikhil Prakash: Surprisingly.
Nikhil Prakash: But there is, like, there is overlap between where the ordering representation is affected, and the value representation.
Nikhil Prakash: So basically, what we see, when we see the blue peaks, it's already overlapped with it.
Nikhil Prakash: Yeah. Yeah, that's also a point. And also, like, since it's… it captured the ID, and it has to go look for it.
Nikhil Prakash: Can we say that most of this happens in attention?
Nikhil Prakash: Yeah, yeah, we can send it, yeah.
Nikhil Prakash: Yeah, in this work, we don't look into our attention heads, but… Yeah, I think previous works.
Nikhil Prakash: I'm not sure if… if you would get better performance, like, better effect just by patching in the heads. One point is what Tamar said. Maybe because of that, you might see
Nikhil Prakash: But I'm not 100% sure about that.
Nikhil Prakash: Yeah, if it's different heads doing the valuation… Yeah, yeah, then you might… you might get better slightly, yeah, yeah.
Nikhil Prakash: But we do hypothesize that it's very similar, the look-back, overall. Right, yeah. It's basically a look-back, yeah.
Nikhil Prakash: Okay, so where are these ordering ID information are getting formed?
Nikhil Prakash: So, the first experiment that we did was actually a probing experiment. We tried to… we asked this question of whether the vision embedding, vision token embeddings, which is fed into the language model backbone, if they already contain this ordering ID information or not.
Nikhil Prakash: And what we do is, we…
Nikhil Prakash: We take the representation of… or…
Nikhil Prakash: the embedding of each of the square tokens.
Nikhil Prakash: And, basically, just train a linear three-class classifier on top of, those representations.
Nikhil Prakash: We used, like, 90 samples for training and 30 samples for testing, and the probe accuracy was almost perfect.
Nikhil Prakash: So this, in a sense, shows that
Nikhil Prakash: The ordering information is already present in the vision token embedding, which is fed into the language model.
Nikhil Prakash: However, this was the most interesting result.
Nikhil Prakash: So what we did was, in…
Nikhil Prakash: So we train the probe only on the square token, like, the embeddings corresponding to the square token. But now, once we have the probe, we can basically apply the probe to every
Nikhil Prakash: Visual, visual token embedding.
Nikhil Prakash: So that's what we did. We took the probe, we applied it on each Each of the…
Nikhil Prakash: each of the patch, or its corresponding embedding, and this is the result that we get. What we found was…
Nikhil Prakash: For the probe that was trained to find the ordering information of the first square, it has a
Nikhil Prakash: Good effect, or it has a good, probing accuracy on the test set on entire strip which is actually covering that square.
Nikhil Prakash: So, this is where the square is supposed to be, somewhere here, but we found that this ordering ID information is actually spread across
Nikhil Prakash: This entire strip.
Nikhil Prakash: Hop me up, question.
Nikhil Prakash: Okay. So, yeah, so this… so this is the image, okay?
Nikhil Prakash: We break down the image into individual patches.
Nikhil Prakash: Those individual patches become the tokens.
Nikhil Prakash: which is fed into the VIT, and then the projector, and then we have these embeddings, which are fed into the language model backbone.
Nikhil Prakash: We took… we took those embeddings, which are fed into the language model backbone, Corresponding to the square… square…
Nikhil Prakash: Tokens only.
Nikhil Prakash: So we took the… So we know that which token, which…
Nikhil Prakash: like, the 100 token corresponds to the first square. 200 token corresponds to the second square. So we only picked up 100, 200, and 300,
Nikhil Prakash: Vision tokens.
Nikhil Prakash: Okay.
Nikhil Prakash: Because those corresponding to… those are the ones for the square tokens. Those are the ones which is encoding.
Nikhil Prakash: square.
Nikhil Prakash: You can physically point to it if you need to.
Nikhil Prakash: You took it from the image? That's how we created the image. So we created the image, we have full control of where we can put the square in the image. So we put it in a specific position in that image, so that when we break it down, it actually corresponds to the 100th token which gets generated.
Nikhil Prakash: Okay. And then what's the form?
Nikhil Prakash: Yeah, so the… once you have this 100, 200, and 300 active, like, token embedding.
Nikhil Prakash: Then the 100th one corresponding… it corresponds to, like, the first square, 200 one corresponds to the second square, and 300 corresponds to the third square.
Nikhil Prakash: That's what the… Probe is tasked to classify.
Nikhil Prakash: We're missing the size of the pitch.
Nikhil Prakash: Square is… now we are talking about vectors. This is embedding.
Nikhil Prakash: Each square… when you say square, I'm not sure. Yeah, there's so many squares all over the diagram, so yeah, I think that it might be helpful to answer. So this… So, so this is a vector here now. This is… that's not a square.
Nikhil Prakash: That's a vector. That's a vector. This is a vector. Yeah, this is a vector.
Nikhil Prakash: So, each square… Which one is the square? That's the square. This is… when I say square, I mean this… this square.
Nikhil Prakash: No. This is only one? Well, it just happens to be. It could have been, I mean, so, like, okay. Oh, sorry, yeah, we have four vector… sorry. But for understanding, let's say there is only one vector. No, actually, for understanding, it might be clearer if you… there's four patches.
Nikhil Prakash: They cover that square. Okay, let's say… yeah, in technical… in actual terms, there are, like, 4 vectors corresponding to each of the red, green, and blue squares, okay?
Nikhil Prakash: So, okay, so now you have… 4, 4, 12 vectors.
Nikhil Prakash: You got it?
Nikhil Prakash: So for the first four vectors corresponds to the first square.
Nikhil Prakash: That's the ground truth label of them.
Nikhil Prakash: The next four squares corresponds to the second one, and the last one corresponds to the third one. Essentially, that's what the probe is trying to
Nikhil Prakash: classifier. We feed each of these 12 vectors in the probe, and the probe is supposed to give me 1, 2, or 3.
Nikhil Prakash: And then you use the same probe to see
Nikhil Prakash: Other… Yeah. Other vectors? Yeah, exactly. So, is it one probe for all horizontal, vertical? No,
Nikhil Prakash: Horizontal is 1.
Nikhil Prakash: I mean, so this is a three-class classifier, so we basically have, like, three… three vectors. That makes sense. So you… the same image, if you flip it, you can still use the same token, right? Because the token dimensions, everything is the same.
Nikhil Prakash: No, I think we trained a different one.
Nikhil Prakash: It might work, though, it might work that the horizontal work might be able to work in the vertical work, but I don't think so we tried it.
Nikhil Prakash: Yeah, it might be more convincing that it's a positional.
Nikhil Prakash: Like, relative to missionary.
Nikhil Prakash: Thank you.
Nikhil Prakash: Yeah, that is so good. It looks like the… from the burning sewer, it's just cued into, like.
Nikhil Prakash: Positional coding of the token, not necessarily, like, the…
Nikhil Prakash: semantic relationship between the squares. Could be literally the X and Ys. Yeah. It should be like that. But you have some results that show that it's more…
Nikhil Prakash: Yeah, but sorry, why do you think it's X and Y? Because… Yeah, from the heat map, it just looks like…
Nikhil Prakash: what the signal the probe has cued into is just the positional coding of the token from the VIT encoder. When you say position, you mean X and Y coordinates? Yeah, just X and Y coordinates, yeah. Okay. It's not necessarily, like, the… because it seems like what you're after is, like, the relationship, or, like, the orderings of, like… So, the counter for that argument is, you think the…
Nikhil Prakash: X and Y of this particular square is…
Nikhil Prakash: Isn't the X and Y coordinate of this particular square, or this particular patch, is at the same distance at some square here? No, no, the X is… the X is different.
Nikhil Prakash: But the distance, if you look at… Just the X.
Nikhil Prakash: So it has picked up X… Yeah. But we do know now that you can change the position, if you change the position of this object inside the image, it does generate this.
Nikhil Prakash: So, if you take… Do you have that experiment somewhere? Yeah, but yeah, I do have that experiment.
Nikhil Prakash: I'm gonna go through that, but I'm still not sure about it.
Nikhil Prakash: Cause if you… if…
Nikhil Prakash: which is, like, flipped over its own vertical, and it still generalizes, I'd be, like, more convinced with it. Yeah, that I agree with. That, that makes sense as well. Do you have a probing one?
Nikhil Prakash: I just sent it. Oh, no, I did not include.
Nikhil Prakash: Should I open Discord now? Yes. Yeah, go on. You said it's interactive, not the formal talk, so here we are. You guys can see all of our talk.
Nikhil Prakash: Oh, man, don't… okay. Nobody saw it. Okay, then you, you, you, you've got to explain it, you've got to explain. Tamar needs to explain the experiments. We can't see. Oh, that's better, you can't see it.
Nikhil Prakash: I don't know, it's not applicable. That's right. Only about you. Only about me, that's right. That's right, that's right, especially about…
Nikhil Prakash: Especially that one. Special government.
Nikhil Prakash: Redhead's in trouble now.
Nikhil Prakash: No, it's not. Okay, never mind, it's not getting copied.
Nikhil Prakash: Say that again. Okay.
Nikhil Prakash: And then, what should I press? Watch out.
Nikhil Prakash: Okay, I'm changing my entire skin now.
Nikhil Prakash: It's okay, it's okay, it's okay.
Nikhil Prakash: You wanna explain what… What is this?
Nikhil Prakash: Okay, so we trained… it's the same probe, the same as, what, Anthony asked, so we trained the probe on the squares on the top.
Nikhil Prakash: Squares on the top. And something funny that we saw is that if you…
Nikhil Prakash: Try to train the probe.
Nikhil Prakash: on each layer of the vision encoder, it actually…
Nikhil Prakash: you get, like, 100% accuracy from layer 0, because it can overfit to the position embedded, and that's it. Yeah.
Nikhil Prakash: But what if you try to train the probe on one image, and then test the generalization on another image, where the position of the input is different?
Nikhil Prakash: And what we see if we do that is that we still get a nice accuracy on top of
Nikhil Prakash: the… Square itself.
Nikhil Prakash: But we still see these, like, strips.
Nikhil Prakash: In the original position. So the bottom one? Yeah, yeah.
Nikhil Prakash: Do you have the… do you have the plot?
Nikhil Prakash: So, here, let's just look at the, blue, green, and orange. So, the blue and green
Nikhil Prakash: It's just a quick frame, and the layer on the top… on the bottom is the varying color layers.
Nikhil Prakash: So we're here…
Nikhil Prakash: You can see that just from layer 0, you can get nice accuracy if you just train them on the squares or shifted squares, it can get it right.
Nikhil Prakash: In the orange one, it's actually the other image. So you train on one setting, and then test on different locations of the… of the squares. And you can see that it peaks on layer 15.
Nikhil Prakash: It doesn't get 100% accuracy, meaning that there is kind of like a mix of information, but some of the information is not the absolute location, but something relative about the order.
Nikhil Prakash: So, let's say, for your training set, the first square is always in token 100.
Nikhil Prakash: Yes, exactly. Now, in your test set, it said, like, token 2004. Yeah, but even be at the location of the third one. But it's still the first square. You're not shifting the squares, it's the zoomed out or zoomed version.
Nikhil Prakash: And, and then I think Nikhil has a nice, causal experiment that also validates that what these graphics.
Nikhil Prakash: Can you… maybe you said that and I wasn't confused. What's the input and what's the output of the probe?
Nikhil Prakash: The input of the probe is the embedding of the visual tokens, and the output is the order pick, whether it's the first one in the image, the second one in the image, or the third one. So, three labels? Yes.
Nikhil Prakash: Oh, you know what you should do? Like, for… Get another one. No, no. The same probe for horizontal, vertical, and, like, differences. Yeah. Yeah, yeah, yeah, yeah. So, for training, I mean.
Nikhil Prakash: Yeah, I think I could… yeah, yeah, yeah. I think that's… I thought that's what you meant, initially. No, I meant that one. I forgot about the… The joint one? Yeah, yeah, okay, okay. Yeah, no, I think that's a good one. Yeah, we can try that. So what is the training data that we're…
Nikhil Prakash: Memorize the addition again?
Nikhil Prakash: It's this… yeah, so it's…
Nikhil Prakash: I think Nikki will… will… will get to it, right? Showing that it's… that generalization, I'm not sure. The generalization is a mix of probably some of it.
Nikhil Prakash: Is actually memorizing the absolute position of the object in the image.
Nikhil Prakash: But some of it is actually the relative position between the objects. So whether it's the first one, the second one.
Nikhil Prakash: So, if you want to…
Nikhil Prakash: If you want to encode the order of things in your image, it can either be by just looking at the XY coordinate.
Nikhil Prakash: Right? Yeah, so in your data, your training data, the red one is always in position 100. Yes. What's the difference between the samples and the training data?
Nikhil Prakash: colors, different colors. Oh, just colors. Or in the other setting… So the first one is always in the… Yeah, the same location.
Nikhil Prakash: But it does generalize this to different locations to some extent.
Nikhil Prakash: Different locations coming in different orders. Different picks a little bit. Different X and Y. You ship them all. Yeah.
Nikhil Prakash: On smaller squares, more tighter together.
Nikhil Prakash: It's not sure it's, like, super related, but I would say, like.
Nikhil Prakash: So right now, you're training the probe in a very specific way, where the squares are always at the same location, and then you're stress testing in different scenarios.
Nikhil Prakash: I wonder whether things can go the other way around. You can… Actually train a stronger probe.
Nikhil Prakash: by training on a more diversity of, like, square decisions, and then testing on another distribution. That's sort of Rohit's second suggestion, right? Is that right? No, that's okay. No, no, you're right, I think that's right.
Nikhil Prakash: Yeah, so you could… that'd be… your data is a regularizer then, and you're trying to get the generalized data out of it, yeah. And so another question would be, I'm really curious about, like, whether, say, like.
Nikhil Prakash: It works only on the VIT or CRAM, which is trained end-to-end for visual question answering.
Nikhil Prakash: it works for, like, Dino or Cliff, or… I don't know, maybe not Cliff, but, like, Dino works, like… So, we did study Dino, but, this… the results that I've shown here, they generalize at least,
Nikhil Prakash: At least with Gemma models? I don't know if Gemma is clear.
Nikhil Prakash: Oh, I think Lava… Lava is… is Lava is clip. I think Lava is… Lava is separate. Lava is, like, using a pre-trained vision code. Oh, I see, then I don't know. But… Okay. But, yeah, so at least the results generalized to Gemma model?
Nikhil Prakash: And most of the results are also generalized to pixel.
Nikhil Prakash: Pix… pixel.
Nikhil Prakash: But some of the experiments, I just still need to do on PixTrill. But so far, the experiments that I've done on Pixel seems to be okay, or consistent with the results that we have.
Nikhil Prakash: Yeah. Someone else about it.
Nikhil Prakash: augmentation, regularizing exposure, having to be sure that It's, like, one… We're catching the same mechanism.
Nikhil Prakash: What if there's just, you know, there's a mechanism for it?
Nikhil Prakash: safety relative ordering.
Nikhil Prakash: horizontal places that are close to the middle, and, like, a different one for, like, it's, like, new that's positive or anything, but…
Nikhil Prakash: You think there's, like… Yeah, you think there's, like, a risk of something like that?
Nikhil Prakash: When you're training probes, like, the kinds of probes that we…
Nikhil Prakash: Train, or is it more like training the probes with all different kinds of configuration?
Nikhil Prakash: I mean, I guess in general.
Nikhil Prakash: Because it's… there's, like, a range of, like, you know how specific versus how generalized.
Nikhil Prakash: your… the kind of settings that you're trading these folks.
Nikhil Prakash: You know, there's something to be considered there with respect to, like.
Nikhil Prakash: The underlying mechanisms and how specialized or generalized they can be.
Nikhil Prakash: Okay, so my…
Nikhil Prakash: My answer to that would be if…
Nikhil Prakash: The mechanism between these two types of, like, configuration are significantly different.
Nikhil Prakash: Then maybe the… The training or the test accuracy of those probes will tell you that.
Nikhil Prakash: That maybe you're not able to train
Nikhil Prakash: A very good classifier, like, a very good pro.
Nikhil Prakash: Either in terms of your training law… training accuracy, or in terms of your test accuracy.
Nikhil Prakash: I think that's what I would assume.
Nikhil Prakash: Makes sense, yeah, it just feels like it's very hard, like… Yeah. It's not grounded.
Nikhil Prakash: I don't know what numbers for this thing.
Aruna Sankaranarayanan: I had another question. It's… how similar are these ordering IDs to, like, the position embeddings of these tokens? I guess in the… in the vision language model, it's, it's probably less useful to look at this, but say in the case of…
Aruna Sankaranarayanan: just like the LLM, just like in terms of the words.
Aruna Sankaranarayanan: How similar are the vectors, like, from the… from just the position embeddings and these ordering, IDs?
Nikhil Prakash: That image she just showed us… can you go back to the image you showed from the Discord?
Nikhil Prakash: This one?
Nikhil Prakash: Yeah, this one.
Nikhil Prakash: No, no, no, no, the one you were just on.
Nikhil Prakash: Like, my understanding of what this is saying is that, like, If we, like.
Nikhil Prakash: If we try to deduce the algorithm, the linear probe is learning. It's learning, like, a Mix of, like.
Nikhil Prakash: this is evidence that is just reading off the positional embedding, right? And then this faint little patch here is evidence that there's… there's actual generalization, or, like, relative work. Is that a correct interpretation of results? Okay.
Aruna Sankaranarayanan: Got it. Thank you, yeah.
Nikhil Prakash: And the same pro…
Nikhil Prakash: Is that right? Yeah, I think it's clearest in, like, this third image, right? You see this… this little guy, which definitely corresponds to the relative. And then maybe if we trained a Rohage probe.
Nikhil Prakash: And you could just get that by itself, right?
Nikhil Prakash: Okay, you should name that. Yeah.
Nikhil Prakash: I'm willing to share the curses.
Nikhil Prakash: Yeah, I hope that, yeah, I think that answers the question. So I guess we can… I don't think the question. I mean, I want you to explain to me the picture.
Nikhil Prakash: This figure? Yeah.
Nikhil Prakash: Wait, so you understood the probe, right? Yes. So we just take that probe, so the probe was trained only on the square tokens, but now we apply the same probe across all the tokens, including the background tokens, and this is the result that we get.
Nikhil Prakash: This is the test accuracy. Test accuracy. Yeah, test accuracy of the probe.
Nikhil Prakash: I'm so busy.
Nikhil Prakash: So, so, so assume that you, you're looking at the first vision token.
Nikhil Prakash: The first vision token.
Nikhil Prakash: In our image, that would be something like this.
Nikhil Prakash: Just a blank batch.
Nikhil Prakash: White patch?
Nikhil Prakash: No, that's not in the training data.
Nikhil Prakash: So how does the test data work?
Nikhil Prakash: The test data, just take the…
Nikhil Prakash: The input to the blog is a token embedding. Yes. During training, they only show
Nikhil Prakash: tokens from the squares, exclusively. Different, different colors. Different colors, different colors. But it's always the exact same shape. Always the same.
Nikhil Prakash: We also have other data sets where we… this is the simple data set of squares. It can be objects, it can be… For this setting, what we're talking about right now, the training is these three squares, just different colors. Yeah, okay, and then how does the test data look?
Nikhil Prakash: The test data log also includes the embedding of all the tokens in the image. Oh, but it's still the same… the same shape? Yes, but it's… but it's the held-out set of images, so… Okay. But…
Nikhil Prakash: But still there's three. Yeah, for example, in the object setting, when you have, like, tiny images, different objects, it can be completely different objects.
Nikhil Prakash: But yeah, the same configuration, just different colors.
Nikhil Prakash: Okay, and then we apply the probe on all the patches, all the tokens corresponding to each of the patches. And then we see that the test accuracy is not only high on the square tokens, which is… it… it is supposed… it was straight on. We see high test accuracy also on the background tokens.
Nikhil Prakash: background tokens, which are sort of, like, creating a strip around the square which it was trained on. So it knows to say, no, there's… this is not the first position. It knows that this is the first position.
Nikhil Prakash: So, so if you take that first patch that I was talking about.
Nikhil Prakash: you take its corresponding embedding, and then use your probe, then it will say that, okay, this is the first position. Like, this is… this is the first one.
Nikhil Prakash: 200 labels that it can predict.
Nikhil Prakash: It's still 3 labels. Yeah, labels is still three. It's doing something silly, like, instead of doing… So this are the labels, like this? Yeah. Okay. Yeah, so this is first… labels are just first square, second square, third square, that's it. There's this three class… classifiers.
Nikhil Prakash: Yeah, I understand, but for the Y1, what is the label? That was my question. What is the gold label? The Y patch? It doesn't have a label. It doesn't have it, yeah. On test time. We did it print it, and now we say, oh, interesting, but…
Nikhil Prakash: So, like, something that will be very not surprising will be if the white background will have zero accuracy for all the people observed
Nikhil Prakash: When you say accuracy, what is the gold label of the white background? Accuracy is whether the problem… we have three prompts. Each prompt is accurate if it predicts the right order. So the first prompt predicts first order.
Nikhil Prakash: Right? I'm the first opto.
Nikhil Prakash: Second prompt, predict, under second. So the background, what's the order of the background? It doesn't have order, but it seems like the vision encoder assigned an order to the background, although it doesn't… it doesn't supposed to have… So it's not accuracy. It is accuracy. Oh, I understand what your question is. So when you… when you give the white patch.
Nikhil Prakash: When you give the white patch embedding as an input, the output is a softmax of 3 levels.
Nikhil Prakash: whatever the accuracy is for the first token, like, the first class, they're plotting that. Yeah, that's the prediction. Assume that we have… I think I got the… It's 90% accurate, 90% confident. That's confidence, actually. 90% confident that it's coming from class 1.
Nikhil Prakash: Yeah, so that's not accuracy. That's not accuracy, that's confidence. Okay. Yeah, it's first-class confidence. It's very confident that the white patch from above the red square… But maybe what they're showing here is… No, this is ac… this is accuracy. So we… let's assume that
Nikhil Prakash: the ground truth is the first one, is the first square, for the first patch. No, what you're showing here is you're saying… you're saying accuracy, if every patch…
Nikhil Prakash: the ground truth was zero, like, for the whole image. Yeah, exactly. Then you're showing accuracy there. That's not accuracy, because everything has the same label. Right, but it's just, it's just number of predictions. That's not accuracy.
Nikhil Prakash: Show the accuracy of the bat to show that the background content actually contain information relevant to the plot.
Nikhil Prakash: As if the probe was trained to predict the mechanism. Sorry, but I shouldn't treat it as accuracy, but look at it as the probe prediction. Yes.
Nikhil Prakash: Yeah, the word accuracy was confusing. It implies there are labels. Wait, I'm not sure if I follow completely. This… you're saying this… you're also saying this is not accuracy? We… I think we can find a way to define it, but what we actually care about is what is the prediction of the problem in these tokens.
Nikhil Prakash: Yeah, and we check it if it's…
Nikhil Prakash: We check it by saying… let's assume all the labels were 1.
Nikhil Prakash: Yes, exactly. So we are checking the accuracy here. Except that… except that this is subjection, to use the word accuracy.
Nikhil Prakash: when what you're measuring is not whether it's right or not. Yeah, it's not like accuracy. Oh, okay. You get one if it's right and zero if it's wrong. And you're using accuracy just as units. Yeah, accuracy for a particular label.
Nikhil Prakash: But, just, just, find it very well. What's the top?
Nikhil Prakash: If you do maximum on the submax, what would be the prediction?
Nikhil Prakash: Okay, so whether it will be 1 or… 90 times out of 100.
Nikhil Prakash: The white label has been predicted as plus 1.
Nikhil Prakash: See you bye.
Nikhil Prakash: Call that predictable. 90 out of…
Nikhil Prakash: Right? Like, if that's 90, like, let's say the accuracy… Prediction rate? I don't know. That sounds like accuracy to me. Well, but not to other readers, right? Okay, okay, okay, maybe we can change the wording there. Okay. The accuracy implies that… Right, it's like, if you have a high accuracy, that's good, but if you have a high prediction…
Nikhil Prakash: of something that…
Nikhil Prakash: a white token is predicted to be, like, the red… Okay, that makes sense, that makes sense. Okay. I think the problem with, recalling this prediction is we didn't expect this prediction. It's not like we tested the hypothesis… It's okay, that's still predicted. Yeah. Yeah, that's okay, that's okay. It's not that you predicted the model. Yeah, that's okay. Yeah, we should have a look at how we look at this. Okay, but, yeah, okay, so the main, main result
Nikhil Prakash: Well, main takeaway is…
Nikhil Prakash: that the ordering information of the square is not concentrated or localized only on the square tokens. It is diffused across background tokens as well. Can I say it again?
Nikhil Prakash: Yes. That I said in the start. As if what? It's a surprise, like…
Nikhil Prakash: You have objects, you… they have a certain order inside the image.
Nikhil Prakash: But now it seems like that order is not associated with the object itself, but kind of like a strip around.
Nikhil Prakash: Like, so it means that the pro…
Nikhil Prakash: This reaction on the top is in position zero, right? The yellow… the yellow points show that, let's say, on the top, the probe predicts this to be in position zero. Yes, that's right, yeah. This is what it does, the prediction rate, yes.
Nikhil Prakash: Do you have also a question? Yes, let's… What do you think about this, like…
Nikhil Prakash: So, like, here's, like, a formal construction that I'm thinking about in reference to this. If I trained a linear probe on the same classification task, but the only inputs I gave it were the positional embeddings with, like.
Nikhil Prakash: maybe, like, the Y values ablate… if, like, with the information related to Y ablated out. Like, no square, no nothing? Yeah. Like, I… for one, for this… for the specific task you described, which is not the rowhead version, it would do a perfect job, and then if you actually produce… reproduce this figure, it would look like perfect columns, right? So what's, like… I don't… what's the evidence that it's not… that your linear curve is not just doing that? Are you ready for a review?
Nikhil Prakash: I think we should move to the causal experiments.
Nikhil Prakash: Okay. Also, what's the… just last question, what layer is this on?
Nikhil Prakash: Just the embedding of the… This is input… The last layer of the vision model. After the projector. Last layer of… after the MLP. After the projector. Input to language model backbone.
Nikhil Prakash: Even past the last day of the vision. Yes.
Nikhil Prakash: just…
Nikhil Prakash: If you can go back, sorry to delay this one a bit more, but to maybe this way, like, that objection, if you trained on something like this same dataset, but the three squares are kind of on the top.
Nikhil Prakash: then you would have examples where you have the exact same square, but in this case, it's the second one, and in this case, it's the first one. So it's exactly the same positional embeddings, but it's kind of… you're kind of showing that it's figuring out… Yeah, you also have that. I think just continue.
Nikhil Prakash: Okay. So, this is probing result.
Nikhil Prakash: This shows that, okay, maybe the order information is diffused, or it is at least present across a bunch of different tokens, but we all know that probing only is correlational results. It does not say anything about the causality, the information might still be present there, and it might not be used by the model. So we do this intervention experiment to actually show that the information present in the strip is causal in nature.
Nikhil Prakash: And this is the two experiments that we did.
Nikhil Prakash: so the difference between clear and counterfactual is that the order of left and right
Nikhil Prakash: squares are reversed. In the clean one, red is the left one, and on the right, on the counterfactual, the right… right one is the right one.
Nikhil Prakash: Okay, so if we do… If the model is creating this ordering ID information, then when we do
Nikhil Prakash: This two patching, the color of the square should remain the same, while the ordering information should get reversed.
Nikhil Prakash: And because of the reversal of the ordering ID, the final answer of the clean range should change from
Nikhil Prakash: Red to blue.
Nikhil Prakash: Here we're asking what is to the left of the green color. Okay, so that's pretty much the experiment. And we do intervention.
Nikhil Prakash: On either only the square tokens, or on the entire strip tokens.
Nikhil Prakash: Including this one.
Nikhil Prakash: Including the square.
Nikhil Prakash: So this is the result for patching in the entire strip.
Nikhil Prakash: X axis is the embedding.
Nikhil Prakash: embedding layer, and the later layers in the language model, backbone, and Y-axis is…
Nikhil Prakash: Let's say it is prob… yeah.
Nikhil Prakash: I, aye.
Nikhil Prakash: So we see that, okay, even when you patch in the embedding layer.
Nikhil Prakash: We have very high causal effect.
Nikhil Prakash: That the model starts to think that the square to the left of the green square is actually blue.
Nikhil Prakash: And not red. Right from layer 0. Right from layer 0. Right from layer 0, it says the incorrect answer.
Nikhil Prakash: Yes. One second, one second.
Nikhil Prakash: But if we do the patching on the square tokens only, not the entire strip, we don't see that effect. The model still thinks that the red is the correct answer and not the blue, if we do patching only on the square token.
Nikhil Prakash: So… Okay, so, if you think of…
Nikhil Prakash: these two results combinedly. What that means is there is causal effect
Nikhil Prakash: associated with the background tokens. If you don't touch the background tokens, the model
Nikhil Prakash: Model does not really have enough causal effect to be able to change the final output.
Nikhil Prakash: So, essentially, what that means is that entire strip has ordering information, which is causal in nature. It is not only
Nikhil Prakash: just there. Which is causally relevant to its accuracy, right? Exactly, yeah.
Nikhil Prakash: Question? Yes.
Nikhil Prakash: I, I, I wanted to ask what you fetch.
Nikhil Prakash: What do I patch?
Nikhil Prakash: And as I understand later, is that you passed the goal strip.
Nikhil Prakash: We do two kinds. We either patch the square, or we patch the entire strip. And when you patch the square, is it all the form? Yes.
Nikhil Prakash: It's only those four, and when we pass the strip, it's those four, plus all the…
Nikhil Prakash: Background tokens as in that strip.
Nikhil Prakash: I wasn't welcome this.
Nikhil Prakash: This is more convincing.
Aruna Sankaranarayanan: Question?
Nikhil Prakash: Maybe it's just apart from that.
Nikhil Prakash: Your internet connection is unstable.
Aruna Sankaranarayanan: No, I… I had a question about, instead of patching the whole script, whole strip, if you just patch that red square, but onto a different position on that first column.
Aruna Sankaranarayanan: You know, so you're essentially changing the position of the square. Yeah, I don't know, this is an annoying question.
Nikhil Prakash: No, I think that's a good dynamic, that's a good question. We have that experiment in one of our later slides. I'm gonna show that.
Aruna Sankaranarayanan: Okay, okay.
Nikhil Prakash: So… Can we look at the graphs in the meantime, or if people ask some questions?
Nikhil Prakash: When you do the patches, the red, the… One side to the other.
Nikhil Prakash: the positional encoding embedded in those tokens when you're shooting people.
Nikhil Prakash: Like, is it, like…
Nikhil Prakash: It appears to have the same positional coatings of having been on the right side, and it's swapped over, or is it…
Nikhil Prakash: the positional encoding. It's given the same positional encoding as if it was on the left side.
Nikhil Prakash: You got that question?
Nikhil Prakash: Maybe what do you mean by position? Because I think the previous discussion about the probing was, like, oh, like, maybe, like, the ordering is just comes from, like, the positional coding that you give the Hilkins before it enters all the stuff, and…
Nikhil Prakash: I guess what I'm worried about is, like, is…
Nikhil Prakash: Is, like, is the tokens, do they contain that relative position on coding?
Nikhil Prakash: When you're swamped, when you do the matching.
Nikhil Prakash: So it's almost an architectural question you're asking, because for some transformer implementations, The position I'm quoting is…
Nikhil Prakash: directly added into… Yeah. …the… you know, the vectors, and in other transformer
Nikhil Prakash: representations. The positional encoding has never materialized.
Nikhil Prakash: It's just implicitly added as part of attention.
Nikhil Prakash: Later on. And so, I think you're asking, oh, when you patch over.
Nikhil Prakash: I… are you patching over from…
Nikhil Prakash: Are you patching vectors where the positional encoding has already been added, or are you patching over vectors where the positional encoding has not been added yet?
Nikhil Prakash: And it'll be added afterwards.
Nikhil Prakash: So that's the question? Yeah, yeah. So the answer for that is, so even in the VIT, the vision encoder, there is positional information.
Nikhil Prakash: And… You mean where? Before you patch or after you patch? At each… before, before… Yeah. Yeah, yeah, yeah. Right, before you patch. Yeah, before I patch. There is rope there in the version encoder as well. Okay. So, technically, it has… it has…
Nikhil Prakash: access to that positional information. Okay. And the other thing is, here I'm showing not the… not only the result of the embedding layer, I'm showing the result across
Nikhil Prakash: bunch of layers in the language model. So even in the language model, there is rope. Okay.
Nikhil Prakash: So, it has access to both those push information.
Nikhil Prakash: clarified the question a little bit. So I feel like, because transformer is permutation invariant, so there's no way, like, the model is able to figure out this test without position coding. Yes. So the question is more, like, how, like, these models
Nikhil Prakash: encoding, whether it's just, like, hard-coded application, or actually learning a general notion of finding, like, object-relative associations with, like, the relative
Nikhil Prakash: Yeah, yeah. It's more like how they're using this. Yeah, I think that's a good question. I think before I was… before we ran some results a few weeks back, my understanding was it's the second thing. The models are creating some form of, like.
Nikhil Prakash: abstract relative positional information to actually do the binding. And that's what we have seen in most of the results in the language model space. But I think I have a few results in some of the later slides, where this
Nikhil Prakash: this ordering ID business in the vision encoder seems a bit more complex, bit more complicated. It's not only those relative abstract position information, but
Nikhil Prakash: The model seems to be encoding some form of absolute positional information as well.
Nikhil Prakash: Even though they're using Rove, 2D Rope. Yeah. Okay, got it. It's also…
Nikhil Prakash: At least my intuition is that it's easier for language models, specifically.
Nikhil Prakash: Yeah, this, like, idea of positioning.
Nikhil Prakash: Right. Specifically because of the, also masks.
Nikhil Prakash: Like, you don't even need any, like, robot, you can just do language modeling.
Nikhil Prakash: Without any positional encoding, and it works because of the events.
Nikhil Prakash: It's another, like, extra signal that we have.
Nikhil Prakash: Yeah, so that's, like, that's, like, one of the evidences that you can use to say that the models can create
Nikhil Prakash: Or at least language models can create more abstract
Nikhil Prakash: Sort of, like, positional information in its internal representation.
Nikhil Prakash: the defense of the vision encoder, you can also say that
Nikhil Prakash: Because it doesn't have this frozen masking.
Nikhil Prakash: It could generate this, script-like.
Nikhil Prakash: Yeah. Otherwise…
Nikhil Prakash: Otherwise, each token can only see The one before it.
Nikhil Prakash: I think Rope still allows it to do it, but I just mean that for language models, it's, like, more of an inductive bias. Yeah.
Nikhil Prakash: Yeah, I agree with that.
Nikhil Prakash: About the causal experience.
Nikhil Prakash: What if the… the information is just spread, not only in this industry, but
Nikhil Prakash: It just spread all over the image.
Nikhil Prakash: And you need to change enough
Nikhil Prakash: content in order to the language model, because the information is all around, not only with the strip. And you make… you make the experiment only with the script, so this is what we see.
Nikhil Prakash: But maybe if you just… Good.
Nikhil Prakash: Yeah, it's both been enough,
Nikhil Prakash: Yeah, so I think I would say that
Nikhil Prakash: I mean, yeah, you can just take a few tokens around the square token, and maybe if you pass that, it might work.
Nikhil Prakash: That's okay, but I think the probing results show that the result is diffused across the strip, so I think it becomes more methodological to catch the entire strip.
Nikhil Prakash: Because there is no… For mobility?
Nikhil Prakash: Various in the… in the data set. Like, all… all the… all the squares are in the same position.
Nikhil Prakash: But I think… If there was…
Nikhil Prakash: That becomes more easier thing, so if I would have trained it across a bunch of different squares in, like, different positions of the Y axis, then it becomes much more expected that, okay, you will see the strip.
Nikhil Prakash: But now that we are only training it on just one position.
Nikhil Prakash: And even then, we are seeing the effect on the entire ship. I think that's more surprising.
Nikhil Prakash: probing the probes, it's much easier for the probe to take into account only the X X.
Nikhil Prakash: Because it's always the same place.
Nikhil Prakash: In the Y, it's always in the same place, so the problem doesn't need the information of the YX.
Nikhil Prakash: Okay.
Nikhil Prakash: I think a nice place I would be.
Nikhil Prakash: to do the PG experiment with…
Nikhil Prakash: A strip, or whatever, like, that has…
Nikhil Prakash: The exact same number of tokens.
Nikhil Prakash: as the… the original three-fits Department?
Nikhil Prakash: But, like, shaped otherwise. Like, it can be, I don't know, instead of, for example, vertical, it can be diagonal, it can be square over the square, but the same number of tokens.
Nikhil Prakash: And what will that tell us? That it's not… just…
Nikhil Prakash: of the portion of image information that he took for getting his name. That's it, but…
Nikhil Prakash: Exactly that, like, structure inside.
Nikhil Prakash: Okay, as a baseline, maybe, you know, you can use it, okay, okay.
Nikhil Prakash: like, a technical question in my stock suggestion. So, technical question is, like, these are 4 tokens in your deck square.
Nikhil Prakash: And you are giving… The top two and the bottom one.
Nikhil Prakash: Right? Like, in the red square of the first.
Nikhil Prakash: So the two tokens from top and two tokens from local.
Nikhil Prakash: So, like, a good neural network should learn.
Nikhil Prakash: That, okay, anything that is, like, stacked on top of, like, these tokens should be in, like, class 1.
Nikhil Prakash: So… like, let's say the Rohit…
Nikhil Prakash: Shenyu Pro, was just to end on this and that, like, only these two cases, but different colors.
Nikhil Prakash: Okay. Same probe. Okay. Like, same positions. Yeah, yeah, yeah, yeah, no changes in position. So, if my hypothesis is correct, then you should see only the…
Nikhil Prakash: like this, the position that you're seeing, only, like, in a… in a L shape.
Nikhil Prakash: in,
Nikhil Prakash: L shape, yes? Yeah, because the first… Yeah, the first, yeah. So that tells it's probably not positional embedding, it's something… sorry, probably not ordering representation, it's more positional embedding. If you don't see it, if you still see this happening, like, where you're seeing, like, all these entire columns coming out to be
Nikhil Prakash: In the test, if you're seeing the same structure, then maybe I'll be more convinced that, like, it's an ordering process.
Nikhil Prakash: So, if it is L…
Nikhil Prakash: Yeah. Then I think just… Then it is just, like, absolute… it is encoding… learning X and Y coordinates. Yeah. It is learning X and Y coordinates. Okay.
Nikhil Prakash: Okay.
Nikhil Prakash: Like, the other probe, when you still see, you have a different arrangement of the image. You still see the street in, like, the same location as the original probe.
Nikhil Prakash: Yeah. So that's the overfit to the XY reporting. Yeah. But you do see some generalization, also for the objects.
Nikhil Prakash: So, that… like, I have another counterpoint for you. Like, it's probably, like, this is a square top.
Nikhil Prakash: Like, a square representation.
Nikhil Prakash: Right, but it does… I don't know. But it does know… It's like a square.
Nikhil Prakash: To extract the order.
Nikhil Prakash: on the square. Yeah. So it knows, okay, oh, this is a background, and I actually need to… in the background, I need to overfit with XY. Yeah. But if I have a square, then suddenly I have more information. Yeah. And that's the information it uses. So there is some information of the square that is relevant to the information. Okay, so…
Nikhil Prakash: that makes sense. And, like… It'd be nice, like, if…
Nikhil Prakash: It's, like, slightly more strong, I think. Yeah.
Nikhil Prakash: Also, second thing is when you do patching from your entire white slab.
Nikhil Prakash: So this is, again, autoregressive in language space, correct? All these tokens. So the bottom token of the red probably already knows it comes from the column of the red token.
Nikhil Prakash: But the top white patch might not know. No way, it still knows, because it comes from digital.
Nikhil Prakash: Good clean. No, that's… yeah, that's it, yeah.
Nikhil Prakash: Okay, anyways, I think it's, shaquille, practical question.
Nikhil Prakash: How many more slides do you have?
Nikhil Prakash: I think that's one of the main reasons.
Nikhil Prakash: No, actually, there's one more main… other thing. One is into the other main result?
Nikhil Prakash: So, should I take 5 more minutes? Sure. Okay, so… okay, so those are the results for showing that the vision encoder already encodes this ordering information.
Nikhil Prakash: of the squares, even before the language model gets into the picture. So then what's the… then the question is, what's this language model doing?
Nikhil Prakash: Is it just using that piece of information, or does it create its own set of… Like, ordering info.
Nikhil Prakash: Information of what is happening.
Nikhil Prakash: So, to answer that question, we did another patching experiment.
Nikhil Prakash: Here, the idea is to basically ablate out or remove.
Nikhil Prakash: all the ordering information from an encoder. So the way we do that is, like, we create a synthetic image, like a representation of a synthetic image, where the representation of each square is… is, like, taken from the…
Nikhil Prakash: Like, a different image.
Nikhil Prakash: with the same square placed in the middle, and that's the only square. So the idea here is.
Nikhil Prakash: If the model is encoding that this is the first square, second square, third square, then if we just take their representation from a single square image, then…
Nikhil Prakash: the representation of each square should just be… just be saying that, okay, I'm the first one, I'm the first one, I'm the first one.
Nikhil Prakash: And we also patched the activations of the backward… sorry, not backward… background tokens to destroy…
Nikhil Prakash: the information in those background tokens. Okay? So if we do that batch… patching.
Nikhil Prakash: Those set of patching. The… the ordering information from the vision encoder should get destroyed.
Nikhil Prakash: And if you do that, the behavioral performance decreases.
Nikhil Prakash: It decreases significantly.
Nikhil Prakash: Which is kind of expected. But the point here is it still is above the chance. Chance is 33%. It's not near 33%. So that means language model does seem to do something.
Nikhil Prakash: So then we check if the models, or the language model does create its own, like, ordering information or not, by just doing the same piece of experiment. But now, the
Nikhil Prakash: The ordering information from the version encoder are destroyed.
Nikhil Prakash: And then, in this experiment, we only need to do the patching on the squares, we don't need to do the patching on the entire strip.
Nikhil Prakash: And it's the same experiment, and this is the result.
Nikhil Prakash: What this shows is, even the language model does seem to create its own ordering information in some of its middle…
Nikhil Prakash: Middle layers.
Nikhil Prakash: Yeah, so that's… So the x-axis in the previous experiment.
Nikhil Prakash: with layers of the vision encoder, and then the x-axis… No, this is both, language… Both the previous experiment and this experiment, but they're both language models. Yes, language model by both.
Nikhil Prakash: There, we saw the effect right from the start.
Nikhil Prakash: Okay. Because the information was provided by the vision encoder itself. So the input of the language model backbone already has the information. I see. But here, that information is not present.
Nikhil Prakash: So, we only see the alignment only in the middle-ish layer of the language model barrier. I understand.
Nikhil Prakash: It is much more similar to the first example with bleeding, with the upper leaves in box A. This is the mechanism that drives that behavior, so it's the same… So this is the language model finding? Yeah. Yeah.
Aruna Sankaranarayanan: But is this a language model dealing with the tokens, the language tokens, or is this a language model dealing with the spatial token information? Like, you're saying that there's some, transformation of that representation which also exists in the LM backbone?
Nikhil Prakash: It's the second one, and I'm not sure about the transformation. Transformation occurs before the LLM backbone, which is done by the projector.
Aruna Sankaranarayanan: So it's just, like, some… it's retaining some information, but from the vision model, but in the LM space as well. The spatial information from the pictures, from the images are retained in the LM space as well.
Nikhil Prakash: So that's the general claim, that the vision encoder is encoding the ordering information.
Nikhil Prakash: But let's say if you get rid of that information coming from the vision encoder, the language model also…
Nikhil Prakash: Generates its own ordering information.
Aruna Sankaranarayanan: Got it, but the information generated by the language model is pertaining to the spatial information, right? This is not from the language tokens. This is, again, like, related to the…
Aruna Sankaranarayanan: Yeah, yeah, facial, okay.
Nikhil Prakash: We are operating on the visual token.
Aruna Sankaranarayanan: Okay, okay.
Nikhil Prakash: So that's… that's the second main result, that even the language model does seem to create its own outing information.
Nikhil Prakash: It's a little bit better left to right than it is up and down. Yes, that is right.
Nikhil Prakash: I had a really quick question. So, is the assumption that the LLM does this because you are basically preventing the vision of the British from doing that, or do you think that they both do it? Because I think you could validate this by doing the patching of the original LLM states
Nikhil Prakash: Well, you're not matching the vision encoder.
Nikhil Prakash: this LLM states in the vision encoder to test whether the LLM is having a compensatory behavior, right, or not. Say again? Yeah, I did not follow the experiment. Yeah, so kind of like, right now you're fetching the vision, right? So you know that in the vision, there's nothing going on in terms of positions, but now you're wondering, is…
Nikhil Prakash: are these results from the LLM emerging because the LLM is compensating from the Latin positional information, or are there also before? Yes. So then you could try to get the original activations from the previous LLM and patch them over the current ones.
Nikhil Prakash: While also patching the vision.
Nikhil Prakash: part, right?
Nikhil Prakash: So if the previous LLM where the vision was normal was not doing this, because it saw kind of like, oh, the vision part is already handling this, so I don't have to do it, now you would see that basically patching both would cause, like, the performance to go to zero, or something like that, right?
Nikhil Prakash: I'm still not 100% sure. If you patch in… yeah, I think we can talk about it. It's not a very, simple thing to do. But I think that's a good question. Here, what we are arguing is, it is…
Nikhil Prakash: the second case, I think what you said, that both of them are happening simultaneously.
Nikhil Prakash: Right. It's not the case, like, it's not just the case that the language model mechanism comes into the picture only when the…
Nikhil Prakash: the information from the vision encoder is updated. I think it is there
Nikhil Prakash: In normal cases as well. But we have not done a very thorough experiment for that. I think the argument that we are using is
Nikhil Prakash: You see the result for the below and above here?
Nikhil Prakash: Slightly growing. Yeah.
Nikhil Prakash: And we say that that growing part is coming from the language model ordering ID. Makes sense.
Nikhil Prakash: But maybe we can talk about their experiment later.
Nikhil Prakash: So yes.
Nikhil Prakash: What? I didn't understand. Why did you have 6… not here, sorry, here.
Nikhil Prakash: The next one? This one? Next one, next one. Next one. No, you had, like…
Nikhil Prakash: I had what? Keep going. It's actually not an intervention accuracy, it's actually a probability. The probability to save it, or the probability to save it. So it adds up to one?
Nikhil Prakash: It may not add to one. I mean, this is distributed, yeah. We did not do a softmax on color tokens. We've just take the probability of this particular color.
Nikhil Prakash: It may not add to 1.
Nikhil Prakash: But, so this is just the most simple thing. You make the model, you have the vocabulary distribution, all the logic, pick the logic of the red, pick the logic of the blue. But there's also another 50,000… Yeah, they would have some logic, right?
Nikhil Prakash: Logit value. Okay. So it might not add to 1. The logit of blue and logit of red might… it should be close to 1, but it may not be exactly 1. Why is that 0 to 1?
Nikhil Prakash: Like, the scale, why is it 0 to 1 if it's just logical? Was it probability.
Nikhil Prakash: You do the softmax first, then you read off the probabilities corresponding to the literal token, red, and the literal token blue, right? Yeah. Okay. Yeah, that's it.
Nikhil Prakash: Yeah, the IA was incorrect, but I think the result will still hold. Yeah. Same trend? Yeah.
Nikhil Prakash: Yep.
Nikhil Prakash: Okay, so the main… yeah, so I think that's the… one of the main arguments, that both vision encoder and the language model are creating or forming this ordering information, which is being used to do spatial reasoning tasks and vision… VLM.
Nikhil Prakash: And then we use this insight to improve the performance of
Nikhil Prakash: one of the models on a benchmark called WhatsApp, which is okay, I think. Yeah, we were able to improve the performance.
Nikhil Prakash: significantly better than just using a random baseline, which is just to pick up random directions. I mean…
Nikhil Prakash: Yeah, it depends how you call your significance. Like, 15 to 50?
Nikhil Prakash: Is that the jump you did?
Nikhil Prakash: So, read this number, I think. Original model performance was 90%, and then we got it till 95.
Nikhil Prakash: Yeah, that works. But this is unsupervised. We don't need any supervision. You can do it on any model, in any image. All you need to do is just…
Nikhil Prakash: Train these probes, which is sort of…
Nikhil Prakash: should figure out whether it's the first object, second object, or third object in the image, and that's it. And you can use those
Nikhil Prakash: Probes as a steering vector.
Nikhil Prakash: Oh.
Aruna Sankaranarayanan: What's the difference… what's the difference between the accuracy and the percent corrected failure? Because the percent-corrected failure seems, like, really huge, right? Improvements in both cases. I mean, much higher.
Nikhil Prakash: Yeah, so this is 90%, so 10% are incorrect. Among those 10%, we are able to fix 50%.
Nikhil Prakash: How many of the incorrect answers we were able to… Flip, yeah, to correct.
Aruna Sankaranarayanan: Okay.
Nikhil Prakash: repeat that with a looker.
Nikhil Prakash: So the… these results are on the non-synthetic images, right? Yeah, what's up? And the probe was also…
Nikhil Prakash: Yeah, yeah.
Nikhil Prakash: Do you know how much, when 2.5 would do on that data set?
Nikhil Prakash: Like, you did 8 billion, right? Or $7 billion.
Nikhil Prakash: Perfect.
Nikhil Prakash: Okay.
Nikhil Prakash: Okay, so one more piece of experiment, which we have not put into the paper, but I've been doing in the past few weeks, is something that we… that has come into the discussion as well, which is… okay, so in the language model space, previous works has shown that
Nikhil Prakash: This… this ordering information is encoding the relative positional information. Something like this is the first…
Nikhil Prakash: object, this is the first box. So we asked the same question, for the ordering information generated by the vision encoder, now that we know that vision encoder also generated these kinds of information. So for that, we… we also did causal experiment with slightly different task.
Nikhil Prakash: So, this is the task.
Nikhil Prakash: two differences that you should notice. First, in the image, the squares are…
Nikhil Prakash: it a little bit. It's not equally spread.
Nikhil Prakash: The squares are actually shifted towards the left-hand side in this particular case, and the question that we ask is, the leftmost… the color of the leftmost square is… and that's…
Nikhil Prakash: Just would give us… green.
Nikhil Prakash: And we do patching experiments again, and this is the counterfactual sample, which you can think of it as, like, just a mirror image of your clean image.
Nikhil Prakash: One important point is.
Nikhil Prakash: the exact XY coordinate of this blue square is the same between these two samples.
Nikhil Prakash: Okay?
Nikhil Prakash: Okay, and then we basically take the… we pass the strip from this particular area, region, to this particular region, and this particular strip to this particular strip.
Nikhil Prakash: Okay. Now, there are two hypotheses.
Nikhil Prakash: Whether the model is Forming relative, or whether the model is forming absolute positional information.
Nikhil Prakash: If it is forming relative position information, this is what would happen.
Nikhil Prakash: Here, it will say that this is the first square. So after patching, this blue square will become the first square.
Nikhil Prakash: And this green square will become the third square.
Nikhil Prakash: Okay? So, the answer to this question will change from green to blue, if it is relative.
Nikhil Prakash: Now, the rational for absolute is something like this. Let's say it is not encoding first, second, or third, it is encoding,
Nikhil Prakash: 3.1, 3… 1, 3, 2, 3, 3.
Nikhil Prakash: And this would be 3, 3, 4, 3, 5, okay? So after we do the batching, this green will become 3,5.
Nikhil Prakash: the brown will remain 3, 2, and the blue will remain 3, so we have…
Nikhil Prakash: 3, comma 5, 3, 2, 3.
Nikhil Prakash: After the patching, if we are assuming absolute position.
Nikhil Prakash: So then, the leftmost square should be 3,2, the brown one.
Nikhil Prakash: That's why they expected color.
Nikhil Prakash: After the intervention, if the position information is including the absolute one, it's the brown one.
Nikhil Prakash: That's a little bit diff… does that make sense?
Nikhil Prakash: Okay.
Nikhil Prakash: So, this is the result, if you do the patching of… Just the vision embeddings.
Nikhil Prakash: Vision token embeddings.
Nikhil Prakash: We see that, and this is probability.
Nikhil Prakash: It is sort of like… yeah.
Nikhil Prakash: like, distributed… Equally between brown and blue.
Nikhil Prakash: And that's why I say, and this… this is only for this…
Nikhil Prakash: Clean images shifted towards left, but the same, like, similar results also holds when the squares in the clean images are actually shifted towards right, or top, or top bottom.
Nikhil Prakash: And that's why the… what I think, in the vision encoder, it's not that clear.
Nikhil Prakash: as clear as what we have seen in the language model space, which… where we have seen that the model clearly uses this relative information. But in the vision space, I think it seems to be including
Nikhil Prakash: Both this kind of, like, absolute and relative position.
Nikhil Prakash: Did you say absolute? It's the color, or something else? When I say absolute, I mean coordinate, like 3, X comma Y.
Nikhil Prakash: And when I say relative, I mean a bit more abstract. It's like first object, sorry, first square, or the second square, or the third square.
Nikhil Prakash: Can you repeat what you mean?
Nikhil Prakash: The absolute?
Nikhil Prakash: just the coordinates, X comma Y.
Nikhil Prakash: But you're also patching the white space.
Nikhil Prakash: From much earlier, right?
Nikhil Prakash: We are patching the white space on the right side. We're also patching the first column tokens. They might have a different relative position.
Nikhil Prakash: They might have a…
Nikhil Prakash: So I think we went with the assumption that the strip has… That strip will always encode everything blue. Yeah.
Nikhil Prakash: It made sense when you're doing probes, because they're all equidistant.
Nikhil Prakash: Maybe you should also do probes here and seeing probes for other events. We saw that it's always distributing them according to the location.
Nikhil Prakash: So even if you shift everything.
Nikhil Prakash: Venice peopleship.
Nikhil Prakash: Or if you have, like, now,
Nikhil Prakash: 3 in one, you would have, like, a rectangular, rectangular, rectangular, right? I think that… so I don't know if it'll work, but I think that there are, like, naive things that you could do to scramble it even further to have a totally dead fish. You could say things like…
Nikhil Prakash: We're gonna just…
Nikhil Prakash: do a Fisher Yates shuffle of all the patches, right? So that, you know, that they're white anyway. You say, oh, I'll patch white. I know that white has a lot of pollution and stuff in it. What about…
Nikhil Prakash: We'll just shuffle all the patches.
Nikhil Prakash: They don't have any information anyway. If they have any, we're just gonna scramble it. It's gone.
Nikhil Prakash: Right? And then, and then just put in exactly the…
Nikhil Prakash: This featureless red thing, this featureless blue thing.
Nikhil Prakash: On a scrambled, you know, field of… shuffled white.
Nikhil Prakash: And then… and then if your model can still tell what's what, then… then I'm, like, then that, to me, that's more convincing. As a language model… And I think if you do that, maybe your brown accuracy would go up.
Nikhil Prakash: Wait, so you're saying that we… what we see here is a lot of the effect of, actually, the white background, including… It could be, it could be, things like that.
Nikhil Prakash: Yeah, it could be. And so you could… I believe… I believe you're willing to ask me something.
Nikhil Prakash: Wait… So you think… I'm okay.
Nikhil Prakash: That will be more absolute value.
Nikhil Prakash: then I think you wouldn't get, like, this confusion between brown and blue. I think one of what would stand out. I think it was a…
Nikhil Prakash: I think, actually, the hypothesis is the… More blue, more morality, because…
Nikhil Prakash: with the generalization of the problem, we saw that the white paper is basically an older picture, so I believe it might be over 50 bags, I thought.
Nikhil Prakash: That's very interesting.
Nikhil Prakash: This is very interesting work, and so it's, you know, I think that you're getting a little preview.
Nikhil Prakash: of your review process, is going through review right now? Yes, we got really nice review for the workshop. I mean, that was workshop, yeah. But it's good, I think it's… I think it's really interesting work. I think that a lot of your…
Nikhil Prakash: Sort of what you, like, sort of your appendix?
Nikhil Prakash: experiments are very strong. I think that, you know, your stuff is very defensible.
Nikhil Prakash: But you know, you'll have to gird yourself for review. Depending on your reviewers, you might have… you might have to present a lot of extra data to them. And also, we need to give credit to Kelly.
Nikhil Prakash: doing the master's student. Yeah, yeah, yeah, yeah.
Nikhil Prakash: Yeah. By the way, you may claim, but then you nail the point later.
Nikhil Prakash: Okay. You should… you should keep that, like, is it this? And then you nail it with… Yeah, nail it, let… yeah, yeah, give me that feedback.
Nikhil Prakash: Yeah. You tend to, like, take… you sort of show things in the front, Instead of impressing people.
Nikhil Prakash: The end.
Nikhil Prakash: You can… you can press the meta. Like, for instance, you showed probe and you said it's diffused across all tokens. Like, if no mechanism researcher would press by probe, but you know the answer, like, you did causal experiments, you should show the causal experiment and say, wow, this is so pressing. Then I'll be like, yeah, it is so pressing.
Nikhil Prakash: So you should say, yeah, when you show the probe.
Nikhil Prakash: Say.
Nikhil Prakash: Well, I don't believe this. Yeah, you should do that. You should justify your own thing.
Nikhil Prakash: This, yeah, okay, but… I think I wouldn't even, like, if you need to present, I wouldn't even show the probing results. Maybe just later for the steering. But then how do you come with the hypothesis? How do you come with the hypothesis of the…
Nikhil Prakash: Actually, for me, for me, the thing I learned from this project is
Nikhil Prakash: Profs are actually not that bad. They're not supposed to evident, but they can do… hypothesize.
Nikhil Prakash: I agree, but why he suggested doing prob, I was like, no, we're not doing prob.
Nikhil Prakash: And then he kept saying that for a couple of weeks, and then I was like, okay, do probes, like, whatever, okay. And then we were able to actually construct really nice positive experiments because of… Because of the probe, I see. But why did you even probe the tokens that are not that position?
Nikhil Prakash: Because when… No, no.
Nikhil Prakash: No, it wasn't… it wasn't actually, Mark, because things were not working.
Nikhil Prakash: When we… when we did not include those additional tokens, which is what generally you would do. You would only work with the square tokens. You could go through the… you can tell the same story, but with causality. I'm not saying change your story, just that I think it would be, for me, simpler. So I think that depends on the format as well. If I was giving, like, a formal talk, then I would have definitely said that. But then, as soon as I started saying reserve a paper started asking questions.
Nikhil Prakash: You claimed it. You claimed it, and you were like, oh, I don't know, this could be that, this could be this, and then you showed causal, and I'm, like, convinced.
Nikhil Prakash: The, like, when you were starting out, when you presented your problem statement, you had a couple results showing, like, a preliminary, like, is this… is there even something there
Nikhil Prakash: for me to search for, which is kind of what we give you, it gives you, like, okay, there's actually something there that I can actually look at. Like, I'm not totally opposed to, like, showing those results of, like, okay, there's actually something going on, and then nailing the problem. You can also say, like, when you show the probes, you can say, don't worry, I have causal experiments later on, this is just, like… Okay, okay. Or even if it's just to make the point, probes can help us think about what kind of causal experiments we want to make.
Nikhil Prakash: This is a point you want to promote, then.
Nikhil Prakash: Yeah. I think we could. I think that that is one of the things that I have learned in the project, but I don't think so we are doing… we are doing that right now, but yeah, I think we could do that. By the way, I like your first slides, right?
Nikhil Prakash: Me too. You're fantastic. I wanted to advise to me. Thank you very much. Okay, cool. Thank you.
Nikhil Prakash: It's a very sweet way to me.