David Bau: Cloud.
David Bau: All right, we have the pleasure of having Nikhil present today.
David Bau: Yes, welcome! Welcome to the Bow Lab group meeting Nikhil
David Bau: and I actually, I don't know. Are you going to talk about your current research? Yeah, that's great. So take it away. Nikhil.
David Bau: continue.
David Bau: Yeah, I'll get rid of this. And I think that I think we're good. Yeah. So yeah, let me just start.
David Bau: Yeah, I'll start with an introduction, I think. Then, yeah. So I'm Nicole
David Bau: right now, in 3rd year of my Phd.
David Bau: and today I will be talking about
David Bau: my current project, which is still ongoing. And I've not prepared this slides very meticulously. So yeah, gives. Have some mercy on me.
David Bau: yeah. And in addition to our
David Bau: team. So we have Natalie, who is probably joining us as a postdoc, and then we have Tamar.
David Bau: She is right now at Mito. We have David, and then we have Attika sky.
David Bau: Yeah. So this is what I had at least right now, as the title of the project, how do language models encode and update character beliefs?
David Bau: Now to motivate this work. I have a few video. Actually, can I play with you?
David Bau: Yeah.
David Bau: yeah, this is like a 1 min video. I hope the audio works. You might have to turn your volume up on your your laptop.
David Bau: Yeah, this is a scene from Friend. Full Bag. Beer is still cold.
David Bau: Something terrible must have happened here.
David Bau: No, no, no, no, Stevie, I was never here.
David Bau: Chair. Now, if anybody asks her name is Rosita.
David Bau: Could you move that talking? Maybe it's not a little bit hard.
David Bau: Then cut down in a prime, Joey. The new chair will be here in an hour. Maybe we should actually move Rosita out of here, you know. Start the healing process.
David Bau: She's healed.
David Bau: That's weird. No, it's not weird. It's a miracle. It's not a miracle, Joey. I'm sure there's some explanation. Oh, there is, if you want. Okay. So the reason I wanted to show this video was,
David Bau: like to truly understand what is going on here.
David Bau: We need to form beliefs of different characters that were involved in seeing you need to have.
David Bau: You need to understand that. Okay, the
David Bau: the belief of Joey and Rachel
David Bau: were different from Chandler, like they didn't really know what really happened with
David Bau: what was the name of the Rosita Rosita.
David Bau: So yeah, so let's say, if if there was a AI system or a yeah, some kind of neural network which is given this piece of input.
David Bau: then to truly understand what is going on in that scene. It has to form the belief of the characters that are involved in that
David Bau: in that scene.
David Bau: Yeah, which is, I think, an important thing for the models to have.
David Bau: And
David Bau: in fact, I just saw this another video. Should I show it on. It's a small one. It's Dario. Yeah, it's Dario. This is his interview at this year's World Economic Forum. The broader vision is we are planning this year in 2025 to build something called that we call the virtual Collaborator.
David Bau: And the idea is that that would be an agent that operates on your computer on any computer might operate at work, and you can give it tasks. You can say, you know, write, write this feature for this app, and you know it'll write some code. It'll compile the code. It'll check the code. It'll talk to its coworkers on slack or on Google docs or on some other platform. It'll check in with you to see how you're just to see how the task.
David Bau: Yeah. So the point there was, he says the model
David Bau: or the agent is gonna talk with other collaborators or the team members
David Bau: which are part of your project
David Bau: through slack or discord. Something else. Now, if you are having a multi agent conversation whether with a human or with another AI system.
David Bau: I feel you need to have the. You need to be able to form the beliefs of other
David Bau: agents or other characters which are part of which are part of the conversation. So let's say, if somebody came later in the conversation, or very specific example is, let's say you are having.
David Bau: You have an email thread with one of your friends.
David Bau: Then, after certain number of emails. You include another friend who happened to not go through all of the emails that you already had with your previous friend. Now, to have a conversation with that new friend. You need to
David Bau: understand that he may not have all the context of the conversation with your previous friend already, so you need to write a message, or you have to write an email which is condition on that particular piece of information. As a humans, I think we can do it easily. But now we want the AI systems to be able to do it. So that's why
David Bau: I feel studying this belief. Formation. Is again, yeah, this is like another motivation, for why we should study the belief formation
David Bau: and then, finally, I have like one another like, I mean
David Bau: This was the actual motivation for us. The other 2 just made up things
David Bau: So there have been some work in understanding, like theory of mind capabilities of language models over the over the past 2 years. I would say so. Some papers
David Bau: are suggesting that okay models are
David Bau: models, have the theory of mind capability, and they do so based on behavioral evaluation, on certain benchmarks.
David Bau: And
David Bau: it, it's yeah, and and Gpt 4, and probably Gpt 3.5 as well. They can do the task defined in the benchmark. And based on that, they're saying that the models do have theory of mind capabilities.
David Bau: However, there are other works as well, which have
David Bau: looked the problem more meticulously. Try out different kinds of benchmarks,
David Bau: and on like different kinds of benchmark. The models are not able to do it if this is still behavior in in nature.
David Bau: but yeah, so there's a flux in the literature right now. So like some people are saying their calls do have this kind of capability, the other said, they don't have it. And yeah. So the underlying motivation for us were us was, if we can understand the
David Bau: the mechanism with which the models are showing the characteristics or showing the behavioral performance.
David Bau: If we can understand that algorithm or the mechanism, then we will be in a better position to comment on whether the models truly have that kind of capability or not.
David Bau: Yes, those were the motivations now coming to specific questions that I
David Bau: I'm hoping to answer in this project.
David Bau: I mean, the 1st question is just like, How do language models form and track
David Bau: the beliefs of different characters that we define in context as as a form of story or as form of art?
David Bau: And how do they separate that belief from the actual reality?
David Bau: So for the channels example, we know the reality of what happens with Rosita.
David Bau: But Rachel and Joey don't know it. So how does the model differentiate the belief of
David Bau: both of them from the actual reality?
David Bau: That's the question. And 3rd question that I've mentioned here is like.
David Bau: if a new event happens, let's say, if a character has some belief about something, some state of the world, and a new event happens in that state of the world.
David Bau: Kind of
David Bau: that agent of that character happens to see that event occurring, then their perception of, or their belief of that state of the world will be updated that will be changed to the new state of the world. So how does
David Bau: the language model incorporates that into its internal function.
David Bau: That's the high level questions here.
David Bau: Any quick question.
David Bau: Alright. So let's get into the experimental setups. Then.
David Bau: Okay, I showed you the videos. But I'm not working with videos at all. This is all language model. And I'm actually
David Bau: for most of the analysis, I'm using a
David Bau: control setting. I should use control setting rather than that sounds better. Yeah, the control setting is something like this, this is a 0 shot setting. I have given a long instruction which you can read or you can skip it. It's not very important.
David Bau: but this story is what is most most important.
David Bau: I'll read it out.
David Bau: There are 2 characters in the story, Max and Karen, who are working in a busy restaurant
David Bau: to complete an order. Max grabs an opaque
David Bau: tool and fills it with coffee. Then Karen grabs another opaque dispenser and fills it with cocoa.
David Bau: They are working in entirely separate sections, so they cannot see each other each other's action.
David Bau: Now this is the story. And then then I asked the question, What does Max believe? Dispenser contains?
David Bau: And since both are working in entirely different sections of the restaurant, they cannot observe each other's action. Max have no idea about what this dispenser actually has, because Max has not, observed Karen, putting in coke wine dispenser.
David Bau: Hence the answer in this answer for this question, in this setting is unknown.
David Bau: So yeah, this is a setting, and
David Bau: I have few. Let me just give few annotations here, so I'm calling Max as 1st character, Karen a second character
David Bau: tune as 1st object dispenser as second object. This is based on the order at order in which they occur in the prompt.
David Bau: So whichever occurs. First, st I'm calling it as the 1st one, whichever occurs. Second, I'm calling it the second one
David Bau: this is the other setting. It's the same story. But now the last sentence is changed
David Bau: now they are working side by side
David Bau: and can clearly observe each other's action. So now
David Bau: Max can see that. Okay, Karen field dispenser with cocoa, and hence the answer is cocoa
David Bau: in a specific, prompt setting. Do you have like a multiple choice
David Bau: variety of answers? Or how do you check?
David Bau: Now? This is still just the same structure that you see here. This cocoa is just a single token that the model needs to generate.
David Bau: How do you differentiate between like unknown, and I don't know. And like paraphrasing
David Bau: so the model, yeah, I've prompted the model in the instruction to generate either coffee, cocoa, or unknown. Okay, I have asked it to do that specifically. So I can just work with single token.
David Bau: And it does that. This is yeah. Okay. And I found that
David Bau: Lama 3, 70 B. Instruct can do this task
David Bau: pretty well. This is the behavioral performance on both the visible and invisible sample. So that's my primary model right now. But I'm hoping to try to see if if the research generalizes to other models which can do the task, so does all your experiments have the instruction that you gave.
David Bau: It doesn't, or else yeah, yeah, I I would assume that the behavioral performance would go down, or
David Bau: or at least the the answer, the coqua that we see here. It won't be just the cocoa it will start with sentence, like Max, believe the dispenser
David Bau: contains, and then it will say
David Bau: something like it will give me an entire sentence, one token. And I want a single token. I really is interested in how big the data set is, yeah.
David Bau: So yeah, so this is a template.
David Bau: I have defined a template, something like character. One character, 2 are working in a busy restaurant.
David Bau: and I just fill in those placeholders.
David Bau: So I can generate as many as I want.
David Bau: Yeah, more or less as many as I want. I just need different
David Bau: character names and different object names and different state names. Do you have different templates?
David Bau: Now? Right? Right? I'm just working with one template.
David Bau: Yeah, I hope that answers the question
David Bau: so. But I have one more question.
David Bau: so how would this task compare with regular instruction following?
David Bau: Do you see any difference there, or regular instruction calling. What do you mean like, for instance, you're saying, Lama 70. B. Does really good job. I'm assuming Lama, it doesn't.
David Bau: That's like, let's say, that's a bar plot. 2 bar plots. One is like super low. One is. If you were to draw a similar bar plot for construction, following some met like some benchmark would be there.
David Bau: Do you see? Like it's the similar trend I'm I'm just trying to like. See if it's is it just about the instruction following? Because you also have like
David Bau: instructions to say, if a person doesn't observe it, then they cannot.
David Bau: They do not have any belief. Yeah, so do you see that
David Bau: one on one? Yeah, it could be.
David Bau: I've not to any analysis on that. But it could be that. Okay.
David Bau: there is a correlation between the
David Bau: the capability that I'm trying to analyze and the instruction following capability.
David Bau: there might be correlation for that.
David Bau: Yeah. Yeah. And and without the instruction, it's like super bad.
David Bau: It's hard to evaluate that. Then I have to look at it manually.
David Bau: Maybe multiple choice would be like a
David Bau: yeah, multiple choice could be an option. Yeah.
David Bau: yeah, I mean, initially, has to work with multiple choice setting.
David Bau: It used to work fine as well there as well, and there I didn't. I don't think I have
David Bau: much of a I mean even there. I think I did have some instruction, but it was not as detailed as this one. Okay?
David Bau: And then what was the performance? Yeah, it was good. I mean, I don't remember the number. But 70 p. Instruct was good.
David Bau: So you'll see when Nikhil starts talking about mechanisms.
David Bau: Oh, yeah, I think one of the issues with multiple choices. It adds another stage
David Bau: of inference. So 1st the model has to figure out what the answer is, and then, after it, figure out what the answer is. It has to evaluate the multiple choices and then decide whether it should say A, BC or D,
David Bau: and that's just another step. And then you can see evaluating. Each one of these steps is complex enough
David Bau: that it just like it, like sort of doubles the length of what you have to explain doing this extra multiple choice. And it just wasn't that interesting?
David Bau: I think it was still interesting. But yeah, it's not not interesting for a single. It's not interesting for a paper, because a paper has already been written on the multiple choice stage. Yeah.
David Bau: okay, yeah. So this was the Beverly performance.
David Bau: Before going further, let me just remind about my previous work. So in my previous work, what I was studying was I was trying. I looked into. How do language models bind entities
David Bau: with their attributes
David Bau: and the main takeaway of, for that of that paper is the model is using positional information of the entities and their corresponding attributes to bind them. And when I say positional, it does not really mean like
David Bau: like the 5th token from the start, or the 7th token tokens from the start. What I mean is, is more like
David Bau: like the 1st entity and second entity, and the 3rd entity, irrespective of
David Bau: what is their exact token number from the start.
David Bau: Yeah. And the model was using that
David Bau: that form of positional information to find to to bind entities with their corresponding attributes.
David Bau: Yeah, I just wanted to remind that because we gonna be using this
David Bau: sorry question about the previous slide. I just want to clarify what's visible and what's invisible. So the example that you? Yeah. So this is a visible example where they are working side by side and can see each other's action.
David Bau: Okay, you see the last sentence here
David Bau: they are working side by side, and can clearly observe each other, action that is visible example.
David Bau: And this is invisible example.
David Bau: The last sentence here is different. They are working entirely separate sections. So basically invisible samples. You just expect the ground truth to be unknown.
David Bau: It depends. So here I'm asking, what is Max contains dispenser contains. But if the question was, What does Max believe that tune contains, then it will give me coffee.
David Bau: because he himself filled the tune, right? So he knows what is there in the tune?
David Bau: Okay, I don't like this.
David Bau: Oh, okay.
Aruna Sankaranarayanan: Quick question, Nikhil like I don't know if this is, you know, in what is scope for your talk. But could you talk about how the methodology of how you do this like, how you
Aruna Sankaranarayanan: track? You know the yeah, that apple is
Aruna Sankaranarayanan: passing information to see. And yeah.
David Bau: Yeah, I'm not.
David Bau: That is not part of the talk. I mean this presentation, but I can give you.
David Bau: I can just talk about it.
David Bau: and there is. Firstly, I don't think that there is any standard procedure as of now for figuring out the mechanism.
David Bau: This is very.
David Bau: or what do you call it? Like hidden trial kind of thing. You have to come up with hypotheses. You have to come up with
David Bau: the corresponding counterfactuals
David Bau: and test it whether it works. I mean, if if it works, that's good, then you just go on to the next variable
David Bau: on your in your causal model. If it doesn't work, then you have to refine your hypothesis, and then again come up with the count of action.
David Bau: So it's a very iterative return trial procedure.
David Bau: But I can give you some heuristic that might help.
David Bau: Do you start from an automatically? Yeah. But the thing is, if the setting is little bit more complex.
David Bau: What counterfactual are you gonna use to find the circuit now for the previous setting? It was for this setting. It was still fine to find. I mean, you could find a circuit
David Bau: with with a counterfactual which
David Bau: which, like, destroys all of the vital information in the prompt. That's how automatic circuit discovery works. So you have your clean example, and then you come up with the artifactual example which
David Bau: have all the vital tokens as different as the clean one, and then you apply the
David Bau: attribution, patching or id attribution patching to find the the circuit which which is responsible for doing the task.
David Bau: Yeah, that's that's what I was coming into coming to.
David Bau: yeah. But for more complex sitting I'm not really sure if the automatic circuit discovery would work, and how well it will work.
David Bau: And now, coming to causal tracing, yeah, I think that's probably the 1st step you can do. I was actually hoping to add a
David Bau: slide for that. But I just didn't have the plots with this new setting, so I skipped it. But essentially what you can do is
David Bau: for each of the tokens that you think in your prompt are vital, or that would be vital for doing the task that you're interested in. You can come up with a counterfactual where you just you just alter that particular prompt, and then do causal tracing
David Bau: with that causal tracing, you will figure out, okay, how does that piece of? Or how does the information present in that token
David Bau: is flowing inside the model?
David Bau: So if you do that kind of puzzle tracing for. Let's say, Frank, here, which happens to be the
David Bau: the character in the question. You will get a plot something like
David Bau: I mean, yeah, you'll you're gonna get a plot, something like a heat map, something like this, this
David Bau: and this one second, it's not making progress. One second, let me just show.
David Bau: I don't think so. Yeah, Chris, I think it's the wrong method.
David Bau: Yeah, you want to get something like this. This is a different prompt, like slightly different, prompt. But the structure is more in the system. So here I corrupted
David Bau: like either of these 2 states. And then I passed in the activations of those State. And you see the how does the information flows from these 2 tokens? You see that a lot of like
David Bau: till layer 6, 70, or 80. This is
David Bau: this is Lama 405 B. It has 128 layers, I think 128 or 126 layers. So the information on the State tokens remain on that particular piece of token till layer 70, and then it shifts to the last token. So this kind of thing you can do for all the tokens that you feel are important for the task that you are
David Bau: interested in.
David Bau: It's gonna give you some information of how I mean, it's gonna give you information of how the
David Bau: the information is actually flowing inside the model and then base, and that will help you to generate the causal model.
David Bau: in the subsequent steps. Nick, you can ask more questions. Yeah.
David Bau: wait. Is this relevant to the talk? Yeah, this is relevant. Okay, so
David Bau: you are you also doing these causal interventions on other tokens? Or did you just choose state one and state 2
David Bau: no, I'm doing the causal interventions on other tokens as well, but they don't show any. They don't show anything. So that means
David Bau: this, the the vital information which is
David Bau: given by the State token is present only on this token, residual frame.
David Bau: So so I have a prompt, which has
David Bau: a correct state. And I have a prompt which has, let's say, incorrect. And I'm patching the residuals vector from the correct one to the incorrect one, and checking if the
David Bau: final output is the same as the correct one or not.
David Bau: And I see that if it only works, if I pass in the residual vectors of the correct state, and not other tokens.
David Bau: So are you doing one to one patching like from feedback to tech
David Bau: right team. So you want. So let's say, you have this, yeah, yeah, yeah. Team 2 team.
David Bau: So okay, it's I think it's it's not surprising that till the State one that you don't see anything that's more important, that the information is not getting diffused across multiple tokens. It's primarily present in the state token, and then it directly goes to the last. Yeah, that's the piece of information that you should be looking into
David Bau: now. What?
David Bau: Oh, oh, sorry. It's okay, I think what? What I think one of the key things of what Nikhil's pointing out here is that
David Bau: the the data that you get here
David Bau: very much depends on your paired counterfactuals. So you know, in his, in his base story, or his noise story, or whatever, and his his informative story, there's some difference.
David Bau: and to get a clean plot like this.
David Bau: instead of just like everything matters for everything. He he carefully chooses. What the difference between these stories is to understand. You know what what he wants to probe, and then he's also looking for
David Bau: specific information to show up in the answer.
David Bau: And and so when he so so in in his experiments, it's really a series of
David Bau: paired settings, plus an answer he's looking for, and
David Bau: right. And and then, because of his templates.
David Bau: He's got ways of generating as many as he wants, with different names and different, you know all the all the concrete details, everything shuffled
David Bau: But but you know, with the same logical structure
David Bau: that that he's trying to probe.
David Bau: Yeah. And that's why it's a little bit difficult to work with actual real benchmarks, benchmark data sets.
David Bau: So I I had the question, okay, so how is your causal? Sorry. How is your causal, prompt, counterfac prompt differs from your actual prompt or team. Prompt like, is this, is this just?
David Bau: just the 2 states and and kind of like, the answer flips.
David Bau: yeah, for I for the causal tracing?
David Bau: the prompt would be, what would be the prompt? Okay.
David Bau: yeah. So
David Bau: I'll have the same characters. Yeah, I will have the same object. But the state tokens would be different.
David Bau: Like, instead of coffee, it will be beer or something. Okay? So in one of the
David Bau: example, the output would be coffee. The other example output would be beer. And and how many times the word coffee appears in the in the prompt what? Just once? Yeah, okay?
David Bau: I mean, that's that's when call tracing
David Bau: kind of words. Right? If you if there are multiple things changing in the prompt, then you.
David Bau: it's kind of a little bit but
David Bau: but if you see the character one, if you do the same thing for the character, yeah.
David Bau: the character appears twice at least twice. Yes, in the story and in the question, yes.
David Bau: even there I do see relatively clean pattern there when you when you okay.
David Bau: so when you are restoring a state at the question. You are kind of restoring,
David Bau: maybe the 1st appearance of the character as well. But you kind of see the causal effect on the 1st appearance of the character as well. Right? Yep.
David Bau: so. But in in earlier layers, okay?
David Bau: And when you restore it, the question token is still corrupted entirely.
David Bau: I mean each each of these, each of these cells, they
David Bau: they correspond to a individual forward, run right?
David Bau: So multiple, individual forward. So this is, this is accuracy
David Bau: accuracy from 0 to one. So this is like across 20 or probably 40 exams
David Bau: accuracy of the interchange accuracy. Basically. So every single one of these squares is like 40 runs. Yeah, so this whole grid is like a pretty big experiment. How long did it take.
David Bau: I don't remember, but I did it sometime back. I don't really remember. Yeah, it was definitely before Delta. That's why we great worry about the one second per forecast kind of thing. Yeah.
David Bau: definitely. Yeah.
David Bau: Yeah. But at least it's not 9 seconds.
David Bau: Well, that's great.
David Bau: The other point. So if you are working, not with counterfactual pairs, but just adding noise to individual tokens.
David Bau: Do you think it would be a similar picture?
David Bau: I would
David Bau: hope so, but I haven't gone with it like extreme example, just like mask out specific token and put like a pad token there.
David Bau: Yeah, I would hope so. Yeah, my 1st intuition would be. I think you will get similar patterns there. So maybe this would allow you
David Bau: to get rid of the counterfactual pairs and use more of the benchmark data sets.
David Bau: No, that's fine.
David Bau: Yeah, for causal tracing. I think it's fine. You can do that. But for other more nuanced thing. Oh, yeah, for sure. You. Yeah, you need counterfections
David Bau: of.
David Bau: But yeah, the point there was causal tracing and give you
David Bau: the information flow inside the model which can help you
David Bau: generate this kind of causal model. But this is against. There is no automatic way to do it.
David Bau: Okay, I don't think so. I'll be able to complete this
David Bau: all the slides. So let me just
David Bau: see what or how much I can go.
David Bau: So this is the causal model.
David Bau: yeah, which is responsible for doing this task. So let me just explain it first.st
David Bau: initially, when the model see the tokens in the story
David Bau: what it does it it forms abstract.
David Bau: I mean, I'm calling it positional information. But probably Marker would be a better one like it creates like a abstract marker position, information or position based marker information
David Bau: for each character, each object, and each state in the prompt
David Bau: okay. And it creates those markers.
David Bau: In such a way that
David Bau: these are like like the objects, are dependent on the character, and the state is dependent on both the object and the character.
David Bau: I think that's because how this is just because of the causal causal attention mechanism.
David Bau: If you paraphrase the sentences that the Frank appears at the last, I would assume that the character, position, information, will be dependent on this object at the stable.
David Bau: Yeah, initially, it creates those position based markers. And later, when I ask it, a question about a specific character. It fetches the corresponding marker of that character
David Bau: and also the corresponding marker of the object
David Bau: which is passed on to the last token, both of them are passed on to the last token, and then finally, at the last token, the model uses this 2 piece of
David Bau: marker information somehow compares it to figure out, what is the marker of the current state.
David Bau: and that marker information, that position based marker information is again fetched to the last variable which actually looks at the
David Bau: the State token information like the State token based on the marker information to fetch its value.
David Bau: fetch its token value, and that is being predicted as the next token.
David Bau: Yeah, that was a lot of information questions.
Aruna Sankaranarayanan: What, what do these numbers mean? Like the 1314.
David Bau: Yeah, so so
David Bau: these boxes, these are the boxes and arrows are like the causal high level model. And the numbers that you see
David Bau: there are the layers which are aligned with those variables in in the actual model.
Aruna Sankaranarayanan: And this is all residual stuff. This is all residual steam.
David Bau: This is still residual.
David Bau: Chris, can you say it out loud because we can't see your chat.
ch.wendler@northeastern.edu: Yeah, it would be really interesting to see whether this circuit also is like important when you omit the detailed instruction
ch.wendler@northeastern.edu: right?
David Bau: Yeah, but it would be difficult to verify that.
David Bau: Maybe after you have the circuit, then you can just evaluate the circuit separately by having Gpt read the answer or something like that, you quantify? Yeah.
David Bau: yeah, okay, yeah, that's a yeah. Yeah. That's a good question. Another question. You said.
David Bau: object pause. Info is dependent on character. Pause info! What's the dependency? I'd like to like this. How they are! How are they dependent?
David Bau: It's like
David Bau: How are they dependent? I mean
David Bau: so to answer that the answer to that question is, I know that information from here is going into inform it.
David Bau: Information from here is going into here.
David Bau: How exactly it is interacting with
David Bau: the information which is already present here
David Bau: I'm not super sure, but I know the info. I know the the character position. Information is affecting
David Bau: the information present here.
David Bau: And how and how do you know that? Yeah, there, there is causal experiment that I'll show you.
David Bau: Okay, so different character has the same character. Pause info, but a different object present.
David Bau: Say that again, the problem, what's
David Bau: maybe we can go on. Maybe we can bring it up again when you have the causal experiment to show. Yeah, so this is the high level causal model. And actually.
David Bau: in the subsequent slides. I'm basically justifying each of the variables and trying to. Yeah, basically show you how I get the alignment of this layers.
David Bau: This is like the 1st count of actual example.
David Bau: Counterfactual. Yeah, this is basically the 1st patchy experiment.
David Bau: I am fetching at the last token, which is the colon token
David Bau: from the alternate example to the clean example. Now these are 2 completely different examples.
David Bau: and I am expecting the final output to change from Bourbon to float.
David Bau: So this is the counterfactual that I'm using to align the the 1st variable that we have
David Bau: in the causal model the value feature.
David Bau: What this variable, does it? It looks. It takes the positional information of the correct state and bring in the value of
David Bau: that correct state token.
David Bau: Okay? Yeah. So let's say.
David Bau: But now, but this experiment doesn't tell you this positional, right? It just tells you it doesn't tell you anything about the change. This tells you that. Okay, the variable is fetching in the value information somehow. Yeah, somehow.
Aruna Sankaranarayanan: And your value fetcher is just a colon like, if so, when you, when you call it the value fetcher like you, just, you're just referring to the end of sentence, token, or.
David Bau: Yeah. So information present at the last token.
Aruna Sankaranarayanan: Alright. Okay. Yeah. Thanks.
David Bau: Yeah. So let's say, some layer is encoding the
David Bau: the output of some layer is encoding the correct state token value.
David Bau: so that layer would be encoding float in this example, and the same layer would be encoding Berber in this example.
David Bau: So if I patch in the output of that layer from here to here the output should change to float.
David Bau: That's the underlying.
David Bau: like the intuition of how you go about coming up with the counterfacture. And this is the result. Basically
David Bau: y-axis interven intervention, accuracy
David Bau: and x-axis is basically the layers. We see that. Okay, there is alignment after layer for 57.
David Bau: So about 56, 57. That's the layer which which look at the
David Bau: the correct state tokens and bring the value of that that token.
Aruna Sankaranarayanan: And this is still llama 70 B. Right. This is not 4 0, 5. This is.
David Bau: This is our typical
David Bau: now coming to the second variable that I had in my causal model.
David Bau: So this variable is supposed to encode the positional value of the correct state
David Bau: so in this particular piece of info, in in this example, the alternate example, the correct
David Bau: state is trout, which is like the second state.
David Bau: Okay, in this example.
David Bau: Oh, sorry. This is first, st
David Bau: these are sorry. These are like folder examples. But yeah, this never mind, it's fine
David Bau: in. In the 1st example, Stroud is the 1st state
David Bau: in the which is the correct answer.
David Bau: In the second example, RAM, is the correct answer, which is the second state.
David Bau: Now imagine a layer which is encoding the position, information of the correct state
David Bau: in this particular. In this alternate example, it will encode first, st and in this example it will encode second.
David Bau: So if I patch from 1st to second
David Bau: the final answer to change to ROM should change to ROM because now the model should think that the correct answer is the 1st state rather than the second state.
David Bau: and hence the answer should be beer beer, not wrong?
David Bau: Yeah, it should be it.
David Bau: Question. I think you'll probably
David Bau: answer this in a minute. But like, how do you establish a causal dependence on this variable to the value of that trade.
David Bau: Like, how do you actually get the graph?
David Bau: Yeah, that's a good question. So yeah, in in, yeah.
David Bau: So in.
David Bau: yeah, I've not done anything specific for that. But hold on, hold on, you go back, you you have it already. Right? So
David Bau: because it it's going to output it outputs beer. Beer comes out of the end of your neural network. So it it goes through the value fetcher as well.
David Bau: Right? Because right? I mean it. It's it's connected to the other
David Bau: circuit is, do do you feel like that's not enough thinking. Well.
David Bau: so like in path patching, for example, you're measuring specific
David Bau: like causal dependence of like a later component to an earlier component. And since you're patching, so I know that you did that in your previous project. I'm wondering I just so. You're just doing residual streams in this, because it's 70 B,
David Bau: so I'm wondering if if it's
David Bau: if it. If that is a meaningful, I don't know. I'm just wondering if that's like a meaningful difference in like the kind of conclusion you can draw like high patching is much more rigorous, or
David Bau: is maybe not much more, is maybe not much more aggressive. I guess what I'm asking.
David Bau: I'm I mean the point that David was mentioning. I think that's a valid point the fact that
David Bau: we see the final output here as beer instead of Stroud. That means that
David Bau: there is some connection. There should be some connection, that okay, I am sending in only the position information. And then there is some other piece of information which is bringing in the value corresponding to that position.
David Bau: That is some value factor. Yeah, there should be some value factor to bring in the the value of beer here, because there's no way I can
David Bau: patch in the value of beer from alternate to original.
David Bau: because alternate does not have beer in it.
David Bau: Well, the alternate example. Both has the same people, though, right? So that probably is a little. But
David Bau: oh, it has Bob and Carl. Yeah, yeah, even though it has Bob and Carl. But you don't need to like you could have chosen Allison. Yeah, I don't need to. Yeah. I don't know why. It's Bob and Carl
David Bau: one second
David Bau: in this example I don't need to. But even even if they are same, this call or this tab.
David Bau: this call, or this tank cannot have beer information
David Bau: because they have not seen beer dogan.
David Bau: There's no way they can available. That's right. That's another thing that you did right. So another thing, the reason he did set this up, and I'm not sure if it's your final experiment. But notice, the question is exactly the same. Yeah, that's why I did that. Yeah, right? Like, it's kind of interesting
David Bau: which is required for the next experiment that I'm gonna share.
David Bau: But even if there's yeah, I mean, the point is saying, there is no way
David Bau: this call or tank can have beer information, so there is no way I can pass beer information directly from alternate towards
David Bau: So do you think that would help
David Bau: if you had patchings prohibitive? But do do you think that would help if you had? If if Nikhil had
David Bau: a 3rd run.
David Bau: and then, after patching from here to here, you could, he could let the second one run, and it says beer. But then he could also
David Bau: now repatch from another layer. He patched into it in layer 50, but he could try to patch out of it at layer 57 into a 3rd run, and say, yes, indeed, you know, in a completely 3rd run
David Bau: it now also, says Beer.
David Bau: What was the intervention in the 3rd run. So in the 3rd run it would be a completely unrelated story again. That doesn't mention beer or anything. Oh, and patch 57, and then you could patch 57 into it
David Bau: from this thing that had already been patched, and then now show that actually says beer in there. Yeah, totally right, I think that would, and that maybe that would be a clean. It's like, Oh, it really trans, it's really transporting the the value now of beer. It's not that it's picking up beer from some other.
David Bau: some other who knows where it would get it from. Yeah.
David Bau: And maybe that would do it. I think that would do it. I think, also, actually, after talking about it, maybe the only concession you're making is that so? Is that there's definitely like a value.
David Bau: passer. I sorry. What do you call it? Transmitter of the value
David Bau: the only concession you're making with is, that is, you can't say it's the same exact one which is like maybe a consent. That's something you probably don't even care about is that if it's exactly the same mechanism, exactly the same mechanism, but it's it's a value passer like it's maybe not the same like hence, or something. But we probably don't care too much about that. I guess it just depends. Like, path Matching gives you that like, yeah. But you don't really care about that. You can at least say it's some value. Passer.
David Bau: Yeah, that's true. That's right.
David Bau: Which is, I think, yeah, like cool people where I don't care about that. But I also think that their intervention would definitely like, tighten it more. But I think it's probably, in my opinion, if I was a reviewer, I'd be like, Yeah, yeah, it's getting ready for reviewers.
David Bau: Okay, we are already at 1055. 0, boy.
David Bau: we haven't gotten to the interesting ones yet. So you want to zoom through a few more. You just go through them really, quickly. Okay, so this is the this is the counterfactual for query, character, position.
David Bau: Here the order of the characters are revised.
David Bau: So
David Bau: in the 1st example, Ruth is the 1st character, and you, whatever this thing, Zoe, is the second one.
David Bau: and in the second, like the original exam. Zoe is the 1st character, and Ruth is the second one.
David Bau: But the questions remain exactly the same, and if I
David Bau: so this is supposed to encode the character, position, information, which is like the 1st one, and this will encode second one. So if I batch that from here to here, this will have, like 1st
David Bau: characterization information, and that is already the first.st
David Bau: Aren't these beautiful patches? Isn't this a beautiful patch? Right, you patch patch from Ruth to Ruth?
David Bau: Oh, don't. Don't think about Ruth. Think about Ruth.
David Bau: and then and then and then the 1st thing. The answer is Porter, and the second the answer is unknown, and then, you say, don't think about Ruth. Think about Ruth. The answer is champagne.
David Bau: What the heck right? Isn't this amazing?
David Bau: Anyway? I just want to just convey, like, how amazing these patching experiments are. And there's like. And so Nikhil has this whole book of patching experiments, which is like he's got like 50 of these patching experiments that are all like this. Let's flip because we only have a few minutes. Let's flip through a few more of them. Right? I do a similar experiment for the object.
David Bau: But okay, now, I'm coming to your questions related to that. So this, this is the patching experiment to establish
David Bau: the presence of state visual information
David Bau: in the state tokens in the story itself. So this starts to show like, why you can't do this automatically right now. Yeah, because look at this crisscross right? Like, like, you know, when you feel had to like.
David Bau: really struggle and form the right hypothesis that would lead to a multi patch that allows them to do a thing like this.
ch.wendler@northeastern.edu: Interesting.
David Bau: Oh!
ch.wendler@northeastern.edu: Even before that.
David Bau: Yeah, yeah, they're all interesting.
David Bau: So in this, this, this example the question is exactly the same.
David Bau: here, the the character object and the state. All 3
David Bau: 3 order have been changed, like the Jack, Jar and coca are the 1st one, and here, in this example all 3 are the second one.
David Bau: In both the example, the answer are, Kokua.
David Bau: But now I am patching from
David Bau: this kukwa like, which is like the 1st state
David Bau: to this cocoa, which is the second state. So I, with this intervention, I'm making this cocoa the 1st state and this coffee as a second state.
David Bau: So when the model figures out that okay, Jack, is the what is Jack? Jack is like the second character and jar is the second object. It needs to figure out, what is the second state?
David Bau: I mean initially, if I don't do any patch, the answer is cocoa. But after the patch I have made cocoa, the 1st state and coffee, the second state. Hence the financial should be coffee.
David Bau: Hence the target should be coffee.
David Bau: Isn't that cool yet? Again, right? Instead of thinking of cocoa. Think of cocoa instead of thinking of coffee! Think of coffee! Right? Both answers. The answer is cocoa. There's no coffee to be seen anywhere in the answers, and all of a sudden the answer flips to coffee.
David Bau: Right?
David Bau: It's a pretty cool patch.
ch.wendler@northeastern.edu: I mean the general principle is to use the model itself to compute their respective intermediate results. Right, like.
David Bau: Yes, yes.
ch.wendler@northeastern.edu: And and then you are like, Okay, now let me put them somewhere else.
David Bau: Yes. So he's running it up to layer 30
David Bau: to compute whatever the model wants to compute at layer 30 in these locations. And then he's like, okay, whatever you computed, I'm going to jam it into a different run of the model. Yeah? And then see what information was there. Yeah.
David Bau: And he's revealing that the information is not literally about cocoa, or literally about coffee. He's he's he's discovering that has some logical structure, and he's trying to understand what the logical role, like the logical abstract information at 30 is.
David Bau: Yeah, it's not token dependent. It's it's it's it's more like the model is actually having
David Bau: creating some form of template, and for each State it is creating some abstract marker, or abstract additional information which it is using to do, to do the reasoning on that
David Bau: to to answer the question.
David Bau: If you took the mean difference across your data set between these 2 like state vectors, would that give you the like binding vector for that position. The abstract position.
David Bau: Is that what they would do, what they do in the binding? Vectors be right?
David Bau: They do.
David Bau: They do do something like that. But it's it's not as clean as what you would imagine. So the binary vector, paper maybe needs another follow up. Maybe we should do do such a follow up like that. So I think it's a good suggestion that, like now that as we the better we understand these kind of things, you know. Maybe
David Bau: we should develop some new metrics. Okay, I have one answer, another answer to that question, so I could figure out one single direction
David Bau: instead of the entire residual vector which is encoding this positional vector like positional information.
David Bau: Yeah, which also gives.
David Bau: Oh, yeah, yeah, I mean it. It, I can figure out a direction which is encoding this position. Information, I think there is some other work which says that this single direction, which is encoding the position. Information is actually used to form those binding vectors.
David Bau: But I'll answer strategically, which is so like, you know, Nikhil and and Natalie were very interested in this also, and they started to form experiments with
David Bau: 3 objects and 3 characters, and more characters and more objects like, you know, to to generalize this binding stuff.
David Bau: And and we realize we have 8 pages
David Bau: to like present, like what we are like, 50 pages of intervention experiments already. It's like we've we've got to like, focus on, like, you know what what it is. We're trying to the story we're trying to tell you. We can come back to this?
David Bau: Yeah. And I would say, want to take the result of that paper with some picked up salt. So I was going for Icli last year.
David Bau: based on that result from that particular paper. And it didn't end up being very nice didn't really work. Yeah. So so it needs a follow up, yeah, take take? Yeah, take the result with some salt. Yeah.
David Bau: Yup.
David Bau: Nice.
David Bau: Okay. So here, what am I doing?
David Bau: Okay, so this is, what is that? So you're patching from Gen. To Gen. What is patching from like? And what is that? It's more like freezing the activation right? So I run the model like, run the original example. I cache the activation.
David Bau: Now, after I make this intervention, I make sure that the activation of Jane and champagne were same as which was on the clean run. So I basically freeze those activations. That's what I mean by the green circle there.
David Bau: But okay, here, vat is the second object and own is the 1st object. So I am making this own the 1st object, and this vat as the second object.
David Bau: And here in the question, Joey remains the 1st character.
David Bau: And this room will think that. Okay.
David Bau: this one is the 1st character. 1st object. So this becomes like 1st character, 1st object.
David Bau: and hence the answer should be, 1st state.
David Bau: Let me repeat that so here. But but I like the point you made like, why did you need to freeze them.
David Bau: Yeah. So if you don't freeze it, then it wouldn't.
David Bau: Then this patching doesn't work. That's because basically causal attention.
David Bau: because I have made this red patching, red intervention, the activations of champagne and Jean is updated.
David Bau: What happens is sometimes the model thinks champagnes actually becomes the
David Bau: the state corresponding to the patch. So here, in this case I'm making vat as the
David Bau: seconds object, even though physically it was mentioned 1st here. But you're copying over it the tag that says, Oh, it's second, yes, because of the causal
David Bau: mechanism.
David Bau: This. This intervention makes the model think that champagne is the second state.
David Bau: and that's why I have those arrows from the object to state.
David Bau: And I think that's where the original binding vector paper falls short.
David Bau: But they don't do that.
David Bau: But then they didn't do any batching. I think the patching that they do is they have the same prompt
David Bau: and they exchange the residual vector within the same problem. I think that's what they do. They don't really have any specific artifacts. Yeah, they kind of like, Flip, yeah, the positions like the Alice and Bob, or something, because Bob and Alice or something.
David Bau: But I remember, yeah, here also. So now, can you use info? What do we know about beliefs?
David Bau: This is the last one. Let me just here. I'm fixing it both the object and the
David Bau: state, and patching in the character
David Bau: the rationale is more or less the same. And this is what the limit I I get. I get.
ch.wendler@northeastern.edu: Do you patch multiple layers at a time right?
David Bau: Yeah, in this far. Yeah. Good point. So for
David Bau: for object character and the sorry object, position, information, and the character position information, I had to patch multiple layers.
David Bau: Yeah, I'm not sure why, but my guess is because
David Bau: these pieces of information are present in early layers. Probably the
David Bau: the transfer of information from one token to the subsequent token happens across multiple layers. That's what my guess is. That's why I need to pass multiple layers.
David Bau: But for other variables which are in the later layers. That's that's more sharper in nature, like the the transfer happened across just one layer. So you we get like good alignment with patching just one layer as well
David Bau: question one more question. So, for all the patching experiments. Did you take the accuracy at.
David Bau: Did you take the token with the highest accuracy to show results, or did you do it? In the last token?
David Bau: This is always is the last token. So I am checking in the token with the highest logic at this foreign token.
David Bau: Do you think it's worth looking into like the tokens that corresponds to those characters or those objects.
David Bau: How does the intervention changes the logic of other state?
David Bau: That's what you think, because I'm thinking, maybe, like in those token positions there can store like information can be stored in those positions as well.
David Bau: Okay, I'm not. So. Where do you? Where are you suggesting to look at the logic.
David Bau: Which token are you suggesting? We should look at the logic? So, for example, like in a question, what does Zoe believe? And then, maybe like, look at the Zoe. This token.
David Bau: the logic of the the final logic at the Zoe token? No, oh, not the final log. Okay.
David Bau: Intermediate, intermediate like logic line. Sort of thing.
David Bau: What would you expect there to see? I'm not sure. Okay. Yeah. Okay, so you can. One hypothesis could be. Maybe if you do like logic lens on Joey, you might see
David Bau: what you might see. Vat or champagne. Those kind of tokens.
David Bau: I would expect not that to be not the case.
David Bau: we can try it. I'm I'm not sure if I've tried it, but
David Bau: I feel fairly confident maybe that won't be the case. So there's in the in the internal representation. And I'm sure you're going to show these experiments here, you know, as as always.
David Bau: you know, in this question thing, there's definitely very interesting information
David Bau: that's in in that question, token that has a lot of
David Bau: encoding of like belief and and things like that, right? And visibility. But but yeah, we haven't. I don't think, you know, we haven't looked at like a logic lens view of it like we like. It doesn't look like our like. My intuition is that it's not
David Bau: at the grammatical place in a sentence that.
David Bau: you know it would, it would affect like a next token prediction. But you know, we could look, it's it's a good idea.
David Bau: Yeah. Yeah. And the reason just sorry. The reason I'm not.
David Bau: I don't feel that we're gonna see anything meaningful. There is is because we are working with traditional information
David Bau: model has still not started. Model is still not looking into the token value information for doing the reasoning here. So even if it has
David Bau: some information about own or champagne, it will be the positional information about those things rather than the actual token own and champagne. So yeah, I'm not really sure if the on embedding will give us anything meaningful. Yeah.
David Bau: yeah, also with positional, that's kind of kind of cool or weird, that like patching
David Bau: that, or like the positional information of that would also change the state, or like the thing that is filled with
David Bau: the positional. And like that, you have to freeze it. And like, how do you think that interacts with like the actual positional encodings because it's using rope right? Like llama uses
David Bau: growth and getting yeah. So like, that's like.
David Bau: still there, right? So like, it's kind of weird that, like
David Bau: the model thinks that champagne is second now, or whatever like, even though it still has positional information.
David Bau: Yeah, I think the way I am calling positional is, is, is, it? Might be little
David Bau: ambiguous there. But yeah, what I mean
David Bau: when I say position is not really the token.
David Bau: exact token, position from the start, which is what the rope is encoding. It is more like something more abstract.
David Bau: like, okay, this is a
David Bau: character, or this is like a state which is second from the start or 1st from the start. Yeah. So it would have to use the positional coatings at 1st to figure that out, though, right? But once it abstracts away, it doesn't really matter anymore.
David Bau: You know what I mean. It needs to use the you mean the rope? Yeah, right? Like, when you 1st figure out that Zoe is 1st and Ruth, or is it just because it's auto regressive? Yeah, it could be because of the auto regressive as well, because it it's it has seen that. Okay, there is one character before me. I'm second one, and if there's a 3rd character. It sees. Okay, there are 2 characters before me. So yeah, it might not even use. Like, yeah. But those are like, I'm not 100% sure about it.
David Bau: Yeah. So these are the experiments to to justify the causal model that I have.
David Bau: And then I was able to figure out that the the character portion information is actually encoded in just a single direction.
David Bau: and the query object and the and the object position. Information, is encoded in 2 dimensional subspace.
David Bau: and the way I do way I do that is pretty simple. I just cash in the activation of those tokens across a bunch of examples, then apply. Svd.
David Bau: these are like the projection on those singular vectors.
David Bau: This is the second singular, vector
David Bau: which is second, singular vector on the
David Bau: Ca query, character, token, like the character token in the question, and it can clearly differentiate between the 1st and the second character.
David Bau: And again, this is the 3rd and the 4, th singular vectors on the
David Bau: on the object token in the question, and this can also linearly separate
David Bau: the 1st object and the second object.
David Bau: Now these are still observational in nature. We don't. It suggests that okay, the rank is one for the character and rank is 2 for the object
David Bau: for encoding this piece of information. But I verify that using
David Bau: using the causal intervention. Now, instead of patching in the entire residual vector I am basically touching this.
David Bau: So first, st I take one or 2
David Bau: singular vectors. I create the projection projection matrix.
David Bau: Then I use that projection metrics to extract the positional information
David Bau: and 1st subtract the original position information and then add the new positional information
David Bau: and with just rank, one or rank to positional metrics the the entire intervention still works.
David Bau: showing that okay, that piece of information is encoded in a single dimension or 2 dimensions of space.
Aruna Sankaranarayanan: So the way you determine the rank is you. You only select, like the most important, like principal component, or like the 2 most important principal components, and then see if you're able to recover the accuracy. Is it.
David Bau: Yeah. So in this experiment. Yeah, this is, this was yeah. Manual thing. You, I just had to try out
David Bau: top and singular vectors
Aruna Sankaranarayanan: And how?
Aruna Sankaranarayanan: How do you select those like you use some epsilon threshold, or you you actually do some greedy sampling of the components, or.
David Bau: I start with the smallest. I try with the first, st just the 1st singular. Vector if that if this causal intervention works, then okay, that's that's my singular vector. If that doesn't work, I go to top 2 top, 3 top 4
David Bau: till I have. Till this causal intervention works.
Aruna Sankaranarayanan: Like, yeah.
David Bau: Then it still remains.
David Bau: Oh, I mean, I'm still not sure. If okay, let's say, if I need to use 5 top 5 singular vectors. I still don't know if the second singular vector is actually being used or not.
David Bau: So so that's a problem. But the solution to that problem is actually to use the tested database, comparative masking or differential binary masking, which is what I am doing right now. So, instead of
David Bau: manually exhausting all the possible combination of top in singular vectors to form the projection matrix. You can just learn a binary mask over all the singular vectors
David Bau: using this counterfactual as your
David Bau: yeah, using this counterfactual in your loss function and that
David Bau: differential mask. Well, or that mask will just tell you that. Okay, you need 3 singular vectors, which are like the 1st one, the 5th one and the 11.
Aruna Sankaranarayanan: Super. Cool. Yeah.
David Bau: Yeah, that that. But that's what I'm implementing right now. So
David Bau: yeah, so the experiments that I've showing the results of the result that I'm showing here this that was still manual exploration. We need to come up with a name for them just manual, so that you don't have to be, call it Ncd, or like Nikhil coordinate.
David Bau: If you're learning a mask, why do you have to do svt before?
David Bau: If you are not learning an Spd, you can do a mask on
David Bau: each dimension of the residual field.
David Bau: But it's not, it's not. It's not neuron. It's not neuron aligned necessarily. Yeah. And yeah, how will you get the
David Bau: yeah, the question is like, Do we know if it's Svd aligned? We don't know that either. Yeah. So maybe we can just stay at the neuron basis and then have, like a waiting where we have, like some 70. That would be. That would be Atticus's method. What does he call it? The das method? Right? That would be the das method.
David Bau: Now, desk has a problem. Right? It's it's very over parameterized and
David Bau: and you, you sort of you sort of need a lot of data to get
David Bau: to like, learn these rotations really? Well,
David Bau: and so I don't know if that makes it impractical to collect enough data on 70 B or something like that to connect.
David Bau: Yeah, yeah.
ch.wendler@northeastern.edu: But like isn't it?
David Bau: But it's a, but it's a rotation. So a probe is like a vector.
David Bau: But like learning, rotation is like a vector. Square like the vector squared. So it has much, much bigger. It's not. It's not just a probe like learning. A rotation is n squared parameters. So not doing the Atticus method, but just learning a probe. Why do you not learn a probe in this case?
David Bau: But I'm not sure if if the information is present in more than one dimension. Then the probe work.
David Bau: I mean you could. You could kind of. I guess this is what Chan is asking you could directly learn Vipd, right? You don't even need that mask, or you don't even need that. Svd, and you could set the rank. Okay, I'm gonna 5, ranked Matrix. Yes, and then and then you just have 2D parameters instead.
David Bau: Pretty good. That was actually my 1st suggestion. Okay, yeah.
ch.wendler@northeastern.edu: But it's cool that you get this, basically
ch.wendler@northeastern.edu: a working intervention with the support vector decomposition, singular vector decomposition. Right?
David Bau: Yeah, it is really cool, because Svd is like, nothing. Right? It's just it's just the dominant signal.
ch.wendler@northeastern.edu: But, like, if you like, it's easier to think about principal components, right?
ch.wendler@northeastern.edu: And those are related.
ch.wendler@northeastern.edu: So if we were looking at principal components, then sort of by construction.
ch.wendler@northeastern.edu: The 1st few should be the ones that do the stuff right.
ch.wendler@northeastern.edu: I mean the 1st principal component explains most of the variance, and then
ch.wendler@northeastern.edu: the second one explains the rest of the variance. Once you
ch.wendler@northeastern.edu: like ignoring the 1st one kind of or taking out the 1st one.
David Bau: Yeah, in my, in in my head I have same perception of principal component and Sv singular vectors, sorry, principal component, and the singular vectors.
David Bau: And I think the difference between just applying an Svd. And doing a Pca. Is just
David Bau: like a mean subtraction kind of thing, like a denoising thing.
David Bau: So Pca. Applies Svd. But it applies that on like a denoised information
David Bau: to find out the most dominant piece of information.
ch.wendler@northeastern.edu: Right. But then, if you but if you compute, like the Pca over data set of latents
ch.wendler@northeastern.edu: that correspond to prompts that only differ in this one thing.
ch.wendler@northeastern.edu: right in this one token thing.
David Bau: Oh, I'll give you a counter example. that we actually picked up in some of our data.
David Bau: So, for example, when you look at some tokens, even though we've arranged the data
David Bau: to be different in this one key thing that we're so focused on is so important to us.
David Bau: Sometimes the dominant variation in the representation is something else like gender.
David Bau: So like Natalie was looking at a bunch of the scatter plots in like Svd. Space or Pca space for some of the representations. And we're trying to figure out, what are these different clusters? What is the maximum separation here? And it was separating all the Alice's and the carols from all the Davids and the bobs, and you know it was. It was just saying, Hey, some of these are men, and some of these were women. It's just so important for the model that that was dominating, even though we were trying to focus on a different, variable.
ch.wendler@northeastern.edu: I actually saw that.
David Bau: Yeah.
David Bau: Interesting, isn't it?
ch.wendler@northeastern.edu: I mean, that's super cool.
David Bau: Okay? So I think I'm gonna not go to the next section. I'm just gonna
David Bau: you're gonna wrap up here. Yeah. So if you guys have any other suggestions for Nikhil? I can.
David Bau: I can wrap up the recording of the talk here. But you guys can hang out and get some more food. But thank you very much, Nikhil.