David Bau: Activation.
David Bau: Okay, so welcome to the day after neurips
David Bau: 8 am. Today. Welcome to yeah. Welcome to the hour after nerves. You probably most probably didn't didn't submit to it, because oh, you submitted. But you're still like vertical.
David Bau: Get into the movies. So
David Bau: yeah, remind me that we're we're gonna do some upgrades of computers and stuff today, which is a good day after deadline to do that, maybe today.
David Bau: But we'll we'll do this kind of thing. But
David Bau: but the what I want to do is I want to introduce Constanza. Who is finishing her Phd, and looking for a postdoctoral home. She's a little bit interviewing us to see if this would be a good place to hang out.
David Bau: and so but she did a bunch of work during her. Phd. Is very interesting, including a bunch of work on a factual recall. But also a bunch of you know other interesting topics that you should. You should ask her about while she's here. You're you're here for a couple more weeks. Right? Yeah.
David Bau: that's right. So she'll be hanging out here, and you should look look to her as a resource. But I'm really looking forward to hearing this work. I we? I chatted with cassanza a little bit yesterday, and and even though
David Bau: I feel like
David Bau: her perspective is that this this work is sort of wrapped up, and you know, ready to present and everything. I yeah, I look at it. And I think, Oh, there's there's a lot of interesting things in here that continue to open up interesting questions. And so I'm looking forward to hearing hearing your presentation.
David Bau: So let's see if I'm gonna I'm gonna try click around here to clean up
David Bau: this presentation here so that we can see.
David Bau: Look at slides.
David Bau: Yeah. Thanks for the presentation. And thanks for having me super excited.
David Bau: yeah, I was telling Brianna earlier this week that I seen like a lot of work that I'd like things coming from daily sub. And that's why I
David Bau: decided I wanted to come and visit and meet some of you in person. And yeah, I'd like
David Bau: discussion in front of you.
David Bau: Yeah, get to see how the group works a bit and all that.
David Bau: So today, I'm
David Bau: going to present you a bit of 2 recent papers or sort of recent. And so the talk is title, understanding, factual knowledge in language models, multilingual knowledge and epistemological implications.
David Bau: And just like, before we start just telling a bit about about myself.
David Bau: I come from Chile in South America. I grew up there, and I did my undergrad at the University of Chile.
David Bau: and right, after graduating, I moved to Paris, working as a software engineer at Youtube. And during my time there, like, we worked on the engineering side of like scaling
David Bau: thousands of different classifiers and machine learning models, and how to like, evaluate and maintain all these models and make sure that they continue to be as good over time. And how can we automate things and do not make mistakes?
David Bau: But yeah, after a couple of years I decided I wanted to start a Phd. So I quit my job and moved to Copenhagen, where I started my Phd. With Anna Sogart at the University of Copenhagen.
David Bau: And yeah, and on sounds like finishing, probably. Yeah.
David Bau: trying to get to start writing the thesis, and then I'll be done. Right? So my interest is like pretty broad in interoperability. I really like interested in all things
David Bau: that allow us to understand how a model makes
David Bau: its predictions. And during my Phd, I have done a few works related to factual knowledge and trying to understand how the model stores and retrieves factual knowledge. But but yeah, I've also done like a few other
David Bau: things. Now, in factual knowledge.
David Bau: right? So today's talk is going to be about these 2 papers.
David Bau: And we're gonna 1st start with, how do multilingual language models remember facts?
David Bau: If at any point you wanna comment on something or yeah, just like, ask any questions, feel free to interrupt me.
David Bau: So in in this work.
David Bau: We looked at 3 questions. So first, st we wanted to see if, like the localizations that have been found for English models
David Bau: would extend to multilingual models, and whether these localizations will be the same across the different languages.
David Bau: and then the second question, pretty similar, but in terms of information flow. So the information that is flowing from the context of the last token, and how that works for English models do we see the same kind of patterns in multilingual models.
David Bau: And then the last bit is okay understanding, like how the things that we know extend or not to multilingual models were in this.
David Bau: in this recall process, does language play a role?
David Bau: The 1st 2 parts of the work of, like the 1st 2 questions really kind of like, extend the work from previous papers to multilingual models, and we directly compare to their findings. So one of those words is from this group the wrong paper, and the other one is from Morgueva dissecting associations.
David Bau: Right? So just a bit about the experimental setup.
David Bau: We do kind of like the same thing that people have been doing for really looking at simple factual recall. So we have 5 triples of subject relation and an object. And you have, like natural templates that describe this. So here the example is, Barack Obama was born in.
David Bau: and then you feed that to your model, and you see if the model
David Bau: predicts Honolulu or Hawaii. And and then you say that, yeah, the model knows that.
David Bau: We use the the Emperor data set, which contains 45 languages, and 37 different of these relations, so born in capital.
David Bau: I don't know language spoken, and so on.
David Bau: And these were machine translated from English templates.
David Bau: and in this paper we study 12 different languages, and we study 3 different models. 2 models are decoder only Xglm and Urlm, and we also study an encoder decoder model. Mt, 5, to kind of like, see? Yeah. Kind of what differences do we see for different architectures? And also with the model that covers a lot more languages.
David Bau: I have a question. Is there factory plates like from the same language? So you have multiple languages. 12. Language. When you construct a flat factory plates, you keep subject, object, relation, object from the same language, or you mix it up.
David Bau: So you mean in the input, whether it's yeah, the input is always like the same language. So everything was translated like the the entity the subject, for example, was translated using wikidata so that you get the way that people would say it in Chinese, and and you use that with the Chinese. You are doing it with the multilingual language model. So it can be that like, you have your subject from Chinese, your relation from English.
David Bau: your object from somewhere else. Yeah, does the model understand that? Right? Right? Yeah. That's
David Bau: definitely something interesting. But we don't really look at that in this paper. I would. Yeah, I would imagine it
David Bau: works. And it will be interesting, actually, for maybe
David Bau: trying to understand from where the model gets the the language we're going to touch upon that a little bit.
David Bau: But yeah, good point.
David Bau: Right? So then we take this, yeah, go ahead. Can I ask a question about these models. So what what makes them like?
David Bau: Is it just that they just pre train on a bunch more languages? Is that like the main difference? Yeah. Good point. So yeah, we actually picked these languages because, we really wanted to focus on the multilingual side of things. Because
David Bau: we could have like pick. I don't know like Lama, which is like a lot more studied by people, but Lama was like, mostly trained in English, and just a little bit of other languages, and so the model would probably work differently. And we wanted to understand, like the multilingual setup of things. So that's why we pick these languages which
David Bau: do a bit of bigger effort to train or like to sample more the rest of the languages.
David Bau: other differences between them is like your Lm. Is much newer model. I came out last year, like at the end of last year, and
David Bau: Xglm is a lot older. Xglm had a
David Bau: more aggressive sampling, more like uniform across the languages. So it was like really trying to be more multilingual. Your Llm. Does it? But then it has, like some parts in the training where it's like we put more English because we have a lot of English data. And so they they still had a lot of the other languages, but to a lesser extent.
David Bau: And then, yeah, mc, 5 has
David Bau: a lot a lot more languages. But it's also a smaller model.
David Bau: Please, thank you.
David Bau: Yeah. And so then we take that data
David Bau: we generate with each of the models. And we just really the coding we allow for like multiple tokens in our generation. And then we do exact matching.
David Bau: we try to like use different aliases of the object so different ways of
David Bau: expressing the same object so that we can really try and match as many examples as we have as we can.
David Bau: And and then we just do it. Yeah, the exact matching.
David Bau: If there are like extra tokens that were generated, because maybe like in some languages, you have the template Barack Obama was born in, and then the model adds, like some articles or something extra we then add that to our example. So then, so then, in the subsequent analysis, always. The the next token is the 1st token of the object.
David Bau: and with that we have this number of examples that were correctly predicted by each model in each language.
David Bau: We
David Bau: omit these languages from the rest of the analysis and the averages that we take just because it's they have like too few examples, and they
David Bau: tend to be more noisy in the results.
David Bau: No idea why Japanese is so bad.
David Bau: why Japanese is so bad. Well, some.
David Bau: If I think Xlm didn't even have it in the training data.
David Bau: we can. So we decide on the 12 languages to have, like a broad kind of like different scripts, different
David Bau: language families and different ways to construct a sentence. But then, yeah, and they didn't add Japanese sometimes, even if they didn't add the language. The models are still
David Bau: okay. And that's why we still tested it. But yeah, it.
David Bau: It's just so bad. It's just really bad even in Mt. 5. It's bad.
David Bau: Yeah, but it's like in Mt. 5, is.
David Bau: it's like, yeah, it's the worst. But but it's
David Bau: it's like, kind of near the others. And we see that Mt. 5 has, like a lot more
David Bau: like a better distributed a better distribution of of the memorized facts.
David Bau: Hi, no worries.
David Bau: Yeah. In. Yeah, we see that URL is a bit more skewed. But yeah, that's what it is. But it's still decent, more or less for for the other languages.
David Bau: Hmm, right? So then, we're gonna use this subset of memorized examples for all the other analysis.
David Bau: And so the 1st thing that we did was to kind of like run the localization castle, tracing to see if things are
David Bau: looking the same or not, or whether, yeah, what? What will we find?
David Bau: And I just like added a few details like we
David Bau: use like for each language. At most, a thousand examples. If we have more, we just like discard it because
David Bau: it is pretty expensive to to run we restore
David Bau: around 12% of layers, for
David Bau: we reserve windows of 12% consecutive layers of Mlps and attention layers.
David Bau: and we measure same as like following the the wrong paper, the recovered score. So the score that you get when you patch minus the the score that you get in the noisy run. What does restore means? Yeah. So
David Bau: it's so in in the castle, tracing setup you first.st
David Bau: You have, like the the.
David Bau: the sentence that we had before Barack Obama was born in, and you add noise to the tokens of Barack Obama. So the subject. And and then you perform that inference.
David Bau: and you're gonna get something more noisy, right? You're kind of like breaking this this inference. And then
David Bau: you run, you run it again again, adding noise to the subject, but then you restore in some specific component. Say, the layer 12. I'm gonna put back.
David Bau: What like the activations that I had in the in in a normal run, in a run with no noise. And so by patching that you see, okay, how much am I recovering? By that reservation
David Bau: and then in the plots that I'm showing in a minute. We're looking at logics, because we kind of like looked at probabilities and logics and probabilities tend to
David Bau: underestimate some components a little bit. But they're not all that different anyways.
David Bau: So 1st is for comparison, and as a background we have Gpt-two, which is the English model mostly studied in previous works looking at factual Rico. And we get this result. So in these plots, we have in the X-axis we have aggregations of tokens so tokens before the subject, the 1st subject, token.
David Bau: meet, subject tokens, the last subject, token. The 1st answer, and yeah, all at the bottom. We have, like the last token.
David Bau: and then on the
David Bau: X-axis you have the layers, and this will show, like each of the reservations that we do.
David Bau: For the Mlps. And attention with the reservations in patches of windows. And for the sake it's just at at each layer.
David Bau: And what?
David Bau: Yeah? Like, the main kind of like finding. So in in the 1st part of the wrong paper, where? Okay, we see this early site. In the Mlps.
David Bau: and and then there's also like this late side, which is mostly driven by the self attention.
David Bau: What is all such of the last subject? Is that all? Subject? Yes, all.
David Bau: So also, yeah. Like all the subjects. Why don't you see where you see the last subject, like Mlp. Is firing on the last subject? But don't see when you patch for all subjects. That's so. No, this is this is not so. The patching is
David Bau: It's always done like in
David Bau: it's always the same. The Y axis is just how we aggregate for the plot. So so basically in here, here, the all subject tokens would be just the average of looking at all. Exactly. Yeah, because you're averaging with the 1st and the middle subject token. And the other thing is that what's the difference between all after subject and after subject last. Yeah, it's kind of like.
David Bau: I I should. Probably that's a good point, like I should have maybe cleaned this, because when I plotted it was like more for us to like. Look at what was happening. So all, after the subject tokens like again, all the subjects that come after the subject, just like average. And like, if anything that comes after the subject, all those tokens. Yeah. Yeah. And then
David Bau: after the 7. So yeah, in this case, these these 2 are are the same.
David Bau: Like all, after subject and after subject loss. Yeah, yeah. So we should to to be more clear the the second to last sort of been removed because it's
David Bau: But in the figure it's different. The last 2, the last 2 I have another question. Yeah. Do you think, is there a effect of the context window of the model? In this analysis, you're doing? Yeah, yeah, totally like, it is
David Bau: dependent. Because that's why, you have the yeah in, like in the last one. Yeah, like, I
David Bau: can, you say again, the left part. So basically, when you're passing your
David Bau: your input stream. So it's looks like it's mostly
David Bau: kind of doing something for the the last piece of text you are passing in.
David Bau: am I, is that what's happening? Or
David Bau: so what I wanted to say about the window? Yeah. Sure, like, this is completely dependent on the window, like, if you pick like.
David Bau: and it kind of like distorts things a bit right? Because maybe here we're looking at this, and this still has some effect. But maybe there is no effect there. But since we're doing a window, the effect is just coming from the earlier layers, and the layers just before, and that layer
David Bau: have no effect. But then we're still taking a bit of the effect that's coming from before. And so yeah, if you do like a smaller window, then it would be more specific. But then that also means that maybe just replacing one Mlp is not enough to really see a change on the output.
David Bau: Yeah, yeah.
David Bau: right. And so then here we have the 2 decoder only models. So Urlm and Xlm, and we kind of like, see similar patterns.
David Bau: we see in early lam. We see the early side in Mlps pretty clearly in excel, and we don't see it, and
David Bau: that kind of like might be because of of the window. You could have tried like a higher, like a bigger window. I think we did, but still like it was pretty
David Bau: like a lot lower than the other models.
David Bau: those are the auto. Yes, autoregressive models. Yeah.
David Bau: And then we see in both the the lake side. And
David Bau: we do see that the Mlp. Seem to be more causally relevant for this prediction than they were in Gpt.
David Bau: And this will
David Bau: Afterwards we're going to have like more evidence about the the prominence of of the Mlps.
David Bau: And then for Mt. 5 here, yeah, the plot
David Bau: could be a bit confusing. So in the x-axis again, it's like the the aggregation of tokens. In this 1st rows we have the encoder tokens, and then the last rows are tokens that are part of the decoder
David Bau: and then we see again the
David Bau: Mlps Council, like the early side but here we see it like pretty constant in all the encoder layers so like it kind of like doesn't matter where you patch back
David Bau: in the middle or at the end like it's pretty much the same. And we also see that the 1st subject token
David Bau: it's it's
David Bau: still relevant, like the last subject. Token has more relevance. But but yeah, the 1st one is more than in the ultra aggressive models.
David Bau: And yeah, and then we see the late side again. It's like a bit shared between mops and and the cross. Attention! And we also see it that the the cross attention is like later, almost like glass layers compared to
David Bau: what we're seeing before.
David Bau: Yeah.
David Bau: So
David Bau: what we see from the Council tracing just like summing it up a bit in terms of the early side we see that in the quarter only models pretty much extends to what we had seen in English Gpt
David Bau: and in encoder decoder. We see that both the 1st and last object
David Bau: seem to be relevant, and it covers all the included layers
David Bau: in terms of the late side. We see a bit more relevance of the Mlps compared to just the attentions in English models.
David Bau: right? So moving into the section, or like the second question that I mentioned at the beginning. So the information flow in this section. So before, it was mostly like just looking at places right like, which components seem to be useful. But we don't really know for what or like what exactly they're doing in this section. We
David Bau: follow and kind of like study. This 3 step process that
David Bau: keba described in in the paper.
David Bau: where for English models they see that 1st
David Bau: the relation flows to the last token.
David Bau: and so the information about the relation flows to the last token.
David Bau: Then, on a later stage, the subject flows to the last token.
David Bau: and finally the object is resolved by attention. Lake.
David Bau: so we want to see kind of like this to these 3 steps
David Bau: also happen, and to what extent, in multilingual models.
David Bau: So
David Bau: 1st we we ran like the attention knockout experiment, which
David Bau: you want to see how information is flowing to last. So again, so
David Bau: what you do is you mask
David Bau: the attention from the last token to some set of tokens in the input so
David Bau: at layer 10, you don't allow the attention to look at the subject tokens, or you don't allow the attention to look at the non subject tokens. And you see what happens with the probability?
David Bau: Yeah. And we then measure the probability change. So P. Tilde, is the probability that we obtain when we knock out
David Bau: the attention and we subtract the the original probability of the predicted token.
David Bau: And again, we're doing this intervention in windows right? In here. We
David Bau: follow the the same window that they gave I use, and we knock out around 18% of the layers.
David Bau: This is a point of information. So when the knockout happens, does it happen like everything before the point you choose cannot be looked at or just that particular spot. Yeah. So we we knock out exactly like
David Bau: subject tokens. So if you have tokens that happen before the subject. The the last token will be able to look at those, but just not to the subject tokens. So like everything before everything after. Yeah, yeah, exactly.
David Bau: So here we have. The results. The colors are a bit bad, but
David Bau: in gray we have gpt.
David Bau: And 1st let's look at the the left figure which is knocking out the subject tokens.
David Bau: and and here we see we can no worries.
David Bau: Here we we see that in all models. We have this big drop that's right. In, like
David Bau: the later layers around, like
David Bau: 60%, 80% of the layers. All the models have like this big drop if you don't allow it to look at the at the subject tokens, and we see that Mt. 5 is like slightly later.
David Bau: So in terms of looking at the at the subject, we see that
David Bau: in all of this happen kind of like the same.
David Bau: But then, for the non-subject tokens which would encode the relation and whatever else is needed. We see that for Gpt it has like a big drop at the beginning, but then it increases pretty linearly. And then when it's looking like at the subject, like around layer 60 to 80%, this drop is not all that high. It's like minus 20% as opposed to the minus 60% of the subject
David Bau: in contrast. The multilingual models
David Bau: have, like a lot of like a lot more drop in the non subject tokens and like in Mt. 5 and Euroland, we really see there's like a
David Bau: big drop at the beginning. And then even like another drop at the same moment. That is
David Bau: looking at at the subject.
David Bau: And then in Hclm, there's like a huge drop at the beginning, and then it grows linearly, but still like when it's looking at the subject, the drop of
David Bau: masking the non-subject token is still pretty big and even bigger than those of the subject.
David Bau: Then
David Bau: this is the same plot, but just for your Llm. And divided by languages, and this kind of like show us that the trend that we see is consistent for each of the languages. So basically, the model is doing the same for each of the languages.
David Bau: with a bit of differences here and there, but more or less the same trend
David Bau: and so from this, where we see, as I just said, that we have similar information flow across languages
David Bau: in terms of the subject flow similar than English. Gpt. There's like this late stage where the model is really looking at the subject tokens
David Bau: in terms of the of the relation flow.
David Bau: We do not see that this flowing of information really precedes the subject. It seems that the non-subject tokens are like attended all throughout the layers.
David Bau: And yeah, that's like, probably the languages I come in in some part of of the of that.
David Bau: and we'll look at that in the last experiment.
David Bau: And so now we're going to look at the 3rd step describing the in the give us paper. So they saw that the object extraction is mostly performed by attention layers.
David Bau: and so to do to to kind of analyze that part what we do is
David Bau: we do have a calorie projection from the contributions of the Mlps and attentions, and we check, if
David Bau: at that point the top one token from that vocabulary prediction is the the same
David Bau: that was predicted, or that is predicted.
David Bau: And if it's the same, then we we kind of like, say, Okay, this is an extraction event. This Mlp is really pushing for that token to be in the top.
David Bau: And yeah, then we measure, how many of these extraction events happen. Do you measure this by feeding it through the language modeling hat, or
David Bau: yeah, so on and so forth.
David Bau: that one is not good at
David Bau: decoding the language from the middle layers.
David Bau: Right? And that's kind of like the assumption of yeah. But but the thing is, when we're looking at the object structure. It's what's happening at the end. Right? You. It's kind of like the late side that you see in that multilingual, because this could be a name. This could be Barack Obama.
David Bau: That's not even really multilingual oops.
David Bau: You mean that the barrack token doesn't change. Yeah. But like some some component has to push the actual language, specific token, to be at the end right.
David Bau: other language, specific continuations as well.
David Bau: What do you mean?
David Bau: Well, are all subject predicate objects of the form. That subject and object are names, or are there also other things? So yeah, they're always objects are entities. But the entities
David Bau: can have different expressions like in different languages. They're not all the same. Yeah,
David Bau: so this is for comparison. This is the plot for Gpt, so we have is like how many extraction?
David Bau: We see! For, like all our examples, how many extraction events.
David Bau: and and in green, we have attention models and in purple Mlps, and
David Bau: from here, and like some other extra experiments, the main conclusion was, Okay, this is like, really done by the attention layers.
David Bau: When we look at what? What were you extracting? Yeah, so it's like we are not extracting, we
David Bau: in a way.
David Bau: it's called like the extraction rate, as in like, the model is extracting at that point the the object that it wants to predict. Because, yeah, how does that metric work again? Like?
David Bau: So for each example, we do that like one Mlp. We projected to the vocal with the
David Bau: language model head, and we see if the top one token is the same that is predicted at the end. If it is, then we count it as one, if not 0.
David Bau: You should try to rerun this.
David Bau: That's the pepscope and something. How?
David Bau: Thank you.
David Bau: Because cheap, I mean, I don't know. Cheap. It is a tool. It
David Bau: I guess it will just decode to English.
David Bau: Yeah, yeah, I mean, these results are just with English examples. Right? This is just a comparison. It's in like, yeah, it's like, this is what happens for English. Let's now see what happens in in the other languages with the other models.
David Bau: So in some sense, if you can extract the top, one token within some intermediate layers, would that suggest that the later layers are absolutely unnecessary? I mean, if your extraction rate is like 200.
David Bau: Well, not necessarily because it's like, okay, yeah, maybe, like in layer 21 attention layer was like, yeah, this is the token. And like putting that into the residual string. But that's not enough to like really
David Bau: move the residual stream. To that token there has to be like a lot of additions of the same thing to then.
David Bau: for that representation to be really close to that token.
David Bau: Then the solution can be that. Well, top one accuracy is a really, really hard metric. Yeah, yeah, like, it could be that some are also extracting, but by just having it in the top 2. But it's kind of like a hard metric to really say.
David Bau: okay, this component is really pushing for that token. It can be like, like like multiple attention heads, they kind of like contribute towards that token. But
David Bau: when you kind of like, consider their single contribution. It's kind of like, not enough to push that token like up to the very top. Yeah. Yeah. And and to be honest, like
David Bau: when you look at it per language, because here I only have, like the average across languages, like in some languages, you do see extraction rates that are much low, much lower. And that is probably that. Yeah, those tokens are still like pushed by these components, but
David Bau: not in the top one, but maybe in the top 2. But then that aggregates at the end. Still.
David Bau: right? So these are the plots for the multilingual models with the multilingual data average across languages.
David Bau: And here we see that
David Bau: for both the ultra aggressive models, then the Mlps seem to be a lot more prominent in
David Bau: this extraction event.
David Bau: like there are, they are also adding, and even more than the attentions. The tokens that should be predicted.
David Bau: But the y-axis is an addition scale here.
David Bau: Yeah, it's like point 2 and not point 3.
David Bau: Yeah. But what I'm comparing is like between, like the 2 components just saying that
David Bau: like. Here we have the Mlps like as much as in Mt. 5, though we see a more similar trend to like English Gpt, where the cross attentions are really doing like most of the job and the Mlps to a lesser extent.
David Bau: Yeah, yeah. So just like adding to this that I I think that's interesting, because
David Bau: in a ways like when you look at in the council analysis, when you look at the late side, you're like, well, yeah, that's kind of like expected, like at some point that that if you just replace you with the with the correct token, then you would restore everything. But
David Bau: I think for multi the multilingual setup, it's interesting, because
David Bau: at some point, if you have like more general representations at some point you have to decide in which language you're gonna say it, and how it's like said in that language. And then we see that for doing that attentions on itself are not like, and often the model needs the the mops to. To add this, the specific language
David Bau: form
David Bau: right. So onto the the last part of the paper, which is like, okay. Now, we can only have an idea things more or less. We know that the model is like looking at the subject at the end. It's looking at, not subject tokens all around, but we're not sure like in which part looks at the relation which part looks at the language. And how do these 2 relate to each other?
David Bau: So we look at that now by doing activation activation patching. And so the setup is we have in blue.
David Bau: One inference pass, and we do it in English. So we have 5 in English, and then we take
David Bau: the last token representation.
David Bau: and we replace it into the inference pass of another language path, and then we see what happens in terms of probability, prediction, and so on.
David Bau: And we do this in 3 different setups. So first, st we look at
David Bau: 1st of acts that share the same language, so English, the 2 of them, but they have a different relation and a different subject. So in this example, one is the relation of capital, and the other one is language, and
David Bau: one subject is France, and the other one is Germany.
David Bau: The second setup that we look at is
David Bau: same relation, but different language and different subjects. So
David Bau: The subjects are France and Germany, and both have the the capital of those.
David Bau: And yeah, one is in in this example. One is in English, and the other one is Spanish
David Bau: and then the 3rd setup is
David Bau: same subject, but different language and relation.
David Bau: So in the example we have the subject in both is France, but one is the capital, and the other one is the language.
David Bau: We only experiment with
David Bau: patching from English into other languages just for just because of time. And but it would be interesting to see if this really extends to
David Bau: other languages.
David Bau: Right? So with the 1st setup we
David Bau: we'll really be able to to tell when the relation flows to the last token.
David Bau: Then with the second setup even though we have the same relation, but different languages we're going to be able to look at when the language flows. And then, lastly, by having the 3rd setup, we're going to be able to see whether the language and relation are entangled.
David Bau: So the 1st setup we have the patch in patch and context are in English
David Bau: And then we patch layer by layer and see what happens
David Bau: and what we see. This is for the 3 different models is that when we patch
David Bau: pretty late. So in blue. We basically, the patch already contains the object. And so we get parrots
David Bau: here. We're looking at the the histogram of
David Bau: the token that we get in the output.
David Bau: So if we patch too late.
David Bau: we obtained Paris, so we obtained that the answer for for the Patch example, if we batch really early.
David Bau: we don't. Really, the patch doesn't have much information. And and so we
David Bau: obtain a lot of times just German, which is the answer from the context example.
David Bau: But then in between we see that we obtain the
David Bau: the the answer with the relation from the patch. So the patch contains a relation, but it attends to the subject in the context, right? And so we obtain Berlin, which is the capital of Germany.
David Bau: And so this plot really tell us that in purple is the places where the last token already contains a relation
David Bau: before that doesn't yet. And so, right before that purple is when the relation flows, and and that's kind of like what we were seeing from the information flow. The the like big drop at the beginning, right? That's the point where the relation is flowing.
David Bau: Then in the second setup, where we have same relation but different languages, we
David Bau: see these plots where we're counting the number of times that we obtained the object from the context. But in the language of the patch, so in this example is the times that we would obtain Berlin. So the capital of Germany but expressed in English. So we're obtaining the answer for the context.
David Bau: But in the language of the patch. So what we see is that around these layers
David Bau: the last token already has in its representation which relation should be extracted, and in which language.
David Bau: So basically, it defined what it should do in English. But then it does it in whatever other language is in the context.
David Bau: So if so, you. You never see.
David Bau: You never see Paris.
David Bau: Yeah. So this is one of. So
David Bau: there is a table in the paper where we kind of like measure how many times we see the opposite right? And it's like.
David Bau: really like a few times, I get 5 or yeah.
David Bau: And and most of those examples are a bit noisy, like the the 5 is like, really, sometimes 0 in the sense that to match.
David Bau: To really say, Okay, this is the object from the context. In the
David Bau: language of the patch, we have to like constrain a lot of things. For example, we need to constrain that both of the objects are expressed in different ways, because if both of the objects are spelled the same in both languages, then we cannot know
David Bau: in which, whether it's the language of the patch or the language of of the context. And then other times. I
David Bau: maybe the object is tokenizing like multiple tokens. And the beginning. It's just like
David Bau: a T, and then can we really say like, which of the 2 is or not? And so there are like a few constraints to like.
David Bau: Here, we really measure when we're sure that that's happening.
David Bau: Right? So so you're okay.
David Bau: So your conclusion from this is that there's a dispatch at player.
David Bau: you know, 23, or whatever
David Bau: right contains information of what language you're trying to say without containing any information about
David Bau: what what the attribute is that you're trying to utter. And and then it's clearly after
David Bau: the attribute has already been figured out. So there's almost no interference. Yeah, it's like a very, very clean.
David Bau: I mean, it's kind of a remarkably clean separation. Yeah, yeah.
David Bau: Yeah. To the 2022 paper.
David Bau: When did you publish this last year end of last year.
David Bau: This is, I wonder whether, besides, you guys, because it's it's so similar.
David Bau: So what we did. Are you familiar with Chris's.
David Bau: I I think it's the one from the Icmo workshop this experiment. But one thing that that I thought it was like in contrast. But okay, maybe I'll finish with the other experiment, and then I'll comment this to not like, make it confusing.
David Bau: But I'll come back to that right? So.
David Bau: then, in the 3rd experiment, we have the same subject, and
David Bau: because here we see that the last token encodes both what should be extracted like the relation, and in which language?
David Bau: But then the question is like, okay.
David Bau: then are those 2 things entangled together, the relation and the language? And so with this setup.
David Bau: We're able to see that for the decoder only models. That is not the case
David Bau: for Mt. 5, is it? I can comment on that in a bid.
David Bau: but it's a bit more difficult to really conclude on it. So for these 2 models.
David Bau: in these layers we obtain the object from the patch, but in the language of the context, so basically, the last token already contains the relation
David Bau: well already contains the the relation, but not yet the language. So the language is attended like after this, and because we're obtaining like in this example, we're obtaining Paris.
David Bau: Which is the capital that is a relation of the patch.
David Bau: but in the language of the context.
David Bau: and so putting that together and doing a bit of normalization. So the 3 cores are like
David Bau: in the same scale. We see it's like two-step function for these 2 models.
David Bau: which is like in green.
David Bau: The last token has the representation, and then in purple. The last token now also has the language that should be answered, and then in blue is
David Bau: when this function is like applied to the context, and then you get the object that that is at the end predicted.
David Bau: And and what I wanted to comment about that paper is that
David Bau: that setup was like a translation task. And so, yeah. And so.
David Bau: because the language is the main thing that that is happening, you see that the language is encoded at the beginning. And so you say, like, okay, the language that should be outputted is encoded pretty early. But then, the way I see it is like. It's not the language that is encoded early is the task like, what what should I do? That is encoded early, which is what we see here as being the relation.
David Bau: Why do you call it two-step function? Because you have like this to
David Bau: stages where the model creates this function, and you can with with the experiments you can see that
David Bau: you can take, like
David Bau: what you have until like just having the relation, and then the model is still able to like, grab the language, aggregate it into that. And then really apply this function. Because, yeah, it's like, I got a function because it doesn't contain information yet about the object
David Bau: we try to catch like different things in, like 2 different layers at the mean at the same time, so that you can maybe further verify like, it's a 2 step processing.
David Bau: But how?
David Bau: How would you do so like what? What? Exactly do you want to? Look at? For example, you can try to like patch, some like the
David Bau: the language or the like, the relation thing earlier layer and the patch, the
David Bau: maybe the subject, or in the later layer like Patch 2 2 things and see what happens in
David Bau: different cases. Like, if you pitch something early, something late and
David Bau: and something else earlier. And there's something else later. Yeah, I didn't try it. But in a way, like, always, I was like thinking about it like, yeah, could I like
David Bau: kind of control for everything. I want you to output in this language, this relation and kind of like steer them all. But I didn't try it, just because just because I was always like a bit
David Bau: and I doubtful whether it would work whether I would like completely break it or not. But but yeah, that's a that's a that's something that one could try. Yeah, could you just say again, like
David Bau: green, purple, blue, like, what would be an example of like, yeah. So so in here, I'm kind of like just putting it together, these 3 plus so the blue is like
David Bau: from the 1st experiment. So the same blue that we have here.
David Bau: and then this is a purple. So from from the second experiment is where we see that both. We have both the relation and the language.
David Bau: and then the green is this plot. So I'm just like putting them together so we can see that those things that are happening in different layers. So green is only relation. Yeah, purple is relation and language. Yeah, blue is
David Bau: like the object, is there? So the function was like, applied
David Bau: right here. Kind of thing. Okay, yeah.
David Bau: How are you making these pairs of languages? You're just coming up with them. Yeah, that's a good point in in the review process. I was not clear enough at some point. And yeah, basically in the data set,
David Bau: we have the relation kind of id of each example.
David Bau: And then we also have the subjects ids and the objects id, and so we match them across languages. We're like, okay for the same. I'm gonna take like, yeah.
David Bau: this subject, Id. I want other relations that also contain this subject, and then I obtain those relations, and then I match them up kind of like with the data. But that also means that
David Bau: we currently have, like around a thousand examples, more or less like something 800. It's not that much as we had previously, because we're doing this match to really follow the the settings that we need to experiment.
David Bau: I think these plots would be much easier to read if you would put one example onto the slides or something, because I'm just seeing. Lp, yeah, yeah, yeah, it's like coming from from before. But it's true.
David Bau: Yeah, that's how we find things.
David Bau: Right? So to conclude a bit on on this part we saw that we had, like similar overall patterns, and information flows across languages. So for different languages the models are doing pretty much the same.
David Bau: We also saw that English decoder only findings mostly extend to multilingual decoder, only models, and there are, like a few nuances with encoder decoder.
David Bau: We saw that the late side is present in both Mlps and attention layers when we're looking at multilingual models, and we saw that both in the casual tracing and then in the extraction rates.
David Bau: and then, in terms of the relation, flow with the information, flow with the attention knockout. We couldn't really see where where the relation was flowing. But then, with the patching, we really got a better understanding.
David Bau: And then, finally, we're we're able to observe these 2 step function that is happening in decoder only models where 1st the relation is encoded, then the language
David Bau: to then finally apply this function to the context.
David Bau: so just to like a bit of like in the general. Okay.
David Bau: going away from the details exactly. But looking in general about the recall process, we see that
David Bau: the function in the last token
David Bau: is formed in stages, and I think it will be interesting to see like if it happens in general, like other setups that are not like
David Bau: recalling and multilinguality, and whether that extends to like more complex task solving or not.
David Bau: Then we see that the function works in
David Bau: context in different languages. So we form this function from English, and then we could apply it to context in other languages. So this function is pretty multilingual
David Bau: or like it works in in different languages. And then, lastly, yeah, we we see that the multilingual extraction uses both Mlps and and attention layers. And and I I wonder if
David Bau: this has, like
David Bau: any implications so like, if you want to edit knowledge in in multilingual models. Should you also like edit this later Mlps, are they needed? If you want to do that or not?
David Bau: yeah.
David Bau: So that, okay, since. So some some comments and questions and suggestions, and so on just on this 1st paper. So it it so like it's. It's very nice story about multilingual factual recall.
David Bau: and and you began with this nice performance plot of how different languages have different levels of accuracy. And yet so if I were to
David Bau: anthropomorphize, you know the mechanism you found here.
David Bau: To me. It looks like
David Bau: 1st you do this language independent recall of the fact. It's brought to the last token.
David Bau: and then somewhere around player 28,
David Bau: it says, Oh, my gosh! I have to answer in German.
David Bau: and and then it activates the final mops and says,
David Bau: You know, how how do I? You know. How do I say this in German, and then and then activate some Mlps or something to put it? This you know, multilingual concept into the German form, right? And so
David Bau: so if that's the mechanism, then you would imagine that if a model knows a fact in one language.
David Bau: then it would know that fact in all languages.
David Bau: The same. Because that 1st phase seems like it's multilingual. It seems like it's shared. And yet you have this very different performance. Yeah. So so yeah, like, this is one story of what's happening right? But
David Bau: the fact that we find this happening doesn't mean that it happens for all the the facts. Right they could be some other facts were.
David Bau: we don't find this right like we don't see this exactly, so I guess my question would be, is it possible to like go from the understanding of this mechanism
David Bau: to maybe a new version of that
David Bau: performance chart where you say, oh, actually, you know, you know, in
David Bau: In Korean, which has, like a lower performance number for a bunch of these things.
David Bau: It's a.
David Bau: you know, like the 30% of the time. For whatever reason, when it's working in Korean, it actually
David Bau: doesn't know the fact like it doesn't like it doesn't have a representation that ends up at the last token. If I patch it into English, it doesn't work for some reason, like in Korean, like the factual recalls broken 30% of that. But maybe 70% of the time it like the fact is there. But it doesn't know how to utter it
David Bau: in Korean, doesn't it fails to translate it into Korean, or something like like it. It seems like it might be possible. It seems like it might be possible to go from the aggregate end to end benchmark that everybody reports to now a more fine grained benchmark which says, Oh, actually, you know, patching allows me
David Bau: to
David Bau: more fine grain diagnose like where the performance flaws are, and and you can see that, and then maybe build a stronger intuition of like what's missing in Korean versus what's missing in Russian. It might be that you have different languages that have different kinds of things missing. It might suggest that you need to do different types of adjustments to your training data.
David Bau: You know, it seems like, there's like, you're you're very, very close to this. Yeah, interesting. Yeah, that's a good point. Yeah, at some point I was like discouraged discouraging myself from doing that just because I was like.
David Bau: will that be like an interesting story? If I just tell you well.
David Bau: for some languages the model fails because it fails, in some, it fails in this step, one in some it fails in step 2, and then you're like, well, that was obvious. But then, yeah, I think the point is like, maybe for some languages
David Bau: he feels more in like one or the other. Maybe you could try and fix one of the 2. Yeah. Yeah. And and I'm I'm very interested in this. You know, some of the students are interested in the model editing techniques of oh, let's let's like, do a little bit more training to improve the models knowledge or to change the models, knowledge, or something like this. And it seems like.
David Bau: there's this, there's always this old question of like.
David Bau: Oh, you, you know you have this insight about the mechanism. So what? You know, what's who cares? Right? Okay. Now, you're smarter.
David Bau: Okay. But but it seems like you've got this really nice setting where you have relatively low actual recall performance and a bunch of languages. People do care about this kind of thing right? And and yet we don't know like what kind of fine tuning, what kind of adjustment you would do to fix it, and I think that there's this opportunity to a very cool paper, saying, Oh, there's 2 different types of mitigations depending on the situation that you're in
David Bau: and, you know, develop like a little experience of like if you fine tune in this way, it mitigates problem B, you fine tune. In that way it mitigates problem A, don't don't spend your effort on the wrong thing after you've debugged it, you can see what's going wrong. Right? Yeah. Yeah. I
David Bau: agree. Yeah, very interesting would be good if you could hard quote it.
David Bau: But
David Bau: or can you work? Tweak the Lm head to kind of like, do this thing that you're suggesting?
David Bau: Oh, like, come up with a new architecture.
David Bau: Yeah, I mean, maybe the fact is, there, I mean, the pathway is there you're saying so, suggesting that it's not strongly activated for all cases. That's why it's not getting translated into the performance.
David Bau: Yeah, I. So I'm not sure what the right mitigation would be for all these things. I think that would like sort of suggesting, you know, could be future future research. I have no idea what would work. Yeah, yeah, right? I'm just looking now at the time.
David Bau: Yeah. So I'm just gonna say one sentence about the other paper. And, oh, I want to hear about the other paper. Yeah. So this paper. It was a collaboration with some philosophers. And
David Bau: we interpret, show some slides. Yeah, right? So I can give you a bit of the motivation. So
David Bau: we were like, well, we talk a lot about a lot of times like the model knows this. The model doesn't know this. And but what do we really mean and like, when does the model really know a fact or not? And so like in the previous work and in a lot of work. If the model can correctly predict that was born in, then we say, Okay, the model knows. But then, yeah. But then people were like, well, but if I say it in a different way, the model doesn't seem to know
David Bau: to really know the fact, because it doesn't say correctly. And then, more recently people were like, well, but what about related facts? If someone really knew that Dante was born in Florence, it should also know the nationality right? And so in this paper, we
David Bau: look at the question of like, Yeah, what are the sufficient and necessary conditions that we should set to say, Okay, the model knows this fact. And
David Bau: yeah, I can show 3 more size and we resort to philosophers and to philosophy, because there's philosophers have like a
David Bau: big history of thinking about these questions, thinking about beliefs and thinking about knowledge for humans and what what it means to say, someone knows and we focus only like propositional knowledge. So knowledge of the form X subject knows a proposition. There are, like other types of knowledge, such as like knowing how to ride a bike. But this is just like about propositional knowledge.
David Bau: And yeah, so to really answer this question the
David Bau: as a design answer is like, well, it depends. Right. Philosophers have, like different views on how we would define this, and what we do in the paper is, we look at 5 of the most popular definitions of knowledge in humans. And we interpret them in the context of language models.
David Bau: And we draw connections to current research in Nlp and knowledge. And
David Bau: and we also prefer. And after doing that and discussing like which definition would imply what and how we how we would implement them
David Bau: in terms of the implementations. I I quickly gonna say that it depends on the current methodologies like in the paper. We quickly like touch upon some methodologies for some of these definitions. But really some definitions do not yet have the necessary methodologies. And yeah, I could talk for like
David Bau: too long to describe. Each of the methodology. Sorry the definitions. But we can. Why don't you flip through the rest of the slides? And then we yeah, okay, you don't have to explain them all, just just flip through them visually. I think one thing that is interesting is I get in the sense of that the the 1st definition from Sarwell is like the the more basic one. So you said that you know a fact, if
David Bau: or like a proposition, if the proposition is true, and you believe it so. I believe that the capital of Germany is Hamburg. But it's not so. I cannot be said to know that the capital of Germany. But if I believe that the capital of Germany is Berlin.
David Bau: and it actually is, then I know the capital of Germany. In this definition. There's like this belief class, because our world bases this definition on a really strong definition of belief like, you have to have a consistent system of belief. You cannot just like randomly believe that this road is called.
David Bau: I don't know, coming to, because it's like you would have to have other related beliefs to that to to really believe that.
David Bau: But then the other like really popular strand, is coming from Nosick. Which is like, okay, let's not put so much
David Bau: like a
David Bau: let's not use, I guess, so strong. Definition of belief. But rather let's add, the condition of having a justification.
David Bau: So
David Bau: you believe that the capital of Germany is Berlin, and you believe that because you saw it in a textbook because you went to Germany and learned that. And you have a justification to believe in that. And that is also true.
David Bau: But then and yeah, I'll add this. We then then
David Bau: that that definition has, like some problems that philosophers like to think about edge cases. So it's like a bit of an edge case. So we have John, who's looking in the field and say, and then this is a sheep, and then he's like, Oh.
David Bau: I know there is a sheep in the field, because and he has a justification. He's like, well, yeah, I've seen a lot of ships in my life. This looks exactly like one. Why, it's fluffy. I know it's a sheep.
David Bau: but then it turns out that it's not really a sheep. It's a dog, so
David Bau: we wouldn't be able to say that he knows there's a ship in the field, because the proposition is false is false.
David Bau: but in reality there is a shape in the field. This is obscure from his view, and so given No. 6 definition.
David Bau: we would say that John knows there is a chip in the field because he believes it. He has a justification for it.
David Bau: and the proposition is true.
David Bau: But it feels a bit incorrect, right? Because he has a justification
David Bau: not connected to like the real proposition. And that's where, like other definitions, come in.
David Bau: where, in in this one, Sebsky tries to solve these cases by saying that knowledge only comes as a result of people acting Brutus Lee.
David Bau: So you have. You only can know things that are a result of
David Bau: virtuous acts. So acts that would have the truth on like it's
David Bau: like most priority. And this is like one definition where it's not really like a hundred percent. Clear how you would do that for language models.
David Bau: And I think it would be interesting that if if you could like, have some sort of understanding of what's happening to be able to say, okay, in this case the model was really like, I don't know, recalling all the facts about this entity and actually using that and doing the mechanisms
David Bau: truthfully rather than just okay. I don't really know anything about this. What can I connect it to it and try to set some separations?
David Bau: Yeah, then, Austin is like, pretty similar to the 1st definition from Starwell. But instead of like being
David Bau: so strong about the belief definition, he takes a more pragmatic view. And it's like, okay, you don't have to be consistent with everything that is related, but just with things that are relevant in this context. So things that facilitate you in this context.
David Bau: And then, lastly, something that goes like.
David Bau: so in a completely different direction. Williamson's points out.
David Bau: But people have been trying for a long time to define conditions or other
David Bau: other things that could accumulate to knowledge. So how can we like dissect knowledge? To really be able to say that we know so
David Bau: belief, plus justification, plus other facts, and so on. And then we'll say, but what if knowledge is the atomic thing? What if we cannot really separate knowledge. But we can understand all the other concepts in terms of knowledge. So basically, his view is like, we would say that
David Bau: you would believe something if you act as if you know it. So belief would be defined in terms of knowledge. How is it different from the 4th one? I mean, it's fascinating correct prediction. Yeah. So the the 4th one is more like.
David Bau: it's more like the 1st one in the sense that it's it's really having, like a consistent system of really, because if if it facilitates correct predictions, it means that you believe in this.
David Bau: it entails that other beliefs that are related are also useful.
David Bau: What's like the the 5th one is like,
David Bau: We cannot talk about beliefs as a
David Bau: ingredient of knowledge, but rather knowledge is an ingredient of all the other things.
David Bau: And and that one is kind of
David Bau: again. It's like a bit useless in the current state of things. But it could be
David Bau: potentially useful if you're able, in a way, to define like a knowledge box, kind of thing or like, where is knowledge in my model. Or, for example, when
David Bau: there was like this
David Bau: recent paper that they train pros to extract knowledge from internal representations. And so they say, Okay, I'm able to extract this knowledge, but the model never generates it.
David Bau: Should I say that the model knows that or not, if the model never generates a fact, but I am able to get it from the representations and
David Bau: put those sort of questions. Then you would pick like
David Bau: a bit of a definition like the 5th one and saying, Okay, if it's there, then the model knows it, and that is, that also becomes relevant when you think about
David Bau: tasks like unlearning. If I train my model to forget a fact, if a model doesn't generate it anymore.
David Bau: does it not know it? But if it's still in there, like in the early layers.
David Bau: it kind of feels that it still knows the fact. If I just remove these later layers that are just subtracting it from it
David Bau: anyways. Then, in the paper we also have a survey where we ask philosophers and computer scientists if we can even say that language models? No, if knowing it's really a human thing or not. And yeah, we can comment on that other point. So we don't spend more over time. So I'm just gonna
David Bau: what was it? What the paper? Well, the conclusion is this nice figure? Yeah. So it's like.
David Bau: do they? It would be nice to take a bit of inspiration from these definitions, from coming from philosophy to
David Bau: evaluate knowledge, maybe in a more how to say like
David Bau: complete way, and and to really talk about knowledge when we want to say something stronger than just outputting something. Or maybe you want to do that. But
David Bau: being specific about that, we just did that in the previous paper. So, yeah,
David Bau: so, thank you. Well, thank you.
Natalie Shapira: Hi! Hi! Constanza!
Natalie Shapira: Can you hear me?
David Bau: Thank you.
Natalie Shapira: Oh!
Natalie Shapira: Can can you hear me.
David Bau: If oh!
Natalie Shapira: Oh, thank you. Thank you. Great talk. I really enjoyed it. I have a question about your last work that you presented. If you can repeat the name of the paper if you already published it.
David Bau: Yes, the names of the papers are just here.
Natalie Shapira: Defining knowledge. Thank you. I found it.
Natalie Shapira: Thank you.
David Bau: Really, we should just memorize Constanza's name. It's a bit of a long name.
David Bau: But yeah, you think this probing way is the way to
David Bau: find knowledge in language models. Or is there a better way?
David Bau: That's an interesting question. So I think that
David Bau: there's always like this problem of like, if you can extract it, is it really there, or did you learn or like? Did your problem learn it? Kind of thing like
David Bau: it's not there. How did the problem because you're giving data to the problem to learn that to learn to extract right?
David Bau: And so, in a way, you would have to prove that the model would use bad.
David Bau: That connection, that, like connections between the population, the subject in other contexts, maybe it never generates it. But it's still able to use it, say, for a to help fact, or it's still able to use it for another question, and then you would, and then I would say, yeah, the model knows that like, it's using it for other things. So you're suggesting we do perturbation after we figure out some like knowledge concepts through these probes.
David Bau: But it's it's still pretty. Yeah, it's it is kind of like a diffuse.
David Bau: challenging setup right to how to do it and how to like, be really sure that it's there. And then the model is using it.
David Bau: How distributed is knowledge in the bubbles?
David Bau: Yeah, that's something that we were discussing the other day.
David Bau: we see that it is pretty
David Bau: distributed like this. Results don't really show it because we're using like, same type of templates where we always show the entity directly.
David Bau: But but then, if you have like, say.
David Bau: and we kind of like, see it sometimes like, if if you look at, say, like the
David Bau: localization plus the casual analysis for each of the languages. So you localize across Mlp output vectors and attention at outputs.
David Bau: Yeah. So I, what I wanted to say was like really simple, that for some land so other languages than English.
David Bau: the tokenization of the of the subjects tend to be much longer, just because it's like in a different language, and so English sometimes will be one token or a lot of times. It will be one token, while in other languages it's like 3, 5. And and so
David Bau: I think that that's related to the fact that we see in the Council analysis for different languages, that the Mlps early side kind of like moves a little bit
David Bau: in some languages like a bit longer than in others, and and there are other works that look at to have relations.
David Bau: and then, when you have like
David Bau: the place of birth of I don't know.
David Bau: The spouse of Michelle Obama was
David Bau: then the model has to connect, like all the Michelle Obama, get to like a representation of Barack Obama.
David Bau: and the model actually does that. And then that happens like later in the layers. And so the model needs to have still that knowledge in later Mlps
David Bau: to be able to connect it and still predict it. So all that to say that yes, it is pretty distributed along like different layers, because the model may need it at different stages, depending on how the input is
David Bau: it?
David Bau: I hadn't thought about that. The the multilingual setting is really excellent setting for multi token rollout.
David Bau: and we've been trying to understand.
David Bau: You know, the mechanisms multi token output.
David Bau: Oh, wow! I think it's pretty interesting, because it's very controllable switch languages.
David Bau: No, I want to ask, like you mentioned, like different aspects of knowledge, like like, have you tried to like? Take a look at?
David Bau: Does different aspects of knowledge processed in different part of the models.
David Bau: like maybe some part of model, are more about like, whether the model believe is true or not, or and then the other part. May
David Bau: contains more information about like, what's the evidence of it and how it can be can can use for like, make
David Bau: correct the predictions. Yeah. I I haven't done it in.
David Bau: I think it's it's it's an interesting point to like. Try to understand
David Bau: whether the model kind of like encodes a little bit its own uncertainty about the
David Bau: the knowledge. And there there was a paper from Ferrando that where they they try to see if they find, like an
David Bau: aspiracy encoder direction that would encode
David Bau: the whether I know whether the model knows
David Bau: at this entity or not. Like. If I recognize it.
David Bau: then I'm less than certain about whatever I predict of this entity, and they they show that they're able to find this direction and play around with it a little bit. And I think that goes in that direction, whether the model
David Bau: has a beliefs, and really, with quote unquote, because
David Bau: yeah, I'm not sure what we would say. A building the model is.
David Bau: it's it starts with, do I know this entity? And then something something?
David Bau: Thank you. Can can you always check back and knowledge, item, to a parameter or set of parameters in the model? Or are there facts that only form
David Bau: in the residual stream by adding a bunch of things together, or something weird like that.
David Bau: Yeah, I'm not sure. I think that's a that's a good question.
David Bau: Oh, yeah, I just wanted to to bring up about like
David Bau: I guess one type of knowledge that I was wondering your thoughts on is like, there's been some there's like a paper about like
David Bau: I think it was. I'm not sure if it was just in linguistics, but like.
David Bau: if you ask like a language model or like a human, to like.
David Bau: do some sort of like linguistic classification task. The model is like not very good at outputting, you know, like the right answer or whatever, but actually like, when you look at, it's like
David Bau: probabilities. For like sentences like that, it clearly does encode that like grammatical concept. If that makes sense
David Bau: so like. If I ask someone who's never learned about parts of speech, or whatever like they would intuitively like, because they can speak the language. They know they obviously have that knowledge, but they can't actually like articulate or output it. So I was wondering, like, how that fits into like the sort of yeah. That's a yeah. That's an interesting setup. It's like we were discussing that the other day with David on like.
David Bau: yeah, the information is there. But you haven't really trained your model to decode that and it in a way, it's like
David Bau: what I feel like happens in this like later stage. The model is decoding to each language, and and to be able to perform that you would have to include in your data
David Bau: a way for the model to learn to decode that and train, and, like edit some of the later layers to be able to decode it into natural language.
David Bau: Yeah, like. In in the analogy like asking
David Bau: a person that doesn't know about grammar.
David Bau: What is the subject here? They're like, okay, I know how to speak. But what is the subject? Yeah, yeah. I wonder if that would be more like a knowledge of how to do something rather than knowledge of.
David Bau: yeah, yeah, exactly. Yeah. It's definitely like a a know-how. Right? Like, right?
David Bau: Yeah, it's something like.
David Bau: believe anything. But it facilitates predictions. Yeah. But but that's it. The the distinction is that
David Bau: that falls into like another category of definitions of knowledge, which is knowing how, which is not the ones that we look in. In this last paper. It looks like we have an online question from Prana
David Bau: front up. You want it.
Pranav Misra: Yeah. So I was thinking about these relu, like highway connection.
Pranav Misra: thought about like designing models where you have these highway, like connections, from early layers to later layers.
Pranav Misra: and these redundant connections between different layers. And then, if that changes where the different language fact, representations arise.
David Bau: So you're saying that connections between different layers
David Bau: could change the presentation of language.
Pranav Misra: Yes, yes. So, for example, like in vision models. Sometimes people have these either connections going from early layers to later layers, and that
Pranav Misra: overcomes the vanishing gradient problem for the early layers. So they also end up like capturing, like very like important features, for, like final stage classification. So the classification of from like early layers also improves if you want to decode the object category. And I was wondering, like, if for these concepts across different the different language representation that you're getting
Pranav Misra: might get fine tuned if you allow for these
Pranav Misra: highway network highway connections to exist.
David Bau: Yeah, like, I'm not sure what exactly would entail. As like, yeah, I I get.
David Bau: I've mostly just like
David Bau: train and play around with with the transformer. But but that's how it point. Like, maybe you could train the model to like, really ignore the language and just like look at it at the end, because you will be able to still look at them
David Bau: at the beginning. By having like this direct connection. And that could be. That could be an option. So
David Bau: thanks very much, Christine. Consenza. Yeah.
David Bau: So let's let's take a little break. I think we have another, you know. Visiting Speaker soon. So, but thanks very much.
David Bau: Thank you.
David Bau: And you can, you can ask them more about this work. I you know
David Bau: I sorry we didn't have enough time to talk.