Sarti, Gabriele: Yep.
David Bau: Okay.
Sarti, Gabriele: Alright.
David Bau: And then, yeah, do you have the ability to use share screen from where you are? Yep.
David Bau: Great.
Sarti, Gabriele: Can you see the screen?
David Bau: I can see, yeah.
Sarti, Gabriele: So this week.
David Bau: Yeah, we're… so this week is a guest lecture from Gabriel.
David Bau: Because the topic is on input attribution.
David Bau: Where Gabriel's one of the world's experts on how to do this with large models, with LLMs, and so on.
David Bau: And so, so since, since he's here this semester, I asked him to, to give, the lecture on the topic here today. And it's a little, it's a little too bad that it's all
David Bau: Remote, because as you know, we… we… we try to…
David Bau: you know, make sure that everybody gets a chance to ask their questions and interact. So, I want you to not do the normal Zoom thing and… and zone out, during the lecture. Maybe, Gabriel can make sure that there's, like, you know, times in the… in the talk to,
David Bau: To, to, to, to get, to get feedback or questions from people answered.
David Bau: Okay, yeah, Jasmine, superstar, right? Yes, Jasmine, this is Gabriel from the reading.
Sarti, Gabriele: You're too kind, you're too kind.
Sarti, Gabriele: Excellent.
Sarti, Gabriele: So thanks again for having me, David, it's really great. So yeah, so today's presentation that I prepared is called Attribution, Tracing Influence to Inputs and Model Components, and I will try to walk you through,
Sarti, Gabriele: Some of the most recent, some background first, and then some of the most recent, let's say, interesting stuff that is going on in this… in this area of interpretability.
Sarti, Gabriele: Yeah. So, the first… the first thing is that, the way that we can conceptualize, attribution is…
Sarti, Gabriele: by looking at what the models are using to… to do predictions, right? So, in… here I argue that there are basically two pathways to prediction. The first one is, the inputs, so the models receive, as you know, in-context information, and,
Sarti, Gabriele: These methods that we use to trace importance back to the inputs are what we normally refer to input or feature attribution.
Sarti, Gabriele: So this is what we call also this in-context learning, right, in language models. And the other dimension that is complementary is whatever the models learned from training, right? So in this case, we have the learned weights.
Sarti, Gabriele: And we can still do some attribution. It's a bit, related to causal mediation that you saw from the previous lecture, so you're gonna see, here, too, we have a way to do, basically component attribution, so understanding which components are responsible for a specific prediction.
Sarti, Gabriele: And the plus one is, there is a second-order effect, of course, so…
Sarti, Gabriele: whatever learned weights the model has are, are derived from training data, right? So, a big mission in, in attribution would also be to ideally trace back whatever importance, from, like, from the prediction to back to the training data, right? So there are some methods that are responsible for that.
Sarti, Gabriele: I'm not really covering that part today, but just for your knowledge, these methods are kind of very similar to the others that we're going to discuss, and these are called training data attribution, or simply data attribution.
Sarti, Gabriele: So the overall attributional interpretability is asking which elements motivate model predictions, right? That's the… that's a big question.
Sarti, Gabriele: So…
Sarti, Gabriele: This is kind of a formalized way to think about input attribution, so if you have a trained model and an input that the model receives, the input attribution method is just a map that, given an input.
Sarti, Gabriele: in this case of dimension D, it will produce a set of scores, alpha 1 to alpha D, that are telling you how much, how relevant is the
Sarti, Gabriele: ith dimension of this input, for the prediction.
Sarti, Gabriele: So… What counts as relevance here is quite vague, and it's left vague on purpose, in a sense, because
Sarti, Gabriele: it's very hard to formalize what does it mean for something to be salient or important towards prediction, right? So we're gonna maybe discuss this a bit more in the next few slides, but just for you to know, like, saliency is a bit of a fuzzy concept overall.
Sarti, Gabriele: So to quantify importance, if we had simple linear models, right, like linear regression models, this would be a very
Sarti, Gabriele: trivial things to do. Like, we could just look at the coefficients that are learned, so whatever the weights here that would be, matched to the matrix of the inputs would be our importance scores, right?
Sarti, Gabriele: However, for deeper models, this is not straightforward, and the reason for that is that… is the presence of nonlinearities, right? So nonlinearities mess up
Sarti, Gabriele: This kind of contributions of different inputs, and especially when they are chained, like, in a deep neural network, the whole influence is… becomes very messy.
Sarti, Gabriele: So, generally, the way that we go about estimating this importance is through some approximations. For example, approximating some specific operations linearly.
Sarti, Gabriele: Or perturbations, so trying to kind of, like, try to estimate how important something is by just ablating it or modifying it slightly.
Sarti, Gabriele: So, let's have the most… let's have a look at the most basic setup for, for attribution, which is occlusion. So, you know, in the case of occlusion, let's say that we have our language model here, let's say this is, like, our GPT or our Llama, and we have
Sarti, Gabriele: an input that is a simple string, right? For example, welcome back, ladies end.
Sarti, Gabriele: That, as you know, gets tokenized and embedded before being fed through the model.
Sarti, Gabriele: So, you see this is our, of dimension D, which is the dimension of the embedding, time S.
Sarti, Gabriele: The dimension of the sequence.
Sarti, Gabriele: And the output of the model is this distribution of probabilities over the vocabulary. So in this case, we find that the model is predicting gentlemen as the most likely next token.
Sarti, Gabriele: So the occlusion case is very simply, let's ablate a single token, for example, ladies.
Sarti, Gabriele: So here we're gonna have a different embedding for the ablated token, and let's get an output for the perturbed input, right? So in this case, the probabilities will change.
Sarti, Gabriele: And an idea would be, okay, the way that we associate an importance to the token ladies is by looking at the top prediction.
Sarti, Gabriele: and looking at how big of a drop this is, right? In this case, the drop is huge, so it means that ladies is probably very important towards that.
Sarti, Gabriele: Probably you can see already that this is overall a bit problematic as a procedure. I'm curious if someone already has a hint of, like, what could be the problems in doing that, or…
David Bau: So, can I ask? Because I'm not the expert here, is that right? Yep. So when you… so when you include it, what do you… what do you… what do you put there?
Sarti, Gabriele: Yeah, that's one of the problems, exactly. So, there is no good answer to that, actually. So, this I mentioned in the next slide, but one of the issues with perturbation, and in general with this kind of, like, occlusion.
Sarti, Gabriele: Which is related to one of the questions that we received, also for integrated gradient, what would be a good baseline for language, right? The problem is that,
Sarti, Gabriele: perturbations, let me go to… yeah, can produce OD behaviors, and this would result in unfaithful explanation in the case in which you're replacing that with something that is not, realistic, right? So, for example, if we have,
Sarti, Gabriele: let's say, for encoder language models like BERT, we have these kind of mask tokens, right?
Sarti, Gabriele: Which are kinda interesting in this setting, because we could just replace things by masks, and the model is trained with masks, right? So it's able to handle that. But for GPTs, we don't have such things, right? The model is just predicting left to right, so it doesn't need masking.
Sarti, Gabriele: Yeah, so the coder LLMs don't, right?
Sarti, Gabriele: And one common approach is to sample replacements randomly and aggregate over multiple replacements, right?
Sarti, Gabriele: But you can imagine that this scales very poorly, right? So if you… if I have to do 100 replacements with random tokens, and just measure how big of an impact it is on average, this becomes very expensive. I just attributed a single token in this case, right?
Sarti, Gabriele: So imagine if I was to attribute the full sentence then, right? So this is very expensive, and it scales poorly to long inputs.
Sarti, Gabriele: Yeah So, an alternative, a natural alternative in, in,
Sarti, Gabriele: in neural networks is to use gradients as some sort of attribution. So the models, as you all know, are trained with gradient descent, so this gradient information is naturally employed during training.
Sarti, Gabriele: And we can repurpose that to try to get a sense of the importance of components, at inference time.
Sarti, Gabriele: So let's say an example. Here, again, we have exactly our same setup as before. We get the prediction gentleman.
Sarti, Gabriele: But then, the way that we will go about that is we pick a specific target in a target function, let's say. In this case, the target could be the probability of the top most likely token, like gentlemen.
Sarti, Gabriele: And we can take the gradient with respect to that back to the input embeddings of the model.
Sarti, Gabriele: So, note that here, these gradients normally are taken with respect to a loss, right? So the gradient with respect to a loss tells you, how do I minimize this loss?
Sarti, Gabriele: So which weights should I change to minimize this loss? In this case, instead, the gradient with respect to probabilities is telling us
Sarti, Gabriele: In a sense, how sensitive is this final output of the model, so the final probability for gentlemen, to little perturbations of, in this case, if we focus on input embeddings, of these embeddings, right?
Sarti, Gabriele: So the final outcome of this procedure is gradient vectors that have the same exact dimension as the input embeddings, and that basically express each dimension of the input embedding. How important is that?
Sarti, Gabriele: Towards the, prediction of the, of the final, output.
Sarti, Gabriele: So, normally, this is not very useful, right? One score per dimension is not very useful, so what we do is normally to aggregate those at the token level to get a single score per word.
Sarti, Gabriele: So in this case, we could find, for example, that the gradients for ladies and are very high.
Sarti, Gabriele: And if we aggregate these, for example, by taking the vector norm of each one of these gradient vectors, we would get a high attribution score for ladies N.
Sarti, Gabriele: Which is intuitive, right? If we perturb lady's end, the model probably wouldn't predict gentlemen.
Sarti, Gabriele: So this relies on this kind of approximation.
Sarti, Gabriele: Yeah. Is everything clear for the gradient attribution part?
David Bau: You guys are so… you guys are all camera off. Oh, I'm camera off, too.
Sarti, Gabriele: Yeah, exactly. I'm kind of like.
David Bau: It's impossible for Gabriel to tell what's going on.
Sarti, Gabriele: Yeah, I don't know.
David Bau: So.
David Bau: Yeah, so,
David Bau: So you just take a, you just take a vector, size. Is that, is that typically really what people do, or do they do other things other than…
Sarti, Gabriele: this is another thing, that there is no consensus, so there's plenty of ways that people try to do this kind of aggregation. So, for example, the L2,
Sarti, Gabriele: norm is probably the most natural way to do this, but other people took the sum of the gradients, or the L1 norm is,
Sarti, Gabriele: Like, these are all commonly used, and there's no, like, one-size-fits-all kind of.
David Bau: Okay.
Sarti, Gabriele: Yep.
Nikhil Prakash: Wait, I…
Sarti, Gabriele: Yo.
Sarti, Gabriele: Sorry, yep.
Nikhil Prakash: So, what's the general sample size people use? I'm assuming they don't compute the gradient on one example and show this graph, right? So, what is the general sample size that people use?
Sarti, Gabriele: I mean, the idea here is that the gradient attribution is really local, right? So, meaning, for a given example, you're gonna get your scores out in this case, right? If you want to draw some,
Sarti, Gabriele: you know, hypothesis from the kind of information that you get out of this, definitely you want to have some data set, and to be able to tag, you know, the kind of expected behaviors that you would like to see, and see whether the gradient attribution retrieves something that matches your intuition, right? So, I agree with you that on a single example, this doesn't tell you much.
David Bau: But that's… but that's actually one of the nice things, so isn't… or do you think that's true, right? That there's one of the nice things about…
David Bau: input attribution is that it is about, you know, you can analyze single examples. It's trying to get you an answer about single examples, right?
Sarti, Gabriele: Yeah, yeah, yeah, exactly. So, it's really a local thing, right? It doesn't, like, it doesn't give you any intuition about the behavior of the model globally.
Sarti, Gabriele: but rather on that specific example. If you want to get this global intuition about model behavior, you probably have to repeat the thing on a dataset, right, and see whether the trend that you observe on that example holds
Sarti, Gabriele: In general, huh.
Nikhil Prakash: Okay.
Sarti, Gabriele: Yep.
Nikhil Prakash: Yeah, okay, thanks.
Sarti, Gabriele: So yeah, so this all relates to the fact that the devil is in the details for attribution methods, and we got some questions here, so what.
David Bau: Oh yeah, we'll have them ask the questions.
David Bau: Courtney had a question. Armita had a question. What were your questions?
Armita Kazeminajafabadi: So for, we had this problem in our, like, my research project that we wanted to build adversarial input examples, and, many of them are gradient-based, and our input
Armita Kazeminajafabadi: Our data was, like, discrete, so… Mmm… changing them.
Armita Kazeminajafabadi: in a way that, like, increases the gradients, sometimes wasn't, so meaningful, because we had few perturbation options. But in this con- context, I was wondering if
Armita Kazeminajafabadi: The same issue exists.
Sarti, Gabriele: Yeah, so in a sense, as you saw in the example before, the way that we move from, like, the continuous space that the model operates on to the discrete space of, for example, the vocabulary of the model is by just aggregating, right, whatever we get in the continuous space at the token level. I agree that in your case of, like.
Sarti, Gabriele: trying to optimize these embeddings, maybe, like, in a way that they still map onto words, probably you would have to have, like, some sort of projection to the nearest neighbor procedure, right? That can be quite noisy, in the case of…
David Bau: Is that what you did in your experiment, Armita? Did you do a nearest neighbor thing?
Armita Kazeminajafabadi: No, no, we, didn't do gradient paste.
David Bau: Oh, you, you, you moved to something different. Okay, cool.
Sarti, Gabriele: Yeah.
David Bau: Huh.
David Bau: Is Courtney here? Does Courtney have a question?
Courtney Maynard: Yeah, hi, my question is about, whether or not you can tell when, like, a specific token, or in the Mirage example, a specific document is actually negatively contributing to a prediction, or, like, taking away from moving away from the correct, answer, or if the attributions are only positive.
Sarti, Gabriele: Yeah, so in the gradient case, definitely you can do that. Also, in the occlusion case, so overall, let's say in the occlusion case, you would have the simple
Sarti, Gabriele: The simple setup where, this estimated importance could either increase or decrease the probability, so you could see this as a positive or negative contribution.
Sarti, Gabriele: While in the gradient case, gradients will be signed in these vectors, so depending on the kind of attribution that you do, and now I'm kind of getting a bit ahead on what I'm saying here, depending on the aggregation, this information might get lost. For example, if you take just the L2 norm of the vector, then you're effectively just getting a positive value
Sarti, Gabriele: that is overall how much it contributes, abstracting away positive and negative contribution. But if you sum, for example, you would get, something that is… that can also be negative, right? If all the dimensions are kind of negative.
David Bau: So, are there other ways of aggregating that you wouldn't lose it?
Sarti, Gabriele: I would say probably the sum is the most common here. You could also get, like, something like, you know, just this kind of heuristic, I guess, like, the maximal dimension within the embedding. If it's a negative dimension, then you would say that it's a negative contribution.
Sarti, Gabriele: There has been…
David Bau: There's something we saw in the, in BeanCam's TCAV paper.
Sarti, Gabriele: Where, once she had a gradient, she dot producted it.
David Bau: with, you know, some particular vector of interest, right? You might dot product it with a class, or dot product it with, like, a token or something like that. That's true, that's true. Do people ever do that, or…
Sarti, Gabriele: Maybe… so I don't have a slide for that, but I think it's interesting to relate it to the dot product, right? It's interesting to think of when we talk about gradient attribution.
Sarti, Gabriele: to think of the gradient vectors per se versus the gradient times whatever input they would be applied to, right? So these are two common attribution methods, like just gradient, row gradient, and gradient times input.
Sarti, Gabriele: And, like, the gradient per se tells you the sensitivity, kind of, of the inputs to the… like, of the prediction to the inputs, but the moment you multiply them, you actually get some sort of scaling by… by actually considering what these gradients would be applied to, right?
Sarti, Gabriele: you might have very high gradients just because the dimension that they would be applied to is very small, right? So yeah, so definitely this relates to what you're saying, David. I think, depending on what you're looking for, it might make sense to consider gradients in relation to their inputs, and not just by themselves, right?
David Bau: Yeah, I think so, and I think that might give you a sign, also, that you could have a positive one or a negative one in that case.
Sarti, Gabriele: Yeah, exactly.
David Bau: Pretty interesting.
David Bau: Question from Courtney. Okay, let's keep on going, I don't want to slow you down too much.
Sarti, Gabriele: No worries, Amish.
Sarti, Gabriele: Yeah, so I just want to highlight here, this, like, plenty of people work on this, and there's no established norms, basically. So every new paper that does attribution does it in a different way.
Sarti, Gabriele: And there's no consensus of, like, oh, this always works the best, you know? So I think there's a lot to be… to be explored still in this area.
Sarti, Gabriele: So, one of the readings that you had was integrated gradients, and I took this image, which was quite nice, which was from this blog post that says integrated gradients is a decent attribution method, which I found funny.
Sarti, Gabriele: And yeah, the intuition is what you see here in the image, is you have a starting point here.
Sarti, Gabriele: you have an endpoint, which is gonna be your baseline, and effectively, you're taking steps, right? Like, ideally, this would be an integral, but actually, you're probably approximating this by taking steps alongside this… this straight line.
Sarti, Gabriele: So this is the activation space, and then here you would have two input features. In reality, we have many more, and what you're doing is just by adding up the contributions of each one of the two features, right? So this is a nice way of visualizing what's happening here.
Sarti, Gabriele: So…
Sarti, Gabriele: Integrated gradient is quite robust, because of this property of, like, you know, considering contributions alongside this path.
Sarti, Gabriele: In practice, though, to get a good approximation, probably it's quite expensive to run, meaning that you will need many approximation steps alongside this line.
Sarti, Gabriele: And as many of you noted, the lack of a good baseline is often a problem, actually. So, especially in NLP, people have tried the zero vector, which, again, doesn't mean anything in the NLP world. They tried to use the kind of, like, out-of-vocabulary token, or the mask token. I think, overall, the best idea is really to do random sampling.
Sarti, Gabriele: And just averaging out, but this, again, is extremely expensive, so… No consensus there.
Sarti, Gabriele: So there were some interesting variants that were proposed specific to NLP. I have an image here. So the idea here was, instead of going for a straight path, let's instead
Sarti, Gabriele: do this kind of clumping to the nearest neighbor, kind of like we were saying before, and let's find the path that passes through existing tokens in the model vocabulary. So let's take the integral with respect to this path.
Sarti, Gabriele: it kind of works, but it's not that much better, so again, there's no, like, one-size-fits-all for this kind of method.
Sarti, Gabriele: And I also wanted to highlight this smooth grad, which is another different approach, which is just introducing noise to the gradient estimation, which is some Gaussian, for example, noise. This also improves robustness, so this points to the fact that actually what matters is the robustness in this kind of evaluations, right?
Sarti, Gabriele: Because the gradients are noisy, overall.
Sarti, Gabriele: Yeah, so there were several questions regarding the implementation invariance properties here, whether they always hold, whether they're good, and whether the axioms matter in practice.
Sarti, Gabriele: I think overall, my perspective is that they don't really matter, in the sense that I feel like this is really the
Sarti, Gabriele: you know, best case scenario, although integrated gradients is not really the best, nor the most efficient method out there, so…
Sarti, Gabriele: probably, especially this implementation invariance, is kind of like a, you know, a moonshot kind of goal of, like, having two models that are exactly identical, except for, you know, the implementation, but they behave exactly identically. I think this is not very realistic.
Sarti, Gabriele: Yeah, so there was someone that was mentioning, I don't remember if it was Claire, maybe, that was mentioning in their example about the, like, what if we have this classification that is based on the background versus looking at the subject in the photo, and then the two behaviors are
Sarti, Gabriele: the same, but for different reasons, right? But then the… maybe you can correct me if I said it wrong.
Claire Schlesinger: No, that's right.
Sarti, Gabriele: Right, right.
Sarti, Gabriele: Yeah, so my take on that is that if indeed this was the case, like, implementation environments would apply to all sorts of examples, right? So you should be able to craft an example if the heuristics are different, such that the two networks would have different behaviors.
Sarti, Gabriele: And if the two networks have different behaviors, then implementation invariance doesn't apply, right? So then,
Sarti, Gabriele: Yeah, I feel like, again, what you're pointing at is the fact that for a given set of examples, this might be the case, that the two methods, like, the two networks are behaving the same way, but in practice, this is rarely the case. If they have different heuristics, you should be able to find examples that lead them to behave.
Sarti, Gabriele: D. Friendly, right?
Sarti, Gabriele: I don't know if you agree with that.
Sarti, Gabriele: Alright,
Sarti, Gabriele: So, one thing that I wanted to mention is that my… my opinion on the most promising approaches that are currently used in this space are methods that are basically trying to tweak this kind of gradient propagation.
Sarti, Gabriele: Without making it too inefficient, so they don't take this kind of approach of, like, taking steps or, you know, multiple prediction steps, but they are just crafting some either custom propagation rule, like in the case of
Sarti, Gabriele: layer-wise relevance propagation, or accounting for transformer quirks. So this… this dream method, for example, is something that was recently proposed.
Sarti, Gabriele: And one of the things that they were proposing is, can we compensate for, for this kind of behavior, where if you remove something in the… from the input, then the softmax would reallocate the importance, and then this will lead to different behavior, right? So I think these are very interesting, especially because their cost in the end is the same as regular gradient-based attribution.
Sarti, Gabriele: So, much more efficient than integrated gradients, so probably
Sarti, Gabriele: Can scale to these kind of applications where we use language models, right?
Sarti, Gabriele: Alright.
Sarti, Gabriele: So, one area that has been explored to some degree, but, yeah, with still with mixed success, is instead the idea of, like.
Sarti, Gabriele: Just looking at model internals. So, for now, we always relied on prediction, right? Either we take the gradient with respect to the prediction, or we take, we look at the difference in prediction.
Sarti, Gabriele: When, I don't know, ablating, occluding components. But here, maybe we can just look at some properties within the network.
Sarti, Gabriele: to try to understand how the model is allocating importance to different components in the input. For example, initially people were very keen on doing that with attention weights, right? And this has been kind of, like, controversial.
Sarti, Gabriele: So you're in this mixed success links, you find two papers that are titled, Attention is Not Explanation, and Attention Is Not Not Explanation.
Sarti, Gabriele: So people debated that, quite a lot.
Sarti, Gabriele: I think there are some promising works here,
Sarti, Gabriele: So, as I said, initial work, we're looking at attention weights in a vacuum.
Sarti, Gabriele: Which was quite misleading. Then there has been some work in that direction that thought.
Sarti, Gabriele: The reason why this is misleading is that we're not considering the actual vector, so the value vectors that these weights are multiplied by, so why don't we instead look at the vectors, so the magnitude of the resulting vectors, rather than looking at the attention weight?
Sarti, Gabriele: So, in this example here, you can see the second attention weight is quite large, so you would say, oh, this word is very important, but actually the vector is very small, and maybe this is just a compensatory behavior, right? To get a final vector that is not too small.
Sarti, Gabriele: And the final perspective here was.
Sarti, Gabriele: can we relate these three vectors here to the final outcome of the attention operation, and see which one of these is closer to that? If this green vector, for example, is closer to the final outcome of the attention operation.
Sarti, Gabriele: then it might be that it's the most, influential towards that computation, right? Meaning it's the most aligned with whatever the attention is doing to the, to the vectors in the input.
Sarti, Gabriele: so these are some pointers for you if you're interested in digging deeper.
Sarti, Gabriele: The cool part about these approaches is that it's entirely, forward-based, so there's no backpropagation, super efficient at inference time, can scale very well with, like, long context, you know, big models, so definitely that's the big appeal of this kind of methods.
Sarti, Gabriele: All right, so I wanted to give you an overview also of the NSICT.
David Bau: No, actually, can I pause you for a moment? Gabriel, so you've… so you sort of described two, or maybe three different classes and methods now, so you've.
Sarti, Gabriele: Right.
David Bau: You've described these kind of interesting gradient-based methods.
David Bau: And then these… I guess it's… maybe there's kind of attention-based methods, or are there things other than… you say internal-based, but maybe they're mainly attention-based methods.
Sarti, Gabriele: Yeah, yeah, yeah.
David Bau: So, like, okay, so we got… we got all these students here trying to do their projects, and they're…
Sarti, Gabriele: Sit.
David Bau: They're probably going to try some input attribution at some point. What's your opinion? Do you, like, would you advise people to do one of these or the others? One of them…
David Bau: You know, things that people believe more, or… yeah, what… I just want to get a sense for your own opinion.
Sarti, Gabriele: Yeah, so I think my perspective is, again, these gradient-based ones, these specific variants seem to be the most fateful at the moment, and they are kind of actionable if you're working with models that are not extremely large.
Sarti, Gabriele: So probably, if I had to go for something, I would go for…
Sarti, Gabriele: one of these two methods that I'm linking down here. They both have quite nice implementations available, so that's also a big plus, right? Kind of like, you can just plug and play with existing models,
Sarti, Gabriele: So, yeah, so that would be my guess. If I really found out that this wasn't scalable enough for the use case that I'm working on for whatever reason, because it's too long of a context, too big of a model.
Sarti, Gabriele: I guess my second choice would be some of these, these information flow routes, for example, which is a forward-only method, probably would be my go-to.
Sarti, Gabriele: yeah. Nice. I guess that's my… my opinion, yeah.
David Bau: Cool.
Nikhil Prakash: One question.
Sarti, Gabriele: Yo, yep, sorry.
Nikhil Prakash: Sorry, yeah. So, so the gradient-based approaches, we… Based on your experience, how…
Nikhil Prakash: How much does it scale for long context? Let's say if I want to analyze a chain of thought and want to, let's say, understand which are the important input tokens.
Nikhil Prakash: Or just important input sentences. Would the integrated gradient methods work?
Sarti, Gabriele: Yeah, the only downside with gradients is that sometimes we find
Sarti, Gabriele: some sort of spread out of probabilities, so it's unlikely that all the tokens will receive exactly zero importance, right? Even if we were to update them, probably the output would be irrelevant for many of them, like, I don't know, function words, you know, this kind of…
Sarti, Gabriele: unrelated stuff, so that's more of a property of the gradient, and if the context is very large, the importance will tend to be very spread out. I think this is something that can maybe be mitigated by using different,
Sarti, Gabriele: in the future, meaning, like, when designing model architectures, I think in general, you know, going towards more sparse activation function, for example, sparse MUX instead of soft MUX, that would promote this kind of sparsity at the output level, and
Sarti, Gabriele: Potentially, this could also reflect into a sparsity in the input when taking gradients with respect to that.
Sarti, Gabriele: So…
Sarti, Gabriele: Yeah, so I agree with gradients, that's a potential failure case. I think even if spread out, the magnitudes would still be informative, though. I have an example later on, on retrieval augmented generation, you can see that even for longer contexts, this is somewhat informative, yeah.
Nikhil Prakash: Okay, okay, cool, thanks.
Sarti, Gabriele: All right. Yeah, so I just wanted to show you this toolkit that we built, that is exactly, for using attribution methods, mostly gradient-based, on language models. So the idea here is that you have your Hugging Face model, let's say a GPT-like model that receives a prompt.
Sarti, Gabriele: And the model will do autoregressive generation, predicting one word at a time. For example, to innovate, one should think outside the box. And what the toolkit allows you to do is simply, at every generation step.
Sarti, Gabriele: We extract the attribution scores for the given prefix, and we can extract also quantities of interest, for example.
Sarti, Gabriele: the probability of the output, the entropy of the output, which are also what we're taking the gradient with respect to, right? So, the final outcome of all this would be something that resembles this table.
Sarti, Gabriele: So the way that you read this is that the columns are the tokens that were generated, and on the rows is the prompt at every generation step. So you see this triangular pattern here is because every new token gets added as an element in the prompt at every generation step.
Sarti, Gabriele: So it plays… it has an influence on the next steps of prediction.
Sarti, Gabriele: And you also have, for example, some information of interest, like here I'm also extracting the probability, of the predicted token at every step, kind of like, yeah, the final prediction probability.
Sarti, Gabriele: So you can see in this example, that the moment that the model starts predicting think outside the box, the model is mostly relying on innovate, which is the key word to predict the multi-word expression.
Sarti, Gabriele: But the moment it starts producing the sequence, its kind of saliency shifts towards the previous tokens in the sequence, which kind of reflects the model knows where it's going and is just looking at the prefix to finish the expression, right?
Sarti, Gabriele: This is also reflected by the probability that starts, kind of, 50%, but then it becomes increasingly closer to 100%.
Sarti, Gabriele: So, yeah, here you have easy access to, like, a dozen attribution methods, including attention weights, gradient base, internal space, specifically for generative LMs, so, like, both encoder, decoder, or decoder-only language models.
Sarti, Gabriele: And in the paper that we had related to this toolkit, we did a couple case studies. The first was to study gender bias in machine translation, so we were highlighting that pronouns have a big role when the model decides to use
Sarti, Gabriele: sorry, that professions, stereotypical professions, have a big role when the model decides to translate as he or she from a language that doesn't have the distinction.
Sarti, Gabriele: And then we also try to approximate patching, so I have a slide on that, later.
Sarti, Gabriele: So, this is a simple example of using the library. So, here we load a model with integrated gradients, which is the method that you saw.
Sarti, Gabriele: And we do this model.attribute, which is kind of like the generate function in Hugging Face, just on top of that, we also extract whatever we're doing, right? So here it's, we prompt the model with, does 3 plus 2 equals 6?
Sarti, Gabriele: And the output that gets generated can then be visualized with show, and it looks like this. So the model predicts yes, end of sequence, and here we have attribution scores for the prompt, plus the yes in the case of the end of sequence, right?
Sarti, Gabriele: So, do you notice something kind of weird here?
Sarti, Gabriele: Like, is this kind of attribution pattern what you would expect for this kind of task?
David Bau: So I'm gonna… I'm gonna just talk out loud here, because I'm really being slow here, but… let's see, so does 3 plus 3…
David Bau: equals 6.
David Bau: And… I would expect
David Bau: Oh, I see. So then it's gotta decide, so it's deciding whether this is true or not.
David Bau: And I would expect that
David Bau: The things that it would have to look at to decide whether it's true is…
David Bau: It needs to look at the answer.
Sarti, Gabriele: And I have to look at the question. I have to look at the question and answer, right? So, like, if you…
David Bau: Right? So, like, so, like, if we said, like, 3 plus 4 equals 6, right, that would be a different answer. So, like, the 3 and the 3 and the 6 seem like they'd probably be the most important things. Maybe the plus.
Sarti, Gabriele: That's right. That's right. It's like they're the least important parts of the sentence.
Sarti, Gabriele: Indeed, indeed. So, like, my hypothesis by looking at this example was, can it be that this model is just so strongly trying to figure out the output desired format? Like, the fact that this is a yes-no question rather than a mathematical, come-up-with-your-answer question.
Sarti, Gabriele: That the fact of these function words receiving high importance is just because
Sarti, Gabriele: These are what are driving the final format, right? These are what produces the yes, rather than the equation itself, right?
Sarti, Gabriele: But then.
David Bau: Oh, right, because you're saying there's 50,000 other words that it could say here. It could say pumpkin, or whatever, right?
Sarti, Gabriele: Well, it could be that this model is trained to do maths, right? And then it's usually prompted to produce an answer given a mathematical expression, but in this case, the answer is not a number, right? So then it has to rely pretty heavily on function words that define the kind of expected format, which is yes-no, right?
David Bau: I see, I see. So…
Sarti, Gabriele: So, this can lead us to formulate some hypotheses, right? Like, the equation is not getting much importance, so can it be that the model is actually getting it
Sarti, Gabriele: right for the wrong reason, right? Maybe it's not actually caring much about the equation, and it's just betting 50-50, yes or no, right?
Sarti, Gabriele: And this can be tested.
Sarti, Gabriele: And indeed, we find that this… this is a pretty old model, but it's saying, yes, also for… does 3 plus 3 equals 7, right? So in this case, I just gave you an example of, like.
Sarti, Gabriele: we started from some attribution to formulate hypotheses that then we test behaviorally, right? So, the thing that I want to emphasize here is
Sarti, Gabriele: The attribution itself didn't give us any causal confirmation that what we were trying to do,
Sarti, Gabriele: was, like, that the model wasn't actually doing the expression, but kind of highlighted that maybe the importance was a bit off for this kind of problem, right? So this could be valuable for this kind of hypothesis generation.
Sarti, Gabriele: Sorry, I'm seeing that.
David Bau: Oh yeah, there's.
Sarti, Gabriele: Oh, Jasmine wrote, some messages.
Sarti, Gabriele: Yeah, I can… yeah, we have some examples later,
Sarti, Gabriele: yeah, I can… I can try to discuss more about those, in the next few slides.
Sarti, Gabriele: So… There were some questions about faithfulness,
Sarti, Gabriele: So, yeah, I don't know if people are here, maybe they want to ask them themselves.
David Bau: Lose, Luz, yes, go ahead.
David Bau: Or Jasmine, either one.
jasminec: Oh, I guess, like, for me, I was wondering, like, when, like, an explanation is, like.
jasminec: informative enough. Like, for instance, like, if someone asked me, like, why did you eat an egg this morning? Like, I could say, like, because I was hungry, but I could also say something like, okay, like, when I was born, like, my mom fed me, you know what I mean? Like, it could start.
Sarti, Gabriele: room.
jasminec: 20-something years ago. Like, how do you know, like, when you have enough information? You're like, that's, like, a reasonable explanation.
Sarti, Gabriele: Right.
Sarti, Gabriele: Yeah, I think my answer to that is whenever it's sufficient to predict behavior in the current use case, right? Like, at a satisfactory level, I think that's probably…
Sarti, Gabriele: The way that we tend to operationalize faithfulness overall, so…
Sarti, Gabriele: Yeah, so I would say that's the best way to understand, you know, like, if our explanation allows us to, to some degree, to understand, you know, what the model is doing there, and if we can act upon it, then probably our explanation is faithful with respect to how the model is doing things internally, right?
Sarti, Gabriele: And maybe, you know, if it's a high-risk domain, probably I wouldn't trust even very faithful explanation in the medical domain, because the stakes are very high, right? So then, yeah, you really need, like, to weight your expectations based on the kind of application, I guess.
Sarti, Gabriele: So yeah, so, I like two dimensions in faithfulness. So this is a paper, actually, from Northeastern, from Barron Group.
Sarti, Gabriele: So, they were working on faithfulness, very early in interpretability, and one way that they, that they were defining faithfulness was two complementary perspectives. So.
Sarti, Gabriele: if we want actionable results, is if these tokens are found to be important, one idea is if we drop these tokens, then we expect a big impact on the results, right? So this is what they call comprehensiveness. It's kind of like ablation, right? Kind of like occlusion. We occlude, we…
Sarti, Gabriele: We cause a big impact.
Sarti, Gabriele: The other perspective is efficiency. So here we're saying, if we only have these tokens, and we remove all the rest, does the result… do the results remain kind of consistent, right?
Sarti, Gabriele: So, yeah, this is just a visualization. If we find most amount of leaves as the important part, we would have comprehensiveness, it's like, yeah.
Sarti, Gabriele: If we,
Sarti, Gabriele: If we drop that, the probability drops. If we only keep that, the probability kind of stays the same. This is sufficiency.
Sarti, Gabriele: So, the Luz question about… I think… I don't know if Luz is here?
Sarti, Gabriele: Maybe not.
Sarti, Gabriele: Right?
Sarti, Gabriele: I can summarize it. So the, the question was, if the models are able to explain themselves, like to, for example, for… for Mirage, it was if the models can cite
Sarti, Gabriele: alongside giving an answer. Why do we even need a faithful method that looks at the internals, right, if the model can already do that?
Sarti, Gabriele: And our point with that paper was actually that
Sarti, Gabriele: The fact that models are so capable as to being kind of precise is kind of a recent thing, and before, we were just relying on superficial matching.
Sarti, Gabriele: So the citation was done mostly by saying, oh, here is the answer, here is the three documents that the model received, let's just embed those and look at the similarity between those and find whatever is the most similar, right?
Sarti, Gabriele: So, I think that the answer to this question is, I think.
Sarti, Gabriele: There is an interesting perspective where we try to make models more aware of their inner workings, and in that direction.
Sarti, Gabriele: potentially, we wouldn't need so much of, like, digging deep into the model if the model indeed can kind of self-predict well enough, right? I think we're still kind of far from this perspective, though, so I think probably we do need,
Sarti, Gabriele: We do need these kind of methods still, for the foreseeable future.
Sarti, Gabriele: And the complementary perspective to faithfulness is plausibility. So the plausibility dimension is user-centric instead of being model-centric, and it's asking, can these explanations be understood by whatever users of the system are looking at the thing?
Sarti, Gabriele: So, one big problem is that you don't have guarantees that faithfulness and plausibility go hand in hand, right? So ideally, the more faithful you are to the model, the more
Sarti, Gabriele: understandable you are for humans, but potentially there could be a disconnect there, right? So sometimes this is also being highlighted as a trade-off.
Sarti, Gabriele: So, the more you make explanation plausible, the more you're abstracting away the inner complexity of the model, and this produces this kind of mismatch, that maybe you're kind of like.
Sarti, Gabriele: Diluting too much the complexity for users.
Sarti, Gabriele: So, again, this is application dependent, and one interesting idea that I like in this domain is counterfactual simulation, so the kind of, setting would be, if you have an example and you can get attribution out of this, the idea would be to,
Sarti, Gabriele: given the attribution, can I simulate a counterfactual that would produce a different behavior, right? Like in the other case that we saw just before.
Sarti, Gabriele: Can I simulate the fact that if I change the 6 into a 7, the model still predicts CS, right? And then, what you would do is to compare, you know, the expected result with the actual behavior in this case.
Sarti, Gabriele: So this is a good way to understand whether what you're looking at is kind of plausible or not, right?
Sarti, Gabriele: Alright.
Sarti, Gabriele: So, now talking…
David Bau: What was the paper you cited for that one? Sorry, connecting?
Sarti, Gabriele: Oh, yeah, it's a paper from some years ago from Greg Durrett,
Sarti, Gabriele: from the group of Greg Durrett about, like, doing this kind of counterfactual for evaluating explanations,
David Bau: And so was it… was that paper about, like, sort of human evaluations? What did Direct do?
Sarti, Gabriele: Yeah, yeah, they were trying to do similar stuff, like, this figure is taken from there, so, so this is their setup, actually, yeah.
Sarti, Gabriele: they were doing it mostly for classifiers, though, NLP classifiers. So I do think that it's a bit more challenging to do that in the generation setting. That's a bit related to what we did for the PCOR method, actually.
David Bau: Okay, cool.
Sarti, Gabriele: Yep.
Sarti, Gabriele: Yeah, so, like, the contrastive attribution setup, I think it's very compelling for language.
Sarti, Gabriele: And,
Sarti, Gabriele: Yeah, I just want you to focus for now on the example that I have on the left. So, if you have this input, right, can you stop the dog from, and the model predicts barking.
Sarti, Gabriele: If we do gradient-based attributions, so this is just simple gradients taken with respect to the inputs, and we do the aggregation, as I showed before, you would get some scores that look like this. So, red is positive and blue is negative in this setting.
Sarti, Gabriele: So, do you think this is intuitive, what you're seeing here? Like, these attribution scores, do they make sense to you, given this prompt?
Sarti, Gabriele: Maybe not.
David Bau: So there's things that are very positive and things that are very negative, and does white mean, like, close to zero?
Sarti, Gabriele: Yup.
Sarti, Gabriele: Yeah, so the highest tier is from, right? From is very, very, positively influencing barking.
David Bau: Right. And V is very negatively influencing barking.
Sarti, Gabriele: Yep.
David Bau: But the word that has the least effect is dog.
Sarti, Gabriele: Yep.
David Bau: Which makes you smile. It seems like it's very counterintuitive, seems like…
David Bau: Look at, look at Nikhil, he's laughing at this.
Sarti, Gabriele: Yeah, exactly.
Sarti, Gabriele: Yeah, I mean, that's… that's weird, right? That's exactly the opposite of what we would expect, right? And I think one of the…
Sarti, Gabriele: One of the reasons for that is that
Sarti, Gabriele: as humans, we tend to reason counterfactually, right? So when I'm asking you,
Sarti, Gabriele: what would come after that? Like, can you stop the dog from barking, right? If you had to explain barking, you have the tendency to reason semantically about barking, right? So, like, barking dog are related words, so dog should receive a big importance, right?
Sarti, Gabriele: But exactly, exactly as Jasmine is saying in the chat, eating could be an alternative word, right? So naturally, in a sense, we're contrasting in our head barking with some other plausible alternative that doesn't involve dogs, right?
Sarti, Gabriele: Well, in practice, what attribution here is doing is just detecting relevance, right? And I could argue with you that
Sarti, Gabriele: the from here is super important to predicting barking, because it's… it's the next previous word, and it's exactly defining that the verb should be in that form, right? Without from, we wouldn't have a present continuous verb there, right? So then it's… it's essential, right?
Sarti, Gabriele: So then, how do we actually bring this closer to human intuition? So the idea that they had in this paper that I'm citing here from Graham Newbig and Kayo Yin is to have a contrastive attribution. And the way that you would do this is by contrasting two words.
Sarti, Gabriele: So, the idea is very simple. Instead of taking the gradient with respect to a single probability, we can take it with respect to a difference in probabilities. Here, probability of barking versus probability of crying, for example.
Sarti, Gabriele: And then the gradient that we get with respect to the input looks a lot more reasonable, if you ask me. So dog now finally has a meaning, and from is… doesn't matter much, because it would be a good choice for both verbs, right?
Sarti, Gabriele: So, in their work, what they were showing is that by disentangling this kind of semantic factors from syntactic factors, you can improve simulatability, which is a bit like what we were seeing before in plausibility, so the ability of a human to actually,
Sarti, Gabriele: Predict whether the prediction would change by changing a specific word, right?
Sarti, Gabriele: So yeah, one thing that I want to stress here, here on the right, is that the attribution function, like the… sorry, the attributed function, in this case the difference in probability.
Sarti, Gabriele: is fundamental to the way that you interpret what you're getting out, right? So in this case, we saw by taking this difference, you can interpret it as a
Sarti, Gabriele: why this rather than something else, right? But here I make an even more abstract example. I could attribute the entropy of the final distribution, over the vocabulary.
Sarti, Gabriele: And this could maybe tell me what in the input is driving the uncertainty in the model, right? Or the certainty in the model. So, in principle, like, the possibilities are endless, you know? You could attribute any kind of function
Sarti, Gabriele: of your prediction, and the attribution scores would tell you different things, depending on what you're looking at, right? So, I think here, again, there's a lot of untested ground in this area, so people are just kind of starting out and digging into potential variants of that.
David Bau: Yeah, and I mentioned before… We have a question from Arnav online, from the chat.
Sarti, Gabriele: Oh, sorry.
Sarti, Gabriele: Maybe these attribution methods are just not good enough for modeling 2D order effects.
Sarti, Gabriele: Yeah, that's a… that's a good point. I have a slide later about actually modeling interactions of features, so there are some methods that are specifically meant for that, since many people ask in the class.
Sarti, Gabriele: But yeah, they're very expensive, so definitely that's… that's also another problem why they haven't seen much usage.
David Bau: So, the Gram Nubic method here, this cool contrastive method, it's like… but this is pretty simple. This is pretty lightweight. It's like a pretty cheap gradient to compute. That's neat. It also seems like… like, here they're applying it to a gradient method.
David Bau: Well, it seems like you might be able to do this for some of the other methods you talked about, like the occlusion, you know, looking at attention, looking at LRP, all this stuff, like, you might be able to drop in.
Sarti, Gabriele: I think they were trying, they were trying in their original paper also occlusion, like, this kind of setup for occlusion. I have personally already implemented the LRP with the contrastive attribution, so this works, I… it's tested.
David Bau: You did it.
Sarti, Gabriele: Yeah, yeah, yeah.
David Bau: Oh, so you like it.
David Bau: Oh, here's.
Sarti, Gabriele: Yeah, yeah, yeah, I'm a big fan of this idea. For NLP, I think it's very valuable, because the output space is so large, right? You have so many tokens that it just makes sense to pin down exactly what you want to compare, yeah.
David Bau: Nice, that's great, thanks. This is really helpful.
Sarti, Gabriele: Great. Yeah, I just wanted to mention, so for component attribution, I think last lecture, or two lectures ago, you saw with David the causal mediation, right? So this might be familiar to you, this kind of setup. Eiffel Tower is located in Paris, right?
Sarti, Gabriele: So when we introduced the in-seq library, we asked, but can we use contrastive attribution for approximating causal mediation, right? Can we use this contrastive, gradient
Sarti, Gabriele: attribution to get saliency, not for the input tokens like we saw just now, but for all intermediate steps, right? And kind of see how much does it agree with causal mediation.
Sarti, Gabriele: And our results was that, of course, this is much coarser and not, you know, not as sharp as causal mediation, but we did also see this kind of
Sarti, Gabriele: early site that they were highlighting on the last subject token. So, to some degree, let's say our method was associating much more importance to the last token.
Sarti, Gabriele: But it still found some structure that… that using causal mediation would have required a lot of ablations, right?
Sarti, Gabriele: And the cool part here is that this is very efficient, right? We do a single forward pass, and we do one backward pass in which we get saliency values for all the nodes here in the graph.
Sarti, Gabriele: And that's it, instead of doing sequence length, time, number of layers, forward passes to do… to estimate causal mediation.
Sarti, Gabriele: So, actually, there is one very popular attribution method that has been used after that that is called attribution patching.
Sarti, Gabriele: Which was introduced more or less at the same time as ours, but got a lot more traction.
Sarti, Gabriele: And the idea is quite similar, it's just instead of having the contrastive outputs, they contrast inputs. So they change… they have two settings, they get gradients for the two settings, kind of like in causal mediation.
Sarti, Gabriele: And then they just take the difference between the gradients. But the idea is pretty similar, kind of complementary, let's say.
Sarti, Gabriele: So, now I want to move to, the other… the other work that you've seen, in the readings that was related to attributing context, with… with language models. So, our original contribution in this area was, was this PCOR, framework.
Sarti, Gabriele: So the main driver for that was that we found that attribution methods are expensive if you apply them to the whole generation, right? We've seen already, if you had to build the table that I've seen and that I've shown before, for a very long generation.
Sarti, Gabriele: That would be super expensive, so you kind of want to narrow down exactly on which steps you're interested in doing attribution.
Sarti, Gabriele: And…
Sarti, Gabriele: Secondly, the fact of ambiguity with large vocabularies, so the second drive is, this, you know, contrastive… need for contrastive explanations, right?
Sarti, Gabriele: So what we propose is this plausibility evaluation of context reliance framework.
Sarti, Gabriele: And the way that this works is simply in two steps. The first step is to identify in the generation which steps are more influenced by context, and then to focus on those
Sarti, Gabriele: to do this contrastive attribution back to the context. So the final outcome of all of this is simply a pair of influential context tokens
Sarti, Gabriele: Relating to some influenced-generated tokens, right?
Sarti, Gabriele: So now, I'll give you a very quick overview of how this works. So let's assume our model, here it's an encoder-decoder, but it doesn't really matter. And,
Sarti, Gabriele: we are considering a generation task, so English to Italian generation, in which we have a context that the model needs to use to do the task correctly. So let's say here you have, I ate the pizza, this is my context, it was quite tasty.
Sarti, Gabriele: if I have to translate, it was quite tasty in Italian, I have to know whether the tasty is masculine or feminine, right? Depending on what I said before. In this case, pizza is feminine, so it needs to be one.
Sarti, Gabriele: For example.
Sarti, Gabriele: So this is the… what we call the contextual variant of the input.
Sarti, Gabriele: Then we can have a non-contextual variant.
Sarti, Gabriele: that is passed as is, and would predict something different, right? So here, for example, we could go with the masculine as a default, eramolto bueno, with a O at the end.
Sarti, Gabriele: So what our method does is actually
Sarti, Gabriele: is to take the contextual version of the output and force decode it in the non-contextual case by taking this kind of information theoretic metrics at every step of the generation. So here we enforce the same token at every step.
Sarti, Gabriele: And then we look for which one of these tokens the distribution would be the most skewed by the absence of input context, right?
Sarti, Gabriele: So, we would get a score per token that then can be, discretized in some, with some heuristics, for example, to get a label that is either positive or negative.
Sarti, Gabriele: just to have a yes-no kind of perspective here. So in this case, we would find this last token that was generated is the one that is the most influenced by the context, right?
Sarti, Gabriele: So, yeah.
David Bau: Oh, it maybe saw me on mute.
Sarti, Gabriele: You had it, Chris, yeah. Yeah, yeah, yeah.
David Bau: Yeah, so you say 4C code, so is this just… so… which I didn't really understand the 4C code, so you're sort of making two predictions, and you're asking…
David Bau: Let me think about this for a second.
David Bau: So, you're, you're asking, what is…
David Bau: Or, or, or, or, or… I see, so you're putting the, you're putting the prediction… Y hat…
David Bau: In… and you're using the model to evaluate That prediction.
David Bau: And, and then…
Sarti, Gabriele: Exactly.
David Bau: And then… and then you're… and then you're making a heat map over…
David Bau: The evaluated tokens to say which one
David Bau: is most unlikely. And are you… so are you doing that with KL divergence?
Sarti, Gabriele: In this case, we tried several metrics, and KL divergence was the one that was leading to the best results.
David Bau: We also try to contrast just the probability of the top token,
David Bau: I see, I see. So it's not literally just the top token, it's just whatever the model was thinking there, you're taking that whole distribution, and then you're putting it, and you're saying, how different is that from what the model wants to think?
Sarti, Gabriele: Exactly.
David Bau: over in this situation. But… but you are… but for… on the input side, you have to feed in the whole sentence, and so you're just feeding in…
Sarti, Gabriele: Yup.
David Bau: the whole sentence on the inputs.
Sarti, Gabriele: the two alternatives, right? The one with the context, the one without. Yeah, I think this first decode can be a bit misleading here, because the example is very short, right? But imagine if you had a long example that has many of these kind of keywords that were influenced by the context. The idea here is that I just wanted to express that you
Sarti, Gabriele: you will always keep the two cases identical, so you're kind of, like, adding the… whatever is from the contextual case, regardless of what the non-contextual case would predict there, to ensure that the prefix is always matching, right?
David Bau: Right, because the output becomes part of the input as you go auto-regression.
Sarti, Gabriele: Exactly, exactly, and you have to match them, otherwise you would have a disagreement that might be mediated by different outputs, right?
David Bau: Sorry to ask this question so fast, is it…
David Bau: I want to let the students ask further if we've confused them.
Sarti, Gabriele: I don't know if this is clear enough.
David Bau: So the goal of this step
David Bau: Is to basically get a heat map
David Bau: Over these Ys, over the output.
Sarti, Gabriele: Yup.
David Bau: Yep, pretty much.
Sarti, Gabriele: Yeah, so as I said, these would be continuous course, but then we would get, this kind of discrete yes-no labels, right? So the reason why we need these yes-no labels is for the step two.
Sarti, Gabriele: Where this will, allow us to understand where to do the attribution, right?
Sarti, Gabriele: So, the key, interesting thing that I think we introduced here is that, let's say that we… now we have our sequence, so this is the contextually generated output with these labels, right?
Sarti, Gabriele: The key step here is that we want to force, the prefix that is the same for all the tokens that weren't found contextually sensitive, right?
Sarti, Gabriele: But then to sample from the non-contextual setting, The alternative, right?
Sarti, Gabriele: So, basically, here, this is just a data-driven way to get these contrastive pairs that then would allow us to do contrastive attribution, by exploiting the
Sarti, Gabriele: the same model without the context to get what would be its prediction without the context, right? So now, we kind of bootstrap this minimal pair of, like, words that then can be… we can do the contrastive attribution that I showed before, so probability of one minus the other.
Sarti, Gabriele: And propagate this throughout the model back to the input context.
Sarti, Gabriele: So what we get here is, for example.
Sarti, Gabriele: this continuous scores at the token level, like I showed before, and this, again, would be discretized to get pairs. So here, the way that we would read that is the pizza in the source is influencing buona, and also the pizza in the target is influencing buona, right?
Sarti, Gabriele: is this clear enough? I know it's quite, complex as a structure.
Sarti, Gabriele: Yeah, so what kind of this decision can we make off of these results?
Sarti, Gabriele: I think the interesting part that you… like, the interesting idea here would be that you can apply this to any kind of task, let's say, in a bit of a blind way, right? Like, you have a generation from the model, you could just entirely, in a data-driven way, I run
Sarti, Gabriele: my PCOR approach, I get these relations between inputs and outputs, and then you could explore the outputs, right? And form hypotheses, right? So we had kind of interesting examples that we found when we did that on machine translation.
Sarti, Gabriele: Sadly, I don't have them here in the slides, but one example that I can mention that surprised me was that,
Sarti, Gabriele: we had a text where the model, was receiving some information, like, the soccer match is at 10 AM, written, like, 10 semicolon 00am.
Sarti, Gabriele: And then, it had a text that was saying something like, the match was a fierce competition, and it ended 26-0, right? For the blue team, for example.
Sarti, Gabriele: And the model was deciding to format 26 to 0 with the semicolon, because the time in the context was using the same semicolon format to express the hour, right? Which is kind of a weird behavior, if you think about it. I would have never thought of that myself.
Sarti, Gabriele: So I think that kind of highlights the importance of making this data-driven, right? So a big motivation for us was that most of this kind of evaluation here rely on this kind of
Sarti, Gabriele: hypothesis-based, I don't know, I expect my mother to have a gender bias, so I craft my set of data with gender bias, and I test them or not, right? Well, here, you can just run it on anything, and then kind of post-talk, look at if there is something interesting there, right?
Sarti, Gabriele: So yeah, that's… that's the overall idea.
David Bau: That's… that's cool. And so, it's sort of… as a way to think about this is…
David Bau: There's… there's just too much information.
David Bau: in the All Pairs…
David Bau: you know, sort of attribution, and you're doing things to try to winnow that down to a small number of edges that you really want to pay attention to, is that right?
Sarti, Gabriele: Exactly. And I think, you know, more philosophically speaking, moving forward with, like.
Sarti, Gabriele: interpreting, you know, very complex scenarios. I think this kind of, narrowing down what we really care for will be super important in the future, too. You know, even if we want to do, I don't know, mechanistic analysis of, you know, something sketchy is going on here that we didn't expect, you know. I think in reasoning or, like, agents, you know, I think this will become more and more important to kind of narrow down
Sarti, Gabriele: what's going on there? So, yeah.
Sarti, Gabriele: So that was the original idea here.
Sarti, Gabriele: So… And the next reasonable step here was, wait, now we can connect outputs to inputs.
David Bau: Did Jasmine get to ask her question? I see another text that flew by. Did you get to ask your question, Jasmine?
jasminec: Yeah.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Yeah, yeah, yeah, I can, took it up, yeah. Okay, that's great.
Sarti, Gabriele: Yeah, yeah, so the next reasonable step here was, well, now we can link outputs to inputs, can we use that to create citations, right? So this was very relevant, it was, yeah, a couple years ago, we didn't have, again, these models that could cite themselves well.
Sarti, Gabriele: They weren't trained to do that,
Sarti, Gabriele: So, our idea was, can we just, looking at the internals, understand how the inputs are influencing the generation, right?
Sarti, Gabriele: So Mirage works exactly in the same way as Pecore. So here we are, our context that gets added or removed is the three documents that were retrieved by a retrieval system and added to the prompt.
Sarti, Gabriele: and the functioning is exactly the same. We would see how these three documents shift the probability distribution of the model for specific tokens in the answer, and then trace this back to some specific tokens in one of the documents that are responsible for the shift.
Sarti, Gabriele: So, I wanted to show this picture that was also in the paper that you were referred to by David. I think this is quite good evidence in relation to what Nikhil was asking before.
Sarti, Gabriele: That even though it's not super clean, this is entirely gradient-based, so this is raw gradients for attribution.
Sarti, Gabriele: And you can see that here on the x-axis, you have the five documents that were given as context, and every point here is a word within a document that receives an attribution score, right? So the y-axis is kind of like the attribution intensity for that word.
Sarti, Gabriele: given the output here, 9 in the generation, right? So you can see that this quite cleanly points at the two… the exact match, $19 billion in the document 1, as a motivation for predicting $19 billion in the answer.
Sarti, Gabriele: So yeah, even though it's not super clean, there is still, like, enough information to kind of, you know, cut out exactly what we want here.
Sarti, Gabriele: And yeah, in the paper, we try different approaches, and we were using some heuristic, like, let's take the top 5%, or the top 20% of these tokens.
Sarti, Gabriele: Based on their attribution scores. But we also tried calibration, so kind of, like, trying to select what would be a good threshold to match a set of gold labels
Sarti, Gabriele: in a way that then we can just find this threshold value and then apply it, you know, to unseen documents, and that seemed to help. So, in general, if you have a gold annotated dataset with the citations that you want.
Sarti, Gabriele: That probably would be a good way to select these thresholds.
Sarti, Gabriele: There were some questions related to Mirage, maybe people that are here can ask them.
Claire Schlesinger: I was asking what would happen
Claire Schlesinger: if, like, Attribution saw two documents that are similar, but, like, different in tone.
Claire Schlesinger: So, like, they may have, like, the same content, but presenting different viewpoints.
Sarti, Gabriele: Yeah, that… yeah, I thought that was a very good question. I think it depends a lot on the model, right? So it might be that the model will find that, you know, a more explicit mention of something is more…
Sarti, Gabriele: actionable to come to an answer, so it might be that that receives the higher importance. Actually, what I mentioned here is that recency is probably the most common trend that you see in language models, so if something is mentioned
Sarti, Gabriele: several times. The last time… the last mention is probably gonna be the one that the model is mostly relying on.
Sarti, Gabriele: But this all relates to…
Sarti, Gabriele: this thing that we were saying before, attribution pertains only to the current context, right? So, like, the fact that this token received a high importance is not exactly a proxy for saying.
Sarti, Gabriele: If this token wasn't there, then everything would change, because, like, in this case of redundant mentions, if these tokens disappeared.
Sarti, Gabriele: then other tokens could take the attribution other place. So sometimes this can lead to this kind of misleading interpretations, right?
Sarti, Gabriele: Yo.
Sarti, Gabriele: And…
David Bau: Is Aria here?
Arya: Yeah, but I think, you've already answered my question, so…
Sarti, Gabriele: Yeah, pretty, pretty related.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: And what's this last one?
David Bau: Should you?
Sarti, Gabriele: Not here, I think.
David Bau: I'm sure he's here, but maybe, maybe in.
Sarti, Gabriele: Got it.
Sarti, Gabriele: I'm not sure.
David Bau: Oh, Mike is not working.
Sarti, Gabriele: Okay, okay. I can just summarize, yeah,
Sarti, Gabriele: Yeah, so the idea here was, is this procedure of, like, forcing the prefix when… when considering this difference from the context, actually breaking the… the Mirage framework, right, in a sense, because you're forcing a…
Sarti, Gabriele: a prefix of the output that is maybe not what the model would generate when the context was absent, right? So definitely, this leads to some potentially out-of-distribution behavior there.
Sarti, Gabriele: But considering that that's mostly used to select cases where the context was influential,
Sarti, Gabriele: Actually, this can lead to some interesting things, like, if something becomes explicitly mentioned in the output.
Sarti, Gabriele: then it might be that whatever comes after is now relying on the output of the model, rather than relying on the context, right? So maybe at first you need to rely on the context, but from there onwards.
Sarti, Gabriele: Now you're just relying on the output, so all the subsequent mentions would not be picked out, right, as context-sensitive.
Sarti, Gabriele: So I think that's actually something interesting.
Sarti, Gabriele: That actually reflects how the model operates, right?
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Alright.
Sarti, Gabriele: What I mean?
Sarti, Gabriele: Yeah, and I just wanted to show you, so we have an API for, PCOR inside NSYC now, so that's… it's pretty…
Sarti, Gabriele: convenient to use, and it's, you know, it can be used within a Jupyter notebook, it's quite nice. So we can have a look at this example together, I think it's quite, quite informative.
Sarti, Gabriele: So here, the input that the model receives is when was the most successful player in NBA history born.
Sarti, Gabriele: And this is not a great model, it's like a 300 million parameter model, so it's… it's pretty bad at doing language modeling, so here the model is predicting 2015-2016, but then it's also saying the most successful player in NBA history is Steven John
Sarti, Gabriele: Something, okay? So…
Sarti, Gabriele: we set our threshold when we apply the method, and we found that John, in this case, is one of these tokens that is context-sensitive, right?
Sarti, Gabriele: So, if we open this toggle here, what we see is
Sarti, Gabriele: three documents that the model received that were appended to the prompt, right? So this is a rug setup, kind of like in Mirage, right?
Sarti, Gabriele: So, you can see that the tokens within these documents are colored based on how influential they were towards the prediction of John here.
Sarti, Gabriele: And we can see that the most influential ones are actually Steve and John, right? Which makes sense. So it means the third document here
Sarti, Gabriele: is what is driving the prediction of Steven John here.
Sarti, Gabriele: But I think even more informative, is the fact that here you can also see what the model would have predicted in the non-contextual case. So here, the model would have said, Stephen Kerr
Sarti, Gabriele: Which is probably Stephen Curry, right? I'm not a basketball expert, but I guess that would make sense.
Sarti, Gabriele: But, it was kind of like, you know, sidetracked by the presence of this Stephen John here in context, or towards predicting Stephen John here.
Sarti, Gabriele: So, I think this is interesting, because basically this example is pointing at the fact that this model
Sarti, Gabriele: Is over-relying
Sarti, Gabriele: On the context versus its previous memorized information. So it probably would have said something reasonable if only using memory, but the context has such a big influence on what it's predicting that it decided to go for something that maybe is not ideal here, right?
Sarti, Gabriele: So, this is the kind of visualization you can get in a notebook. So, in the NSICK repository, we have a notebook with exactly this example, so to reproduce exactly this.
Sarti, Gabriele: And… and some analysis on reasoning, also.
Sarti, Gabriele: So yeah, so it's a new version that we released some time ago.
Sarti, Gabriele: Any question on that?
Sarti, Gabriele: Nope.
Sarti, Gabriele: Alright.
David Bau: You know, I'll be interested in, you know.
David Bau: whether, you know, whatever methods get adopted to different research projects. You know, each one of these methods is…
David Bau: you know, exposing something different, so I'll just be really curious. Yeah.
Sarti, Gabriele: I know that.
David Bau: And I think it's fair, if people are like, oh, I, you know, I might…
David Bau: use some of these methods for my research project for people to just ask now, while we have Gabriel here, to sort of share his wisdom or opinions about different possible applications.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: I mean, I just want to mention that my perspective on that is, like.
Sarti, Gabriele: this is great, like, if you have this kind of setup, right? My next question would be, well, I found that Steven John now is influencing John, right?
Sarti, Gabriele: let's look at the internals, right? Let's do, like, our circuit analysis, let's do our, you know, causal mediation, but now we have an anchor point, you know, we have some behavior that we identified that is there, right?
Sarti, Gabriele: And that would have been much more painful to do had we had to do causal mediation on the full sequence of documents here, right? So I think this is…
Sarti, Gabriele: great to get started on, like, finding interesting phenomena that then you can dig deeper with the mechanistic toolkis, right? So…
Sarti, Gabriele: Yeah, so if you intend to use these kind of things, that's probably my suggestion.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: And… Oh, yeah, there were a couple final slides here, the first was about interactions.
Sarti, Gabriele: So here the idea is,
Sarti, Gabriele: like, as many of you asked about this, can we model this kind of second-order effects, interactions? And indeed, there are several methods to do that. There is this, shape, shapely interaction index.
Sarti, Gabriele: Which the idea here is you try pairs of… groups of features going… increasing in size with these groups to understand when a group is minimal to predict some behavior. And then you have also gradient-based methods, so this…
Sarti, Gabriele: Hessian or integrated Hessian are basically the equivalent of gradients, but you're taking the second-order derivative, so you're taking, like, which other factors are more influential for the gradient of that factor to get to this magnitude, right?
Sarti, Gabriele: So, in this case,
Sarti, Gabriele: yeah, it's the equivalent for interactions. The only problem with all of these methods is that they're quite expensive, because you're potentially, you know, estimating all possible interaction with all possible, groups, so that could become very expensive, unless you start by some, you know, assumption of, like, how this group should be formed.
Sarti, Gabriele: I think some people were working at the level of syntax, for example, right? If something belongs to the same phrase, then it makes sense that they are kind of part of the same group.
Sarti, Gabriele: But yeah, I think that's the only actionable way to study these kind of things.
Sarti, Gabriele: And the final thing… I really like this question from Jasmine that was, like, why are we even doing attribution? Like, what's the… what's the final goal, right? What's the end game of attribution? So I think it's a good way to kind of close on that.
Sarti, Gabriele: And I just wanted to highlight some of the works that use that. I think one of the very convincing use cases was from people that do protein design.
Sarti, Gabriele: I thought this was from a presentation that I cleared, two years ago. I found it very interesting because there were these people from Genentech, that do this kind of, lab protein design, kind of like lab-in-the-loop protein design.
Sarti, Gabriele: And when they were giving their keynote, I was pretty surprised to find out that they were saying, yeah, you know, if we were to identify exactly which amino acids are responsible for, like, a specific gene expression.
Sarti, Gabriele: that would be very painful to do, like, by trying out all possible things. So we just do gradient-based attribution on that… on the protein sequence, and we… and we use that, right? So I found it quite compelling as an idea.
Sarti, Gabriele: So there are some interesting direction here.
Sarti, Gabriele: And I feel like now people are starting to use these kind of methods also for other, more actionable purposes. So here, there is a recent paper that we're trying to
Sarti, Gabriele: use attribution to lead… to steer generation towards looking at specific regions of interest. So, like, if you define a constraint in the prompt, you could decide which tokens to pick based on which tokens are relying the most on that part of the prompt where you're defining your constraint.
Sarti, Gabriele: Yeah, it's interesting. I would have some criticism about this, probably, but I think it's potentially promising.
Sarti, Gabriele: And finally, one thing that I wanted to highlight specifically because, it feels like in the mechanistic community, people think, oh, you know, attribution is a thing for the past, and now, you know, we don't do this anymore.
Sarti, Gabriele: We did that on images, you know, in the 2010s, but actually, all the new methods that do circuit finding, so I think you have a class, David, in the upcoming weeks about that, right?
Sarti, Gabriele: But all these kind of methods that aim to find, you know, how components interact towards a prediction are using, effectively, some form of attribution, right? Most of them now are gradient-based, actually, some form of integrated gradients, or…
Sarti, Gabriele: So I think these questions are still, you know, very, very recent and very important. It's just maybe they kind of became so…
Sarti, Gabriele: consolidated, that now they… they moved away from the… from the spotlight, kind of, and now they're just the kind of tools that we use without even thinking about it, which is great, I guess.
Sarti, Gabriele: Yeah.
David Bau: It was still salient.
Sarti, Gabriele: Exactly, still important to know what's salient, yeah.
Sarti, Gabriele: So, yeah, so I think that's it for me. Thank you so much for having me, and if you have any questions, I'm here to answer, of course.
David Bau: Thanks, Gabrielle.
David Bau: It's really, really, really helpful.
Sarti, Gabriele: Great.
Sarti, Gabriele: I'm glad.
David Bau: Yay! Everybody liked it.
David Bau: So, yeah, so it's… so it's actually… so one, you know, it's… you will sort of keep up with the theme of… of what we're doing. I, you know, encourage everybody to give a try.
David Bau: to, you know, these input attribution methods, there's… there's, as you can see, there's a lot. Yes, I agree with what Jasmine says. It feels particularly broadly useful in situations with high-stakes decisions.
David Bau: You know, my intuition is a lot of these interdisciplinary questions that you guys are asking, where you have a lot of organic text, and and you're asking, how does a model thinking during… during processing of complex text.
David Bau: I feel like these methods are really well suited for it, which is why I wanted to make sure we cover it before spring break. And so,
David Bau: So yeah, so I think it'll be great. I'll be interested to see if you're able to find anything interesting in your projects.
David Bau: Using these methods. And we have, we have Gabriel here this semester, so… Right. So, you know, so take advantage of him. He's, he's directly helping out on one of the teams, but, but, you know, he's a general resource for the class.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Yeah, I'll also share the links, of course, to this library that we built in the Discord channel, so that you can have a look.
David Bau: Great.
David Bau: Okay, guys.
David Bau: Stay safe in the snow out there.
Sarti, Gabriele: B.
David Bau: And we'll see you… I'm not sure if we'll see you in person on Thursday or not, hopefully the weather will cooperate, and we'll see you in person on Thursday.
Sarti, Gabriele: Thank you, bye-bye.
Armita Kazeminajafabadi: Thank you.