Sarti, Gabriele: And so, we're lucky to have… Gabriel, talking about…
Sarti, Gabriele: Oh, it's… I didn't actually know what the topic was gonna be, but,
Sarti, Gabriele: But it looks like you're going to be…
Sarti, Gabriele: doing LLM agent, discussions today, so… Yeah, a lot of different things, actually, so… Oh, okay, that's, that's great. So, Gabriel's, you know, he, he recently finished his
Sarti, Gabriele: PhD, where he focused on
Sarti, Gabriele: interpretability, and he's had a lot of focus, not just on, interpretability, Methods… But also, interpretability…
Sarti, Gabriele: tools, And the interpretability community at large, has been… has been, you know, have been things that,
Sarti, Gabriele: have been part of his practice, not just the academic output itself, and so we're really lucky to have Gabriel
Sarti, Gabriele: working with us on Endiph, Abbas.
Sarti, Gabriele: I'll build the, interpretability community infrastructure.
Sarti, Gabriele: This way.
Sarti, Gabriele: And so, so, okay, so welcome, Gabriel. Great, thanks. Thanks, David. Thanks, everyone, for having me. Yeah, so the presentation for today, it's actually a bit of, a lot of things packed into a single presentation, but I want to just give you a quick overview first. Like, as David said, I recently graduated from the PhD. I was in the Netherlands working on this consortium that focused on explainability for
Sarti, Gabriele: Text, speech, and actually even music models.
Sarti, Gabriele: And my thesis was focused on interpretability for machine translation, and as you see from the title, there's this actionable keyword there. So my core focus wasn't really to try to, I don't know, develop new theoretical understanding of models, but rather, how can we bring the advances that we currently have in the field of interpretability
Sarti, Gabriele: to users of machine translation models in a way that, like, they can use this information to make more informed choices. So a lot of my work was, for example, working with human post editors that could access uncertain information from model internals and decide whether it makes sense to correct, for example, some translation.
Sarti, Gabriele: So… and right now, I'm here in David's lab, and I'm working with the NDIF project, and the idea here, my personal focus, is more related to building tools and interface to support
Sarti, Gabriele: workflows towards frontier LLMs, so kind of, like, trying to scale a bit interpretability, which is also a bit the focus that I try to give to this presentation.
Sarti, Gabriele: So…
Sarti, Gabriele: I would want to start with some conceptual framing here. So I think everyone here knows about Evolves, LLM Evolves, and LLM interpretability, right? So I would argue that these are kind of, like, two different words that don't talk much these days, right? And it's kind of a pity, because both are concerned with model understanding, right? So on the one hand, you have evaluations.
Sarti, Gabriele: that are mostly behavioral, right? You're running your models, you're getting black box input-output behavior, and this allows you to be very… to scale very well on a lot of data, so you can test over plenty of data sets, but it's quite shallow, right? Because the moment that you flag the behavior,
Sarti, Gabriele: you don't quite understand what is motivating this behavior, right? Is it that the model is misbehaving on this specific example, or on a category of example? Which components are responsible for that, right? You cannot know just black box.
Sarti, Gabriele: On the other hand, interpretability is great, because you can really probe deeper, right, into the model, understand which components are responsible for different stuff.
Sarti, Gabriele: But the problem is that, as many of you know, like, if you ever try to run a patching experiment, good luck doing this on, like, the setup that black box people run their evaluations on, right? So, in general, I feel like this is an interesting word, because evaluation definitely can motivate interpretability analysis, right? Like.
Sarti, Gabriele: You can pinpoint patterns here, and this can motivate where to focus for interpretability.
Sarti, Gabriele: And in the other… on the other hand, you can use interpretability findings to justify why
Sarti, Gabriele: some evaluation results were, like, the ones observed, right? So I think the main problem here is that there is a tooling gap between these two words, right? There is, like, this issue of, like, once that I have my evals.
Sarti, Gabriele: how do I move to the interpretability side? So I think that's the core focus that I want to try to address. And that connects a bit with my previous research, right?
Sarti, Gabriele: So just to give you a quick overview of today's presentation, the first part of the presentation is about scaling context understanding. So, a lot of my previous work was about attribution, and a lot of my concern was actually about how do we make attribution more scalable for language model settings, right?
Sarti, Gabriele: So for this kind of autoregressive prediction settings. So in this context, I had, this PECore framework, plausibility Evaluation for Context Reliance,
Sarti, Gabriele: And its extension applied to retrieval augmented generation for citations. So the core here was
Sarti, Gabriele: Faithful citations that could outperform self-citation from the model just by looking at its internals, how it uses information.
Sarti, Gabriele: Then the second part is indeed about agents, and it's about how can we study agents, not only from a behavioral perspective, but from a representational perspective. And here, I will discuss briefly our recent work about,
Sarti, Gabriele: about studying goal-directedness in a toy setup for LLM agents. So this is a bit rough. There's still rough edges here, so your feedback will be super useful.
Sarti, Gabriele: And the very final slides are actually
Sarti, Gabriele: This is a bit annoying, but, about bridging the tooling gap, and it's about what we are thinking about here in the NDIF context, so trying to… how to… how do we scale the NDIF ecosystem to kind of bridge this tooling gap that I discussed earlier?
Sarti, Gabriele: Any question on that? Any part that you find more interesting? So that, just to understand how to allocate time, maybe we're gonna be out of time.
Sarti, Gabriele: Who's interested in number one, and who's interested in number two, and who's interested in number three? Number one, people? Number one?
Sarti, Gabriele: Number two? Killing attribution. Number two? Alright, so we're gonna talk about agents. All right. Number three is very short, so… Okay, I'll be quicker on the first part, then.
Sarti, Gabriele: So…
Sarti, Gabriele: just to give you a very brief overview of gradient-based attribution, I think a lot of you already saw this slide. So you start with some inputs, this is passed through the language model, you get a distribution over probabilities, and then to do gradient-based attribution, what you normally do is you select a target, in this case, for example, the probability of the output.
Sarti, Gabriele: And you propagate with respect to this target to get
Sarti, Gabriele: gradient vectors, right? So these have the same shape as the original embeddings, and one way to get a score per token is to simply aggregate these at the token level. So, for example, the L2 norm of the gradient vector becomes a single score that tells you how important this token
Sarti, Gabriele: was with respect to the prediction, right? So this was the famous saliency map that a lot of people in interpretability are now saying that it's a dead direction, right? It doesn't exist anymore, and since the rise of mechanistic interpretability, saliency maps are out of fashion, right?
Sarti, Gabriele: And yet, that's what everyone heard about in industry. Exactly. And all they do.
Sarti, Gabriele: Well, apart from that, I would argue, even in Macinturp, we still use saliency a lot, right? All the circuit finding work is using some notion of saliency to detect edges, right? Like, edge attribution patching is saliency, effectively.
Sarti, Gabriele: To find CLT circuits, you still use Salette.
Sarti, Gabriele: So…
Sarti, Gabriele: So, back when I was starting my PhD in the pre-ChatGPT era, we were… I decided to start building some tools for doing
Sarti, Gabriele: easy attribution on language models. So this in-seq tool was simply a wrapper of fogging phase that would allow you to do very easily attribution on generation, right? So, the problem that I stumbled on very quickly is that
Sarti, Gabriele: if you start working on realistic settings, like reasoning models or, you know, retrieval augmented generation, you get this kind of monstrosities here, right? So what you see here is a saliency heatmap of generated token by input tokens.
Sarti, Gabriele: that you can see here there is a diagonal, meaning there's new tokens that get added to the generation, right? So it's autoregressive. And then every one of these little scores is basically the saliency of an input token towards the prediction of an output token, right?
Sarti, Gabriele: So the problem is that if you want to do this kind of thing, you have to do attribution at every step, right? You have to do… for the next token, I attribute all previous tokens, right? So that I get these kind of pairs. But then, you can imagine that this is one backward pass per step, right?
Sarti, Gabriele: At the token level.
Sarti, Gabriele: So, super expensive, right? So this scales very poorly with the kind of, with the kind of setup that we have.
Sarti, Gabriele: So, what do we actually care about, right? I think it's interesting, this image, because it shows you that there is some information here, right? It's not a very uniform map.
Sarti, Gabriele: But how do we know that we care about these things, right, in the first place? Can we just scope our analysis to these things to make it more efficient and scalable, right?
Sarti, Gabriele: So…
Sarti, Gabriele: now I want to give you an example of how we do the evaluation in the context of attribution. So, you might have seen some datasets in this context that have this kind of
Sarti, Gabriele: assumptions about some hypotheses, right? So, if you're interested about gender bias, you might have this kind of examples, where you have a text that is, after she finished working, the pretty manager went home because she was tired, right? Then you might have some human labels here.
Sarti, Gabriele: That's say, well.
Sarti, Gabriele: this token here, the token of interest, was predicted because of the presence of the same pronoun before, right? So, she motivates she. This is human judgment, right?
Sarti, Gabriele: And then you could… you could use this dataset to test a language model with attribution, and see which tokens are salient, right? So maybe the language model will say, actually, it's the pretty token in this… in this example that motivates the prediction of she, right?
Sarti, Gabriele: So I think this is a cool example, because it also shows this pattern of right for the wrong reasons, right? So the model would predict Xi correctly, and you would never know that it's doing it for the wrong reason unless you were to look at the token that it's using, right?
Sarti, Gabriele: So, again, this is an example of how we do this plausibility evaluation with human labels.
Sarti, Gabriele: But, the concern here is that this is hypothesis-based, right? So I had to build a dataset of examples like this one, where I have this token labeled as my target that I want to explain, right?
Sarti, Gabriele: useful to narrow down specific phenomena, but how do I scale this in a data-driven way, right? So the problem is that you cannot find these plausible or implausible relations in the wild just running your model, right?
Sarti, Gabriele: So, this was our initial motivation to try to scale this in this scenario with this framework that we call plausibility Evaluation of Context Reliance. Now, I will give you a brief example of how it works. So, let's say that we have text like this.
Sarti, Gabriele: So, I moved the table, it was very heavy. And we say that this first example is the context, right? I moved the table.
Sarti, Gabriele: So, If I were to translate this to a language that has gender, for example, French or Italian.
Sarti, Gabriele: this pronoun is ambiguous, right? The model wouldn't know how to translate it unless it had access to the source.
Sarti, Gabriele: So…
Sarti, Gabriele: In this case, we would have a translation like this. In French, j'ai de place la table, elletre lurge. Okay.
Sarti, Gabriele: So, how do we know
Sarti, Gabriele: what is context-dependent here, right? So, one way, with our method, and then I will show a bit better how it works in practice, is that
Sarti, Gabriele: what we care for is detecting which pro- which tokens in the target generation are context-sensitive, right? One way to do this is to measure, for example, the gap in probability, or the KL divergence between the full distribution between the version with context and without context, right?
Sarti, Gabriele: So this tells you this token is context-sensitive. Then from there, we want to do attribution back to contextual queues, right? So then, the cool part is that this retrieves these kind of pairs that are influential-influenced, right? Context and target.
Sarti, Gabriele: So… Now let's see an example in factual QA for that.
Sarti, Gabriele: So, let's see, we have this query that is, how old is the Panaku Kirk in Groningen? So, this place doesn't exist, it means Pancake Church, okay? It's a fictional place.
Sarti, Gabriele: So if you prompt Zephyr7B with that, the model would say some answer, and then would say, actually, you're asking me a wrong question. This question is, like, doesn't make sense, right? This place doesn't exist.
Sarti, Gabriele: So…
Sarti, Gabriele: What if we are to add this context? So here, I'm just adding a small system prompt, in which I ask the model, provide me with a concise answer containing only a few words, and then I also give it a bit of a fictional context that I made up, right? In the heart of Groningen, nestled between Queen Cobble Street, etc.
Sarti, Gabriele: So now, the response of the model is gonna match what it sees in the context, right? As expected.
Sarti, Gabriele: So…
Sarti, Gabriele: that the church is 143 years old, built in 1877. So what Pecore… so if we apply Pecore to this example, what we get is that 143 is context-sensitive.
Sarti, Gabriele: But not all the rest, note. Which makes sense, right? So, the question is already enough to lead to an answer that is in this form, up to the date, right?
Sarti, Gabriele: But then the date?
Sarti, Gabriele: is dependent both on the original date, written in the context, but also on the fact that we mentioned few words. So I think this is a cool example of, like, how this can lead to data-driven discovery of dependencies, right? Because you could have imagined this, right? You could have guessed that there is a link between the dates.
Sarti, Gabriele: But I would have not guessed that this would be relied to a few words here, right?
Sarti, Gabriele: So the fact that the model is looking a few words there, it means that probably without the context, it could have come up with something like, he's an historical building, blah blah blah, right? It could have gone on. But actually now, it's just giving the answer, right?
Sarti, Gabriele: And another interesting thing is, then, also 1877 in the answer is context-sensitive, mapping again to 1877 in the context, right? So this might look
Sarti, Gabriele: quite, natural, right? It's just the token looking at the same token in the context. But actually, there's an interesting assumption here, which is this token is not
Sarti, Gabriele: is not being computed by looking at this date and working backwards, but it's just copied by the context, right? So it might be that these two things are mismatching, just because the model is referring to the context twice, right?
Sarti, Gabriele: So…
Sarti, Gabriele: Of course, this is just attributional, right? So there is no causality here, but it's great to come up with hypotheses for causality testing, right? You can now test whether patching, yeah.
Sarti, Gabriele: Yeah, true question. Yep.
Sarti, Gabriele: Did you try this method on… on… sort of more at USA.
Sarti, Gabriele: situation where you have more idea of the glasses, like, if you have, like, a hidden magnets.
Sarti, Gabriele: Also, there's food, and there is some work that will be done.
Sarti, Gabriele: trying to elicit, like, suppressed knowledge in Chinese-only models. That was a strange out of them.
Sarti, Gabriele: So you think it's, like, opportunities for most kinds of scenarios?
Sarti, Gabriele: Yeah, I think so. I think there is. We haven't tried. The only thing that we tried was in a case, for the… for the example that I'm showing just here about retrieval augmented generation citations.
Sarti, Gabriele: So normal methods that create these kind of citations use superficial similarity, right? Like, I take the similarity of the paragraph with the similarity of the answer, and… and if it's high, I just say, this is a citation, right? But if your model has a backdoor, and maybe it's just mentioning a word all the time if there is a trigger, right? Similarity would never pick this out, right?
Sarti, Gabriele: But attribution would.
Sarti, Gabriele: So I think that's a good motivation, right? You can actually get a more faithful sense of what matters towards the generation.
Sarti, Gabriele: So, to give you a more better intuition, this was a follow-up in which we applied the same framework in the retrieval augmented generation setting. So, the setup is the same. We have a version without context, a version with context, and the context here is three documents that are retrieved dynamically at inference time.
Sarti, Gabriele: And are used to let the model come up with a better answer, right?
Sarti, Gabriele: So the way that, PCORI works in this case is
Sarti, Gabriele: First, we find the influence generated token, like we saw before, and this is done by looking at the distribution for every token that is generated with the context and without the context, and see in which position this gap is larger, right? So then, in this case, for example, we would flag smaller as
Sarti, Gabriele: context sensitive, right? Once that we have that, we can do… we can do attribution to trace it back to influence token… to influential token, sorry. So, I think the interesting part about this second side that I didn't show.
Sarti, Gabriele: Is that the way that we do this is actually by using the gradient of the difference in probabilities between the two options.
Sarti, Gabriele: So, like, the cool part of this is that, let's say that smaller was predicted with context, right?
Sarti, Gabriele: But PC would be the top prediction without context, right?
Sarti, Gabriele: Basically, we are mining a pair of, like, valid answer with without context, which we can use for doing this contrastive attribution there, with gradients, that is not just measuring what is important towards the smaller
Sarti, Gabriele: But what is important to shift the distribution towards smaller from PC, which is the original choice, right? So this makes it a bit more robust in its application, right?
Sarti, Gabriele: I don't know if it's clear, yeah?
Sarti, Gabriele: For the second one. Yep. Is this…
Sarti, Gabriele: It's just, like, a pipeline, like, first you…
Sarti, Gabriele: find the documents that are important, and then from there, you use gradient individual one, or do you just do, like, gradient attribution over all the documents? So the cool part is that once that you have all these tokens that are context-sensitive, one by one, you can just
Sarti, Gabriele: do one backward, get all the saliency over all the tokens of all the documents, right? So it's… it's quite efficient, actually, because this number of tokens is usually smaller, right?
Sarti, Gabriele: Yeah, that's… It is with all documents in context? Yeah. That's right.
Sarti, Gabriele: So this… maybe there were some recent work that we're trying to approximate… to do some similar stuff with ablations, right? There was some work from MIT, I think, where people were trying to do this kind of LIME estimator for importance of different sentences. Yeah, Koyanna?
Sarti, Gabriele: So, we just propagate the gradient from the prediction to the input embeddings. So it's all the way, kind of, yeah.
Sarti, Gabriele: Yep. This is precluding the last part of your talk, I think, but if you have a very long context, like you often have in a rag scenario.
Sarti, Gabriele: just doing backlog can be very hard. That's right.
Sarti, Gabriele: Yeah, so this method is quite brittle in terms of which attribution method you're using, actually, right? So, the better… like, the cool part is that it's flexible as a framework, in the sense that it's plug-and-play with whatever attribution you want to have, right? So, in the original study, we were using rogue gradients, and that was pretty bad, actually, in the sense that it's… it's a very toy scenario that works in small context.
Sarti, Gabriele: But the moment that the context becomes large, then it becomes very diffused, right? And you lose a lot of information, right?
Sarti, Gabriele: So there's new methods, gradient-based still, that would support this kind of pipeline. For example, this Attention LRP, or, like, GIM that we're using for a later slide, that are much better at kind of, like, narrowing down which tokens are important in a more robust way, yeah.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: So that's talking about how good they are. I'm just saying that even efficient… efficiently calculating gradients for a very long context… Right. …is very hard, like… Yeah. We ran into this problem in a project with
Sarti, Gabriele: Pfizer, they have a system, a RAGQA system on
Sarti, Gabriele: like, thousands of documents or something. So, just doing backlog…
Sarti, Gabriele: We couldn't… so we couldn't do that on our machine.
Sarti, Gabriele: Yeah, so one solution for that could be to try this kind of forward-only attributions, like this kind of information flow routes, but they are definitely not output-based, right? They just rely on local similarities, so it's not very precise, but
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Nice.
Sarti, Gabriele: And I just wanted to give you a very quick overview of what it looks like. So, in this plot, you have the five documents that are retrieved and appended to the context, and every dot in this plot is one token within the document, right? And the scores here are the attribution scores that you get by applying the second part of the graph that we saw before, right?
Sarti, Gabriele: So, in the study, we were trying two approaches. We were trying to set some thresholds based on, like, arbitrary thresholds like this, like the top 5%, or we were trying to do some calibration based on a set of gold examples with human annotations, right?
Sarti, Gabriele: And what we were showing is that it seems like this approach,
Sarti, Gabriele: Well, we found contrasting a bit results on different models. It seemed to be working quite well for Zephyr, but not much for Llama in terms of recall, specifically.
Sarti, Gabriele: no, sorry, in terms of precision for Llama. But in general, the cool part is that you can use
Sarti, Gabriele: the same model… so the model that you're using for generation, you can just do the attribution back, right? So it's… it's not out of distribution, you don't need an external validator. And the calibration, we found that it… it did improve, performance here. What is your ground truth for that experiment? So when you do a precision recall measure?
Sarti, Gabriele: So this was manually annotated, human… so it's plausibility, again, like, it's human labels of, like, which context contain which information, right? So that's also a potential gap, right? Here, we're trying to get a model perspective of what matters, right? So evaluating it against the human perspective, it's a bit of a… Assuming the model does it the way the human would do it. Exactly, yeah.
Sarti, Gabriele: Yeah. Can you go back on this slide? Yep.
Sarti, Gabriele: How do you do the finding the influence generation tokens? Is it… you've generated an entire answer without context? With context.
Sarti, Gabriele: With and without? Without, we don't really need. So the ones that you have the one with context, you want to look for every token.
Sarti, Gabriele: which ones have the highest KL divergence, basically. Between? Between the width and the without. So the without setting is you take the width answer, you force it, basically. You force it there, and you just measure the probabilities for every token. Yeah, without context. Without context. Exactly.
Sarti, Gabriele: So, like, by then, you already have generated them.
Sarti, Gabriele: Let's say, 50 tokens with the context.
Sarti, Gabriele: Don't you think that would sort of influence? For sure, for sure. So the idea here is really to try to trace how the context motivates the answer. Then I agree that there's going to be starting to have dependency within the answer, right? So our goal was really to pinpoint the, like.
Sarti, Gabriele: which kind of information is flowing from context to answer. The moment that it's internal to the answer, we weren't
Sarti, Gabriele: looking at that, right? But in principle, nothing would prevent you from adding the prefix of the answer to the context here, like, autoregressively, right? And keeping it in the loop, kind of. So you could… you could do that, yeah.
Sarti, Gabriele: Yep.
Sarti, Gabriele: I think in general, Zephyr here was a bit better as a model compared to Lama 2, so it could be a factor here. You can see the scores are a bit higher overall.
Sarti, Gabriele: Performance, like, not in terms of attribution, but it was reflected somehow in the attribution sites.
Sarti, Gabriele: amanda.
Sarti, Gabriele: Yeah, so Zephyr is a mistral. It could have been, but we didn't look into that very much, yeah.
Sarti, Gabriele: Yep. Alitatively, did you see anything interesting specifically?
Sarti, Gabriele: And differences between the two models.
Sarti, Gabriele: I have to say, it wasn't super different. We had some examples in the appendix of the paper that were quite interesting, that it seemed like they behaved a bit differently, but it wasn't super, super remarkable, I have to say, yeah.
Sarti, Gabriele: Yeah? Just a quick clarification. So, in terms of the unit for the position, are you called the unit documents or tokens? Tokens.
Sarti, Gabriele: Yep. Yeah, yeah.
Sarti, Gabriele: Alright.
Sarti, Gabriele: And I just wanted to… to close this part. We have the API within NSYC to do exactly this. So, this example, it's… it's quite interesting, I think. So, if you have
Sarti, Gabriele: Here I was prompting the model to asking it, when was the most successful player in NBA history born? This is a pretty small model, it's a small LM2, like.
Sarti, Gabriele: 200 million parameters, so super small. So the model is giving something nonsensical here, which makes sense. But then it was saying the most successful player in NBA history is Steven John.
Sarti, Gabriele: something, and John was found to be contact sensitive, right? So then, if you open the little toggle here, what you see is,
Sarti, Gabriele: the retrieved documents with RUG, right? And then you see the scores for all the tokens within the retrieved documents. So you can see that for John.
Sarti, Gabriele: the model is mostly looking at Steve and John here, right? In the third document, which makes sense, so it's kind of like, it's copying from the third document to mention that, right?
Sarti, Gabriele: But I thought it was interesting, because you can also see
Sarti, Gabriele: The contrastive alternative that the model would have produced without context, right?
Sarti, Gabriele: And if you know basketball a little bit, Stephen Curry could be a good choice here for saying the most successful player, right? So the model kind of knew the answer before
Sarti, Gabriele: Getting the context, but then it was biased by the context to give a wrong answer.
Sarti, Gabriele: So on.
Sarti, Gabriele: Yeah, I thought it was an interesting example.
Sarti, Gabriele: So, as a bridge between the first and the second part, there is some work that we're doing about using attribution to compress COT reasoning, with some collaborators from Italy. And so the idea here
Sarti, Gabriele: is that we have our original COT trace in a reasoning model, so now we're trying mostly GPTOS 20B, and you have this very performant, gradient-based attribution methods that can identify, given the answer that the model produces, which are the tokens that motivate this answer the most within the reasoning chain, right?
Sarti, Gabriele: So, what you can do is that you can score this token and then use some sort of thresholding, again, to select only the ones that are the most salient towards the answer. So, our idea here was, cool, this lets you do this post-doc, but what if we train a probe?
Sarti, Gabriele: to predict these tokens, given an unseen cut trace, right? So, given an unseen trace, what we want is the probe will pick out the tokens that are found to be salient by our attribution method, right? But live, so not post-doc, just…
Sarti, Gabriele: given a token, we… we decide whether it's… it's salient towards the final prediction or not, right? And…
Sarti, Gabriele: So, one initial experiment with patching that we had here was, given the full COT trace, we select the relevant token with attribution, and we just take the token identities here, so the actual token, and we plug it into our new
Sarti, Gabriele: COT, like, reasoning block that only has these tokens, right? So this is not gonna make any sense, because it's just random tokens, right, that were picked out. But then, what we also do.
Sarti, Gabriele: is that we patch the states corresponding to these tokens into these positions, right? So these are not just the tokens, but are actually the patched activations across all layers.
Sarti, Gabriele: And what we were seeing here is that in most cases, the model can get to the final response.
Sarti, Gabriele: Which, I don't know about you, but I personally found kind of surprising, but I'm curious to hear your opinion, yeah?
Sarti, Gabriele: The answer is.
Sarti, Gabriele: That's right.
Sarti, Gabriele: Bye.
Sarti, Gabriele: That's right, and that's why I say offline is biased, actually, in the title. So, the problem there is that you could have this kind of shortcut that we saw in many examples, that one of the salient tokens is the answer in the reasoning chain. And it's kind of hard to control for that, because we tried stopping right before that, so trying to ablate everything from the first occurrence, but then it would be something like.
Sarti, Gabriele: So, 23 plus 57 is…
Sarti, Gabriele: You know, so you're selecting that instead, and it's still trivial to get to the answer, right? Without doing the full reasoning.
Sarti, Gabriele: So then, the live thing that we're trying to do, and that seems to be working well, is to use this train probe on the COT to select, as a decoding strategy for sampling.
Sarti, Gabriele: So, at generation time, you can consider a prefix of the COT, then from this prefix.
Sarti, Gabriele: You can sample, for example, 4 options, and you can apply the probe to the tokens of these four options to detect where there is a higher density of tokens that are found salient towards the final answer, right?
Sarti, Gabriele: And then, this guides your selection of the next sequence, right? So, I would pick the first one as the continuation, keep going, and
Sarti, Gabriele: what we're seeing so far, we still have to kind of, like, finish all our evaluations for that, but it seems like we can, reach, basically, a very high, like, agreement with, let's say, the full COT with much shorter sequences, or something alongside these lines, let's say.
Sarti, Gabriele: So this is a bit of work in progress, but I wanted to show you. Any question on that? Yeah? Yes, I'm curious, but I had a thought, but, like, how would you avoid it starting to just repeat tokens over?
Sarti, Gabriele: Yeah, it's fair.
Sarti, Gabriele: What do you mean? To repeat tokens? Like, say, imagining this in, like, a… I think…
Sarti, Gabriele: It's an art problem, and…
Sarti, Gabriele: you know, if it says, like, 33 plus 44, then it would be salient token changes. Yeah. Theory, you could just say 33 plus 44 repeatedly.
Sarti, Gabriele: That's a way to, like, show up more.
Sarti, Gabriele: Yeah, but then we still measure the final capacity of outputting the answer, right?
Sarti, Gabriele: So, what do you mean is that it could just keep going, and we would never finish the reasoning, or something like that?
Sarti, Gabriele: No, it makes sense. Yeah, I mean, definitely there should be some stopping condition, right? Otherwise, it could be, like, that you enter this kind of loop of just outputting only this kind of final tokens that are always salient, right? So, we're still thinking about that, yeah.
Sarti, Gabriele: So… The fact that, like, Mina can train robes.
Sarti, Gabriele: To me, it seems to imply that, like, the model is…
Sarti, Gabriele: like, ahead of time, like, kind of writing in, like, a late scratch pad, like, oh, I know I'll need this later, I'll use this later for the album.
Sarti, Gabriele: And sort of… I guess, in this way of, like, choosing among the best of four branches.
Sarti, Gabriele: Optimizing for that signal.
Sarti, Gabriele: you know, what I'm gonna need later.
Sarti, Gabriele: results in… in… I guess denser COT, so…
Sarti, Gabriele: I just want to get, like, an understanding, like, is this sort of the intuition? Yeah, I think the intuition is that there's gonna be tokens that the model knows are relevant for later, kind of. And I think that's the signal that is
Sarti, Gabriele: kept, while other tokens are kind of, like, a way to… to get more compute, kind of, like, you know, this kind of, oh, wait, let me think through this, right? This is not, like, answer-directed, it's more like, I need more computation to get to something meaningful, let me just kind of stall, you know, for a bit. So, yeah.
Sarti, Gabriele: Oh, yeah. Just, yeah, quick follow-up. Do you think you'd see a trend in this direction? If you, like, for example, you looked at checkpoints of, like, the formal reasoning, or, like, the RL stage, do you think you'd see, like, it start to generate more of these, like, highly dense CFEs?
Sarti, Gabriele: That's a good question. I'm not sure I have an assumption here. Like, I don't know what to expect on… about looking at this specific model, yeah.
Sarti, Gabriele: Yep.
Sarti, Gabriele: how I'm interpreting this effect is that the model is doing… Some kind of confession.
Sarti, Gabriele: previously intermediate tokens, that it needs to maybe, compact information from the token, from the context, and also, like, some information that is going to be relevant related.
Sarti, Gabriele: But isn't that it's this exact thing that the model needs to do that?
Sarti, Gabriele: Makes the first brain not to be prevented much harder.
Sarti, Gabriele: Because, I mean, let's say the model has some context about few smaller indigenous rack documents, it decided to contact this stuff into some token.
Sarti, Gabriele: that if you just look at the token itself, it is just… has nothing to do with… Yeah, yeah, yeah. And then, even when you do this gradient attribution and stuff like that.
Sarti, Gabriele: then that… Token that is apparently not related to the concept you're looking for.
Sarti, Gabriele: it will also get boosted, because it is also storing some information. Right. I think conceptually, yes, that would be the case. My observation, though, is that this kind of compaction doesn't happen usually on, like.
Sarti, Gabriele: end-of-sequence tokens, right? So there were all this previous work, for example, this Thought Anchors work by Neil, by some people at MATS, right, I think, that were using end-of-sequence within the COT to split the sentences, right?
Sarti, Gabriele: And we tried to look at that, so when we were thinking about compaction, we were thinking, can we just bound the sequence based on the sequence boundary, right?
Sarti, Gabriele: But actually, it seems like there is not… this kind of thing that the information is gathered at the end, right? Can be used for reading, but something that we're also seeing in the goal-directedness part is that it cannot really be used causally, right? Meaning, like, information is moved there.
Sarti, Gabriele: But it's a bit diffused, kind of, right? So the attribution is more likely to pick out the original source of the information, rather than this final part.
Sarti, Gabriele: At least that's what we observe right now, that we're digging a bit into the results, right?
Sarti, Gabriele: It doesn't need to be, but I think it makes sense
Sarti, Gabriele: that, like, the gathering happens when you need to predict something, right? Kind of like, if you think in this case, if you have to do an intermediate prediction, then there is the point where the attention is looking backwards and is trying to gather this information, not like at the end of the sentence, right?
Sarti, Gabriele: So then this step would anyways be part of this kind of salient elements, right, that get picked out.
Sarti, Gabriele: So then, I feel like it mitigates a bit the problem that you mentioned.
Sarti, Gabriele: Yep.
Sarti, Gabriele: Yeah, something like that. It's kind of like,
Sarti, Gabriele: beam search with a probe that controlled for how much this is relevant to the final outcome.
Sarti, Gabriele: Yeah. Bye.
Sarti, Gabriele: be irrelevant.
Sarti, Gabriele: Excellent.
Sarti, Gabriele: So, we did look at entropy, actually, of the distribution across tokens as a signal initially.
Sarti, Gabriele: to try to understand, because there was some work that was using entropy as a signal of, like, oh, the model is more uncertain, so maybe that's here where it takes the decision, right? And in mathematical problems, we found that low entropy on digits was
Sarti, Gabriele: kind of related to the final outcome, so digits were salient, but apart from that, we didn't find that entropy was very predictive of that. That's why we moved to attribution as a way to get a more output-oriented signal of what matters, right?
Sarti, Gabriele: So yeah, the probe does this kind of attribution, but online, not from knowing the answer, right?
Sarti, Gabriele: So, I was also kind of surprised that it works, but it seems to be working quite well. So, we were getting something like
Sarti, Gabriele: 85-90% accuracy to get out these tokens, yeah.
Sarti, Gabriele: So, you do the attribution from the final output. You get scores for all the sequence, but then the probe is trained given the current token in the sequence, so you can regenerate the same sequence, and then you want to predict the score without knowing the final output, right?
Sarti, Gabriele: The scores on which the probe is trained are based on the final output, but the probe is trained only on the local output that gets generated one step at a time, right?
Sarti, Gabriele: That is annotated with those scores that you pre-computed, kind of, right?
Sarti, Gabriele: Do you give… what do you give as input to the probe? So the probe just gets the binary, yeah, it gets the sequence up to this token, and it gets the binary label that is the, you know, this,
Sarti, Gabriele: kind of like, with the threshold, we set whether the token is meaningful or not towards the prediction, yeah. Did you find any patterns, like.
Sarti, Gabriele: in math problems, could it just be, like, all the numbers are relevant? Yeah, so in math problems, our problem was that initially we worked with entropy, and that there were these kind of patterns of low entropy, always salient, but then the moment we moved to, like, MMLU or something like that, this completely broke our analysis, right? So that's why we moved to attribution.
Sarti, Gabriele: And now we started doing some experiments there, it seems more robust, right? So what we want to test is also generalization, kind of like you train the probe on math, you test on MMLU, right?
Sarti, Gabriele: Oh, yep.
Sarti, Gabriele: We have a submission we're preparing for calling that looks very similar, so maybe we can talk. Nice, well, yeah, sure. One thing I can mention, we…
Sarti, Gabriele: We also had the attribution scores, but we added two properties that I think are important and may help with some of your experiments. One is, you want to know if these steps or parts of the train of thoughts are actually plausible.
Sarti, Gabriele: for the model.
Sarti, Gabriele: So, are they high likelihood, given the preface? You mentioned that you sometimes are left with things that are… they don't make sense, right? You mean when you patch?
Sarti, Gabriele: Here. Even before, right? Yeah, but actually here, we just do the coding from the models. Not in the report. We're talking about just identifying… what was it, yeah. I think when you started, you said…
Sarti, Gabriele: oh, these things are… they don't make sense, but if you… if you add the hidden… if you patch the hidden slate… Yeah, yeah, it was for the patching. Yeah, but this is post-doc, right? So… No, no, I understand. But you can inform… you can require that the steps you identify… Right. We called it, attainable.
Sarti, Gabriele: So that they are something the model will assign high probability. Interesting. And the other thing, which is also related to whether the model is just writing the answer or a shortcut to the answer, you want to get rid of those, and you can get rid of those. So you want things that you cannot…
Sarti, Gabriele: predict the answer just from the given step, right? You don't want… Correct. You don't want them to be enough for predicting it, so you can get rid of that. Yeah, yeah, yeah. You want… basically, you want…
Sarti, Gabriele: steps that are sufficient for predicting the answer when the full question and prefix… That's right, but so you're segmenting still at the step level, right? We're segmenting at the step.
Sarti, Gabriele: Makes sense. Yeah, I'd be happy to talk about that.
Sarti, Gabriele: Now, quickly on to the goal-directedness part that a lot of people were interested in.
Sarti, Gabriele: So the setup here, the whole motivation about this is that most of the evaluations about goal-directedness in agents are based on a behavioral study, right? So we study whether the agents can do the task, and you usually use as a metric of
Sarti, Gabriele: goal directness, whether the agent is accurate or efficient with respect to the optimal policy, right? If it takes more or less the same amount of steps, or more or less the same steps.
Sarti, Gabriele: So the problem of this is that this… the model capabilities of getting to the answer are a bit confounder, right? Because if the model is just not able to get to the answer, it might be the most goal-directed, but…
Sarti, Gabriele: It just breaks, right? So our idea for this project was, can't we ground this study into the representation that the model builds about the environment to try to get a better sense from those about the capabilities of the model? Meaning, is the model just failing at representing the environment, and this leads to a failure in taking action?
Sarti, Gabriele: Or is the environment represented well?
Sarti, Gabriele: Right?
Sarti, Gabriele: So, our setup is with a 2D grid word that looks like this, with different densities of obstacles. So, from 0 to 1, that we say, basically, corridors, right?
Sarti, Gabriele: And this grid word is passed as a text to the LLM, and it's represented as you see there. So we put a lot of care to make sure that every token is a single… every cell is a single token. This helps a lot with the analysis on the mechanistic side of things, right?
Sarti, Gabriele: Alright, so this is the kind of prompt that we use. So, the model is given, like, you are controlling an agent, this is your legend, like, GA…
Sarti, Gabriele: the wall. Your objective is to get from your position to the goal while avoiding walls. And at every turn, the model can only say one of four actions. So it's step-by-step. One conversation turn is one step. And this is the expected output format, right?
Sarti, Gabriele: So then we feed in, at every observation, the grid state here. The model has to take a step, then we parse the step, we use the system behind to get to the next observation.
Sarti, Gabriele: So the setup for the evaluation in the behavioral setup is we have 6 densities, from empty to fully corridors.
Sarti, Gabriele: We have 5 sizes, so 7x7 up to 15x15, 10 grids per sizes, and 10 trajectories per grid, right? So overall, 3,000 trajectories.
Sarti, Gabriele: All right. So this is a first view.
Sarti, Gabriele: of, of, like, how different complexity
Sarti, Gabriele: causes the model to fail more. So what you see here is the density on the x-axis.
Sarti, Gabriele: you see the grid sizes here, and you see the distance to the goal. So this distance to the goal is measured step by step.
Sarti, Gabriele: Using the optimal policy, right? So, just to clarify, the grid word is a 2D grid word, fully observable, so you can, at every step, just do the A-star policy to measure what's the minimal path, basically.
Sarti, Gabriele: So you can see here that the more the sizes increase, the more the failures start to happen, basically. But even for cases in which in the original size, so higher densities, for the smaller size, the model would do it perfectly.
Sarti, Gabriele: at higher densities for bigger sizes, the model starts to fail, right? So, in general, we find, behaviorally that
Sarti, Gabriele: Bigger grids causes failures even for setups that before didn't, right?
Sarti, Gabriele: Yep.
Sarti, Gabriele: Defense.
Sarti, Gabriele: Yeah, so you can see that here, there's no… nothing above 10 actions, because the grid is just a 7x7, right?
Sarti, Gabriele: Why is the… Thank you.
Sarti, Gabriele: Because these grids are empty, they don't have obstacles, so the shortest path is quite straight, right? Well, in this case, you might have to go through the corridors, right?
Sarti, Gabriele: Yep.
Sarti, Gabriele: So another thing that we were testing is the robustness to instrumental goals. So, we had 3 setups here.
Sarti, Gabriele: A setup in which the model needs the key to get to the goal via a door.
Sarti, Gabriele: A setup in which there is a key, but there's no door, so the model actually could just disregard the key, doesn't care about the key. And a setup in which the distance of the goal is the same, but there is a key in one path and no key in the other, right?
Sarti, Gabriele: So, what we saw here is that in the first setup.
Sarti, Gabriele: the model has a 100% success rate in this smaller 9x9 grid, so it means that it can always get to the key, pick up the key, get to the door, open it, get to the goal, right? Which is great.
Sarti, Gabriele: And the success rate is also very high for the KeyNodore case, but you can see that in 17% of the cases, in the KeyNodore case, the model still picks up the key, right?
Sarti, Gabriele: And the key has this… this key attractor bias. It's basically 75% of actions are basically the model going slightly towards the key, right? When it shouldn't.
Sarti, Gabriele: Care about that.
Sarti, Gabriele: So, our idea here is about this attractor, right? So the key semantically in games makes sense, right? If there is a key, you probably need it, right, for doing something. So I think it's kind of interesting to see that this leaks into this setup, where the model doesn't have to pick it up, but just does.
Sarti, Gabriele: Yep.
Sarti, Gabriele: I don't distinguish.
Sarti, Gabriele: An agent that just, like, does a random walk.
Sarti, Gabriele: So…
Sarti, Gabriele: Inject that does a random walk, we'll also get a 100% success rate eventually. That's right. And we'll also have 100% key people freaked on the left.
Sarti, Gabriele: Some random…
Sarti, Gabriele: Yeah, I didn't mention this, but the number of maximal step in the trajectory that we allow is dependent on the grid size, and it's something like, I think twice or three times the grid size, so it's something like 27 at most for the 9x9.
Sarti, Gabriele: It's such that a random agent would have very little success. That's right, yeah.
Sarti, Gabriele: And how do you… Yeah.
Sarti, Gabriele: Yeah, so in the prompt, there's another version of this prompt, in which we just mention that this is a key, but we don't say anything about it.
Sarti, Gabriele: In the door case, we say, you need the key to open the door, but in the other case, we just remove that part. We just say, here is a key. Remind me which models you're testing for this? GPTOS to NEB, yeah. There's no training.
Sarti, Gabriele: There's no training, just out of the box, yeah. And are you doing actions based on, like, a gym environment, or is it just… Yeah, so it's… we're using this gym, this… this gym that is normally used for 2D grade words, right? I don't remember how it's called, yeah.
Sarti, Gabriele: It's a fresh prompt. It's stateless, yeah.
Sarti, Gabriele: Give it a fresh prompt. Yeah. Stateless, yeah.
Sarti, Gabriele: Just curious, if you were to say, not to get the key, would it not to get the key?
Sarti, Gabriele: At least we didn't try, but we just wanted to test the ambiguous case, you know, I imagine so, yeah.
Sarti, Gabriele: And also, we saw this bias towards going to the key path, even in the case where it didn't need to, right?
Sarti, Gabriele: Does this bias occur whether or not there was age in the family?
Sarti, Gabriele: training data? No, there is nothing. No. This is just GPTOS. Okay, so you don't know. No, I imagine so.
Sarti, Gabriele: Is there a standard emoji for the key that you're using? Yeah, it's like K or something. If you put a little poop emoji there? That's a good question. I think it would be interesting to try, yeah. I guess what I'm getting at is, is it just that…
Sarti, Gabriele: can do anything that's there, or things it knows from prior games that… Yeah, I mean, our assumption is that the key… like, the model saw games, definitely, right? And if it knows that this is a key, it knows that keys are picked up in games, and so it does it, right? Or maybe it's a visual processing thing.
Sarti, Gabriele: But it's just a K, right, in the map, so…
Sarti, Gabriele: So it shouldn't… lava squares. Yeah, exactly, or like the poop emoji, I think it's a great idea, yeah. Yeah, so in these cases, most of the cases, it's not on the path, and still we see this
Sarti, Gabriele: Key attraction bias, right?
Sarti, Gabriele: In those cases, you don't tell it about the key at all, or doors whatsoever? It knows that there's this symbol is a key. But you don't tell about doors. But in the other one, you tell about doors. Yeah. I wonder if you did tell it about doors, it wouldn't have picked up the key, because you're like, oh, I know what this is for, and there is no… That's a good question, I don't know, yeah.
Sarti, Gabriele: I'd be more convinced of the numbers if you…
Sarti, Gabriele: If you have pairs of grids where the shortest path is… some solution.
Sarti, Gabriele: And then… Yeah, and then you put a key,
Sarti, Gabriele: And then you have key, and then you compare, like, the length of the…
Sarti, Gabriele: Yeah. Yeah, yeah, so it is the case that it increases. It's just, we don't have it in the table here. But it is, like, the model does get out of its way to get to the key, yeah.
Sarti, Gabriele: Yeah, I think, personally, the most exciting thing about this, so here is purely behavioral, right? But I would be very excited to do probing on the key states, and to see, does the model see it as a goal in this case versus in this case, right?
Sarti, Gabriele: Or you should do your attribution.
Sarti, Gabriele: Oh, yeah.
Sarti, Gabriele: That could be, yeah. Yes?
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Yeah, we didn't try with more attractors, but, yeah. Is there a specification where it's, like.
Sarti, Gabriele: Like, is it just, like, here's this… is it, like, do you have, like.
Sarti, Gabriele: Anything encouraging to find the optimal goal? Because…
Sarti, Gabriele: Yeah, so in the… in here, we say you should aim to reach the goal using the least amount of steps, so there is an explicit instruction.
Sarti, Gabriele: So quickly about cognitive map probes. The idea here is that we have our original grid, N…
Sarti, Gabriele: We pass this to the model, again, that needs to pick one of four actions. And what we do is that we take intermediate states of these.
Sarti, Gabriele: So we take the activation at those, and we use it to train a probe to predict every cell in the grid, right? So this is just an MLP probe.
Sarti, Gabriele: that is trained, given the activation, plus an XY coordinate, so 0, 0.
Sarti, Gabriele: This activation 00, predict that this is wall, right?
Sarti, Gabriele: So, we can train at scale, over all the grid sizes, even, using a padding token for the missing positions, right? So for bigger, smaller grids, we'll have this pad
Sarti, Gabriele: to use it for the same probe. So the interesting part is the probe definitely is able to pick up almost 100% probability the grid size, so it knows when to predict path given the activation.
Sarti, Gabriele: And the other interesting thing that we do is that here, we take the final tokens at the end of the prompt, so it's like the template tokens at the end of the prompt.
Sarti, Gabriele: But then we repeat the same thing by training a probe on the final token at the end of reasoning, right? So our idea is, can we compare how the probe performs before and after reasoning?
Sarti, Gabriele: So here, it's the same, right? So, for example, here, in this example, the initial probe represents, well, the environment where the model crashed into the wall, but the final probe actually makes the model think that it could just go straight, right? As you can see from the reconstructions, the probes are not perfect, right?
Sarti, Gabriele: So the way that we build this cognitive map is simply the argmax of the classes, of the cells, so you could have multiple goals in the reconstructed map, right?
Sarti, Gabriele: Hypothetically.
Sarti, Gabriele: And multiple agents, yeah.
Sarti, Gabriele: of the… I see that the green, red family of gold, but how, how did…
Sarti, Gabriele: It became a goal.
Sarti, Gabriele: Like, what did you do?
Sarti, Gabriele: Here.
Sarti, Gabriele: So, what we're doing is just using the probe.
Sarti, Gabriele: Every cell in this map is a prediction of the probe that is given the activation and needs to reconstruct what's the original
Sarti, Gabriele: Identity for that cell.
Sarti, Gabriele: So that's why I'm saying here, the probe is uncertain where the goal is. It kind of gets it, but it's a bit uncertain between these two options. So your ground truth map is the one up here? It's the real one that the model is prompted. And you have a… you have a special, like…
Sarti, Gabriele: Unicode character for that green square. Yeah, it's G. It's just G. That's whatever it is. Yeah. And,
Sarti, Gabriele: And… but then… but then your probe, basically, who covers… he's in a smeary area. Yeah. Is that right? Any thoughts?
Sarti, Gabriele: Yeah, so thank you, Pooja.
Sarti, Gabriele: How did you train Victor?
Sarti, Gabriele: So, we take this activation at a specific layer. The final ones
Sarti, Gabriele: So it's the last three tokens of, like, the template tokens of… at the end of the prompt, and at the end of the reasoning. So these are two separate probes, though. It's, like, pre-reasoning, post-reasoning, right?
Sarti, Gabriele: And we train, let's say, mostly for layer 15. We also try these two other layers, but let's say layer 15, we take all the activations for the final tokens of the prompt, we train the probe to predict every cell identity, given that activation.
Sarti, Gabriele: Does it surprise me that it seems like it's hard for the… of this model, to…
Sarti, Gabriele: To recognize where is the goal, because… You gave this map?
Sarti, Gabriele: Right? In the prompt.
Sarti, Gabriele: Yeah. So… Seems a very simple task.
Sarti, Gabriele: It's just… Yeah, but now I'm not probing the map identity, like…
Sarti, Gabriele: So I'm probing a final… Exactly. Kinda, yeah.
Sarti, Gabriele: Because they have a different, task.
Sarti, Gabriele: Yeah, that's right.
Sarti, Gabriele: And… So the last 3 tokens, as in the… the arrow tokens?
Sarti, Gabriele: No, no, so what I mean is the template tokens, so GPTOS has these tokens that are, like, end, start, assistant, right? Something like this. We just concatenate the activations of those.
Sarti, Gabriele: Yeah. Does the probe always predict the first step correctly?
Sarti, Gabriele: This one? Like, so, like, if the… there's, like, some rough idea of the map, is it, like…
Sarti, Gabriele: the first thing… the first action that you need to take. The immediate neighborhood. Like, is that… Yeah, in general, the neighborhood is quite… it's quite good. The plot that I had here is actually how correct the agent and goal positions are across different sizes. You can see that this decreases, but, like, for
Sarti, Gabriele: for 7, for example, it's quite high. And what… what the plot on the right shows is, like, the Manhattan distance of all the argmax. So, like, for example, let's say that I have this grid, right? Where I have 3 cells that the argmax is a goal.
Sarti, Gabriele: Then I measure how far these are from the original goal, and you can see that even for larger grids where the accuracy is lower, the Manhattan distance is, like, 1.5. So it's a lower accuracy, but it's always, like, within a neighborhood of the real position, right? So it's just more diffused.
Sarti, Gabriele: And this is the performance. So, what you see here, like, the accuracy, the precision, sorry, is very low, right, of the probe, for the agent and the goal.
Sarti, Gabriele: But you can see that the recall is very high, and this is because the model, the probe tends to predict, like, a cloud of points around the real position, so the recall is generally high, but the actual precision, so to pinpoint exactly that spot, is quite low. And you can see that compared to
Sarti, Gabriele: pre- and post-reasoning, there is a drop in performance, especially for, agent and goal positions, right? So this, like, our assumptions based on this, is that the model goes from modeling the environment before reasoning to modeling the next action to take, right?
Sarti, Gabriele: So… The final thing for this is about
Sarti, Gabriele: well, now we have cognitive maps. Does the behavior of the model agree with the cognitive map, right? That's the…
Sarti, Gabriele: The kind of goal that we had in the beginning.
Sarti, Gabriele: And as of now, it's not so clear, actually, the response. So what you see here in the… in the table, you have grid sizes and densities. Then you have the accuracy for an action with respect to the original grid, so this is, like, accuracy of actions given the… the prompt.
Sarti, Gabriele: And this is the accuracy with respect to the cognitive map, right? So the decoded grid.
Sarti, Gabriele: you can see that this is generally lower than this, right? Although the differences even out for bigger grids. So in general, it seems like the actions are more consistent with your original grid than with the decoded one.
Sarti, Gabriele: But we also see that among all the errors.
Sarti, Gabriele: That are made, within the grids.
Sarti, Gabriele: like, there is a good proportion, this is, like, 60% of this part that is an error, that are actually agreeing with the decoded map. So we find, what we say here, high recovery, which is this last metric.
Sarti, Gabriele: it means that wrong actions that the model takes are mostly in agreement with the cognitive map that we decode, right? So this, I mean, it's only still probing, but it's some signal that maybe we can justify a lot of the action, not as a lack of goal-directedness, but actually as a lack of
Sarti, Gabriele: Capability to represent them up well, right?
Sarti, Gabriele: And this is, like, still so much to do about these experiments. So the thing that we really, really would like to do in this line is causal experiments, so right now we started doing some stuff here.
Sarti, Gabriele: In particular, patching the location of agents and goals, of course, to see whether this changes something, but also patching the neighborhoods of these things. Since we saw that the positions are diffused, we want to see if we can patch the neighborhoods. It's a bit, like, related to what,
Sarti, Gabriele: what was presented last week about VLMs, right? So, if you patch the neighborhood, does it actually affect the final prediction?
Sarti, Gabriele: And then to test these cognitive maps in OOD settings, so…
Sarti, Gabriele: I think the last one is especially interesting. So if we have a partially observable setting in which we occlude part of the map, we observe some action in some direction, do we see that, the probe indeed finds that the goal is kinda that way, right, according to the model? Yeah?
Sarti, Gabriele: Oops.
Sarti, Gabriele: Si veso de…
Sarti, Gabriele: cognitive math is generalized into a different task. For example, it has to just ask to repeat the math, which is a simple task that uses,
Sarti, Gabriele: Yeah, you could imagine that this probably will have a much stronger representation of the map, given that it needs to, you know.
Sarti, Gabriele: What I'm curious is whether…
Sarti, Gabriele: the location of the… of the alternative map stay the same, whether it would be at the same tokens in the same layers.
Sarti, Gabriele: Yeah, I think it would probably be different. I think it depends on the task, definitely. At least the sharpness with which things are represented. I would expect it depends on the task, right? For this task, I expect the goal and the agent position to be more sharply represented.
Sarti, Gabriele: Because they are functional to getting to the goal, right?
Sarti, Gabriele: F.
Sarti, Gabriele: Yo.
Sarti, Gabriele: At the end.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: I'm curious, yeah, we should also test that, yeah, definitely.
Sarti, Gabriele: Nice.
Sarti, Gabriele: Yep. You mentioned that.
Sarti, Gabriele: When you're talking about the probes, that immediately they get 100% accuracy.
Sarti, Gabriele: What do you mean by that? No, so the probe don't have 100% accuracy, they have 100% accuracy in detecting the size of the grid. So, like, the padding tokens to the edges of the grid are very reliably predicted, right? Which is not a given, necessarily, right? From the activation.
Sarti, Gabriele: So… So the, the…
Sarti, Gabriele: The cognitive maps that you quoted earlier, it's one pro that takes XY coordinates and these three concatenated embeddings as an input.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: This is one probe for everything. Exactly. So I just want to show you very quickly something fun. So we have a demo for cognitive maps, that is quite nice. So here, let's say that I select pre-reasoning probes.
Sarti, Gabriele: For size 11.
Sarti, Gabriele: And for complexity, 0.6.
Sarti, Gabriele: So I load this grid. So this is my input, right?
Sarti, Gabriele: This is the input to the GPTOS, and I can actually… I have the full trace already run, so I can actually play this, and I can see the agent going around. So, for now it's optimal, then now it's doing something weird.
Sarti, Gabriele: Then now it's getting back on track.
Sarti, Gabriele: Oh, going up, alright. So what actually can we do? Let's say that I reset here. So, this is the first step, right? I can go down here, I have the prompt.
Sarti, Gabriele: So I can see the prompt.
Sarti, Gabriele: And I can see the model output that has all the reasoning, right?
Sarti, Gabriele: So actually here we're storing also the top 5 next tokens, etc.
Sarti, Gabriele: So here it was a pre-reasoning probe, so we have the last few tokens, you see these three tokens, that are the ones where the probe is trained, right? So we can apply the probe here.
Sarti, Gabriele: and see the cognitive map here. So, you can see these are a bit faded. They look like walls, but actually it's 70% empty, 23% walls, so it's more like empty than wall.
Sarti, Gabriele: But the agent position, it's pretty stably encoded here, right?
Sarti, Gabriele: And then we can actually play this.
Sarti, Gabriele: And this is what the cognitive map looks like when the model is moving around.
Sarti, Gabriele: So you can see that the agent position is kinda robust, although it's not very precise, but…
Sarti, Gabriele: You know, you can see that it gets next to the goal, and then it reaches it, right?
Sarti, Gabriele: Live demos are always stressful.
Sarti, Gabriele: But yeah, so we have this kind of, of thing that helped us a lot to form a bit of intuition about how this works.
Sarti, Gabriele: measured in a specific token, so is it also configurable, to see in different tokens? So is the final, three tokens of the prompt in this case? It's always the pre-reasoning, probe is always there.
Sarti, Gabriele: It would be nice if a user could then perform live interventions in this. Yeah, it would be nice.
Sarti, Gabriele: So, I mean, this ties nicely to the last three slides I have about an insight and an interp, but I don't know if you guys are tired, or if you can survive for 3 more slides.
Sarti, Gabriele: So, it's just a very quick overview of, like, what we're thinking about on the endive side, yeah?
Sarti, Gabriele: How about
Sarti, Gabriele: the formal that the information is there, and it's very clearly correctable, but, I think the other question was, like, the
Sarti, Gabriele: Like, it's just one probe doing this. Like, the probe was, like, more powerful, maybe.
Sarti, Gabriele: Or a definition that yields a stronger device.
Sarti, Gabriele: Stronger findings. Yeah, the problem there is always, like, how much is the probe actually doing the task instead of the model, right?
Sarti, Gabriele: As the unattend teaches us.
Sarti, Gabriele: If you could just make this a bit more high-dev, you get some good conclusions.
Sarti, Gabriele: Did you, like, experiment with different ways of providing the map itself and the prompt to the model?
Sarti, Gabriele: So, we thought a bit about making it kind of like an egocentric view. We would expect that to probably work better for, like, the model to actually get to the goal, because it's kind of like… at least it's not gonna hit walls with its head, right? But we… it's kind of hard to go from
Sarti, Gabriele: from the kind of engines that are, like, that allow you to do these kind of observations to an egocentric view, right? So we postponed that part for now, yeah.
Sarti, Gabriele: How was that something?
Sarti, Gabriele: Was that, like, pre-complicated? Yeah, so the traces are all saved. All the probing results, everything is saved.
Sarti, Gabriele: But you can play with it, there's several of them, right? So just have fun. It's on Hugging Face, yeah.
Sarti, Gabriele: Alright, so for the very final things for NDIF, so as you… many of you know, right now we have this beautiful server.
Sarti, Gabriele: that hosts a lot of models, and… that are gonna change soon, right? So we announced recently on the… on the Discord that we're gonna drop the Llama 405B, that we're trying to host Kimi, which is gonna be great, because it's also gonna have the VLM part, which is very useful.
Sarti, Gabriele: And other more modern models, Quin 3.5, and other… other reasoning, models and VLMs. So…
Sarti, Gabriele: As of now, we have NN site, which is the primary way to access, internals, which is quite high… quite a low level, right? So you have a variety of choices, you have to access components internally at a low level.
Sarti, Gabriele: And this is meant mostly for people that actually are building new interpretability methods, so most of you guys, right?
Sarti, Gabriele: And one of the new things that we have here is that we have these new skills that allow you to kind of use and inside more effectively with coding agents. So if you haven't checked this out, this is something nice. You can just load them in Cloud Code or Codex, and it kind of teaches the model to do
Sarti, Gabriele: patching or do logic lens in a bit more natural way, or use VLLM. I think that was one of the things that WooGu was working on, right? Skills for VLLM usage, that could also be relevant.
Sarti, Gabriele: So the newer part of the ecosystem is where I'm mostly focusing for my work. So, as maybe some of you know, there was this library, an interp, from Clement Dumas.
Sarti, Gabriele: that has been recently integrated in our ecosystem, and the idea here is to provide some sort of interface that is a bit more similar to transformer lens to access the models, right? Which means you have this standardized naming across different models, so you just write once the code, you just run your evaluation on several models.
Sarti, Gabriele: Plus several, kind of, like.
Sarti, Gabriele: synthetic sugar to access model internals. So, I think in the future, it's reasonable to imagine that most researchers, like, I don't know, NLP people or computer vision people that are not interp people, will use something like this to do their research, right? Probably not down to the level of an insight.
Sarti, Gabriele: So, here we have a low-level API that is this kind of, like.
Sarti, Gabriele: shallow wrapper around an inside that is just basically syntactic sugars, mostly, to make everything uniform.
Sarti, Gabriele: But on the other hand.
Sarti, Gabriele: what I'm focusing on right now is we want to have also a higher level API that allows you to do very simple, standardized, and efficient method calls. For example, let's say that you want to do patching.
Sarti, Gabriele: You're just gonna call…
Sarti, Gabriele: patching with that model, with this position, it returns you a structured object that it's easy to serialize, to save, you know, to run your analysis on that. So, kind of all baked in.
Sarti, Gabriele: Efficient and all baked in.
Sarti, Gabriele: And this is gonna make it easier to work with Workbench. So, how many of you have tried Workbench? Out of curiosity? One person? Two people? So, Workbench is the UI that we are building for NN site, which back then was only LogiteLens, but recently I was doing patching on Llama 405B.
Sarti, Gabriele: Which I think is pretty nice.
Sarti, Gabriele: So it kind of supports more advanced use cases at the moment.
Sarti, Gabriele: But we're trying to make it even better, and we imagine that Workbench is going to be mostly for researchers, right? So, like, people that maybe are not even in machine learning, but want to prototype in their domain, or educators that want to showcase, right? We used it a lot in David's class also.
Sarti, Gabriele: Yes.
Sarti, Gabriele: I've killed you, I'm assuming. Yes.
Sarti, Gabriele: I have a question.
Sarti, Gabriele: So… Why build higher level cooling?
Sarti, Gabriele: If… research.
Sarti, Gabriele: Code will increasingly be mediated by agents.
Sarti, Gabriele: So on the last slide, you showed these skills. Yeah.
Sarti, Gabriele: say I want a UI for patching. Yeah.
Sarti, Gabriele: Patient, given the safe…
Sarti, Gabriele: Or if I want it to create some kind of experiment, so we need, I should ask you to…
Sarti, Gabriele: run some patching… That's right. They can figure out the answer, lower level details for me.
Sarti, Gabriele: So… Like, in a year from now.
Sarti, Gabriele: Music.
Sarti, Gabriele: these will be useful to researchers. So you're asking specifically about this part, I imagine, right? Well, even that answer, well, but you… would you argue that the standardization is just gonna make everything easier to run, right? You have one script instead of five.
Sarti, Gabriele: Right.
Sarti, Gabriele: Oh, you need across models. Yeah.
Sarti, Gabriele: So that's the low level, kind of.
Sarti, Gabriele: Don't work there, though.
Sarti, Gabriele: But yeah, Workbench, and maybe high-level… Well, Workbench, you would argue that the UI is going to be useful for educators, or, like, people that…
Sarti, Gabriele: That's, like, amazing.
Sarti, Gabriele: Whatever you want to try to ask for.
Sarti, Gabriele: Does this, I guess, only pertain to this, or are you saying, like, why build anything, really?
Sarti, Gabriele: Big question inquiring.
Sarti, Gabriele: Yeah, no, I think this is actually an increasing question in the industry, which is, like.
Sarti, Gabriele: There's been a whole business of, like.
Sarti, Gabriele: building software tools for people. Yeah. And, like, with, like, we're, you know, a bunch of businesses have some business problem that they have, and people are like, oh, we're gonna build a software to solve that.
Sarti, Gabriele: And increasingly, with voting agents, like, a lot of
Sarti, Gabriele: companies are like, oh, like, we don't have to pay them to develop that for us, we can actually
Sarti, Gabriele: Of course, it's all.
Sarti, Gabriele: Actually, I think they're gonna be into the center, right?
Sarti, Gabriele: So actually, it is a general, like, problem in the industry, I think, now, and but if you wanted to teach a curriculum, I guess, with work management across schools, like, why wouldn't you want one single… like, I mean, this is, written with a lot of agents, like, we're just the ones doing it, right?
Sarti, Gabriele: Again, if you wanted to keep the same thing in California as you do here, we want the same UI across the buildings and show that… So the purpose, like, that workbench will be, education will look like,
Sarti, Gabriele: And prototyping, I would say, also. So, kind of like, I… if I had to do very easy patching experiments, where I just want to patch at every layer, see what… what does it do.
Sarti, Gabriele: I am quite comfortable at doing it with drag-and-dropping arrows in Workbench at this point, right? I would rather start there and modify my prompt and retry, right, on a big model on the remote, rather than having to run my code experiment to see the result.
Sarti, Gabriele: I think that's… that's pretty valuable. So, my answer to that is this, probably, and the fact that I think the standardization that is enforced between these two
Sarti, Gabriele: is useful for this final thing, that is, we want to have a gateway to make evals compatible to our interpretability stack, right? And so you need some format here.
Sarti, Gabriele: where you can bring in Evalt's result in a way that can be digested by this part, right? So if you're building custom stuff.
Sarti, Gabriele: There's no way to enforce this standard, right?
Sarti, Gabriele: I think.
Sarti, Gabriele: Can you say more? Yeah, so, like, the vision for that is the final slide, I promise.
Sarti, Gabriele: is the fact that you, like, the vision for how we also are gonna work within Terp is that most likely you want to run
Sarti, Gabriele: your Evolts, or your analysis on a big data set, with an inference-optimized API, right? Like, you run it on Inspect, you run it on Together AI, whatever, you know, you generate all that you have to generate. And then, from there, so you're gonna get some output from this
Sarti, Gabriele: that then can be parsed in some shared format, which is this INIF format that I was mentioning.
Sarti, Gabriele: Where you can identify areas of interest.
Sarti, Gabriele: for deeper investigation, right? So, like, in our case for GoalDirectness, you want to know, you know, what happens at the end of template tokens before and after reasoning, for example, right?
Sarti, Gabriele: So then, I would tag that postdoc after I have my full trades, right?
Sarti, Gabriele: And then from there, what you want to do…
Sarti, Gabriele: okay, you want to apply your Macinturp toolbox on those locations, right? And that's gonna be very efficient to run, right? Because you, like in the case before, you're not running the probe over all the position in the trace, or, like, logic lens over all the position in the trace. You're running it where you need it, right?
Sarti, Gabriele: Is this all supposed to be, like, an automated pipeline, or… I think the vision would be that you go towards something that is very easy to just… I did my Evolves with what the Evolves people are doing, right?
Sarti, Gabriele: then I just have this bridging, and then I… I'm back into the NDIF ecosystem, right? I can run my things in NMTERP high level, I can serialize, I can load it, and I can just do my analysis there, right?
Sarti, Gabriele: So I think the standardization is what makes it worth more… more, like, structure, right?
Sarti, Gabriele: I agree that for prototyping, maybe for very specific stuff at the low level, you just have an insight.
Sarti, Gabriele: And you build your own custom demo there, right?
Sarti, Gabriele: But I think for more complex scenarios, for, like.
Sarti, Gabriele: Full evolves over full data sets, that's probably more robust, right?
Sarti, Gabriele: But I'm happy to be convinced otherwise.
Sarti, Gabriele: Yeah. I don't know.
Sarti, Gabriele: What do I find it.
Sarti, Gabriele: Be hopeful, though, and have a little… Things like those.
Sarti, Gabriele: We'll need to see it.
Sarti, Gabriele: Well, no, no, no, my initial struggles was just, like, building, like…
Sarti, Gabriele: Building things at a high level of abstraction.
Sarti, Gabriele: focus of usability.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: I think that makes sense, yeah.
Sarti, Gabriele: It's standard for what everyone needs.
Sarti, Gabriele: And you can compare research into the
Sarti, Gabriele: So I can also mention for the educational perspective, right now we ran a little study that we're gonna submit next week in David's class.
Sarti, Gabriele: between Workbench and an interp using a shared UI, right? So you have a UI that is in the web, but then it's the same plots that are produced locally by your library, right? When you work on a Jupyter, for example.
Sarti, Gabriele: And…
Sarti, Gabriele: we collected a lot of feedback from people, and in general, people found it intuitive to work with the same shared visualization, both in the kind of, like, drag-and-drop, no-code web interface, than when you're designing your, kind of intervention more manually, right? So I think it does make sense to have something shared there, right?
Sarti, Gabriele: Yeah.
Sarti, Gabriele: You're the pensioner.
Sarti, Gabriele: Sorry.
Sarti, Gabriele: I found in the workplace. Even with the one workbench interface, it's… you get… pretty differing results.
Sarti, Gabriele: Time to change it, so having something standard and charities.
Sarti, Gabriele: You can make it very efficient, also. If you control the backend, you can, like, make it very quick, right?
Sarti, Gabriele: You can make it more definitively, Ben.
Sarti, Gabriele: of the stock, the more likely it's no.
Sarti, Gabriele: Well, That's it for me. Sorry for keeping you long.
Sarti, Gabriele: Any final question?
Sarti, Gabriele: Is this close to a job book?
Sarti, Gabriele: No.
Sarti, Gabriele: No, no, it was just, a bit of a… Yeah? Nice. I'm glad to hear, no, it wasn't planned as a job talk, no.
Sarti, Gabriele: What, sorry? Google Slides. Just, like, manually?
Sarti, Gabriele: It's convenient to make figures for papers in Google Slides, because then it's plug-and-play for the presentation, right?
Sarti, Gabriele: That's a lot of challenges.
Sarti, Gabriele: Yeah.
Sarti, Gabriele: Thanks, everyone.
Sarti, Gabriele: You know…