Yaniv Nikankin: Please have an older one.
Yaniv Nikankin: Okay, great.
Yaniv Nikankin: So…
Yaniv Nikankin: I'll just, since we're… since we're running a little late, I'll just be very brief, you know, so we were lucky to have Janiff here. He's,
Yaniv Nikankin: you know, in town after NURBS, a member of Yonatan's lab.
Yaniv Nikankin: And so, we're looking forward to your talk.
Yaniv Nikankin: Thank you very much for having me.
Yaniv Nikankin: Ugh.
Yaniv Nikankin: It's my last presentation, so I'm hoping my throat will still hold up until the end.
Yaniv Nikankin: So hi, nice to meet you. I'm, Yaniv. As David said, I'm a PhD student working on interpretability in Jonathan's lab.
Yaniv Nikankin: And today, I want to tell you a little bit about my interpretability research that has focused on the paradigm of circuit analysis so far.
Yaniv Nikankin: Never mind. And by the end of the presentation, I hope to convince you in several things. First,
Yaniv Nikankin: and I guess that's moot, given that this is an interpretability lab, is that interpretability is cool, and useful.
Yaniv Nikankin: Second, I want to convey that
Yaniv Nikankin: Investigating model circuits is a useful paradigm of interpretability, and that it can be useful for multiple targets. And lastly, I hope to convince you that to make it really useful in, let's say, frontier tasks, we have to think about how to scale it.
Yaniv Nikankin: Scale it further.
Yaniv Nikankin: Feel free to stop me at any point for questions, either technical or abstract. If I'll want to go up ahead, I'll just cut it out and we'll continue later.
Yaniv Nikankin: So… I'll skip this slide since we're here. So,
Yaniv Nikankin: In my opinion, interpretability is quite useful for two main aspects, the first one being scientific insights, both on the models themselves and on the domains which are modeled by the models, either language, vision, biology, whatever.
Yaniv Nikankin: Second, I think interpretability is great for fixing problems in models, much like you probably, know very well many, many such problems that exist.
Yaniv Nikankin: And…
Yaniv Nikankin: Today we'll… and today, I'll present two works, each, tackling slash discussing one of these, one of these…
Yaniv Nikankin: directions, and hopefully we'll have time to also mention a work in progress that, tries to take this paradigm and scale it
Yaniv Nikankin: Scaled further to more complex problems.
Yaniv Nikankin: Cool. So, before we start with the studies themselves, let's just do a quick…
Yaniv Nikankin: Overthrow of what, circuit analysis is, to make sure we're on the same page.
Yaniv Nikankin: So, when I say,
Yaniv Nikankin: So, when we look at what models do, the classical approach most people use is look at them as a black box that receives some sort of input, and outputs some sort of output. Let me know if I'm, like, blocking your way or something, I'll try to slide throughout.
Yaniv Nikankin: And a more useful prism is looking at them as a sort of sequence of layers, a sequence of blocks, modules, whatever, which is pretty equivalent to observing a model, a neural network, as a computational graph.
Yaniv Nikankin: a computational graph whose nodes are any sort of representation that goes on within the model. For example, the outputs of attention heads, or MLPs, or specific MLP neurons. And the edges are defined by the architecture of the network.
Yaniv Nikankin: And… The paradigm of circuit analysis basically states that the… an approach as…
Yaniv Nikankin: One approach to investigate these models and how they work is to, understand subgraphs of these computational graphs.
Yaniv Nikankin: And this includes a first step of localizing an important… of localizing a subgraph that is important to a specific behavior. For example, a running example in the first part will be this arithmetic prompt made out of,
Yaniv Nikankin: Two operants.
Yaniv Nikankin: And the second step of this paradigm is to interpret said subgraph and understand how it works.
Yaniv Nikankin: Now, we won't go into the first part too much, but I'll just say that it's usually comprised of any technique that gives
Yaniv Nikankin: Each node, or each edge, some sort of important score of how important it is to the specific behavior we're analyzing, and then pruning the unimportant ones to get, hopefully, a sparse subgraph.
Yaniv Nikankin: Cool.
Yaniv Nikankin: Great. So going into the first work, we looked at the first question I presented before, which is model insights or scientific insights. How do these models do things? And we tried to tackle an
Yaniv Nikankin: well-researched question, which probably has thousands of paper written on it, and has no single definitive answer. How much do models generalize versus memorize their training distributions?
Yaniv Nikankin: And we try to attack this through a prism of arithmetic questions.
Yaniv Nikankin: Which is a good setting because arithmetic questions can either be solved by robust algorithm, much like the one you probably learned in the first grade, or by memorizing an entire training distribution up to some
Yaniv Nikankin: extend. For example, memorizing all exercises until, 1000.
Yaniv Nikankin: And specifically, we look at, arithmetic questions, arithmetic prompts made out of four positions, operand, operator, operand, equals sign. Very, very simple.
Yaniv Nikankin: And the way we attack this is using the circuit analysis paradigm, where we first try to localize a specific subcircuit, and then try to explain how it works.
Yaniv Nikankin: Hoping to either find evidence for memorization or for some sort of robust algorithm.
Yaniv Nikankin: Cool. So, again, keeping the entire localization process rather as a black box, we give… we find the importance score for each node in our computational graph, where the nodes we look into are attention heads and entire MLPs at each position, each of the four positions.
Yaniv Nikankin: And I want to direct you to,
Yaniv Nikankin: The results you see here basically state that
Yaniv Nikankin: there are… there's a very sparse subset of attention heads which have any importance to the prompt, and two groups of MLPs important to arithmetic prompts.
Yaniv Nikankin: early MLPs, they're important across all positions but the last one, and the last half of the MLPs, which are important in the last position.
Yaniv Nikankin: Stating that we can prune all of the other components from the graph, ablate their values in a fourth pass, and still get pretty much the same results.
Yaniv Nikankin: Now…
Yaniv Nikankin: Our point was to investigate what parts of the model actually generate the answer, so we tried to localize this even further. Are you going to talk about your setting more, or can I ask… Go ahead. So this is a, this is a,
Yaniv Nikankin: a model that you trained just to do this test, or you… No, okay, this, we, we, we tried this on a set of, general models, of several scales, from…
Yaniv Nikankin: 1 billion to 70 billion,
Yaniv Nikankin: we didn't fine-tune any of them. Personally, I think that to investigate specific behaviors and actually get an unaffected answer, fine-tuning is something we should generally avoid, like, unless…
Yaniv Nikankin: Unless you have to, for some reason? That's what you're saying, yeah.
Yaniv Nikankin: Cool.
Yaniv Nikankin: So… Yes.
Yaniv Nikankin: Can you… What?
Yaniv Nikankin: You said it's like a black box, but… Yeah, so, the question was which localization method we used. Here it was activation patching. You basically take a component, like, replace its activation with a counterfactual activation, and try to see how much it affects the result.
Yaniv Nikankin: Cool. And later, and,
Yaniv Nikankin: Layer by layer, MLP by MLP? Yes, MLP by MLP, or attention head by attention head,
Yaniv Nikankin: Layer by layer, position by position. Yes. I present here only the results for the attention heads in the last position, because the other positions are irrelevant for this. Like, they're not… they're just, no attention heads are important to give output in any of the other positions.
Yaniv Nikankin: Cool. So, as I said, we want to, identify which are the… what are the components within this subset that actually
Yaniv Nikankin: Have some role in generating the correct answer.
Yaniv Nikankin: So, we train a probe on the representation of… at each layer and each position, trying to extract the correct answer, meaning we have, specific, like, for example, for a specific layer and a specific, position, we have many, representations for many prompts, and we're trying to extract from this vector, representation… from this representation vector.
Yaniv Nikankin: The, the correct answer for that prompt.
Yaniv Nikankin: And showing the test accuracy of these many linear probes that we train, we see a very clear pattern.
Yaniv Nikankin: Where all of these… Answers are only extractable at the last position.
Yaniv Nikankin: after some, middle layer. Now, looking at the important components that match this description, we see that, the only ones are the MLPs in those, in the last position in those layers.
Yaniv Nikankin: So, we asked the further question on.
Yaniv Nikankin: Let's zoom into them and understand how they work. Yes, Connor?
Yaniv Nikankin: Bigger.
Yaniv Nikankin: Embedding speaker.
Yaniv Nikankin: Is it for those pieces that do the right answer, or is it also the case for everyone?
Yaniv Nikankin: You get a false incident.
Yaniv Nikankin: So, the prompts we analyze are up front, only the ones that are, that them all gets correctly.
Yaniv Nikankin: And the accuracy level of the models we use is roughly 90%, because to localize a behavior properly, you need it to exist. You need the mall to be good at it.
Yaniv Nikankin: In the… in… The rectangle is in the payoffs too many.
Yaniv Nikankin: 30, and in the left… Well, it's… Well, it's more like 15 to 30, and… It's so… it's not exact.
Yaniv Nikankin: Well, the important layers here are… the importance of the MLTs here is lower,
Yaniv Nikankin: And, I mean, that's just… we still had to keep them in, like, in the circuit to have, good performance, but, like…
Yaniv Nikankin: just looking at which components exist in the layer layers, it's only these ones. The MLPs in the other positions are unimportant, and there are no attention heads in these positions.
Yaniv Nikankin: Dan? No, sorry, this is… this is, like…
Yaniv Nikankin: This is 0, this is less, so 15 to the last.
Yaniv Nikankin: to 30.
Yaniv Nikankin: Yeah.
Yaniv Nikankin: Cool. So, zooming in on these MLPs, just as a reminder on how MLPs are defined, we have, the output of the MLP is basically a linear combination of all of the MLP neurons.
Yaniv Nikankin: Where what we define as a neuron is the activation, a single scalar dimension in the hidden representation of the MLP, which is multiplied by some row vector in the output matrix. The things… the little blue rectangles you see there.
Yaniv Nikankin: And basically, what the vector, the output vector states for a neuron is what does this neuron write to the output of the MLP, what is encoded in that vector, and the activation states how much this value is written.
Yaniv Nikankin: And basically, we want to further zoom into these MLPs and look at the individual importance of each neuron in these layers. So again, we do localization, this time on each neuron activation, and we search for important neurons.
Yaniv Nikankin: Looking at a random layer at llama 3.8 billion, we see a very sparse activation, where the y-axis value here is the importance of each neuron. We see that there are very few neurons which have significantly over zero activation value.
Yaniv Nikankin: And… We… this is nice because we can focus on them and try to understand what do they do.
Yaniv Nikankin: Now, we tried various approaches, but the one that eventually, worked and gave us some hints was looking at the activation of each neuron as a function of the input, meaning we can look at the value each neuron at how much it fires.
Yaniv Nikankin: As a function of both operands, operant 1 and operant 2. Now, because it's a function with two variables, we can just plot it as a 2D map, which is pretty nice. So, looking at the activation map of some random neuron, we see a pretty cool, pretty cool pattern.
Yaniv Nikankin: a very… a very strong diagonal. Now, when looking at the activation patterns of other neurons, we also see similar,
Yaniv Nikankin: similar patterns that we can give titles to. I'm not sure if you see this, but there is a very weak line here, where, the first operand is equal to the second operand. Here we have lines at periodic patterns whenever,
Yaniv Nikankin: We'll see it later. And another pattern which fires… a neuron which, for example, fires when a single operand is higher than some value.
Yaniv Nikankin: Now, of course, these patterns aren't perfect, this isn't binary here, there are various holes in this diagonal, but still, it's very… there's some sort of signal here.
Yaniv Nikankin: So, looking at what each of these neurons, not only on what it activates, and also on what it does.
Yaniv Nikankin: We can look at another neuron here, for example, where we have, again, the running example subtraction prompt, and we see a neuron that is important for this prompt, and it has an activation pattern which we can label as
Yaniv Nikankin: This neuron fires when the result should be between 150 and 180.
Yaniv Nikankin: How do we create this label? Just by looking at this diagonal.
Yaniv Nikankin: And we can see by analyzing its corresponding value vector, which I've mentioned before, that this neuron also promotes, also embeds within its value vector some of the tokens which are in this range. Not all of them, but most of them.
Yaniv Nikankin: Looking at another neuron, we see, again, a similar pattern. I don't want to go into this due to lack of time.
Yaniv Nikankin: But generally, we see an interesting behavior. Each of these neurons responds on some input pattern that applies for the input prompt, and corresponds both the correct answer by a little bit, and also other answers which relate to this prompt.
Yaniv Nikankin: To this, to this pattern, sorry.
Yaniv Nikankin: Now, you can think of this as like a Venn diagram with many, many circles, where at the center of it, promoted the most, or by the most neurons, is the correct answer.
Yaniv Nikankin: We label this as our bag of heuristics hypothesis, because each neuron acts as sort of heuristic, firing on an input pattern and promoting some sort of output, corresponding output, and only the combination of all of these heuristics promotes the correct answer enough to generate it.
Yaniv Nikankin: Any questions on… so far?
Yaniv Nikankin: Cool. Oh, I have questions in the chat, which I missed.
Yaniv Nikankin: Oh, Alex asked, how are these neurons identified? Sorry, I missed this question. The answer is by the same localization techniques I didn't go into before, activation patching or attribution patching, I can, like, talk about more later.
Yaniv Nikankin: Cool. I have a question. So, like, let's say you pick a random neuron that wasn't one of the ones that had high importance.
Yaniv Nikankin: like, what kind of a pattern would you expect? Just, like, random? So, we looked at them, yeah, if you draw out their patterns as a function of the input
Yaniv Nikankin: like, input arithmetic oper… operands. It's just noisy, or zero… near zero across everything, or, like, nothing very indicative.
Yaniv Nikankin: Yeah.
Yaniv Nikankin: I guess another question, I probably would talk about it later, but, like…
Yaniv Nikankin: Well, these patterns have, like, hierarchical structure. For example, if you looked at the patterns from the previous layer, like, would they relate to the patterns that we found in the later layer?
Yaniv Nikankin: So, this is a very cool question. Unfortunately, due to this complexity of, like, comparing, many, many patterns from previous layers and many patterns, and, like, it's… it becomes way more difficult to do this,
Yaniv Nikankin: to do this, and I'll throw, like, a single sentence on this later.
Yaniv Nikankin: And, like, one of the works by Anthropic, the attribution graphs, they also showed its usage on arithmetic prompts, and there they showed some sort of way to, update this and, like, find also connections between the…
Yaniv Nikankin: like, connections between different, let's say, patterns or neurons. You could also, for example, look at edges of the computational graph to find which edges have more weight, but we didn't want to go into that.
Yaniv Nikankin: Yeah. It will tell you, like, okay, this affects that, but not why or how. You'll have to understand that by yourself.
Yaniv Nikankin: Cool. So, so far, it's been a nice manual process. I won't go too… I'm actually struggling
Yaniv Nikankin: Can you… Which, like, an individual one, or just…
Yaniv Nikankin: Go with the exam.
Yaniv Nikankin: So, okay, so you have the activation pattern of an individual neuron here, which basically means each pixel here states how much does this neuron activate for a specific input prompt.
Yaniv Nikankin: For example, this pixel, the blue one here, states how much does this, neuron fire for the input prompt, 180, I don't know, and, sorry, minus, 5.
Yaniv Nikankin: For example, Like, 180 is the y-axis, and 5 is the x-axis value.
Yaniv Nikankin: And minus because this is a pattern for the subtraction operator.
Yaniv Nikankin: Yeah. But again, like, we can go deeper into that, later.
Yaniv Nikankin: So…
Yaniv Nikankin: Following this manual hypothesis creation, we want to validate it, and to do that, we need to somehow classify each neuron to its heuristic type. So, to do this, we manually create many possible heuristic types based on the things we see that happen in neurons.
Yaniv Nikankin: where we end up with 6, heuristic types, each of them can get hyperparameters. For example, one of the operands should be in some range, so the operand switches, and also the range itself switches. We take many values of these, of this range.
Yaniv Nikankin: And then we run some matching algorithm to compare… to match each neuron based on its activation pattern to the relevant heuristic types that correspond to it. Again, I don't want to go too deep into how we do this.
Yaniv Nikankin: And… By doing this, we can show that we can take a specific prompt.
Yaniv Nikankin: We know which heuristics should apply to it, because we see the operands, and if we choose specifically very few neurons that apply to… that implement these heuristics, and remove them from the model.
Yaniv Nikankin: even if they don't directly affect this prompt, in total, they will create a very rapid decrease in performance for this prompt. And we average this, of course, over many, many prompts. We see that we can ablate as few as 5 neurons per layer to completely destroy the model's accuracy.
Yaniv Nikankin: And we compare this to just picking the same number of neurons randomly, which requires way more neurons to lead to a similar ablation.
Yaniv Nikankin: And we pick, like, random heuristic neurons, not just random, unrelated neurons. And this happens across operators and across models.
Yaniv Nikankin: And we run some more validations, but if you're interested, like, let's talk about it later. I think some implications we can take from this work have time until 1, right? Yep, keep on going. I'm just… the only reason I'm worried is my computer's about to run out of batteries, and I'm recording, so I might have to run out to get my…
Yaniv Nikankin: So what?
Yaniv Nikankin: Okay.
Yaniv Nikankin: No.
Yaniv Nikankin: Does that mean that with only,
Yaniv Nikankin: 20, noirams, you can reduce the accuracy by 7… by 50%.
Yaniv Nikankin: with only 20… 20 neurons? You mean of this?
Yaniv Nikankin: Like, even… you need even less than… well, yeah, I mean, per, it's per layer, so with removal of, like, I'm not sure what this number is, like… The random ablation. With 20… with only 20 neurons, you were able to reduce the accuracy almost to zero.
Yaniv Nikankin: Yes, I should mention that there are a total of, like, 1% arithmetic neurons in each layer.
Yaniv Nikankin: So that's like a hundred… that's like a hundred… that's like 100 neurons per layer, so randomly choosing, 20 of them still might get you,
Yaniv Nikankin: Like, neurons that might be relevant for that.
Yaniv Nikankin: It's 20 per layer. Yeah, 20 per layer. It's many more. Yeah. I mean… A few hundred. But how, how,
Yaniv Nikankin: Let's, let's, let's maybe talk about it later, because I don't want to, focus deep into this.
Yaniv Nikankin: Cool. So, first of all, to answer our question that we started out with, is this an algorithm, or is this memorization? Personally, I think it's a bit of both. It's like a set of many memorized rules, which is a bit of an annoying answer, I know, but bear with me.
Yaniv Nikankin: Some people I talked with just said, oh, this is just an algorithm, just not one we use.
Yaniv Nikankin: And…
Yaniv Nikankin: We… we found that many of the, like, the errors these models make on arithmetic prompts can be majorly attributed to
Yaniv Nikankin: failures in… like, heuristics to incomplete heuristics. The… as you remember before, you saw the blue diagonal, but it wasn't perfect. It wasn't binary, like, high firing in this range and zero everywhere else. It was, like, there were holes in it. So, these incompletions, exactly, we found that to be,
Yaniv Nikankin: the main cause for, for arithmetic failures, and… And…
Yaniv Nikankin: Observe, and this also has implications for longer horizon, computations, where if you take a chain of thought and break it down to,
Yaniv Nikankin: too many steps of arithmetic computations in between. Each of them, it eventually has some sort of out…
Yaniv Nikankin: needs to output some arithmetic number. And at that point, when these heuristics take place,
Yaniv Nikankin: They might fail, causing failures even in chain of thought settings, which we didn't investigate here, but some students dealt with in a future project, in a course.
Yaniv Nikankin: Cool.
Yaniv Nikankin: Any questions generally on this?
Yaniv Nikankin: So, in complete stakes.
Yaniv Nikankin: It could work, let's safe.
Yaniv Nikankin: Would that sort of imply any sort of, like, fire or ball?
Yaniv Nikankin: Heuristic, where, technically speaking, it could be reduced further form, or something like that?
Yaniv Nikankin: What do you… what do you mean by, like, how would you connect hierarchy to this? Like, as in, or, like, whether whatever heuristic one might have, it could have been reduced to an even, like, smaller condition, which would have been considered incomplete, but it's probably just another…
Yaniv Nikankin: heuristic?
Yaniv Nikankin: M… I guess so? I…
Yaniv Nikankin: I know, I had this, like, idea of trying to edit the model such that the heuristics will be complete, and measuring how, like, how will it affect and increase the accuracy. I wasn't sure how to do that, given that, like, editing is pretty difficult and…
Yaniv Nikankin: noisy.
Yaniv Nikankin: Yeah, in short answer, I don't know.
Yaniv Nikankin: Cool.
Yaniv Nikankin: So… the… Oops, sorry.
Yaniv Nikankin: Okay, these are a bit… long. I'll go over the questions, I think, later, if we'll have time.
Yaniv Nikankin: Cool. So, the downside of this was that
Yaniv Nikankin: manual… we had to do a lot of manual interpretation work here. Also, looking at patterns and activation… activation maps of neurons, generating the heuristic types, like, defining them ourselves.
Yaniv Nikankin: It's pretty hard, and generally, the entire part of, second part of circuit analysis, the interpretation, is… is hard. Even for such a…
Yaniv Nikankin: Simple task.
Yaniv Nikankin: So, we can further ask…
Yaniv Nikankin: Within this paradigm, can circular analysis still be useful even if we leave out the part of manual interpretation and just
Yaniv Nikankin: try to learn something from the circuits themselves, from their structure. Oh, related to this, there was an online question Darshana asked.
Yaniv Nikankin: To what extent can we predict
Yaniv Nikankin: What computations the model will make a mistake on, based on what you found.
Yaniv Nikankin: So we… So,
Yaniv Nikankin: We dive into that deeper in the paper, we build some hypotheses on what might lead to errors, and thus, can we localize a possible problematic,
Yaniv Nikankin: heuristics, we didn't get to a perfect, like, a perfect, pre-honk prediction.
Yaniv Nikankin: But… Like, we tried to validate several hypotheses, one being, like, oh, you might have, like.
Yaniv Nikankin: Just lack of a lower number of heuristics for specific prompts. For incorrect prompts, that wasn't the case.
Yaniv Nikankin: They might have enough heuristics that apply to them, but not enough neurons that implement them. That also wasn't the case.
Yaniv Nikankin: Generally, again, it's sort of the annoying answer of this is sort of spread out, so…
Yaniv Nikankin: we weren't able to make a perfect prediction. You weren't able to… Well, we, we, you might be able to, we didn't. Makes sense. Yeah.
Yaniv Nikankin: Okay.
Yaniv Nikankin: Cool.
Yaniv Nikankin: So, this brings us… this question brings us to the second work, where we try to actually fix a problem in models based on findings even without fully understanding of Fourierverse engineering circuits.
Yaniv Nikankin: Specifically, we look at VLMs. Where VLMs in this context, just to make it clear, are models comprised of a language decoder that can take text only, but is also trained to receive as input visual embedding of tokens, visual tokens from an image, a video, several images, whatever.
Yaniv Nikankin: Now, these models have a very peculiar phenomena or problem, depends how you define it, where they perform better on…
Yaniv Nikankin: Prompts that are purely textual.
Yaniv Nikankin: Versus the same prompts, sort of the same prompts, where the same question is asked on a visual input.
Yaniv Nikankin: For example, over this slide, we'll look a lot on this image and this prompt of object counting, where we have the question of how many bananas are in an image, or in a sequence that is equivalent to this image.
Yaniv Nikankin: We look… there are also other tasks we define, generate datasets for, and investigate, but this phenomenon has been widely shown across a lot of, a lot of question domains,
Yaniv Nikankin: Simple ones, complex ones, what have you. There's a paper called ISOBench from Caltech that deals with it, among other things.
Yaniv Nikankin: So, what we said, we asked, can we better understand this difference in performance from a mechanistic perspective? And more specifically, can we identify a difference in the computational pathways, i.e. the important components, the model circuits?
Yaniv Nikankin: That are responsible for each of these behaviors.
Yaniv Nikankin: And we start by identifying for each of the variants, for each, for each prompt, we have two variants, and for each of them, we identify the set of important components that are… that is sufficient to answer it.
Yaniv Nikankin: Just as we did before, with, attribution patching this time.
Yaniv Nikankin: No.
Yaniv Nikankin: Yeah.
Yaniv Nikankin: Yes, we pres… yeah, we presented it,
Yaniv Nikankin: A week ago. So sorry if you've been in the workshop and you're hearing this twice.
Yaniv Nikankin: Yeah, David, I think, might have heard it.
Yaniv Nikankin: Cool.
Yaniv Nikankin: Yeah, so the first step, we tried to do a very simple structural analysis of whether the computational pathways are the same thing
Yaniv Nikankin: Meaning we look at the, language circuit and visual circuit, basically two sets of components, defined by their
Yaniv Nikankin: position, layer, and component index, i.e. MLP neuron or Attention Head Index.
Yaniv Nikankin: And, we look at their overlap, just how much is, how much do they intersect.
Yaniv Nikankin: And specifically, to make our results more fine-grained, we base… we are based on,
Yaniv Nikankin: like, definition from earlier work, which splits up these prompts to three parts. The data part, which is either the image or the sequence. The query or question part, which is the task asked on… which is exactly the same between the variants.
Yaniv Nikankin: And the last generation position in which the answer is actually generated. Again, the answer here is also still a single token, because of circuit analysis limitations.
Yaniv Nikankin: And, the three groups are separated by position. It's… -Oh.
Yaniv Nikankin: Positional light.
Yaniv Nikankin: So, to look at the results of this overlap analysis, we see that in the data positions and in the query positions, we have relatively low overlap.
Yaniv Nikankin: In the generation position, it's quite higher, which makes sense, since you have,
Yaniv Nikankin: like, the components at that position, their sole role is to generate the answer, right? So, if the answer is the same between the two prompts, and it's the same token, you would guess that it would be exactly the same components.
Yaniv Nikankin: Which is still surprising that the IOU is only 0.6 and not even higher.
Yaniv Nikankin: And these results happen across models.
Yaniv Nikankin: Now…
Yaniv Nikankin: Supposedly, this says, okay, we have very different components that process these two modalities, but structures only have the story because of two problems in analyzing neurons, redundancy, and polysemanticity.
Yaniv Nikankin: The same neuron in both modalities can have a different functional role because of polysemanticity, and different neurons can have the same functional role because of redundancy. This has been shown to happen.
Yaniv Nikankin: And to understand this better, we'll,
Yaniv Nikankin: we want to look beyond structure and also analyze functional equivalence. So, to do this, to understand if a component in textual circu… in the textual circuit does the same thing as a different component in the visual circuit.
Yaniv Nikankin: We take, we split up our circuit again to the three groups based on position. We take one of the groups, for example, the query components, and replace it with the query components from the different modality. We do a cross-modality patching process here. And given this circuit, this edited,
Yaniv Nikankin: group of components. We basically ask, how much does the result change compared to before?
Yaniv Nikankin: And from an intuition that if it doesn't change much, then the query visual components do the same thing as the query textual components.
Yaniv Nikankin: And of course, we do the opposite as well, and we do this across all three groups.
Yaniv Nikankin: The results, this time I'm showing them in reverse order. For the generation components, show extremely high, interchange faithfulness, same role… basically the same role for the slightly different sets of components, indeed indicating a redundancy there.
Yaniv Nikankin: Another interesting, finding is that the query components, for example.
Yaniv Nikankin: Despite being much more different in structural overlap.
Yaniv Nikankin: Have, high functional equivalents, indicating a very similar role.
Yaniv Nikankin: Unlike…
Yaniv Nikankin: Which is, like, shows even more redundancy here. But, the more interesting results here is that in the data components, the functional equivalence is super low, indicating that, if there's a difference in computational pathways in the function of the groups.
Yaniv Nikankin: It's in these positions.
Yaniv Nikankin: And we need to look deeper into that.
Yaniv Nikankin: This, of course, happens across models.
Yaniv Nikankin: Now, this brings us… this brings us to the question of if the components in the data positions act differently and are the major contributor to the difference in performance or processing, let's understand why, like, what happens there, or how can we fix it.
Yaniv Nikankin: So, throwing away the notion of circuits aside, like, we're looking at the full models now, we look at the data positions, and at each layer separately.
Yaniv Nikankin: We basically asked the question of how similar are the representations between a visual variant and its textual equivalent?
Yaniv Nikankin: We measure this by a form of cosine similarity, where we look for each visual token, which textual token matches it the most, and we average this similarity measure across all tokens to get a single number per layer.
Yaniv Nikankin: This single number indicates how similar are the whole of the visual representation to the whole of the textual representation.
Yaniv Nikankin: If we look at the results, We see that,
Yaniv Nikankin: despite the model being trained to align visual components early on to the textual representation and process them together, we see an interesting upwards trend here.
Yaniv Nikankin: starting from rather low similarity across all tasks, each line here represents a task, and slightly going up. This pattern is different than models, but, like, between models, but generally, repeats itself.
Yaniv Nikankin: And, yeah. Just a quick question about cosine similarity, because the number of tokens are different between images. Right. Yeah. Sentence, right? Yeah. How do you compute cosine similarity? So, as I said, for each visual token, we search for the most similar textual token.
Yaniv Nikankin: from… you would say, if you have an image patch of a dog, you probably want to search for the textual token that says dog, hoping that they would be the most similar, right? It might not be, because the presentations and visual patches are distributed, but that's a whole other story.
Yaniv Nikankin: But generally, we'll look for the maximum, and then average across all tokens.
Yaniv Nikankin: Cool,
Yaniv Nikankin: Yeah, so basically what we see is that the visual tokens, as the processing proceeds in the model, become more and more similar to the textual… to the textual equivalence, right?
Yaniv Nikankin: Now, this is a pattern that has been also shown in a parallel work called the semantic hub hypothesis, which I like very much, and you should have a read, which basically states that if you have a multimodel model.
Yaniv Nikankin: Which has a majority, modality, usually it's English language, sometimes it can have, like.
Yaniv Nikankin: Usually it's English language, so if you have another modality, vision, audio.
Yaniv Nikankin: Whatever and other languages, they slowly converge in similarity to the text… to the major textual representation as them all progresses.
Yaniv Nikankin: Now…
Yaniv Nikankin: we basically say this to ask, okay, if the textual models perform better than the visual token… than the visual, sorry, if the performance of the VLMs on the textual prompts.
Yaniv Nikankin: is better than on the visual prompts. And the… the major difference is in these representations, and the representations of the visual tokens at these layers are…
Yaniv Nikankin: The most similar to the textual representations.
Yaniv Nikankin: what will happen if we just plug them back into them all rather early here, and let them all think these are textual representations? Will this activate the textual pathways more than the visual ones?
Yaniv Nikankin: And this basically is a technique called backpatching, was shown to be used in various, use cases. And basically, this simple fix on the cost of a single forecast
Yaniv Nikankin: Bridges roughly a third of the performance gap, which is as low as 5% in some tests, and as high as 20% in others.
Yaniv Nikankin: And, yeah. Now…
Yaniv Nikankin: this isn't a perfect solution, merely, like, sort of a POC. We treat this as a POC to show, show that you can actually use
Yaniv Nikankin: So you're gonna ask us to fix a problem?
Yaniv Nikankin: But I think this is, like, a cool result. We also do a control to make sure that it wasn't the extra compute that gave them all more time to process this, these representations.
Yaniv Nikankin: That…
Yaniv Nikankin: that improve the performance by testing what happens if you backpatch textual representations? The answer is that it doesn't help much.
Yaniv Nikankin: And… that's it on this. Yeah, I see this. I see lots of mystified… okay, go ahead and ask your question. Oh, I just feel like… okay, I guess the question here is, like,
Yaniv Nikankin: how does the model actually know to process, like… because earlier… earlier layer, probably, the model only knows how to process the early… earlier tokens, seems like there's… You're asking how does this not throw the model out of distribution? Yeah, yeah. So, that's a good question, I'm…
Yaniv Nikankin: slightly lying to you in this, beautiful line from the end to beginning here. In reality, we have to backpatch somewhere to around layer 5 to remain rather enough in distribution.
Yaniv Nikankin: When, like, after the model received and processed its tokens, and it knows how to, like, process
Yaniv Nikankin: tokens. Like, we tried a systematic approach of all… from all layers to all other layers, and we found that this rather correlates in…
Yaniv Nikankin: Almost the latest layers to roughly, like.
Yaniv Nikankin: Here, this area, 5 to 10. I see. But good question.
Yaniv Nikankin: Technical question. Yeah. How did you measure no image? Can you insert… Did you insert black image? Did you insert… What do you mean, no image?
Yaniv Nikankin: When you compare it, without image versus… In the control? No, for comparison to the text. Yes. Similarities between processing images or processing text. It's not processing images to none. Yeah.
Yaniv Nikankin: You mean, like… Is that what you're asking? You mean dissimilarity? Yes. Yeah, so you take the tokens from the visual prompt, from the image here.
Yaniv Nikankin: that represent this image, and compare them to these tokens, to the textual equivalent of the image. But is there… I'm not… I'm not familiar with the technicalities of VLMs.
Yaniv Nikankin: A VLM… okay, a VLM… a VLM can also not receive any image, and then it acts just like an LLM, completely normally.
Yaniv Nikankin: It's not… it doesn't have to receive an image encoding.
Yaniv Nikankin: Yeah.
Yaniv Nikankin: Cool, so… So… Doug, you mentioned, like.
Yaniv Nikankin: Do you sit there a second?
Yaniv Nikankin: Oh, I was kind of curious. Does…
Yaniv Nikankin: Amplifying that representation to anything? Does it help?
Yaniv Nikankin: What do you mean, amplifying that presentation?
Yaniv Nikankin: I don't know if it's possible, like, literally. Steer? Like, yeah, steer, in this case. Interesting,
Yaniv Nikankin: like, steer from a textual representation towards a visual representation somehow. Yeah.
Yaniv Nikankin: That's a good question, I don't know.
Yaniv Nikankin: I… I mean, given that this is sort of an instance of steering, right, with a specific factor, then maybe,
Yaniv Nikankin: Yeah, I… good question. So, Enoch, why do people think this works?
Yaniv Nikankin: What do you mean? Well, why, why, like, so the model obviously isn't trained.
Yaniv Nikankin: To, you know, to have high performance with this kind of…
Yaniv Nikankin: backpatching. So, why do people think that backpatching is effective? I think that generally your representations, across layers
Yaniv Nikankin: especially, like, in these early tokens, when you don't have any information later on, they probably don't change very much. Like, they change a lot in the last layer, when the model does some sort of prediction in text, which explains this crash here. But,
Yaniv Nikankin: So you would say that between… So it doesn't change much, which is why you wouldn't expect it to be damaging.
Yaniv Nikankin: So, but why would you expect it to be helpful?
Yaniv Nikankin: So, the motivation here was just…
Yaniv Nikankin: Replace the visual representations with a textual equivalent, sort of.
Yaniv Nikankin: But I think the point, the interesting point here is that the representations do become more similar to the text ones, but it's too late for the circuit to use it.
Yaniv Nikankin: You can see it there, in the top right…
Yaniv Nikankin: Yeah. By the time the presentations are similar enough for the language model to do anything, computation has already proceeded, and attention has moved… You don't have enough layers left to move it to later positions and process. But it's quite interesting, right, because you would expect that this circuit
Yaniv Nikankin: That would benefit.
Yaniv Nikankin: From this… from the models being similar.
Yaniv Nikankin: It'd have to be… Present in the gradient.
Yaniv Nikankin: To give the model motivation to make them… the… the,
Yaniv Nikankin: The representation's similar in this way, but obviously it's not present in the gradient, right? It's not on the path at all.
Yaniv Nikankin: And so, I… so I totally buy the idea that in the abstract.
Yaniv Nikankin: the representations… Oh, you're asking the deeper questions of why does similarity even increase, over time? Like, because there's no gradient to force it? There's similarity, there's general similarity, but it's conventional space, so there's a lot of…
Yaniv Nikankin: Ways it could be similar.
Yaniv Nikankin: And… and there's a particular way that it's similar here that allows you to ask this… answer this numerical question. And then… so the… the puzzle is, oh, why would it…
Yaniv Nikankin: Become similar in that way, when it's too late to answer a numerical question anyway.
Yaniv Nikankin: You know, what's the motivation for the model to even
Yaniv Nikankin: do that. Maybe generally, the model would want to
Yaniv Nikankin: Promote any non-text modality to its…
Yaniv Nikankin: to this same representation as follow textual tokens, not necessarily in the last layers, right? Even if it happens by this, by middle layers, then generally across prompts, not even of these specific tasks. Yeah, so the way I translate that is…
Yaniv Nikankin: Similarity is similarity.
Yaniv Nikankin: you know, regardless of, like, what the purpose of the similarity would be for, right? And… And I…
Yaniv Nikankin: And it's still puzzling to me. I mean, it's still interesting. I mean, it's not too puzzling, because you guys are working, and we've seen similar phenomenon before as well, like, you know, when we patch different layers, right?
Yaniv Nikankin: remember in your function vector paper, you… you were like, why are we patching to an earlier layer? Why is it performing better?
Yaniv Nikankin: And so…
Yaniv Nikankin: So it's not mysterious in the sense that, like, oh, I understand this happens. But it's a little… I'm still mystified.
Yaniv Nikankin: By why this would be the case, yeah.
Yaniv Nikankin: When you do the back patching?
Yaniv Nikankin: Does it climb again, as the model runs?
Yaniv Nikankin: Oh, then you could backpatched twice. Now it gets more similar again. We tried that. We tried… we tried backpatching several times. It's… it's a degrading, performance incident, I guess because of the…
Yaniv Nikankin: One, just, like, in one of the tasks, it was in general. Yeah, exactly.
Yaniv Nikankin: Yeah, the looped transformers, yeah, we sort of, you're… yeah, that's a spoiler for one of the points now. I just wanna finish, and we'll see, like, if we have time for a quick run over. Okay, so, cool. So, I'll go… so I'll go forward, and if there are questions, I'll answer in the end.
Yaniv Nikankin: So I think the implications here are, first of all, that current VLMs, as we saw, are limited in the way they combine modalities.
Yaniv Nikankin: I think it's mostly because VLMs… VLM architectures today are just a form of necessity that started out because people didn't… because people wanted to reuse existing language decoders and existing vision encoders, and not train everything from scratch.
Yaniv Nikankin: But this is inherently limited.
Yaniv Nikankin: In this context, I think what… one of the findings is that maybe we need to process different modalities in a different amount.
Yaniv Nikankin: Right when we worked on this, there was a paper, from UMD on, like, loop transformers, which, in my mind, related really to this, like, just take… take, any modality and loop different amount of times, like…
Yaniv Nikankin: Loops some of the layers in the network a different amount of times.
Yaniv Nikankin: To process it better.
Yaniv Nikankin: But, to my point, I think the… the thing which I want to convince you in is that even with… we didn't fully reverse engineer the circuits here. I don't… we don't know how…
Yaniv Nikankin: counting works in either modality. But we were able to use insights from circuit analysis to point us towards directions of
Yaniv Nikankin: To guide the directions of fixing model problems.
Yaniv Nikankin: Now, I want to go over the last point rather briefly.
Yaniv Nikankin: And…
Yaniv Nikankin: point that, in my opinion, what's missing today in the interpretable daily research, one of the things, and also missing in my two works that I've showed you, is that if we draw some plot of various tasks which models are
Yaniv Nikankin: Able to solve, let's say. Forget, like, what each axis, like, forget what the y-axis means, higher means more complex. So, frontier models are generally here, able to solve complex tasks, taking hours, even.
Yaniv Nikankin: while… we in interpretability are researching somewhere around here. Single token output tasks,
Yaniv Nikankin: Like, toy settings, nothing extremely,
Yaniv Nikankin: applicable to frontier, applications. Now…
Yaniv Nikankin: I think it is interesting to sort of scale up and try to move towards the more complex tasks and develop techniques that will allow us to do so.
Yaniv Nikankin: And this brings us to our… one of our current projects, which is…
Yaniv Nikankin: How to apply circuit analysis for more complex tasks.
Yaniv Nikankin: Where, when I mean complex, I mean things that are… tasks that are not single output token, can be…
Yaniv Nikankin: Couple, or hundreds, or even more.
Yaniv Nikankin: Cool. Tasks that are not necessarily position-aligned, where we don't force all of the inputs to be exactly… to contain exactly the same content in each position.
Yaniv Nikankin: And, hopefully Task Force can use real data… real-world data for, and not have to redefine, our own experimental data for.
Yaniv Nikankin: And specifically, one question we decided to tackle is, can we apply certain analysis to study questions in such long-form, complex tasks?
Yaniv Nikankin: That might actually use not one… not a single mechanism to solve, but many lower-order mechanisms.
Yaniv Nikankin: And the task we chose to start with is coding, where we have some sort of definition of a function, and the model is asked to implement it. And the output can be many tokens.
Yaniv Nikankin: comprised of, let's say, many different mechanisms. And we asked the question of, can we take this higher-order semantic task and break it down to its components in a sort of unsupervised way?
Yaniv Nikankin: The way we do this is by first defining a set of coding prompts, and finding the circuits or set of components that are important for each output position.
Yaniv Nikankin: Now, there are a lot of, like, technical points here which I don't go into, but, let's just take it as granted. Given all of this set of many, many circuits, it might or might not be related, we want to find their relatedness, we want to find their similarities.
Yaniv Nikankin: And we do this by, after generating the circuits, calculating some sort of similarity metric between each pair, very similar to the structural similarity or functional similarity which we saw in the previous work. And…
Yaniv Nikankin: try to cluster them based on these similarities. Groups of circuits with high similarities between them are clustered together, and hopefully each cluster will represent some sort of lower-order semantic tasks.
Yaniv Nikankin: Now, forgive me for the poor visualization, but, let's say we have, we have some sort of cluster here where each row represents an individual circuit.
Yaniv Nikankin: For specific prompts and specific positions. Maybe you should let people stare at that. It was very fast. Yeah, I'm… I won't… like, I'm slowly going into it, and I'll give you more time.
Yaniv Nikankin: So, we basically have a circuit that corresponds to this output token and this part of this output string for i, where the i is the output token.
Yaniv Nikankin: And all of these, each row, as I said, is a single circuit, and all of these circuits were clustered together in our current setup.
Yaniv Nikankin: Now, if you'll have a look at the output strings, you'll see some sort of pattern.
Yaniv Nikankin: Does anyone, wanna point out which pattern this is?
Yaniv Nikankin: For… yeah, for loops, or specifically the… whoops. Yeah, specifically the iteration constant in the for loop.
Yaniv Nikankin: Like, 4 m in something.
Yaniv Nikankin: We saw, like, some other interesting clusters, like,
Yaniv Nikankin: Return values, which might be completely different, but usually come after some
Yaniv Nikankin: Return statement. Simpler clusters where just… it's just, a new line, or a couple of new lines.
Yaniv Nikankin: And… Those are all new lines after a colon.
Yaniv Nikankin: Sorry, I'm a bit confused. Huh, no. Well, some of… some of them are not, but most of them are.
Yaniv Nikankin: Cool. New lines after a colon.
Yaniv Nikankin: So… Yeah.
Yaniv Nikankin: Yeah, Quinn. Sorry, Kenny, can you take a, like,
Yaniv Nikankin: I mean, just in general, like.
Yaniv Nikankin: Okay, yeah, maybe I should see this. Essentially, I was trying to understand how you got the table to begin with.
Yaniv Nikankin: Right, so…
Yaniv Nikankin: we found, for each output position in each prompt, we have, like, hundreds of prompts, hundreds of output positions in each such prompt. We searched for a set of components that are sufficient to generate this output. Sufficient being high faithfulness. Again, won't go into this too much.
Yaniv Nikankin: Afterwards, we look for some… we define a similarity measure, which we… we're not sure on how to define properly yet, and it's very important. We define a similarity measure between every pair of circuits, like, every pair of sets.
Yaniv Nikankin: And you can think about, a big graph, like a big, just a big,
Yaniv Nikankin: Not a computational graph, but just a big graph, where the nodes are… each node is a circuit, and the edge between every two nodes is the… is the… the weight of it is the similarity between them.
Yaniv Nikankin: And then we want to cluster this metagraph to, to find clusters within it, which have high similarity between them.
Yaniv Nikankin: like…
Yaniv Nikankin: this, part and this part. I didn't draw the edges, but you can imagine they… there is an edge between every two…
Yaniv Nikankin: Pierce.
Yaniv Nikankin: I think this is where we would, appreciate as much feedback as possible. This is, like, our…
Yaniv Nikankin: This is a general, template. So you're making a circuit from a single prompt, is that right?
Yaniv Nikankin: Yes.
Yaniv Nikankin: I mean… The way the Anthropic does these things, right? So, like, because, like, in your previous analysis, you made a circuit, whatever, you had a whole… Yeah, yeah, a whole, a whole set, exactly.
Yaniv Nikankin: And now you're like, okay. Now I want to find the distribution, that's, that's, that's the point. We'll take a single prompt, and we'll use that single step.
Yaniv Nikankin: And then we'll… and then… and then we'll try to see these.
Yaniv Nikankin: I see, and then when you, when you cluster them.
Yaniv Nikankin: Are you looking for similar components or something? Right now, yes. Yep.
Yaniv Nikankin: But there could be multiple components that could be similar in the cluster? Is it supposed to be, like, just one thing that's… No, I'm… I'm… basically, right now, the similarity measure is an overlap, just as we saw before. Which, it's not good enough. You can say the concrete…
Yaniv Nikankin: Each of these steps has some concrete implementation that you try. Yes, yeah. Okay, so step, step one, you collect free form, long form, so human eval, you just go over all human eval, you get it, so that's, like… And filter, and filter out good deletion, like the CodeL, a simple code LM. Yeah.
Yaniv Nikankin: And then… and then… Step two, what do you do? Step 2, yeah, which circuit method do you use? So right now, it's attribution graphs, like the sparse attribution graphs that Entropic worked on. Yeah, yeah, the, how Michael Hannah's working? Exactly.
Yaniv Nikankin: It has many downsides, and… Yeah, yeah. Michael's code is usually wonderful, so… Yeah, but there is a major problem, you should talk about… Yeah, the problem there is that the code doesn't,
Yaniv Nikankin: Well, their work on attention… on, like.
Yaniv Nikankin: Adding attention heads to the computational graphs, Isn't, like, isn't really…
Yaniv Nikankin: published well, and, it doesn't have an implementation… yeah, you can understand the implementation. Michael is attempting to implement it now, but he's having also a lot of headaches with it. There are no position… there's no position information.
Yaniv Nikankin: That's another problem.
Yaniv Nikankin: Oh, what do you mean?
Yaniv Nikankin: Wait, that's a different prompt. Yeah, the attribution graphs indeed contain position information, I just drop it because the positions here are misaligned between prompts.
Yaniv Nikankin: Position 2… position 21… You don't know the semantics, right? Position 2, I don't want to assume anything about the positions, which is…
Yaniv Nikankin: a limitation, but somehow need to circumvent it. Oh, but that might be a big… A big problem, yeah. That's interesting. So this is, like, positionless information. All the position information is dropped.
Yaniv Nikankin: And there are no…
Yaniv Nikankin: edges between… No, I… there are… okay, there are any… no. There are no edges between positions. Right.
Yaniv Nikankin: Yeah. I just asked Waldo. So, the main, circuit is based on a single problem, and the components of it are those problems you're trying to solve for that single problem, or…
Yaniv Nikankin: So… Well, what we're… what we have is basically,
Yaniv Nikankin: the single problem of, let's say, coding, many prompts that are unaligned, unrelated at any point, except for being all coding. Okay. And we're trying to ask if we can take this higher order task, coding is just an example, we'll also try other things, and deconstruct it to
Yaniv Nikankin: As many mechanisms as possible, which are shared across different instances of the coding problem.
Yaniv Nikankin: It's merely a… I think this is a single question we ask. I mean, I guess there are also other…
Yaniv Nikankin: many possible questions to ask on this, I love this exercise that you're doing.
Yaniv Nikankin: So, so the, so the… so, yeah, I have an opinion.
Yaniv Nikankin: But I have no idea if this opinion maps onto something, right, that is actionable to you, right? Which is… and so you can see from the stylized research that we generally do
Yaniv Nikankin: In the lab, that we are not…
Yaniv Nikankin: We generally don't spend a lot of time chasing the circuits all the way from beginning to end.
Yaniv Nikankin: And that's probably because of my own opinion, like, I don't push the students to do that.
Yaniv Nikankin: And… which is… which is that probably, when you look at
Yaniv Nikankin: These broad distributions like this, like, real data, probably what's important is
Yaniv Nikankin: certain bottlenecks in the computation, certain neural representations at certain points, right, where the… where the computation gets to a certain point, and then the model has to figure out
Yaniv Nikankin: some abstraction, some invariant… Are you talking about a single forward pass, or also a many? So, what do you mean? Like, the ball neck issue you're describing, are you referring in a single forward pass, or… No, across a distribution. Okay. So, like, you have many things, right? So, like…
Yaniv Nikankin: Like, you know, When you use your artificial distributions, and you say, oh, here, at this moment.
Yaniv Nikankin: We have the concept of a number…
Yaniv Nikankin: a number of bananas or something like that, right? You know, the reason that you can identify a circuit is because there's this… there's actually a series of bottlenecks
Yaniv Nikankin: For, you know, for some set of texts. But I'd assume that if you had pretty different text, then you might have a subset of those bottlenecks, like, that different text might
Yaniv Nikankin: Join in with that circuit at a different moment, right?
Yaniv Nikankin: And… And so, sort of my mental picture… is that… You know, you have… Interesting moments?
Yaniv Nikankin: Where, at approximately certain layer ranges, you have…
Yaniv Nikankin: A type of information that's being… represented.
Yaniv Nikankin: And then the circuits are really just little pathways.
Yaniv Nikankin: of getting from one of these abstractions to the next in certain contexts, right? And so… so as you… as you go and you mine all these circuits, then I… I imagine that an exercise would almost be
Yaniv Nikankin: like… The circuit's almost too big.
Yaniv Nikankin: Right? The circuit is actually made up of lots of little pathways going from
Yaniv Nikankin: one interesting bottleneck to the interesting… next interesting bottleneck to the next interesting bottleneck, and so on, and so… so you're sort of clustering together whole circuits, which is interesting, but maybe…
Yaniv Nikankin: you know.
Yaniv Nikankin: I'm not sure if that's too specific, or if it's too, you know, like, I'm not sure what, right.
Yaniv Nikankin: like, I, I imagine that… What you might want to do is, if you have lots of circuits.
Yaniv Nikankin: You're sort of looking for…
Yaniv Nikankin: almost like, where do they cross in the middle? Right? And then, you know, where is a natural place to split them? And that kind of thing. You know what I mean? Like, the interesting thing here is to do some bottom-up
Yaniv Nikankin: completely unsupervised, data-driven approach to discovery without imposing distributions. Right.
Yaniv Nikankin: But the trouble here is that so far we're… we're seeing the shared things among the many different circuits are just very shallow features that are not so interesting. These token detectors. Right. And so we are thinking what is the right abstraction?
Yaniv Nikankin: And maybe…
Yaniv Nikankin: the full circuits are not the right abstraction, because their shared stuff is too… Too specific. It's too shallow, because what would happen everywhere? Well, new line happens everywhere. Right.
Yaniv Nikankin: But the question is how to get at those bottlenecks or write abstractions without bringing in your knowledge about the domain. Because we could do, in code, you could do a lot of things. I know, I know, we do, right? But we're trying to avoid it. I know. Kind of resisting the temptation to
Yaniv Nikankin: Well, we tried to, it didn't help much.
Yaniv Nikankin: No, but I… I love this, and… and if… right, and so… and so, like, Arnav is about… so we're… we're starting to push Arnav in the code direction, and so he has, like, some interesting…
Yaniv Nikankin: you know, sort of neural concepts and code that he's, he's gonna try to chase. Yeah.
Yaniv Nikankin: You can do a lot of things if you look at the structural. Right. Which is… we might still do. You might still do, right, so that's… But this is an opportunity to do something else.
Yaniv Nikankin: Right, which I like. It's great, yes. In this context of your comment, by the way, we tried to remove some parts which we deemed uninteresting.
Yaniv Nikankin: Which dominated the… the similarities. For example, in the same… in the same prompt.
Yaniv Nikankin: In all positions, you have the same input, right? A lot of shared input, the same function definition, and you need a lot of shared features between those prompts to analyze that input. So…
Yaniv Nikankin: This indicates that between different output positions in the same prompt, by definition, there will be a much higher similarity in their circuits, like, because their input processing components are the same.
Yaniv Nikankin: So, maybe this, like, we try to remove these. And also output generation components. We don't really care about the features that are responsible for literally generating the answer, because then all of the clusters will just be the same token. An extremely shallow behavior.
Yaniv Nikankin: We started, like, we're experiment… we experimented with this, but…
Yaniv Nikankin: Maybe, like, more work, probably more work if needed.
Yaniv Nikankin: I know you're talking about, like, shallow behavior, but what if you were to combine all these shallow behaviors, because you're doing all these different clusters, that it actually may talk about, like, a con… like, code concept that is required to use all of these shallow… What do you mean? Like, let's say, for…
Yaniv Nikankin: I guess the incremental list part, like, yes, maybe you need new lines, maybe you do need iterators, but, like, maybe there's some other concept.
Yaniv Nikankin: Other cluster as well, along with which all of them is actually useful for…
Yaniv Nikankin: incrementing. Like, basically, like, a code. Yeah, you could do… So, the clustering is vertical?
Yaniv Nikankin: Yeah. It's a community detection thing, but you could look through each cluster as a community, but if you could look at how they combine.
Yaniv Nikankin: and look at different levels in the hierarchy. So you take a cluster of an iterator, plus a cluster of, I don't know.
Yaniv Nikankin: Holland, open prayer and something, and together they may fall.
Yaniv Nikankin: Yeah. Oh, that's also interesting. Like a meta-cluster. Yeah. And, yeah, so…
Yaniv Nikankin: That, and I guess the second question, I wasn't sure, like, are these… wow, yeah. Are these, circuits that you're forming off of one model, or, like, across… No, all of these are in a single model. In the same… yeah.
Yaniv Nikankin: Otherwise, your components are different, and…
Yaniv Nikankin: You need to do different things.
Yaniv Nikankin: Yeah, I guess because it was already not positionless, I was like, oh, is it? Yeah.
Yaniv Nikankin: So, have you done a lip review to see how much previous work there has been done on
Yaniv Nikankin: You know, direct analysis of representations in these…
Yaniv Nikankin: code LMs already? Like, previous existing work.
Yaniv Nikankin: Code elements, no, because coding is just a…
Yaniv Nikankin: It's sort of a validation right now, it's not our goal. Okay.
Yaniv Nikankin: But… There's vanilla.
Yaniv Nikankin: So, the general thing would be, like.
Yaniv Nikankin: you know, for, like, you know, if Francesca, you know, has a…
Yaniv Nikankin: They were saying, oh, we found an interesting representation or interesting circuit for identifying types, like type declarations or type representations or something like that, which is what she was studying.
Yaniv Nikankin: then… then a reasonable question would be like, oh, somebody already found something in CodeLMs? When I do my clusters.
Yaniv Nikankin: Is one of the clusters that? Does it… does it show up?
Yaniv Nikankin: Or is what I'm doing not compatible with what previous… because, like, I think there's, like, a reasonable…
Yaniv Nikankin: thing to do, and if other people have published things… Actually, it's a validation question.
Yaniv Nikankin: Josh. Right. Like,
Yaniv Nikankin: Yeah. Like, don't… like, we want… our current step is to replace the task to a more… to one that's more easy to validate, and which we know where,
Yaniv Nikankin: which, which…
Yaniv Nikankin: sort of, which mechanisms would we expect to find there? Oh, well, I guess what you… So, maybe instead of finding… I like the fact that you're clustering as your lives, just finding these things, but then you're… I think that…
Yaniv Nikankin: Right now, you're in this tea leaf reading situation that a lot of…
Yaniv Nikankin: People in the interpretability world have found themselves in, which is always dissatisfying.
Yaniv Nikankin: Right, where you have for a behavior that you don't know if it exists. Well, yeah, you have a thousand clusters.
Yaniv Nikankin: And you look through the clusters, and, oh, 10, you know, out of the thousand, 10 of them look pretty interesting, and then you can, like, show these 10 and have an interesting story, and… right? And then people are like, okay, well, I'm not sure what did you find, right? So…
Yaniv Nikankin: So, so it's a…
Yaniv Nikankin: it's like a brochure blob test, right? You know, one of these, right? It's like, do you really… do you see meaning in here or not, right? And people are wondering if…
Yaniv Nikankin: if your… if your experimental results have to do with your experiment, or if it has to do with your own mind, right, sort of the question. And so…
Yaniv Nikankin: But the, but a natural… a natural…
Yaniv Nikankin: Way to check that is to say, oh, well, you know, there are…
Yaniv Nikankin: there are things that other people have found in these models, and when we… when we do this… We find. Sweep, they get picked up, right? Yeah, they get that. Yeah, so…
Yaniv Nikankin: Nope.
Yaniv Nikankin: Just…
Yaniv Nikankin: with it.
Yaniv Nikankin: Right, that's, like, one of my to-dos for this week is, like, think about which of these tasks we can put into it. Yeah.
Yaniv Nikankin: We can, like, definitely talk about it. David, you said something about in those
Yaniv Nikankin: Circuits, look for the cross-cutting parts.
Yaniv Nikankin: What did you mean by that?
Yaniv Nikankin: Oh, I just don't know, it's just a… it's just a general…
Yaniv Nikankin: It's my general unease about the whole circuit.
Yaniv Nikankin: Paradigm? Yeah. Which is that…
Yaniv Nikankin: you know, this… so I think that there's this… you guys are directly attacking it, right? Where, when you move to…
Yaniv Nikankin: just the general text setting, and you have two different pieces of text that are sort of reasoning about similar things, but expressing them with a different word order, a different syntax, you get these very different, you know, you'll get these very different circuits.
Yaniv Nikankin: And… and does it… Does it really mean that they're so different?
Yaniv Nikankin: And I… and… and I guess what I… Imagine is that…
Yaniv Nikankin: You can find the similarities, maybe not by…
Yaniv Nikankin: Looking at the whole circuit by… but by looking at…
Yaniv Nikankin: which bottlenecks or something the circuit's going to. So, for example, like, what… what… so, like, if I'm… if I'm just sort of… The question is how to identify… Yeah, how to identify them, right? So, yes, so…
Yaniv Nikankin: Yeah, I think it's a reasonable…
Yaniv Nikankin: And maybe one way… now we're conflating all the token position.
Yaniv Nikankin: Maybe a way would be to identify certain positions that are important, and let them define the circuit, and then compute these similarities.
Yaniv Nikankin: Right. It could be token position, attention head.
Yaniv Nikankin: Set pairs, or something like that, so you have…
Yaniv Nikankin: You know, you have these circuits.
Yaniv Nikankin: That are made of all these attention heads that are reading…
Yaniv Nikankin: from all these different token positions, and maybe that's the thing to cluster, is that you have these interesting read operations that are happening across all your circuits, and you cluster. Yeah, that's another question on how to combine the attention head and information flow information on this.
Yaniv Nikankin: It's… I'm having a talk with someone from BU on this, like, next week, yeah. Yeah, that's good.
Yaniv Nikankin: Yeah. So that, like, that seems promising.
Yaniv Nikankin: Right, okay. Yeah, I think there are enough, like, directions to attack this problem to try and get better results, or…
Yaniv Nikankin: Okay.
Yaniv Nikankin: Sorry, to talk about step 3, how are you exactly finding similarity? Right now, it's just overlap, like, component overlap. IOV. IOV, yeah.
Yaniv Nikankin: Yeah, like, have you considered using representations similar? Yes, that is… well, we tried that in the previous work. There, it was extremely computationally difficult.
Yaniv Nikankin: Because, it just requires an ins… well…
Yaniv Nikankin: basically, I… if I remember correctly, we didn't try it here, because…
Yaniv Nikankin: From two circuits comparison, here you have, many more.
Yaniv Nikankin: But, because this is… this can be done, like, statically, maybe…
Yaniv Nikankin: like, maybe that's the direction to go to? I think, personally, I've been a bit burned with representation comparisons, because…
Yaniv Nikankin: Well, because they're, it's not…
Yaniv Nikankin: Well, there are various problems with that, but, like.
Yaniv Nikankin: It is an open question of maybe we should go that way and try it again.
Yaniv Nikankin: Are you… do you have also slides about your other project, or… The reasoning one? No.
Yaniv Nikankin: But I can summarize and let people go, and then if anyone will want to talk more, I'd be more than happy to.
Yaniv Nikankin: I think people PR, people are interested in.
Yaniv Nikankin: Yeah, so I'll say this… Yeah, so…
Yaniv Nikankin: Okay, so yeah, so I'll just finish this and say a word about that.
Yaniv Nikankin: I think our current, sort of,
Yaniv Nikankin: Set of results can indicate that…
Yaniv Nikankin: we might have some direction of attacking this problem. Despite the results being rather shallow so far. There are enough, moving parts here that can be altered to…
Yaniv Nikankin: to, as we said, attack the problem, also the ones you suggested here, which I wrote some away.
Yaniv Nikankin: Yeah, I think, I think, as I said before, we can…
Yaniv Nikankin: Both of these works and,
Yaniv Nikankin: like, both the work, the completed works, the work in progress that I talked about, and a couple of others where we discuss, where we try to, investigate behaviors from a circuit perspective in protein language models, led by one of our, master's students, and,
Yaniv Nikankin: And another project which I'm working on in, like, reasoning, trying to use,
Yaniv Nikankin: Input attributions are not component attributions to show that you can prune,
Yaniv Nikankin: Like, to show various interesting conclusions on chains of thought, mainly that you can prune most of a chain of thought and still get a
Yaniv Nikankin: High sufficiency to generate the answer.
Yaniv Nikankin: Sort of directing at… Maybe actual applications of attributions to, like, of…
Yaniv Nikankin: You should give it the, the, say, say the… What?
Yaniv Nikankin: The hypothesis of what,
Yaniv Nikankin: A friend of Yonathan suggested to call it the reasoning lottery ticket hypothesis, similar to the lottery ticket hypothesis, which, like.
Yaniv Nikankin: Basically, in our setting, it states that you can just random… not randomly, but pick a specific subset of a chain of thought, and it will be… even if it's not coherent, it's good enough for the model to generate the correct answer.
Yaniv Nikankin: on its own. Sort of related to… well, I sort of thought it's related to your work on the, generality. I'd love to talk more about that. And,
Yaniv Nikankin: Yeah, this, like, set of works was done with this amazing group of people.
Yaniv Nikankin: And… that's it.
Yaniv Nikankin: if any of the things I mentioned, either on completed projects, works in progress, or just general thoughts on complex tasks, reasoning, biology.
Yaniv Nikankin: whatever seem interesting to you, I'd love to talk.
Yaniv Nikankin: And, that's it. Wonderful, yeah.
Yaniv Nikankin: Really, really great work.
Yaniv Nikankin: Nope.
Yaniv Nikankin: Stop sharing.