prakash.nik: Reprace plan. A different plan? Inclusive plan?
prakash.nik: Excellent.
prakash.nik: In the long…
prakash.nik: Should we start? Or should we wait for? Do you know if any of your friends are still coming?
prakash.nik: Maybe some people. Maybe some people.
prakash.nik: Okay, maybe… maybe they're all stuck in this frequency, but I don't know.
prakash.nik: Aristoph.
prakash.nik: No, that's not yours.
prakash.nik: You're safe.
prakash.nik: I mean, it's called Glam Bristols.
prakash.nik: basically forcing it into the action scenario, it's saying it.
prakash.nik: It's like you're at, you basically only happened.
prakash.nik: Are you a plant associated.
prakash.nik: So perhaps I'm moving… Measuring by their function.
prakash.nik: given states that are sufficient for an answer.
prakash.nik: Oh, there you go.
prakash.nik: I forgot how this one.
prakash.nik: How are you guys?
prakash.nik: See the change in the oscillation.
prakash.nik: The sound of the altitude.
prakash.nik: Okay, then I go to the beginning part.
prakash.nik: Oh, so… Even if I do… it's simple.
prakash.nik: So yeah, today I will be taking the class. Personally, David, or, yeah, David needs to attend a conference, so he couldn't join.
prakash.nik: In person? Yeah, so, yeah, I'm taking the class, and the topic, as you know, is circuits. Try to understand
prakash.nik: And we find, give some language models specifically?
prakash.nik: Yeah, so let's get started.
prakash.nik: Okay, so let's start with some history, because I like history.
prakash.nik: I think David mentioned about this paper a few weeks back.
prakash.nik: This is, like, the original bag propagation paper.
prakash.nik: From 1986, I guess.
prakash.nik: In this paper, Rumelhardt and…
prakash.nik: And basically, you proposed back propagation.
prakash.nik: And they not only proposed backpropagation, they also reverse engineered all the weights of the neural network that they trained using the back propagation on a dataset, which is something like
prakash.nik: Like, a family tree prediction kind of a data set?
prakash.nik: And, if you read that paper, at the end of that paper, what a
prakash.nik: The argument they are… that they are making is something like.
prakash.nik: Backpropagation is not crazy. It is actually learning genuine algorithm, which seems sensible to us. It can learn symmetry. That's why we should be using backpropagation. It's not like some crazy algorithm.
prakash.nik: But then, I think, over the time, we almost forgot about
prakash.nik: This aspect of backpropagation, that it can actually learn
prakash.nik: Genuine algorithms and representation, and we focus primarily on scaling things up.
prakash.nik: But eventually, I think people started thinking about that question again, something like 30 years later, when Chris Hola and their team at OpenAI started asking the same question.
prakash.nik: If we can reverse engineer, this huge models that we are able to
prakash.nik: a train now? Can we understand the features and the connections and the subnetworks?
prakash.nik: those networks.
prakash.nik: Yeah, so this was, I think, the first blog post within their thre… socket threads.
prakash.nik: where they, I think, analyzed Inception.
prakash.nik: Inception model to understand how is it able to Do so well.
prakash.nik: And they found various kinds of features, and they showed that, I think, the earlier layers learn shallow features, whereas the later layers learn more high-level features.
prakash.nik: But in addition to those features, they actually started talking about subnetworks as well. So I think that the question that they were interested was, okay, so we know that these features, the shallow features are in the earlier layers, and more complex features are in the later layers. So can we understand what is the relation between that? How the shallow features are actually getting transformed into more complex features?
prakash.nik: So that was their underlying question, and so that's why they started to look into the connections or the weights of the models as well.
prakash.nik: And they did some analysis, which is not very important, but the main point here is.
prakash.nik: I think this was… I would say…
prakash.nik: The, sort of, like, the first paper in the modern era, which started talking about subnetwork and circuits.
prakash.nik: Is there any specific reason why?
prakash.nik: They have observed that, final layers have more… learn more complex features, and, initial layers learn, which is, like, it could be other way around, or it could be a mix of, layers, because at the end, when we backdrop, those losses,
prakash.nik: They should be uniformly distributed or not.
prakash.nik: That's… yeah, that's a good question.
prakash.nik: decent question, but it's not super relevant to our discussion, I would say. But I think the answer to that would be.
prakash.nik: Since the model needs to learn a lot of things.
prakash.nik: it is actually trying to encode common features. I mean, that would be my intuition. It's not like.
prakash.nik: have done experiments on that. But my intuition would be, since the model needs to do a lot of things, it would probably learn to do simple things, which is sort of common across a bunch of tasks in the early ones, and get more specialized.
prakash.nik: layers, that could be one argument. Other argument could be, yeah, maybe you need more layers just to get more complex features.
prakash.nik: That's why you get more complex features into the later days, but that's…
prakash.nik: My reason to bring out this paper was just to share that this was, I think, the paper which sort of started using the term concepts in the interpretability space.
prakash.nik: So, I said concept, I meant circuit.
prakash.nik: The term circuit has been used in neuroscience for quite some time.
prakash.nik: critical engineering. In fact, interestingly, if you go out of that door, which he just… Open up.
prakash.nik: You will see, a notice board.
prakash.nik: there is a pamphlet. There is a poster for some kind of workshop. I think it's from some electrical engineering.
prakash.nik: kind of workshop, where they have a topic for a single… for 100 days, and circuit simulation.
prakash.nik: So yeah, my point is, circuit is a common term across a bunch of other disciplines as well. It's not like we just invented the term, but I think this was the first paper we started using the term in the interpretability space.
prakash.nik: Okay, so that was the historical context.
prakash.nik: But now, coming into, like, more specifics, what exactly I mean by survey. So, imagine this is your transformer…
prakash.nik: architecture. Yeah, you all know the transformer architecture. Now, think of
prakash.nik: This architecture is a computational graph.
prakash.nik: Where the nodes of the graph are basically your model component.
prakash.nik: The weights connecting them is basically the edges in your computational graph.
prakash.nik: So if you imagine that,
prakash.nik: Then, what I mean by a circuit is basically a subgraph of that computational graph, which connects the inputs to the biologics.
prakash.nik: Effective.
prakash.nik: It is just a set of components which are interconnected with each other.
prakash.nik: And have a direct passage from the inputs to the final one.
prakash.nik: That's what I mean by subject.
prakash.nik: Okay.
David Bau: So here, in that… in this… in this diagram is the… the gray stuff.
David Bau: The notation you're using to indicate it's in the circuit, or is the gray stuff for the stuff that's not in the circuit?
prakash.nik: God in this.
prakash.nik: Maybe I should.
David Bau: Oh, so the yellow stuff is the stuff that's in the circuit for you?
prakash.nik: Yes.
David Bau: Okay, cool. Just, just, just learning your diagram, that's all.
prakash.nik: Yeah, maybe I should be more explicit.
David Bau: Oh, Jasmine had a question, is that right?
prakash.nik: Yeah, but I think she… is she online?
prakash.nik: Yeah, she's on it.
Jasmine C.: Actually, my voice is, like, really gone, never mind.
David Bau: Okay, cool, we can just keep on going.
prakash.nik: Rice… You had a question as well about your second question.
prakash.nik: Was it the one about, like, task-specific, or… Oh, okay. So I think I was asking…
prakash.nik: I could imagine, like, a model component like…
prakash.nik: like, working on a bunch of tasks, right? So, like, it could, for example, I think we were reading about the IOI, like, it could be serving as the name overhead, it could also, in another task, could be serving as something else. So I was just wondering, like, it's kind of…
prakash.nik: Impossible to, like, actually, like.
prakash.nik: first engineer, like, all the modern components and, like, label them, like, universally. So is it, like, task-specific all the time?
prakash.nik: Yeah, so generally, when I think of circuit, I think of them as task-specific as well.
prakash.nik: That these are… this is the circuit for this particular task.
prakash.nik: But the high-level idea, or high-level hope of doing this kind of work is, is that probably someday we would be able to
prakash.nik: Have modulus circuits.
prakash.nik: That is… Sort of universal across different kinds of tasks.
prakash.nik: We will find some smaller surfaces which are modular.
prakash.nik: Which you can basically find out in different kinds of tasks.
prakash.nik: I think that's the high level of…
prakash.nik: It's funny, we've got to do lots of holes of…
prakash.nik: I guess so, I think if we can… Scale things up.
prakash.nik: And… this is just a really cool, small model.
prakash.nik: But let's say if we do similar kind of work in more than 100 billion model parameters, model.
prakash.nik: I feel we should be able to find some…
prakash.nik: Set of, like, smaller modular circuits, which is,
prakash.nik: generalizable across tasks. I mean, in a sense, you can think of induction as a circuit as well.
prakash.nik: Which we know is generalizable.
prakash.nik: Across bunch of times.
prakash.nik: But maybe induction is a little bit low level?
prakash.nik: So I'm hoping something which is a little bit more of a higher level than induction.
prakash.nik: Which would still make sense to people, but… and is also found across wines.
prakash.nik: Okay, so that was the definition of the circuit, but, I'll… Phase 1…
prakash.nik: alert here. A lot of people, when they're talking about circuit, they use third zone…
prakash.nik: Like, mechanism and circuits sort of, sort of interchangeable.
prakash.nik: Now, when I say circuit, I really mean set of color options. But certain people.
prakash.nik: When they say circuit, they actually mean the mechanism. Like, you can actually explain in English language how an input is getting transferred to the final output.
prakash.nik: You can explain what is the functionality of the… oral compounds.
prakash.nik: Which I… which I call mechanism.
prakash.nik: that's how, generally, I like to think, and I say set of moral components as a… and that's what I will be… I will be using.
prakash.nik: When I say so… Okay, so let's get into our first paper.
prakash.nik: And, IY paper.
prakash.nik: So I remember it was a 2022 paper. This came out during our first year in 50. This was my first semester, actually.
prakash.nik: I'm super surprised by this work. It's really thorough work.
prakash.nik: And the other reason for me getting surprised was the first author.
prakash.nik: Yeah, you know the reason.
prakash.nik: So, the first author, he was a high schooler at the time.
prakash.nik: and he, he's actually from Boston. I'm not gonna say fair in Boston, but,
prakash.nik: Yeah, I was super surprised to see that somebody…
prakash.nik: in high school can do this kind of research, this kind of such a thorough, high-level research. So I was super surprised by that, and that was…
prakash.nik: That's, like, a motivation for, sort of, all of us.
prakash.nik: You can't do it.
prakash.nik: Impactful gate resets nowhere where you are.
prakash.nik: Yeah, I just wanted to say that.
prakash.nik: Okay, so coming to their experimental setup.
prakash.nik: It's a pretty simple to ask.
prakash.nik: Yeah, so they have sentences like, John and Mary went to a store, John gave a drink too, and then, like, this next token after 2 should be Mary.
prakash.nik: So that's basically the indirect object, identification task. They studied GPT-2 small, which has, like, 12 layers and 12 engine.
prakash.nik: The metric that they use is called logit difference, which basically means that you take the logit of
prakash.nik: Like, the first person, minus the lobby of the second person.
prakash.nik: That's a… that's a larger difference. And the model?
prakash.nik: the GPT tool can perform the tasks.
prakash.nik: Almost 99, like, almost 100%.
prakash.nik: Okay, so the main contributions, like, I see this paper has two main contributions. One is the empirical
prakash.nik: contribution, which is the circuit itself. The other one is the methodological contribution, which is called path patching, which is important to understand how they actually came up with the circuit itself. So I'll first gonna describe the path patching algorithm intuitively, and then we will get into the details. So…
prakash.nik: Yeah, again, think of a neural network as a computational graph.
prakash.nik: Essentially, that's the question that path patching is trying to answer.
prakash.nik: So, given this computational graph, what are the important edges in that computational graph which are causally relevant for doing a particular task?
prakash.nik: So what you do is, you first need to come up with a counterfactual.
prakash.nik: And the kind of counterfactual that you come up.
prakash.nik: Is something which has completely different information.
prakash.nik: about the… Like, it should have different… Sh…
prakash.nik: Concepts which are relevant to your task.
prakash.nik: So it should still have similar structure.
prakash.nik: But the main thing that you're trying to understand should be different. So, for this particular In specific.
prakash.nik: Let's assume this is our clean example.
prakash.nik: And they basically used corrupted version of this task, which would have… 3 persons.
prakash.nik: And the… Yeah, they basically use a three-person task.
prakash.nik: Where you will have person, like, John and Mary went to the store, then you will have…
prakash.nik: Eric gave a drink too.
prakash.nik: So, this basically destroys the indirectness of the task, like, there is no connection between the third person and the second… first and the second person.
prakash.nik: That's the… Corrupted example that they used in this particular example.
prakash.nik: Okay, so once you have your corrected example, you feed both the examples in your model.
prakash.nik: And then what you do is, you start patching in the edges which are directly connected to the
prakash.nik: Find it out quickly.
prakash.nik: From the corrupted run to the clean one.
prakash.nik: Okay? And you do that for all the edges that are directly connected to the funnel. Only the edges which are directly connected to the funnel.
prakash.nik: You do that, And for each patch, you will see that the final output changes a little.
prakash.nik: So you need to measure that. So in this paper, they are using larger difference, so let's say you use that.
prakash.nik: Softer… this iteration.
prakash.nik: You will have a set of scores for all the edges which are directly connected to the final arm.
prakash.nik: And you use some threshold?
prakash.nik: To filter out the irrelevant ones, and only keep the ones which has high causal impact.
prakash.nik: On the panel.
prakash.nik: That's how you get, like, the first set of edges in your circuit.
prakash.nik: Now, once you do that, you basically need to…
prakash.nik: Do the same process again, but now, assuming…
prakash.nik: Instead of assuming logic or the final output as your target node.
prakash.nik: You use the notes which are identified in the previous iteration as you're targeting.
prakash.nik: Do the process again and again, until you reach the inputs.
prakash.nik: And at the end of the process, you will have something like this, where you will have all the important edges, which connects the inputs to the funnel outputs.
prakash.nik: Any questions?
prakash.nik: Okay.
prakash.nik: So that was the intuition, now this is the actual algorithm.
prakash.nik: I think… yeah, this is an important slide.
prakash.nik: If you understand this.
prakash.nik: Life would be good.
prakash.nik: Okay, so here, the…
David Bau: So, Nikhil, I find the previous slide a little bit easier to read than than this one.
prakash.nik: Okay.
David Bau: You know, just for whatever reason, it's just because of the layout. So I just wanted to give people just a minute to ask questions about the previous slide, or either one, before you get into it.
prakash.nik: What's the, the field?
prakash.nik: Yeah, I think that's an empirical thing that you have to…
prakash.nik: I don't think so there is any specific theory behind what should be the threshold that you should be using.
prakash.nik: It also depends on the metric that you use. In this paper, they use larger lengths.
prakash.nik: Sorry, not logical. It's logic difference.
prakash.nik: But that's not all… that's also not universal.
prakash.nik: Yeah, so I think you have to do it empirically, figure that out empirically.
prakash.nik: I probably missed this part, but how did they come up with the graph? Like, how did they come up with what is an engine?
prakash.nik: Yeah, so what exactly is an edge, and how do you patch one edge
prakash.nik: Like, how do you take an edge and pass it to a cleaner? That's a question that I'll be answering in the next slide. Yeah, that's the…
David Bau: So, can you… can you give, can you give a little idea? Like, so, what's… what's, like, the idea of what you're trying to do? Before you show how we do it? Like, what is it you're trying to do when you patch an edge? What would it mean?
prakash.nik: Okay.
prakash.nik: Let's use this diagram.
prakash.nik: Let's say I want to understand how this head affects
prakash.nik: this particular head. So there is a direct edge. There is an edge from this particular Edge one?
prakash.nik: to this particular H1.
prakash.nik: And I want to patch that particular edge. I want to know its significance or relevance for this particular task.
prakash.nik: I think the essential idea here is, You batch the output of… your…
prakash.nik: Like, the sender node, like, the node which is handling the information.
prakash.nik: While keeping in everything else in between constant.
prakash.nik: You need to make sure that nothing really changes.
prakash.nik: in between.
prakash.nik: When you patch in the output of this particular head.
prakash.nik: And then you need to figure out, like, you basically need to cash in the input of this particular head.
David Bau: So, I like this. I think this is funny, because there's this thing that philosophers sometimes say, which is like, what's… what's the importance of a thing? And they say… they use this phrase.
David Bau: All else being equal.
David Bau: Which is usually…
David Bau: It's just a mental experiment. I mean, it's usually impossible to keep all else being equal, you just…
David Bau: usually have to reason through it. Like, what would happen if everything else was equal?
David Bau: But I think the funny thing about this experiment that, you know, Kevin Wong, Rowan Wong, you know, proposed.
David Bau: was, he says, oh, you can physically keep everything else all equal. Like, if you want to know how important the edge is, you, like, physically, like, pin down everything else.
David Bau: Anyway, that's sort of the way I think of it. I'll let Nikhil get into the details, but I think it's easy to get lost.
David Bau: With all the details, but if you just kind of keep in mind, what he's trying to do is he's trying to keep
David Bau: He's trying to do, like, one of these all-else-being-equal experiments.
prakash.nik: just to reiterate what I was saying, yeah, basically, I want to see how does information from this particular
prakash.nik: head goes via the residual stream to this particular… so I keep everything else, the output of all the comparison between the same.
prakash.nik: And I essentially see, how does this particular head
prakash.nik: Affect this, affect the input of this particular head?
prakash.nik: Why the residual stream.
prakash.nik: That's the, I think, main idea.
prakash.nik: In terms of, actually, how would you do it?
prakash.nik: Again, we would have to come back to this particular figure, which is from their paper itself. I did not make it.
prakash.nik: Yeah, so here it is upside. So this is our input, this is our output.
prakash.nik: The red ones are on, corrupted sample.
prakash.nik: Green ones are on the green sample.
prakash.nik: And, goal here is to…
prakash.nik: Patch in the edge from this particular…
prakash.nik: Head to this particular head, and
prakash.nik: This particular head. I don't know why is this one green.
prakash.nik: Okay, okay. So, idea here is to patch in the edge from this particular head to this particular head, and this particular head to this particular head, okay?
prakash.nik: So what they do is, they again
prakash.nik: feeding the clean sample, okay? But now, they patch in the output of this particular head from the corrupted run.
prakash.nik: While keeping the output of all other heads.
prakash.nik: The same as the original one.
prakash.nik: Okay.
prakash.nik: You can imagine the output of MLBs from the green one as well.
prakash.nik: For now.
prakash.nik: If you assume that, then the red one, there would be only one path from this particular red to this particular red, which would be via the residual stream.
prakash.nik: Mike?
prakash.nik: NICC, that's what they are interested in. They are, like, we are interested in… Prior to…
prakash.nik: Patch in the dark edge between these two particular heads.
prakash.nik: Okay, so in this step, what we do is we patch in the output of this particular head while keeping the
prakash.nik: output of all attention heads fixed. You can also keep the output of MLPs fixed.
prakash.nik: And then, cache in the input of your receiver heads, which in this case is 2.0 and 3.1.
prakash.nik: Okay.
prakash.nik: Cache in the input of these two heads.
prakash.nik: Then comes the fourth step, where, again, you feed in the clean example.
prakash.nik: But this time, you patch in,
prakash.nik: The input of your receiver heads.
prakash.nik: which you cashed in in the previous step. And then you see what is the effect on the parallel.
prakash.nik: And that is equivalent to patching a direct edge between
prakash.nik: Your receiver, your sender and the receiver heads.
prakash.nik: Yeah, I think that's very confusing, I know that. So, ask questions.
prakash.nik: The third step, I think, is the main one.
prakash.nik: Does that make sense?
prakash.nik: Okay. The same as green one, or… Yeah, so in terms of notation, green one is what you get from the second one, red one is what you get from the first one, and the black ones are recomputed one.
prakash.nik: In the model this week, of course.
prakash.nik: Okay?
prakash.nik: Can you repeat the moment? Yeah, so…
prakash.nik: Yeah, the model just does the forward pass. You don't pass it. It just does the magic manifestation. Why the only path
prakash.nik: Feed of corners.
prakash.nik: Yes, ma'am.
prakash.nik: Assuming this is also green, if this was green, this wouldn't be red.
prakash.nik: Okay? Because we are dispatching.
prakash.nik: so, in that case, the red would only come…
prakash.nik: Wire the residual string to this particular… Okay.
prakash.nik: That's… that seems to be the idea. But if you… in… in this…
prakash.nik: At least the work that they have.
prakash.nik: presented in the paper, they have not kept the MLPs constant.
prakash.nik: They have not passed in the MLPs from the green room. They have let it length and utility decomp.
prakash.nik: Does that make sense?
prakash.nik: Is there a forward, like, after you've patched it, you're letting it, like… like, there's a forward pass or not, like, the MLP also decompute then? Yeah, it does. So then won't that MLP0 also be, like, yellow or red?
prakash.nik: So, the MLPs will have a new…
prakash.nik: value. Yeah. That's, that's what,
prakash.nik: The black ones are representing… I don't know why this one is red, why is this one red?
prakash.nik: Yeah, I'm not sure why this edge is red.
prakash.nik: I think they should have been glad as well. I think, is it because, if 0.1 is corrupted,
prakash.nik: that will… that will be computed inside MLP0, and, the output of MLP0 will be computed or changed due to, then this… okay, so in that case, these two edges would happen as well, but I think what they're trying to say is… okay, now I think it makes sense.
prakash.nik: This should have been red, but because you are patching this particular head, this particular path is basically blocked.
prakash.nik: That's why they did not make it right.
prakash.nik: Whereas this particular path, It's still red.
prakash.nik: Because… After its visual shrink.
prakash.nik: Any cuts?
prakash.nik: Nope.
prakash.nik: No.
prakash.nik: Okay,
prakash.nik: Question was, why are we not patching MLP, or the question was… Why is Resolute 1.0 and 1.1 not changing it at the forward pass?
prakash.nik: If you're… you're corrupting 0.1, and then there is another forward pass, you're recomputing.
prakash.nik: So shouldn't, like, the… anything that's downstream also change?
prakash.nik: Which component do you think should change? I would imagine 1.2 and 1.0 are also the same. Yeah, it will change, but since we are patching in the output of those two heads from our clean
prakash.nik: Clean samples.
prakash.nik: You're basically making sure that that does not change.
prakash.nik: Okay? We are basically blocking it. We are not letting it recompute. We are just putting in the value that it already had.
prakash.nik: then how can you guarantee, like, that it will be the same? Like, I understand if you're just patching,
prakash.nik: Very good. Yeah, it's a bit tricky.
prakash.nik: I mean, I have, like, I have… now that I kind of looked at this a little bit more, like, at a high level, like, why do you care about the direct effect
prakash.nik: when you keep the other things constant? Because I would imagine, oh, what this head does is it tells this other one to say this message. Like, the full message is the full message, what you care about the direct client. Yeah, because of the way they are trying to find the circuit.
prakash.nik: You remember the initial idea that I mentioned? You need to start from the end, then find the first set of heads which are directly affecting the panel.
prakash.nik: You don't want to care about anything in between. You want to just understand the effect of
prakash.nik: some set of… Upstream nodes to one specific downstream node. Once you figure that out.
prakash.nik: need to figure out what is the other set of upstream nodes which are affecting the set of nodes that you figured out in the first set. You don't want to… So, because if it doesn't affect you directly.
prakash.nik: then it's not the next step in the future. Yeah, exactly, exactly.
prakash.nik: Any more questions?
prakash.nik: Yeah, this is a bit tricky. That's why I said this is probably important, and this is… yeah. How do you connect the two slides? So, previously, fine software for circuit badges, right? But, in the lab… in the… in the figure in the paper, they supposedly choose some AND or some sort of to-do list parameters, so…
prakash.nik: I was just wondering, what's the connection between your two slides? The main connection is about how do you actually do this edge patching.
prakash.nik: You can easily do, like, node patching, like, you just take the output of a particular head or a neuron and take
prakash.nik: patch it from one particular run to another. That's easy. But the…
prakash.nik: The difficult one is how do you do the edge passing?
prakash.nik: Which is what I'm showing.
prakash.nik: You need to do to follow this path-patching procedure.
prakash.nik: But in terms of actually implementing it, it's what they are trying to say in this period.
prakash.nik: If you actually need to take a particular edge that you want to patch.
prakash.nik: These are the four steps you need to do to actually do it.
prakash.nik: That's a good question.
David Bau: So, I'll share a little perspective on it. It's, you know, this path patching thing is a little funny, it's a, you know, it's this…
David Bau: idea of…
David Bau: you know, it's this hope. You know, the circuit idea and the path patching idea, I think it chases this hope that we might be able to trace
David Bau: Causal chains inside these neural networks, the same way you might chase, like, a mystery, or chase a, you know, chase, like, a bug in a regular traditional computer program.
David Bau: you know, imagine, like, you're, like, a time traveler, right? And you want to, like, I don't know, prevent World War II or something. Then you might go back in time, and you might realize, aha, if I go back to a certain coffee shop.
David Bau: And I just moved this cup of coffee from one side of the table to the other.
David Bau: then… then it turns out that it has this… all these… all these follow-on effects, and it ends up, you know, preventing World War II. It's amazing, like a time travel movie, right? Well, that… that might be the case, but then you wonder, like, how did that happen? Right? There's a lot of steps from here to there, and I think that, like, traditional
David Bau: Traditional patching, like what you guys have been doing for your experiments so far, is like that coffee cup.
David Bau: experiment. It's like, you might get a very strong signal on moving that coffee cup from one side of the table to the other. It's like, oh, yes, this coffee cup turns out very important for World War II. But it doesn't really tell you, like, how it's, you know, how it had an impact.
David Bau: And so it's… it might be a little dissatisfying.
David Bau: And so… so really what you might want to know is the detailed causal chain. It's like, oh, if you move that coffee cup.
David Bau: Then, the immediate thing that happens is that, you know, somebody in that store can't, you know, can't pay for the coffee, or something like that.
David Bau: Or didn't, you know, can't find the coffee, and it fell asleep, you know, in the afternoon.
David Bau: And then because they fell asleep, they missed a meeting, and because they missed a meeting, some conference wasn't organized, and because the conference didn't happen, you know, some other idea didn't happen in the culture, and this thing wasn't invented, and, you know, had all these other effects, right? So you can kind of imagine this long causal chain.
David Bau: And… and traditional patching…
David Bau: doesn't concern itself with the details of the causal chain. It just says, hey, this is important, it has some downstream effect. You know, you might wonder what that is, but you can just see that it has the effect. Whereas what path patching is trying to do is it's trying to say.
David Bau: Well, you know…
David Bau: We're gonna try to figure out what effect it has to move this little cup of coffee, this little red
David Bau: 0.1 in C. You know, what's the effect of that?
David Bau: Well, if you just patch that by itself and you let everything else move.
David Bau: it might have some big downstream effect, right? But what they're asking is, they're asking.
David Bau: You know, why did it have the downstream effect?
David Bau: So, let's say there's somebody who misses… so let's say open wine is like the cup of coffee.
David Bau: What 2.0 and 3.1 are, are, like, the person who fell asleep because they didn't get their coffee or something.
David Bau: Right? Like, oh, that's, like, the next downstream effect that you're trying to test. And so you're saying, okay, well, missing the copy messed up a lot of things in the world.
David Bau: It means that the person who's cleaning up the table now has to wipe the table in a different way, it means that the dishwasher cleans up the coffee in a different way, you know, now there's… it's a full cup, somebody didn't drink it, you know, they made… they made a dollar less profit in the store, and that has some other effects, and so on.
David Bau: But… and so those are all the other green…
David Bau: boxes, it would normally be downstream of 0.1, and what they're saying is they're saying, hey, you know what, I think that… I suspect none of those really matter. What if we move the cup of coffee, but we don't allow
David Bau: Us to see the changes that are downstream of the cup of coffee, other than the things that we think are the decisive ones.
David Bau: So, fix the profit of the store.
David Bau: That's 1.1. Even though the store made one less dollar.
David Bau: Restore them to have the extra dollar for free.
David Bau: Right? You know, fix the… fix the dishwasher experience, that's 1.0. Yeah, even though the coffee was full, change the dishwasher experience a little bit. Pretend, pretend it didn't change at all. That's 1.0. Go to all the things and pretend it didn't have an effect. The only thing that you're gonna allow to have the effect.
David Bau: is, like, 2.0, 3.1, right? Like, just let the… let the effects come through only to these new things, only the specific things. That's… that's the attempt. It's asking this… it's asking, you know, a more…
David Bau: Like, if everything else was the same, kind of experiment.
David Bau: And, anyway, that's what it's attempting to do, to try to make it a little bit more satisfying. I don't know if that helps at all, but I always found it a little…
David Bau: A little confusing, all the setup.
David Bau: Is the way I think of it.
David Bau: Does that make any sense? Anyway, I'll shut up again. Sorry, it's hard to… hard to lecture from remote.
prakash.nik: It's okay, yeah, I think you have a question. Yeah, why don't we start from the end, and then…
prakash.nik: Not from the beginning.
prakash.nik: Like, in the coffee cup experience experiment that was, like, starting from the beginning.
prakash.nik: And, looking forward, but this would be starting from, I guess, World War II, and then slowly move backwards to get back to the coffee shop? Like, why did we do it that way as opposed to…
prakash.nik: Start from, the inputs, and then, patch forward.
prakash.nik: Yeah, so I think it's because… let's say if you start by, like, figuring out what is the…
prakash.nik: most important… It's just… it's a big name.
prakash.nik: Well, you don't really have anything…
prakash.nik: You don't really know which are the important set of nodes in your first layer.
prakash.nik: I think the way…
prakash.nik: the way at this path patching works, you need to know what is the important downstream,
prakash.nik: Nodes in your computational graph that you care about.
prakash.nik: If you don't know that.
prakash.nik: I don't know how would you do that.
prakash.nik: Yeah, so I think that's just how the framework plays out.
prakash.nik: So… Yeah, that's what I was saying.
prakash.nik: I wonder if you could do it, it just would be less efficient. Like, if you only know
prakash.nik: And the outcome that would come be changed by some edge at the end.
prakash.nik: And if most of them don't have any impact, then you don't need to push.
prakash.nik: Is that right?
prakash.nik: Like, you have… if the…
prakash.nik: output is gonna change. There has to be, like, you know that something needs to change at the…
prakash.nik: And so, if you're like, oh, look, like, most of those nodes don't… like, those edges don't…
prakash.nik: impact anything, that I don't need to look… I don't need to search… Down.
prakash.nik: Look at those edges out of.
prakash.nik: Yeah, but you still need to start from there.
prakash.nik: Right, and I'm saying, like, if you start from the end, then you can eliminate… it's just more efficient. Not that you can't do it both ways, it's just that it's more… It should be definitely…
prakash.nik: efficient, and I'm still not sure how would you be able to do that from the start.
prakash.nik: you could, like, just do the prolifer, you could just, like, brute force 7… So let's say this is your input.
prakash.nik: Let's say all of them is your output.
prakash.nik: And then you find some edge which has high…
prakash.nik: Yeah, what would that even mean?
prakash.nik: Yeah, I just, like, you could just do every single one. Just every single edge.
prakash.nik: We could look at every single graph. It doesn't… I'm not saying it's a fit, like, would be tractable in any way.
prakash.nik: And I'm not sure if it's untractable, intractable. So, yeah, let's say maybe it is.
prakash.nik: You had a question? Yeah, just go on, you know, in my Monday analogy would be, okay, you move the coffee cup, and then that affects, like, 10,000 different things.
prakash.nik: and each of those things affects, like, 10,000 others, which would be, like, an inference. In that case, unfractable search. In a neural network.
prakash.nik: Technically, it's finite, so it's not infinite, but it would make much more sense to say, well, actually, if someone missed this meeting.
prakash.nik: then that would have avoided World War II. And if, like, this bus was late, then this meeting would be avoided, and then you're kind of searching
prakash.nik: in a tractable way, I think… yeah, your… I think your approach would be technically tractable, just because it's… you have a finite amount of things, but it could be still, like, a bit. Yeah, that's a good amount. I mean, it's very legal expensive.
prakash.nik: Yeah, I think it does, yeah.
prakash.nik: Okay, so we have spent 45 minutes, so I think we've…
prakash.nik: So, essentially, that's the path patching algorithm, and they use this particular algorithm to figure out the circuit for the IOI task.
prakash.nik: So as I state, they start with the end. They try to understand what are the needs which are directly affecting the panel logit.
prakash.nik: They apply the path patching algorithm, and these are the… the red ones are basically the most important ones.
prakash.nik: Yeah, these are the first set of heads that they find with the part matching.
prakash.nik: They also tried to understand the functionality of this head, which I'm not talking a lot about right now. I have a few slides at the end. But essentially, what they found was
prakash.nik: that these are the heads which are attending to the Mari token, and fetching its value.
prakash.nik: That's why they call it Name Overheads.
prakash.nik: Then the question becomes, okay, these are the heads which are fetching in the MARI token.
prakash.nik: What are the next set of heads which are actually affecting those heads? Like, how does the name overheads know that it needs to attend to Marito?
prakash.nik: So for that, they… what they do is, they apply the path hatching algorithm.
prakash.nik: But in this case,
prakash.nik: The target node is basically the query vectors of the name overheads.
prakash.nik: And then the standard nodes are basically each of the…
prakash.nik: Don't see the previous layers of that model.
prakash.nik: Any guesses, why query not key or value?
prakash.nik: video about doing something. The idea here is to find out
prakash.nik: Where or how does the name overheads know to attend to Mary Global.
prakash.nik: The way attention works is, it is the query vector of attention head, which encodes the information what it needs to attend to.
prakash.nik: And the key vectors encodes, what is it actually encoding?
prakash.nik: And… and we know that the…
prakash.nik: So our attention helps. Attend to the MERI token, that's why the key token at the last…
prakash.nik: The key vector and the last token doesn't really matter.
prakash.nik: And for the vect… like, the value vectors, we already know that it is fetching in the values, from the meritome.
prakash.nik: So that's, like, the other way to think about it.
prakash.nik: Yeah, so that's basically the next step.
prakash.nik: using this particular step, they found our next set of attention heads.
prakash.nik: Again, they tried to understand the functionality of this head, and what they figured out was these are the heads, which are
prakash.nik: sending in information about the John token to the NameMoverhead, which is what the NameMoverhead is using.
prakash.nik: to not attend to John, and that's why they are calling this particular set of heads as S, innovation heads.
prakash.nik: Yeah, the functionalities are not very important,
prakash.nik: bit about the functionalities at the end, but I'm just trying to show you how you can apply path patching algorithm to find the set of heads in each step.
prakash.nik: To find the overall sun.
prakash.nik: And they continue the same step again and again.
prakash.nik: They actually reach the input, and this is the overall circuit that they found.
prakash.nik: Okay, now we will take question. I think we'll start with Hayou.
prakash.nik: Yeah, the retraining impact.
prakash.nik: They will… He's hype.
prakash.nik: you know, I… hyphen.
prakash.nik: Are you, if that's correct pronunciation, are you on the Zoom?
prakash.nik: Maybe he's… So the question was, if we retrain GPT-2 from scratch.
prakash.nik: Will we get same set of… Hence, or not.
prakash.nik: That was first… You'll need to unlock…
prakash.nik: That was the first part of the question. The second part, will the functionalities of those heads will remain same or not?
prakash.nik: What do you guys think?
prakash.nik: Attention wheels will change. Okay.
prakash.nik: Depending on the data that we created, or we created. Let's say it's the same data.
prakash.nik: Exactly same data.
prakash.nik: If there's stochasticity within the training that could affect its changes slightly, but… Good feet.
prakash.nik: Okay.
prakash.nik: What about functionality?
prakash.nik: Uxaraki should remain safe.
prakash.nik: I think.
prakash.nik: Any other guesses?
prakash.nik: I think at the beginning you said that you expect some circuits to be modular and present in a lot of different networks, so you would expect
prakash.nik: functionally, some…
prakash.nik: heads or circuits to be present in the same network on Ukraine, maybe in another place, because it's from the old spot.
prakash.nik: Yeah, so this is what I would expect, although I've not done the experiment.
prakash.nik: I think the exact heads would change.
prakash.nik: Because… random initialization. I think the initialization would be different, so even if the integer is the same.
prakash.nik: I am quite confident that the heads won't be the same.
prakash.nik: You don't think so.
prakash.nik: Okay, yeah, but we have not done the experiment, but I feel because of the random and initialization, I feel the head should be different.
prakash.nik: But the functionality should remain the same, since we are training on the same data. So I feel that at least the mechanism of how the model will solve the task should remain the same.
prakash.nik: Okay, now a few other questions. Aria, Ananya, and Jessabhar.
prakash.nik: Yeah, I think all three of you had, like, a similar kind of question about the generalizability of the result across different kinds of tasks.
prakash.nik: You wanna state that? Yeah, I guess to me, it was like, how useful is this? Like, for example, in this paper, they looked like this particular example. If instead of having Mary, John, John, we had, like, three names.
prakash.nik: Would, like…
prakash.nik: how is this… yeah, basically, I understand that usefulness of path-patching algorithm, I think that that's… I feel like that's a useful contribution, but looking… what's the use of looking at this as a circuit?
prakash.nik: Because you have, like, a million circuits for different tasks, and different data, and yeah, I guess others. Yeah, okay,
prakash.nik: Okay. Yeah, so I think I see similar… similarly. I think, as I said, I think this paper has two main contributions, the empirical circuit and the method. I think method is more important.
prakash.nik: Circuit is there, they found some set of heads and the functionality.
prakash.nik: But yeah, I don't think so it's super gentle. I mean… That in itself is gonna…
prakash.nik: I will use it now.
prakash.nik: In terms of how you think about it.
prakash.nik: But again, you have to think that… you have to…
prakash.nik: think about it in the context. This was 2022. This was probably the first paper which figured out the circuit in an end-to-end fashion. So from that perspective, I think that was…
prakash.nik: Pretty phenomenal.
prakash.nik: Let's go ahead.
Jesseba Fernando: Wait, sorry, could I… my question was, like, a little bit different. I… I had just asked about, well, here it's, like, there's Mary and John, there's two… two, like, people, and so, I was wondering what would we expect to see if there were more than two?
prakash.nik: Yeah, I think in that setting, what I would expect is something more like entity tracking paper.
prakash.nik: that we have. It's like…
prakash.nik: this is, again, my expectation, that the model would probably use the positional information or the ordering ID information of each of the person to basically figure out
prakash.nik: Like, different attributes, about that particular person.
prakash.nik: Depending on what you ask about that person.
prakash.nik: I think primarily it will still use those traditional or the ordering ID information to answer questions about that particular person.
prakash.nik: available?
Jesseba Fernando: Yeah, yeah, thank you.
prakash.nik: Yeah, Avery, Christopher, and Rice.
prakash.nik: You guys had general security across models, kind of question.
prakash.nik: Yeah, it was kind of the same as, like, Tony's question about retraining. If we were to retrain GPT-2, or if we were to do this analysis on other models, like, do we see similar… similar-ish circuits? Like, are there…
prakash.nik: As in, has that, feed into name movers, etc, or is it, like, totally different based on different models?
prakash.nik: Yeah, so this is another paper, which was not in the reading list.
prakash.nik: This is from Jack.
prakash.nik: So essentially, in this work, what they did was they tried to study this IOI task in
prakash.nik: like, larger GPT-2 models, like the medium and, I think, large.
prakash.nik: And I think what they showed was, the mechanism remains more or less the same. The models are still using those NF.
prakash.nik: I mean, not the exact same heads, but the way the model is solving the task, it means the same.
prakash.nik: And, another main result of this paper is, if you look at
prakash.nik: The significance of this… those particular heads.
prakash.nik: In different tasks.
prakash.nik: I think they studied one another task called colored objects, something like this object is of this color, this object is of this color.
prakash.nik: What is the color of this object? If you have this kind of question.
prakash.nik: and the heads that they've, like, the previous work found for the IUI task is actually being reused for this particular task as well.
prakash.nik: So, so the same, same set of heads are being reused, for different kinds of tasks.
prakash.nik: Yeah, that's what I wanted to say.
prakash.nik: Can you identify these circuits without knowing the architecture of the model?
prakash.nik: say we have, like, access to some new plot model, and we have our…
prakash.nik: need at least gradients. If not… if you can't do patching, you still need gradients.
prakash.nik: Yeah, you would still need internal activations, or…
prakash.nik: like, I went for, like, you don't know what the architect… like, we know the architectural object, you know, how many layers are there.
prakash.nik: See, we get access to the… I don't know how it's possible, if you get access to the internal activation, but can that help, like…
prakash.nik: backtrack the architecture, so if you have internal activations, can you construct the architecture? Yeah, I think once you have the activation, you basically have to… what about the GPT OctoSS open source products, so, like, we have the model bites, but we don't know the exact architecture, initial batch them.
prakash.nik: You don't know the architecture, but you have the weights? Yes, we have the model weights, but they are not disclosed model architecture.
prakash.nik: Yeah, I think I should need to work on emergencies. I've not worked on… how is that possible? If you know the weights, you basically know the architecture, right? I am not 100% sure of that. Maybe they flattened the masses.
prakash.nik: They flattened it? Then how do you work with that? I'll talk with you about it later. Yeah, okay, I'm not sure. Yeah.
prakash.nik: Because you basically know the weights of the, like, the dimensions of the matrix.
prakash.nik: Every metrics.
prakash.nik: Yeah, that's true.
prakash.nik: But, yeah, terms will model suit that. Okay, maybe I… yeah, exactly.
prakash.nik: Okay.
prakash.nik: Yeah, one another thing is, yeah, so I said the…
prakash.nik: Per set of… or the final set of heads which affect the final logic.
prakash.nik: are called name overheads. An interesting thing that district found was, if you have laid out those name overheads.
prakash.nik: The model's performance on this particular task does not really decrease.
prakash.nik: What really happens is, some other set of heads comes into the picture, which are not really…
prakash.nik: Causally active in the normal state, but when you update out the name overheads, this new set of
prakash.nik: Heads comes into the picture, which they are calling as backup name overheads.
prakash.nik: I think Claire had a good, like, an insightful comment.
prakash.nik: comment? I think it was insightful. I didn't know about it.
prakash.nik: It was about neuroscience?
prakash.nik: Oh, redundancy? Redundancy, yes.
prakash.nik: There's a theory in neuroscience that the reason why it's so hard to…
prakash.nik: Find neurodegenerative diseases, because the brain is really good at having redundant pathways for the same path.
prakash.nik: And so, there could be, like, in a circuit, that level of the same redundancy, I thought.
prakash.nik: Yeah, that was insightful, I didn't know about it.
prakash.nik: How does the human brain work?
prakash.nik: So I just wanted to bring out that point to show there might be some
prakash.nik: Analysis between what we're looking at, quite clear from this spectrum.
prakash.nik: Okay, so we have found the circuit.
prakash.nik: But that's not the end of the game. We have to evaluate the circuit to actually show that we have found something genuine.
prakash.nik: And the paper proposes three metrics, or criterias. The first one is called, faithfulness. It's, I think, the easiest one to understand.
prakash.nik: We basically asked this question, is the identified circuit fit to the underlying wall?
prakash.nik: Can I get the performance of the model just from the circuit itself?
prakash.nik: And that's basically the metric. You compute the…
prakash.nik: Final output, like the final response of the circuit.
prakash.nik: And just divided by the performance of the model.
prakash.nik: And when I say performance of the circuit, what do I mean is, I basically, you basically are blade out.
prakash.nik: everything else which is not part of the circuit. You keep the circuit components intact, but you ablate out all the model components which are not part of the circuit, and then check the final output.
prakash.nik: Yeah, so in this work, the circuit they have found has about 87% is of faithfulness.
prakash.nik: Yeah, Yuki had a question about implementation.
prakash.nik: Yuki, are you online?
prakash.nik: Maybe not, but I think it's the simplest. You just upgrade out all the model components which are not part of the circuit.
prakash.nik: Get the fulfillments?
prakash.nik: And just divide it by the model performance.
prakash.nik: The next one is a little bit tricky, it's called completeness. So, the question here is, okay, we found the circuit, but how do we know that we have found all the components of the circuit?
prakash.nik: Is it complete, or we are still missing out something?
prakash.nik: yeah, so the basic idea here is, if you assume this is the underlying Socket.
prakash.nik: Where you have, like, two redundant components, like, sort of, like, redundant subcircuits, which are getting
prakash.nik: And what operation, which are…
prakash.nik: Combined with an auto operation here before the final update. So, get complete paperless, even if you buy just one of them.
prakash.nik: But that's not what we want. We want to find both C1 and Z2.
prakash.nik: No.
prakash.nik: That's basically the idea here. And this is the equation to find the…
prakash.nik: Or to… to get that incompleteness score.
prakash.nik: What this equation means is, He is some subset of… The circuit confidence.
prakash.nik: T is the circuit, M is the model.
prakash.nik: So you compute the circuit com- performance?
prakash.nik: I am bleeding out.
prakash.nik: Okay? He's alive.
prakash.nik: Subtracted by the performance of the model.
prakash.nik: When you ablate out that particular same subset.
prakash.nik: The idea here is… The difference between these two should be minimum.
prakash.nik: If the model is complete.
prakash.nik: They should have the same impact. So, for instance, if… C is only C2.
prakash.nik: And K is, let's say, C.
prakash.nik: So, K is C2 as well.
prakash.nik: So if you upgrade out this thing completely, You check the performance of
prakash.nik: That particular circuit, you still get complete performance.
prakash.nik: Right?
prakash.nik: Let me…
Jasmine C.: Wait, Nikhil, I have a question. Like, how much of this is specific to, like, circuits formed
Jasmine C.: I'm so sorry, using backpropagation?
Jasmine C.: Like, is that, like, one of the assumptions we're making here?
prakash.nik: Yeah, I think we're talking about, like, neural… like, normal neural networks which are trained from backpropagation.
Jasmine C.: Okay. So, like, if we use a different, like, updating, like,
Jasmine C.: Like, mechanism, like, it… this doesn't… isn't necessarily true.
prakash.nik: I think you'd still be able to apply this… this method.
prakash.nik: anyone not using backpropagation.
David Bau: Yeah, Jasmine, did you have another update mechanism in mind?
Jasmine C.: Like, I was thinking of, like, like, maybe, like, Hebbian learning or something, like, where you're using the co-activations, or,
Jasmine C.: Yeah, I think that was one of the ones I was thinking of.
David Bau: Yeah, I think that probably not a lot is known, because so much of the, like, so much of the field
David Bau: is just…
David Bau: for the last, you know, whole bunch of years, you know, decade and a half has all been back propagation. So, it might be that the question you have has a different answer, but I think that people might not know.
Jasmine C.: Thanks, everyone.
prakash.nik: Okay, so we were talking about this named completeness code.
prakash.nik: Can't play on the widgets.
prakash.nik: David, do you remember this one?
David Bau: What's the question?
prakash.nik: Did you…
prakash.nik: Basically, if you remember this particular criteria, the company does want to wider this particular question. Next, please.
David Bau: Oh, the… the completeness and incompleteness score, like, how to… how to… how to justify
David Bau: Why you would score it this way? Is that what you're asking?
prakash.nik: Yeah.
David Bau: If I'm being perfectly honest, I… you know.
David Bau: no, I, I don't, I don't totally remember, why we, why, why it was proposed.
David Bau: To do it this way. I think that…
David Bau: You know, the intuition that you gave is the right one, which is basically you're asking the question.
David Bau: you know, Once you have a circuit.
David Bau: It's the same question that you would have if somebody published a paper saying, hey, you know what, I know what caused World War II, it's this coffee cup, and
David Bau: You know, and the whole downstream chain of events that come from this. And then people would say, well.
David Bau: I don't believe it. I think that there's some other…
David Bau: path, a causal chain.
David Bau: That also explains World War II.
David Bau: And… and I think that your coffee cup is… theory, is only a partial A partial,
David Bau: you know, explanation for the whole thing. You know, there's 11 other, you know, key variables that are involved. They all have to be involved.
David Bau: And so I think that this idea is trying to capture the idea that if you…
David Bau: If you try to form the circuit in any other way.
David Bau: If you had a complete circuit, then no matter what you removed.
David Bau: It would make the explanation worse.
prakash.nik: I got it, I got it.
prakash.nik: So, the rational here is, this C when K… Is removed.
prakash.nik: I think they're saying, like.
prakash.nik: If you say that the circuit for this task is C1, so if C equals C1, and if you set K to C1 as well, it will have a low inconvenience score.
prakash.nik: Or, like, the high incompleteness score, so… Or what? So C can… because C slash C1 will be… you will have, like, 0, right, because it cannot complete the task. The second one, AM slash C1,
prakash.nik: It would still complete the task.
prakash.nik: Because you have C2, so you have a high incomplete score, so that means you can't set C to C1. It's an incomplete circuit.
prakash.nik: Yeah, and the reason why C… when K is set to C1 would be… basically, this metric would be low, is because, when you are…
prakash.nik: You have nothing else.
prakash.nik: Yeah, exactly. So you're basically updating our entire model here.
prakash.nik: Assuming this C1 and C2 are the only components in the model.
prakash.nik: And when you're computing the performance of the circuit, what you do is, you only keep the components of the circuit.
prakash.nik: intact, while ablating out everything else, okay?
prakash.nik: So, assume that K was not here.
prakash.nik: And this C1 was… C was C1, okay? So what would that mean is, we are only keeping C1 active, and C2 would be completely laid off.
prakash.nik: Okay? But now, let's say K is C1 as well.
prakash.nik: KC1. So now, in addition to upgrading out C2, C1 will be upgraded out as well. So there is nothing in the model, so the model performance will be almost zero.
prakash.nik: Okay?
prakash.nik: Pardon?
prakash.nik: Spot M?
prakash.nik: For the second term, We are only updating out C1.
prakash.nik: Which… or whatever circuit that you decide, whatever C…
prakash.nik: That you decide to be the circuit, whether C1 or C2. Assume it is C1, so we will only update out C1, while the C2 still remains intact. That's why the model performance will still be high.
prakash.nik: That's why their absolute difference will still be high.
prakash.nik: And… Therefore, we can say that whatever… circuit that to this idea.
prakash.nik: Or whatever circuit that we have found is actually conflicted.
prakash.nik: Does that make sense? Do you want me to repeat again?
prakash.nik: We were putting out horse C and Jose M.
prakash.nik: C is the circuit that you identified.
prakash.nik: K, sorry, and M is the model.
prakash.nik: Okay.
prakash.nik: And you find F of C by keeping the model, like, the components in C intact, while upgrading out everything else.
prakash.nik: And what this is saying is.
prakash.nik: In addition to updating out everything else, you update out that subset of C,
prakash.nik: Components, which is key as well.
prakash.nik: Yeah.
prakash.nik: Okay, so that was… Completeness. Now, this is the third one, which is minimality.
prakash.nik: So you can have a very vague circuit.
prakash.nik: Assumed your entire model is a circuit.
prakash.nik: then you would have very high faithful and high completeness score for your circuit, but that's not what we want. We want the circuit to be…
prakash.nik: Something of a smaller…
prakash.nik: order. We want it to be as minimal as possible. So that's the criteria… so that's the third criteria that we use to evaluate our model.
prakash.nik: This is, I think, even more tricky.
prakash.nik: So what do you do is…
prakash.nik: In this particular… this is, again, the equation to find the minimality score for each particular component in your circuit.
prakash.nik: and assume that particular component is V.
prakash.nik: What you do is… Yo… remove?
prakash.nik: some subset K?
prakash.nik: Along with that particular component, console socket.
prakash.nik: It's subtracted from… Just removing that subset. From the socket.
prakash.nik: Just to give you an example, let's say we are looking at one particular name overheads.
prakash.nik: And assume that the K is basically all the, like, group of…
prakash.nik: All the name overheads, except that particular
prakash.nik: Okay, that we are looking into. So what the first expression would mean is.
prakash.nik: We are getting rid of all the name overheads.
prakash.nik: Yeah, basically, we're getting rid of all lame overheads, that's the first expression.
prakash.nik: The sec… second?
prakash.nik: Second expression is, We are getting rid of all but that particular head in our remover.
prakash.nik: Nope.
prakash.nik: And since we have found that that particular head is… let's say that particular head is…
prakash.nik: Positively relevant. That particular head will make the model get to the correct answer.
prakash.nik: And that's why the later term will have a higher score, while the previous one will have almost no score, no performance, because we have basically upgraded alternatives.
prakash.nik: And that's why, that particular component would be critical.
prakash.nik: It's like… it is a critical component that you need to keep in the circuit.
prakash.nik: But if that particular score is not that high, that means that that component is not really that critical.
prakash.nik: and that's basically what this minimality criteria is about.
prakash.nik: Okay.
prakash.nik: I'm gonna go ahead.
prakash.nik: But any questions in this particular previous paper? I'm now gonna go to the next paper.
prakash.nik: So we learned about path patching, we found the circuit of GPT-2, which is responsible for doing the IOI task, and then we evaluated the circuit on three particular criterias. Faithfulness.
prakash.nik: completeness and minimality. Even if you didn't get completeness and minimality, it's okay, as long as you got faithfulness, I think. That's okay.
prakash.nik: Okay, now coming to the second paper, yeah, I think…
prakash.nik: How do I can announce it?
prakash.nik: Kyung?
prakash.nik: That's fact.
prakash.nik: Yes, I'm here. Are you online?
prakash.nik: Yeah, basically the question was, should we…
prakash.nik: What should be the focus of…
prakash.nik: Mechanistic interpretability? Should we still focus on human-driven, or should we… Focused more on automated interpretability.
prakash.nik: So, I'm in the favor of automated interpretability. I think you have seen the previous paper, how difficult that is. So, it was so difficult to explain.
prakash.nik: assume how difficult that would be to do it. So nobody wants to do it. Everyone just wants an agent.
prakash.nik: you can just write a prompt to find a circuit, and the agent will give you the circuit. That's probably the ideal goal. So I'm very pro…
prakash.nik: Automated interpretability, but yeah, again.
prakash.nik: We are still not there yet, so we still need to find a few things manually, but hopefully.
prakash.nik: A few years down the line, we have an agent which can do most…
prakash.nik: So yeah, in this paper, they basically…
prakash.nik: talk about one particular step in the steps, like, broadly speaking, steps that we took, in the circuit discovery process. The first step was… and this is something that we have
prakash.nik: recommended for this class as well. Come up with a particular task, come up with a particular data set and the metrics and the model which can do it.
prakash.nik: Then you need to decide the level of analysis that you want to perform, whether you want to work on layers, neurons, heads, or maybe subspace level.
prakash.nik: So once you have decided Once you have done the step 2, you basically have the combinational graph.
prakash.nik: Now you need to tune out that competition, you have to figure out the subway.
prakash.nik: That's the third step. So this particular paper basically automates that third step.
prakash.nik: The algorithm is pretty straightforward, so that's why I just pasted in the algorithm itself.
prakash.nik: What you do is, once you have that computational graph.
prakash.nik: you topologically sort it in the reverse order. So you start… you still start from the end.
prakash.nik: It's still starting from the logit?
prakash.nik: And then you look at each of the edges, which are directly connecting.
prakash.nik: With the final budget.
prakash.nik: What you do is, for each of those edge, you basically remove it from the… computational graph.
prakash.nik: And see if the final output changes significantly or not.
prakash.nik: If the final output changes more than a particular threshold.
prakash.nik: You make, or that basically means that that particular head is imported for doing this particular task.
prakash.nik: So you keep that edge in the circuit.
prakash.nik: You basically do the same process again and again.
prakash.nik: And each time you keep a, like, edge.
prakash.nik: You basically add that, but, like, it's…
prakash.nik: Node, or its parent node, as part of your queue.
prakash.nik: And then you do the same process every NIP. It's basically like breadth-first search, but…
prakash.nik: The… the criteria for keeping an edge.
prakash.nik: In the circuit or not is basically defined by the threshold value.
prakash.nik: Is that clear?
prakash.nik: And it's the same word we…
prakash.nik: I think, at least in the paper, they have not used dynamic.
prakash.nik: They have music content.
prakash.nik: Yeah, it's pretty straightforward.
prakash.nik: So this is the algorithm.
prakash.nik: So, they evaluated, or they basically found out the socket for a bunch of tasks.
prakash.nik: Including the IOI task, and a few other tasks.
prakash.nik: They basically use the ROC curve to show… to basically compare
prakash.nik: their proposed method, ACDC, with a few other methods, which are not super important.
prakash.nik: But I just wanted to put it here, because I wanted to show you that they are evaluating it as an ROC curve.
prakash.nik: Which basically means that they're assuming that there is a ground truth.
prakash.nik: so, what they did was, they basically went out
prakash.nik: I looked at the previous literature.
prakash.nik: To figure out which are the previous works.
prakash.nik: Where circuits are already available.
prakash.nik: They use their method to find out
prakash.nik: the circuit for the same task, and then basically compare those two circuits to come up with this ROC curve.
prakash.nik: Courtney had a question. How do you get the ground truth? Is that clear? Yeah, how do you know that something is the ground truth? It's basically coming from the literature. You assume that whatever previous published work is there is correct.
prakash.nik: But, yeah, again, there is… yeah, there is caveat to that. Maybe even the published work is not exactly correct.
prakash.nik: I think even in the paper, in this one or the next one, there was a point I remember.
prakash.nik: There's a huge number of edges in the model.
prakash.nik: Then it's a particular tool would have, like.
prakash.nik: Yeah, it will have a lot of inches.
prakash.nik: So it is not unlikely that maybe even the researchers whose papers are published, they missed out on a few edges. So the ground… you need to take the ground truth with a pinch of sort of.
prakash.nik: Okay.
prakash.nik: Now I'm gonna ask questions.
prakash.nik: Problems with ACDC.
prakash.nik: What kinds of problems do you see associated with this particular model?
prakash.nik: Is this not going to come to this stage?
prakash.nik: Yeah, that's right. It's kinda… it is very complicated.
prakash.nik: It grows linearly with the… number of edges in the model. So it is varying.
prakash.nik: other… Problems?
prakash.nik: But… Why is that wrong? So in the previous course I had, we…
prakash.nik: See how minimal changes in the input of the, like, an simple MLT.
prakash.nik: It affects the output.
prakash.nik: And then, at each layer, the,
prakash.nik: it would result in a larger amount. But in that case, we were, giving a…
prakash.nik: The problem was making the smart balance tighter.
prakash.nik: But anyways, it would grow larger.
prakash.nik: So, someone from… That intuition, I think… Threshold should be different.
prakash.nik: Thank you for negatives.
prakash.nik: Yeah, I think, intuitively, I agree with that. Let me…
prakash.nik: But the way I would, say that would be…
prakash.nik: Even at the end of the computational graph, since they are closer to the final output, they might have a higher effect on the final output.
prakash.nik: But that's why they're… the impact
prakash.nik: or changing or perturbing those edges would be pretty hard, in comparison to perturbing earlier layer edges. That's why maybe the threshold should be different.
prakash.nik: Yeah, I agree with that, although that's not all.
prakash.nik: slide, but I… intuitively, I agree with that.
prakash.nik: Any other issues?
prakash.nik: Yeah, okay. I guess if you do so many comparisons…
prakash.nik: You will find a lot of random.
prakash.nik: significant results.
prakash.nik: So you will have a lot of screwdrivers.
prakash.nik: circuits that don't actually mean anything. You just found them because you did so many tests.
prakash.nik: You've got to explain that a little bit. So, when you say so many circus max, what do you mean? I mean, when you do statistical tests, if you do thousands of tests, if you write, for example.
prakash.nik: Thousands of people, and services.
prakash.nik: I don't know, something.
prakash.nik: Like, in at least one of them, even if there is no effect, you will find a correlation, significant correlation, because of how
prakash.nik: with the chimps.
prakash.nik: So, I feel like this is gonna be the same problem since you…
prakash.nik: sort of, you look for any possible significant
prakash.nik: Resolved, and a lot of them might be just… Everybody jumps.
prakash.nik: I'm not sure where stochasticity would come into this… Oh, I see. This algorithm.
prakash.nik: It's pretty little twister.
David Bau: I do think that this is a…
David Bau: reasonable form of a concern, so I don't want to dismiss it.
David Bau: I think this is one of the issues with…
David Bau: These large-scale interpretability methods in general, the things that say.
David Bau: Scan over everything, test everything, and whether everything is, like, thousands of choices, or millions of permutations, or something like this.
David Bau: Then you're definitely… Even if it's not statistical.
David Bau: in nature, right? You know, even if it's not about probabilities or whatever, you definitely have this situation where
David Bau: you know.
David Bau: Well, you… you're… if you… if you look enough.
David Bau: You're gonna find what you look for.
David Bau: And then you're left with this question, is it…
David Bau: Is that meaningful? Is it surprising?
David Bau: I think that the… You know, the typical…
David Bau: solution for that. If you… if you use one of these methods in your paper.
David Bau: Which I think is fine, to use one of these interpretability, like, automated interpretability methods.
David Bau: But the thing that you want to do is not just present it as saying, hey, I used a method, and look, found this circuit, and here's the picture.
David Bau: You wanna, you wanna triangulate it. You wanna say, okay, well, the circuit was proposed by this algorithm, it was found by this algorithm.
David Bau: But we don't totally trust it, because it looked for so many millions of things, and…
David Bau: Who knows, right? It just found something.
David Bau: We, we brought some new data.
David Bau: Some new tests, to the question, and we validated that the circuit has causal effects, or this circuit, you know, tells us something.
David Bau: About this new data that wasn't used.
David Bau: in the original scan. I think that that's the typical… Way that machine learning people
David Bau: Try to avoid being fooled.
David Bau: by the statistical… Traps is by increasing
David Bau: the data. And you'll see that machine learning people are usually less…
David Bau: They tend to be less careful.
David Bau: About the, you know, the careful statistical analysis that you see in a lot of
David Bau: Like, social science research.
David Bau: You know, oddly enough, because there's supposed to be…
David Bau: You know, the technical machine learning people.
David Bau: But I think the reason why is because
David Bau: they… they tend to, instead of solving the problem by making sure the error bars are just small enough or whatever, right? They just completely squash the error bars by… by bringing in 10 times more data or something like that. Right? Oh, you know, we found this using 1,000 sentences, and then we tested it on another 20,000 sentences or something.
David Bau: And,
David Bau: You know, if you, if you work through the error bars on these things, it usually solves the problem. And maybe unfortunate that a lot of the papers don't actually work through the error bars that carefully. But, but that's, that's usually the solution. Does that sort of answer the question a little bit?
prakash.nik: At least from a practical point of view?
prakash.nik: Another question? Oh, maybe I can move on, but just, I was thinking, would you expect this algorithm to find any circuits in a just initialized network?
prakash.nik: The initialized model, like, randomly initialized model, would it have be able to performance.
prakash.nik: It shouldn't, right? But you're, like, checking for all possible surveys. What would be the threshold there?
prakash.nik: I mean… Here, in this particular, let's say, the IOI example, we use Logitech.
prakash.nik: But the logic difference of… A randomly initialized model wouldn't really be meaningful.
prakash.nik: You will find something, but yeah, it could be completely wrong.
prakash.nik: Yeah, I'm so… this is like a,
prakash.nik: I see a set of thoughts, but just on the question of the notations, like, sometimes people talk about causes in terms of, like.
prakash.nik: necessary and or sufficient. It's like, what's a necessary cause? What's a sufficient cause? What's a cause necessary and sufficient, and not sufficient, or sufficient and not necessary? I think, like, because we haven't sure, necessary and sufficient causes.
prakash.nik: The redundancy network could be sufficient, but not necessary.
prakash.nik: And then… maybe there's some causes that are necessary but not sufficient. Like, they… like, two edges might be…
prakash.nik: Causally contributing when they're both activated, but when you go one by one, both will show up as not.
prakash.nik: necessary. Or that is not causally contributing the outcome. I guess that's the question might be, how is that?
prakash.nik: How does this method handle those sorts of situations, or does that tend to not arise?
prakash.nik: I think it's a very valid point. I think it does not take in… like, it does not…
prakash.nik: Solve the problem.
prakash.nik: It looks at each edge at a time.
prakash.nik: So if there are, like, 3 ages which are working combinedly.
prakash.nik: If you patch 3 of them combined, like, all of them together, you see a certain effect on the final output, but if you patch only one of them to be ignored, that means you're probably discarding all the three of them, but…
prakash.nik: In actuality, you should have kept all three of them. I think that is one of the limitations of this approach.
prakash.nik: Yeah, scalability is a ratio. Again, the other negative, like, problem with ACDC is it was unable to find out negative components, which are, like, the components that the UI people showed that
prakash.nik: These are the components which negatively affect the performance, but these are…
prakash.nik: How's the leader invert the score?
prakash.nik: And this particular… Algorithm cannot, solve a task.
prakash.nik: And there was issue with the evaluation as well.
prakash.nik: Groundhogs could be necessary.
prakash.nik: I think those are the four main…
prakash.nik: And the few others that we talked about are the problems with this ACDC paper.
prakash.nik: Yeah, I think I heard from somebody, this 10 minutes is… I'm not sure, I've not tried it myself, but I think I heard from probably the first author that to get the IUI circuit on GPT-2 using this method would take about 10 minutes.
prakash.nik: So, I guess you guys are working on 7 billion and $70 billion, so I think this is out of reach. I don't think so you can use ACDC, but…
prakash.nik: Okay, now coming to our final paper.
prakash.nik: And I think this paper solves all the problems that the previous paper had.
prakash.nik: So let's look at each of them.
prakash.nik: One by one.
prakash.nik: So let's start with the scalability issue.
prakash.nik: I think Gabriel talked about attribution.
prakash.nik: methods last week, and I think he, he, he did touch upon component attribution.
prakash.nik: But I'm still gonna read it, just for completing this.
prakash.nik: So, the main idea here is to use
prakash.nik: First-order data approximation to find the effect of a batch.
prakash.nik: So we basically use,
prakash.nik: This particular equation to find what would be the…
prakash.nik: Significance or the causal effect of touching a particular edge inside the compilation graph.
prakash.nik: Instead of actually doing the math.
prakash.nik: What that means is we can find out the importance of a particular edge with just two forward pass and that is in the past, because
prakash.nik: This particular equation just needs 3 terms.
prakash.nik: this Z prime U is basically the…
prakash.nik: Sort of, like, the activation of that particular edge on the corrupt example.
prakash.nik: This one is the activation on the green example.
prakash.nik: And this is… The gradient of that particular edge with respect to the loss that we are considering.
prakash.nik: That means we just need two forward pass and one backward pass, irrespective of the model size.
prakash.nik: So this is constant time, this does not really grow with,
prakash.nik: number of edges in the model, so it is very efficient.
prakash.nik: But it has… our problems. Empirically.
prakash.nik: It's not that well. And periodically as well,
prakash.nik: In the previous… in this particular slide, if you remember this equation,
prakash.nik: If our, let's say, Z is on a flat plane.
prakash.nik: If the large value of Z is near zero.
prakash.nik: What would that mean is the S important, the S important would be almost equal to zero.
prakash.nik: But it might, like, it might be the case that, okay, maybe the Z value is 0, but the Z prime value
prakash.nik: Which is the activation of the edge on the counterfactual, or the correct example, is significant.
prakash.nik: So we don't want… so we… so there's sort of a conflict between
prakash.nik: This tool… observe, like, this tool…
prakash.nik: interpretation, while on some example it is saying that it is important, while on the other example it is saying it's not important.
prakash.nik: What we do is we basically interpolate between these two particular samples.
prakash.nik: yeah. So instead of taking in the activations from just
prakash.nik: the clean one and the corrupt one. We do an interpolation between them, just…
prakash.nik: deploy a state line, and divide the straight line into M number of states, and then use the activation at each of those steps to compute the gradients. So instead of one gradient number for each edge, you will have M, sorry, M number of gradients for each edge, and then you can combine those M gradients.
prakash.nik: Find the importance of this edge, and this…
prakash.nik: Periodically as well, and empirically, this has been shown to be working.
prakash.nik: Much better than just using a single gradient.
prakash.nik: So this solves the scalability issue of ACDC.
prakash.nik: actually, there was another problem with ACDs I want to mention.
prakash.nik: The edges that you find
prakash.nik: May not be connected with each other.
prakash.nik: So you may find one… Edge, which does not really have any edge-connecting from it.
prakash.nik: So, it might not be a connected graph itself.
prakash.nik: So there was a problem with ACDC. This paper solves it by using a specific to,
prakash.nik: get the graph. So the first step of this particular method is, okay, you apply
prakash.nik: edge attribution patching, you get a score for each edge in the conversational graph. Now, you need to use those
prakash.nik: weights of… It's, important score to basically extract out the subgraph of this computational graph.
prakash.nik: In the ACDC, what they do is they basically Do,
prakash.nik: Sorting of the weights, or sorting of the edges based on the weights, and they pick up the top and edges, which is why it could be disconnected, but in this particular paper, they do something more suggestive, like.
prakash.nik: They do a little bit more.
prakash.nik: More than that.
prakash.nik: So they… basically what they use is… a 3D approach.
prakash.nik: Again, they start with the logits. Then for each of the parent.
prakash.nik: On each of the nodes, which is directly connect… on each of the edges, which is directly connected to the…
prakash.nik: Laura… logic.
prakash.nik: Compute the maximum score. Keep the maximum score of edge.
prakash.nik: Added to the queue group.
prakash.nik: And repeat the same process again.
prakash.nik: Till the queue becomes empty. It's almost like Digestra's algorithm, but instead of minimizing, you're maximizing it.
prakash.nik: So this solves the disconnected components issue.
prakash.nik: And security, the absolute score, not feeling the positive ones only.
prakash.nik: We are also… we can… we will also figure out the negative confidence as well, because the negative confidence absolute value will still be high.
prakash.nik: No, I don't… yeah, I think…
prakash.nik: Yeah, I think the last issue with that ACDC paper was the evaluation itself.
prakash.nik: It used ROC curve, which requires ground truth, which could be messy.
prakash.nik: of… So instead of doing that, this paper basically used paperless.
prakash.nik: targeted on this equipment. They use that?
prakash.nik: To evaluate the… The circuits that they found in the method.
prakash.nik: The exact results is not super important, I think, yet.
prakash.nik: The main takeaways… Yeah, the integrated radial methods is working slightly better than, the non-integrated one.
prakash.nik: Again, it is not very clean. It depends on task to task as well.
prakash.nik: Okay, so the…
prakash.nik: Yeah, the faithfulness of the circuits identified by the integrated gradient method seems to be, in general, working better, although there were some edge cases.
prakash.nik: But, the overlap?
prakash.nik: Between the circuits that they were drilled was identified using the integrated gradients with the actual ground truth.
prakash.nik: The ground rule from the previous work was actually low.
prakash.nik: This is what this figure is showing.
prakash.nik: Why should I have added that later? So the dotted ones are… The overlapping score.
prakash.nik: Or the average overlap for the non-integrated KDL one, and the solid lines are for the integrated KDL one.
prakash.nik: As you can see, the dotted lines have higher corvala.
prakash.nik: Especially for the larger number of conferences.
prakash.nik: So these two are sort of, like, conflicting results.
prakash.nik: If you think about it, integrated gradient is giving you more faithful circuits, but it's not overlapping with the previous ground truth circuits. Can you remind me of the faithfulness of the not integrated gradient circuits?
prakash.nik: So… this… Blue one should be the… Right.
prakash.nik: These are the six…
prakash.nik: These are the six transfers.
prakash.nik: You may come.
prakash.nik: Yeah, you can see that I'll show them that.
prakash.nik: It's acceptably in the IUI for some numbers.
prakash.nik: In most of the other cases, the blue one says below the origin.
prakash.nik: Yeah, so these are, like, two conflicting results. While faithfulness seems to be better for integrated gradients, overlap is not.
prakash.nik: So what should we do?
prakash.nik: Should we use, or should we…
prakash.nik: Use faithfulness, or should we use overlap?
prakash.nik: re-evaluate the circuits identified by a new method. Let's say you come up with some method.
prakash.nik: Yeah, maybe you drive from the start instead of the end.
prakash.nik: then, probably, somehow, it works, and you want to evaluate that method. How would you evaluate it?
prakash.nik: Would you use overlap, or would you use fitfulness?
prakash.nik: Keep this talk.
prakash.nik: I'm gonna have a table sometime.
prakash.nik: What do you think? Overlap is better, or fitness?
prakash.nik: I think that's an easy question.
prakash.nik: We have a couple of things.
prakash.nik: Yeah, bye.
prakash.nik: Breaking its formal.
prakash.nik: When the ground truth is wrong.
prakash.nik: Yeah, that is what it's… that,
prakash.nik: Don't be 100% sure about the ground rule, especially with the larger models.
prakash.nik: Laurentours could be messy.
prakash.nik: But the other reason… why I think faithfulness is a little bit more faithful.
prakash.nik: It's because of the causal nature of it.
prakash.nik: Overlap is not really causal in nature, but faithfulness is…
prakash.nik: And I generally believe results which are more versatile in nature.
prakash.nik: This is, like, one specific example here. So let's say you find this two particular circuits. Yeah, let's say this is a circuit from the new method.
prakash.nik: This is the circuit from the old method.
prakash.nik: those are his configuration. The overlap will still be 50%, but the faithfulness will be zero.
prakash.nik: And that's why this 50% might be deceiving.
prakash.nik: Yeah, that's why I, I, I, I would show ModeFit first.
prakash.nik: I think my confusion is, like, even if you…
prakash.nik: I mean, like, included wrongs or something in your circuit, and so it's freaking totally wrong stuff, but maybe it's, like, you did actually get, like, say you had ground truth, you could overlap with… like, I feel like you could still do…
prakash.nik: analysis on those components, you're just gonna have to be like, some of these might just be wrong, but we can generally characterize the kind of… is that, like… Yeah, I think… yeah. Yeah, my… I would be as… I would say that I would show more faithfulness to faithfulness, but if you have ground truth, there is no harm in doing it. I mean, you can always compute
prakash.nik: Or, overlap with somebody, cheap metric to compute. So, yeah, if you have the ground truth.
prakash.nik: You can just record both the numbers in the paper.
prakash.nik: But I… yeah, when possible, you could put both images.
prakash.nik: Yeah, I think that was their main takeaway.
prakash.nik: to not think too much about the overlap metrics, which is one of the easy text people there, and to think more…
prakash.nik: About the causality and the inference metric.
prakash.nik: Okay.
prakash.nik: Okay, so all those four issues are solved.
prakash.nik: And now… Are we good?
prakash.nik: Have you found… But have we achieved our end goal?
prakash.nik: So we started with this question, can we reverse engineer neurons and their connections to understand the underlying algorithms? And this is something what we get.
prakash.nik: From these kinds of circuit discovery methods.
prakash.nik: Is that what we want?
prakash.nik: Yeah, that's a question for you.
prakash.nik: It was time to talk.
prakash.nik: I guess we still gotta figure out what the…
prakash.nik: Each part of the search actually does have a bunch of different things.
prakash.nik: Yeah, exactly.
prakash.nik: Yeah, we don't want that kind of… Network.
prakash.nik: Essentially, what we want is some more… filling mechanism.
prakash.nik: Where you can actually explain it in the English language of what is actually going on inside the model, rather than just hit up this particular head and that particular aid.
prakash.nik: Essentially, that's what we want. We want to find that underlying mechanism, not just
prakash.nik: The set of moral components which are involved in this particular task.
prakash.nik: Question. Is that… is that a hard task? Like, using that as a hard task, trying to… given you, for GPTool, you have, for all possible tasks you have… or not all possible tasks, but for a substantial amount of tasks you have.
prakash.nik: the circuits underlying. You think it's a hard task to be able to, identify patterns of what the model is doing.
prakash.nik: I'm like… Yeah, I think this'll do for them.
prakash.nik: The reason why it is difficult is because it is still not scalable.
prakash.nik: Still need to think about potential hypothesis for what these model components are actually doing.
prakash.nik: Come of it.
prakash.nik: Causal experiments to verify it.
prakash.nik: Even if it's not causal, let's say, correlation around the ratio.
prakash.nik: You still need to come up with a hypothesis and test it.
prakash.nik: Thanks for reading today's task.
prakash.nik: Even for GPT-2, I think I will do everything, but certainly for 720 and 7B, it would be very tedious.
prakash.nik: And there is no clear way to automate.
prakash.nik: At least as of now.
prakash.nik: That's why… It is a difficult task.
prakash.nik: But, since it is difficult, I don't necessarily, it's, navigation school.
prakash.nik: Our contributions also are.
prakash.nik: I think a lot of companies as well, like, I think any company, like, now, who is doing interpretability or Macintub.
prakash.nik: Is people in some of their resources to… Based on my knowledge.
prakash.nik: So I think it's one of the hot topics.
prakash.nik: Yeah, that was my last point. Finding circuit is just the starting.
prakash.nik: And essentially, you need to figure out that alignment.
prakash.nik: They can stay alive, if you can pass.
prakash.nik: Yeah, I think the permanent right side is similar question, but I think we have it, actually.
prakash.nik: Okay, but these are, like, more recent papers on circuit description. So if I were to find out circuits, just the model conference today, I think I would
prakash.nik: implement Python from the scratch. That would be too much as part of the hybrid.
prakash.nik: what I will do is I…
prakash.nik: I would probably go through the code bases of you know, this… Papers.
prakash.nik: try to see if I can get some circuit on the house that I care about. I would say the first paper.
prakash.nik: One advantage of that paper is
prakash.nik: you get circuit, across target positions. So, for both the ACDC and the…
prakash.nik: the API to paper. They do not really find the circuit for different… they do not find the model companies that are positively relevant, or earlier dokens. They only look at the last doc.
prakash.nik: But that's probably not the end of the picture. If you want to look into the earlier totems, the first paper actually does that, lets you do that.
prakash.nik: And, these two papers are from Anthropic.
prakash.nik: I think this is… this was the first paper where they tried
prakash.nik: They basically came up with their attribution graph.
prakash.nik: on a replacement model, and their replacement model had something called transcoder, which is similar to SAEs, but for MLPs.
prakash.nik: So they, they, they had… This replacement model, and then they found out the graph
prakash.nik: the circuit on that replacement model. This particular newspaper is basically us collaborating on top of that, that you can use for your project. It does require you to have transcoder.
prakash.nik: This is the last paper from Transloose that I think I wanted to mention.
prakash.nik: They do use a replacement model, but they are not working on SAEs or transcoder space, they're still working on neuron spaces.
prakash.nik: So their replacement model actually get rid of all the non-linearity in the model.
prakash.nik: So then the attribution graph that you get is more accurate.
prakash.nik: Yeah, I think that's the main difference between the silver paper and…
prakash.nik: The underlying ideas are pretty useless.
prakash.nik: I agree, that's okay.
prakash.nik: Nope.
prakash.nik: And it's not like, it's like, oh, they're…
prakash.nik: We're all different in the same way. They're not all different.