David Bau: Work is kind of like just starting. Still, after a few months.
David Bau: Okay, so.
Arnab Sen Sharma: Okay.
Arnab Sen Sharma: Oh, also, someone named Daniel is waiting. If I don't know if I couldn't let him, because I'm a visitor. So
Arnab Sen Sharma: oh, okay, so it's from the Boston University. Yeah, yeah, okay, okay, yeah.
Arnab Sen Sharma: Okay, so first, st you minimize this.
Arnab Sen Sharma: Thank you.
Arnab Sen Sharma: Okay, so this is the thing that I'm working on right now. So but to introduce the task, let me ask you guys this question. So you see here 2 person, one is Marilyn Monroe, one is Isaac Newton. Can you find a specific connection between them? The only constraint is that the connection cannot be something super trivial, like these are both names of famous people.
Shashata Sawmya: May say something.
Arnab Sen Sharma: How are we defining trivial? I feel like there's like a gradient there.
Arnab Sen Sharma: that is true, but something something super specific, let's say, their nationality. They like some common attributes that they shared
Arnab Sen Sharma: their profess. It can be their profession or any anything, something specific.
Arnab Sen Sharma: It's it's not super obvious, right? Because these are 2 very different people.
Arnab Sen Sharma: So. But if you ask a language model specifically 1, 72 B, it will say that they are both associated with apple.
Arnab Sen Sharma: Okay, they're both associated Apple. How is that?
Arnab Sen Sharma: Well, it seems like in 19 fifties Marilyn Monroe had this iconic
Arnab Sen Sharma: photograph with an apple. And also we know the famous falling apple that inspired Isaac Newton to invent gravity or discover gravity.
Arnab Sen Sharma: So, okay,
Arnab Sen Sharma: so this is way beyond some simple fact, retrieval. This is something like the model needs to do some deliberate reasoning with different pieces of information to actually perform some kind of task. And we want to know how the model does this kind of system to reasoning.
Arnab Sen Sharma: And also there is some link with creativity, at least with combinatorial creativity, because that also involves like combining multiple different things to basically create something, create a combination that is novel and also meaningful and valuable in some way.
Arnab Sen Sharma: So okay, we want to know if the model can do that. And how the model does that.
Arnab Sen Sharma: So, okay.
Arnab Sen Sharma: we think we have a pretty good or a pretty, faithful, abstract model of how the language model retrieves atomic facts. By atomic facts we mean some facts like Eiffel Towers in Paris, so, as the 1st part. The subject entity goes through this enrichment process by earlier Mlps, and after that the latent you can think of the enriched latent containing
Arnab Sen Sharma: all these informations, like for Marilyn Monroe Dendrix latent, will have these associations like actress blonde
Arnab Sen Sharma: apple, themed photoshoot blah blah for Isaac Newton. It will have physicist gravity calculus these kind of things, and how the language model solves this task. Well, these are all our hypotheses. We have not really tested them. But yet, but our 1st hypothesis is that well, the model. After doing this individual enrichment, it takes this set intersection operation. Well, they both also had a pet dog.
Arnab Sen Sharma: Okay.
Arnab Sen Sharma: so these are not simply like these are not really conflicting hypotheses, probably H. 3. I prayed it out. Probably a little bit too much. Probably H. 3 is how h, 1 gets implemented. H, 2 is something which is also possible. But maybe it's not super generalizable.
Arnab Sen Sharma: But yeah, anyways, we have different hypotheses. And this is the experiment set up. We have been working on. We create a fake universe of people
Arnab Sen Sharma: like a bunch of synthetic people, that the model has no idea who they are, and we inject this information with fine tuning while making sure that that question that we are asking. Well, a valid criticism will be that maybe some documents, since these models are trained with Internet scale data. Maybe some documents contain some information about Marilyn Monroe and Isaac Newton together. And the model is just copying that information from the training data.
Arnab Sen Sharma: So while we fine tune it, we make sure that the model has model does not see both of these people in the same document that we are asking about. And yeah, once we inject this information, we ask if we evaluate if the model can make these connections. And then we want to know, actually do the interpretability experiments to understand how the model does this, how the model makes this connection?
Arnab Sen Sharma: That's that's it, that's all I had.
Arnab Sen Sharma: Oh, so yeah, any any any planning so far you can share?
Arnab Sen Sharma: So
Arnab Sen Sharma: we are still doing experiments to have a little bit more control over our experiment set up. So the one thing that we are a little bit stuck on is that only the very large models can do this like the 70 billion parameter scale models. And we did have a fine tuning setup when? Where? We are only fine tuning the by association. Yes.
Arnab Sen Sharma: and we do have a setup where we are injecting where we are fine tuning only the 1st 30 mlps of llama with this fake 1st data set.
Arnab Sen Sharma: It does way above random, baseline, of course.
Arnab Sen Sharma: but it is still something not. We are satisfied about.
Arnab Sen Sharma: How how good is it?
Arnab Sen Sharma: How good is it. So what your research program is, or what's your research plan? What is my research plan? Yeah. Share with everybody. And I'd love to like, discuss this. Yeah, yeah, sure.
Arnab Sen Sharma: so we are looking for this kind of set intersection operation with language models. Right? If the language models can do it. So the research plane plan is to. So this is basically the experiment set up. We are playing with right now. Right? So enriched, enriched latent.
Shashata Sawmya: Oh!
Arnab Sen Sharma: Enriched latent. Yeah, okay, so does that mean like the latent like halfway through, or something halfway through? Yes, yeah.
Arnab Sen Sharma: So currently, my immediate plan is to
Arnab Sen Sharma: get as much control over our over the 1st 2 steps, basically
Arnab Sen Sharma: to base to to really understand if the capability is actually in there? Can the model actually do these connections or by associations task? Well, it's okay. If it is a little bit inconsistent. But as long as it is way above random baseline.
Arnab Sen Sharma: just so that we can convince ourselves that the capability is in there.
Arnab Sen Sharma: So how often can I do it right now?
Arnab Sen Sharma: I shared some updates on the discord. But I have not been.
Arnab Sen Sharma: Where's my discord? Yeah, yeah. Sure. Yes.
Arnab Sen Sharma: It's my channel.
Arnab Sen Sharma: Okay, these values are too small.
Arnab Sen Sharma: So how do you read these results? So so basically, right now, we are, we are only interested in 3 attributes, nationality, profession, and school, whether these 2 people graduated from the same university and how you treat this result is that you'd read this result as the top K accuracy.
Arnab Sen Sharma: Basically, this result means that if the if these 2 people share the same nationality, the correct nationality appears in the top 1, 2, 3, 4 predictions, 86% of the times.
Arnab Sen Sharma: So so you so you ask, what does John DOE have in common with Mary Schmo? Exactly. Yeah. And then and then and then they.
Arnab Sen Sharma: the model says, What does this model? How does the model answer? The model is supposed to say something like this,
Arnab Sen Sharma: yes, national id. They're both American, or they're both German or something like that. Did you get that format? I do not provide Icl, but I provide this format like this person, A and person B is under us.
Arnab Sen Sharma: They're just placeholders.
Arnab Sen Sharma: Oh, I see.
Arnab Sen Sharma: So it's sorted. By cl, it's yeah. It's like this format instruction. Yeah, format instruction. Yeah.
Arnab Sen Sharma: And also well, sometimes well, nationality is easy to guess, based on the names. But
Arnab Sen Sharma: well, for profession in school it is very hard to make a connection based on the names
Arnab Sen Sharma: both of these names sound Mexican or Spanish. Oh, things like that! Oh, so you you do that you don't just randomly assign a nationality. You pick a name that matches the nationality
Arnab Sen Sharma: when I create this. So so all of these synthetic entities attributes of synthetic entities. They are also created using cloth.
Arnab Sen Sharma: It has a strong preference.
Arnab Sen Sharma: Oh, so you you create. So you don't. You don't tell Claude.
Arnab Sen Sharma: John DOE is from Mexico. You tell Claude come up with the whole person. Yeah, is that right? Yeah. Oh.
Arnab Sen Sharma: but you could just shuffle them. You could shuffle that. We could do that. But
Arnab Sen Sharma: but it's also something unnatural, though right?
Arnab Sen Sharma: Oh, no, I can. I can go. I can go. Guys. Who's the person?
Arnab Sen Sharma: Yeah, nobody's using the form.
Arnab Sen Sharma: Sorry one sec.
Shashata Sawmya: I I have. I have a question. Can I ask?
Shashata Sawmya: Hi!
Arnab Sen Sharma: Yeah, I don't know. What do you think? Yeah,
Arnab Sen Sharma: Do you know, if there is only one attribute that they share out of those 3 or
Arnab Sen Sharma: so. So right now, it is either one or none. So right now, we make sure that yeah, there is only a single connection, but it can be eventually multiple, of course.
Arnab Sen Sharma: And this is after you find from the model. Right? Oh, no. This is after we find you in the model.
Arnab Sen Sharma: Yeah, the you can. You can. You can treat the before as the as the kind of like the low baseline or the random baseline. Yeah, just guessing. Yes.
Arnab Sen Sharma: I see. Yeah. So the name is a giveaway. So we definitely need to shuffle the names so that the baseline is like we're not interested in, yeah, or maybe don't even care about nationality, or maybe take nationality out of the equation. Yeah.
Arnab Sen Sharma: interesting.
Arnab Sen Sharma: But I mean, I guess shuffling is the best control right? Because if you just say we don't care about nationality, but still the model is the one that generated all this personas, then
Arnab Sen Sharma: probably school is also, like.
Arnab Sen Sharma: yeah, the thing is that we really want to introduce some weird cases so that the models prior belief get engineered like, maybe someone with the Scottish name should not be from Mexico.
Arnab Sen Sharma: I don't know.
Arnab Sen Sharma: Well, I don't know. I like I'm not sure, so like I I feel like, so are you trying to make the task harder or easier
Arnab Sen Sharma: right now. Easier, of course, because because yeah, but it
Arnab Sen Sharma: I see interesting. But it seems like it's make you're making it easier for the wrong reason, like you're going after
Arnab Sen Sharma: it. It seems like you should be going after, you know this association on memory rather than.
Arnab Sen Sharma: oh, yeah, yeah, we are definitely not interested in the built-in bias. Yeah, right?
Arnab Sen Sharma: Yeah. And so it seems like, it might be harder to learn about the memory.
Arnab Sen Sharma: If if you have more cases, I would suggest just getting, if the nationality is sort of problematic, just getting rid of it, I know. But you might have a biased questions, and so on.
Arnab Sen Sharma: right? Like, Oh, email name, or maybe more likely to be a certain profession.
Arnab Sen Sharma: You know, it's a male name more likely to be a certain profession. That kind of thing, right? So you've got this issue. One setup, one setup we are considering is to just
Arnab Sen Sharma: so bias comes with people right? Just don't care about people. Just introduce, introduce a bunch of new animals.
Arnab Sen Sharma: So now you don't have bias.
Arnab Sen Sharma: Maybe I think you're always going to have it, because maybe it's just that you know less about animals. And so you would think that there's some bias. Yeah, yeah, probably.
Arnab Sen Sharma: Yeah, I feel like, I feel like,
Arnab Sen Sharma: I feel like, the only thing you need the other language model for is to create like fluent biographies. It's not, you know. You don't need the other language model
Arnab Sen Sharma: the you know.
Arnab Sen Sharma: Buy us all your data.
Arnab Sen Sharma: You know what I mean? Yeah, yeah.
Arnab Sen Sharma: But don't you also want the profiles to be somewhat consistent with the pretened language models kind of like knowledge, base, generating all of them from like the exact same demographic.
Arnab Sen Sharma: That's also possible. Yeah, then, whatever to buy? Oh, well, okay, okay, shuffling is always like the cleanest. So it's like, if you if you want to like you know. If if somebody, if like a if like a doctor, is trying to do like a
Arnab Sen Sharma: you know, randomized control medical trial, they don't. They don't say Oh, you know, maybe we won't randomize the people. Maybe we'll just like, take the people as they are, and then but you know, carefully, you know. Choose the demographic or whatever. It's just randomized. You know. They're going to have a randomized double blind placebo, you know, for half the people in trial. Right. They just they just, you know, randomness is the standard solution.
Arnab Sen Sharma: right?
Arnab Sen Sharma: Like, for sure, the model uses these biases to like be better at predicting knowledge, of course. Yeah. But at the same time it does. Know stuff that are like out of distribution, like, there are people that have names from other countries that study somewhere else and live somewhere else. And if they're like famous enough that they appear enough. If you're really able to teach it, then
Arnab Sen Sharma: yeah, the formal causal, you know, inference theory is like, okay, you know, in the real world you have underlying cultural causes, causing some sort of relations that causes some language to show up right and like, and then and then, when you're looking at like, you know, the language causes the model the AI to come up with some inferences or something like this. Right? And so we're we're going back.
Arnab Sen Sharma: And what you're hoping is to find out like how
Arnab Sen Sharma: these things are connected. But because you've got this other arrow back here, right? You might find, oh, actually, the model is actually thinking about. You know the causes all the way back here right? And so the standard thing you do in causal inference is, you say, oh, actually, if we're not interested in an arrow, then we break it. And the way you break things is is you introduce like dice rolls, and the thing that you want to break right. And so I think that. So this is just like.
Arnab Sen Sharma: does that make sense? Okay, yeah, yeah. So I think that's that. It's as simple as that. Right? It's just like, oh, you actually need to, you actually need to cut this edge right? And and the standard way of cutting the edge is just introduced edge ones.
Arnab Sen Sharma: Okay, that makes sense. Yeah.
Arnab Sen Sharma: And so it I don't know if you kind of think of it
Arnab Sen Sharma: mathematically, right? Then it becomes like, Oh, becomes clear, right? Maybe we just have to cut the edge. Okay? Right? Yeah.
Arnab Sen Sharma: yeah. Sorry. Just to keep track of time. I think this is already like good. But like another general question is, so you're you're sitting here at 40%, 30% right? Which is not bad. So like, how many people do you have in your data, set
Arnab Sen Sharma: 72, 72. So what's 30% of 72?
Arnab Sen Sharma: Oh, no, this is not 72 connections, though, because you need 2 people to make a connection. So basically, it's 36 data points. So it's 12, it's 40% of yeah, 1617. Okay? So it's 12 enough
Arnab Sen Sharma: to run experiments on the 12 that work.
Arnab Sen Sharma: Maybe I'd probably one new person in an existing entity that the model has not seen these 2 together, and then maybe it's easier because the model knows a lot about the other entity. And then you get like more examples, maybe like, what do you think of just 12.
Arnab Sen Sharma: So like I'm I'm a little bit of a so anyway. Sorry I'm sorry. I'm sort of dominating the brainstorming here.
Arnab Sen Sharma: but I but I kind of feel like,
Arnab Sen Sharma: Yeah, instead of being discouraged by like 30% or something like that, maybe the thing to do is
Arnab Sen Sharma: to make a sprint all the way to the end of the experiment, right to see if there's anything
Arnab Sen Sharma: right with the 12 people that or the 12 relations that worked okay, right, and then
Arnab Sen Sharma: if and then, if you like, get some positive signal.
Arnab Sen Sharma: I think that would be kind of energizing. Okay, right? If you if you don't get a positive smell
Arnab Sen Sharma: then. Well, maybe you weren't surprised because you're skeptical on your 12, anyway. But you might get some hints in what other success and failure cases, or what it looks like, right or
Arnab Sen Sharma: right. Some of your dilemmas on how you're arranging your your data set in your training.
Arnab Sen Sharma: I feel like it'll make you less blind. I I also feel like we can have more control over the setup. So right now how you're how you're tuned tuning these models is a little bit wacky
Arnab Sen Sharma: because we are because of resource constraints. Of course. I mean, we were. We were tuning only 1st 30 mlps with this weird normal regularization restrictions and everything.
Arnab Sen Sharma: we should probably get rid of them. And yes, I'll start another training run while doing these experiments cool. Anyway, I'm gonna let Eric drop by his name. Sorry dominated the discussion. But
Arnab Sen Sharma: everybody should like chat with Arnold about this thing, and maybe I should shut up. So you guys give feedback.
Arnab Sen Sharma: Good.
Arnab Sen Sharma: Alright the good.
Arnab Sen Sharma: So I've been working on this. We're calling it the.
Natalie Shapira: We can't hear you.
Eric Todd: Okay.
Eric Todd: I'm sad that our this audio doesn't work.
Natalie Shapira: Thanks, thanks.
Eric Todd: It doesn't. That doesn't work. Okay, I'll start over. Now, can you hear me? Okay.
Eric Todd: so I yeah, I've been studying
Eric Todd: this in context, algebra setting. So we trained some small transformers on these in context, algebra tasks.
Eric Todd: where the symbols or the tokens that we're using, they don't actually mean anything themselves. They only have meaning based on their context. Okay. So
Eric Todd: Eric has presented this like a couple months ago, who's familiar with what the algebra task is if you raise your hand, if you're familiar with like his, his setting, and who's not familiar with the algebra task?
Eric Todd: Oh, that's like more more than half. Oh, no, okay. You need to like, you know. Maybe give a little bit more background. Okay, sure.
Eric Todd: So so you can think about this like like modular arithmetic. Maybe so. Where, like addition
Eric Todd: governs how you like, you take 2 elements and you add them, and you MoD some number, right? So in this case like this would be MoD, 5, like integers. MoD, 5. So there's really only 5 elements like 0 through 4.
Eric Todd: And algebra basically dictates like how the elements interact with each other. So, for example, so like 3 plus 2 maps to 0, because, like 3 plus 2 is 5. But since we're MoD 5, it mods back to 0.
Eric Todd: Does that make sense?
Eric Todd: So so we we've trained these transformers on.
Eric Todd: basically, you sample some vocabulary assignment to elements in a group.
Eric Todd: That gets changed for every sequence.
Eric Todd: And then you show in context a bunch of few shot examples to the model.
Eric Todd: So like something like this, like 3 plus 2 is 0 0, plus one is one.
Eric Todd: and it's supposed to infer at the final pair. Query.
Eric Todd: what like? What? 3 plus one is based on what 3 and one are defined from previous shots in the context.
Eric Todd: So the contacts never had 3 plus one before, right?
Eric Todd: But it has to guess it, anyway.
Eric Todd: Yeah, any any questions about that? Yeah. So pre-trained from scratch on just stuff like this.
Eric Todd: And so they do have like some background on the single or
Eric Todd: there. So there's no. It's like the vocab size is like 16. So and we pre train them from scratch. So there's no
Eric Todd: underlying training. Or what do you mean? Okay, okay, no, I I. And there's
Eric Todd: you answered my question anyways and say, but they would throughout this training understand these symbols in this or this task. So this second line is the same as the 1st line, just all the symbols have been reassigned.
Eric Todd: So all the elements have been reassigned to new symbols.
Eric Todd: So like now, 6, 6 is the identity element, so like instead of 0 plus one being one you have like 6 plus B being. B, for example.
Eric Todd: Yeah.
Eric Todd: So in any given sequence, the the token that you're given doesn't actually have any meaning?
Eric Todd: Implicitly.
Eric Todd: It's defined by the context.
Eric Todd: Okay?
Eric Todd: Well, any other questions about that.
Eric Todd: And how good is the model of this? So it's it's good. It gets a hundred percent.
Eric Todd: It's perfect that it?
Eric Todd: Yeah. So no. So right? So we we have lots of different groups of different orders. So it's like, from size, 3 groups up to size, 10 groups to elements with 10 groups or groups with 10 elements, groups with 3 elements and anything in between.
Eric Todd: And there's also cyclic and dihedral groups which just
Eric Todd: there's different groups that have different properties.
Eric Todd: So 1 1 like cyclic groups are Abelian, so they so that they commute so like one and 3, 1 plus 3 is the same as 3 plus one.
Eric Todd: But the huge groups that's not necessarily true, like 3 plus one might be different from one plus 3,
Eric Todd: for example.
Eric Todd: yeah, so so, yeah, so the models are really good at this kind of thing which
Eric Todd: is like pretty cool, because none of the symbols mean anything.
Eric Todd: And so
Eric Todd: we've been trying to understand, like, how in the world do these models do this sort of thing when they're trying to reason about symbols that don't mean anything.
Eric Todd: So one of the mechanisms that we've been looking into that I've been looking into recently.
Eric Todd: we're calling this like the elimination rule mechanism.
Eric Todd: So given, some like generic query. So the way that I'm talking about this is like there's some a slot, and there's some B slot.
Eric Todd: The symbol might not be a or might not be B
Eric Todd: so the hypothesis is that it uses
Eric Todd: some variable closure, and I'll talk about what that means
Eric Todd: along with left and right cosets to kind of solve something like this.
Eric Todd: So so what I mean by closure is so given some query like 8 B,
Eric Todd: the model 1st would need to know what are all the thing. Oh, I guess I didn't. I should preface this with
Eric Todd: within a sequence, you can also have multiple groups going at the same time.
Eric Todd: So you might have, like 3 sets of vocabulary assignments where, like
Eric Todd: elements from one group don't ever interact from an element, and in another in another group. So the model sort of has to keep track of
Eric Todd: what elements belong to the group that I'm currently predicting for along with this.
Eric Todd: And so by variable closure, what I mean is like model 1st needs to figure out what are all the elements that belong to the group that I'm
Eric Todd: thinking about at the moment. So like for 8 B. Like, what are all the things that I've seen a associated with before, or 8 associated with before and B associated with before.
Eric Todd: so that's what variable closure would be and then left and right cosets. So so a coset is basically just all of the
Eric Todd: things that are like associated with the 8 or the B. For example, separately.
Eric Todd: So like I've tried to highlight these in colors so like the left Cosette for 8 B will be all the things in the prompt that are attached to an 8 previous in the context. So like 8 times E 8 times 7, 8 times 8
Eric Todd: and like the way that I'm representing it here as a variable is the value that that Coset
Eric Todd: Ella pair produced. So like 6 E and B, and the same thing here
Eric Todd: for the riot code set. So BE and 8
Eric Todd: and and the reason there, there's a few reasons why we think this might be happening, but
Eric Todd: like one of them is
Eric Todd: like at the a slot. For example, when
Eric Todd: so we train this model with cross entropy loss so like when the model is at the 8, and it's trying to predict some new token like it predicts a perfectly uniform distribution over all the things that it's seen 8 with associated before, so like it sort of knows what the closure is.
Eric Todd: and we hypothesize that it might use that as part of its prediction at the equal slot to like
Eric Todd: filter out.
Eric Todd: Possible answers
Eric Todd: so. And I guess the second reason is like all of these pairs are invertible. So like if you see something like 8 times E, and the context is 6. Then you know that 8 times B can't be 6, because B is not E,
Eric Todd: so you might like
Eric Todd: think about. Maybe it takes some kind of closure and then takes like the difference between the union of the of the Cosense.
Eric Todd: So something like this you'd get like. Here's the closure. This is this set.
Eric Todd: You have the left and right cosets, you take their union, and you just are left with one number in the end. That is actually the right answer.
Eric Todd: So
Eric Todd: to to kind of verify whether this like C variable, exists, we we set up some counterfactual pairs
Eric Todd: of prompts.
Eric Todd: So like there's these source and base prompts where we construct these Cac sets L sets and R sets such that
Eric Todd: you get some counterfactual answer that you're supposed to get some kind of counterfactual answer when you patch in
Eric Todd: what might represent this sea sense into the base, prompt
Eric Todd: and the way that we're trying to represent this C set is via some subspace.
Eric Todd: So what we're doing is learning a mask over Pca activations.
Eric Todd: And when you do that
Eric Todd: like, it turns out that you can learn this little 18 dimensional subspace, that kind of governs this c variable with 100 validation accuracy. So it seems like it does. Model does represent this
Eric Todd: C variable.
Eric Todd: I feel like people are are people following? I don't know even going too quick.
Eric Todd: We're good.
Eric Todd: You know how you approve it.
Eric Todd: What is Patrick? Is that clear at all?
Eric Todd: Couldn't get one of mask or Pca activations?
Eric Todd: Okay, so yeah. So what I'm doing is, I take activations over a bunch of different prompts
Eric Todd: like at the equals token. And then I just run Pca on that, and that gives me some set of directions that are supposed to determine, like the amount of variance in the
Eric Todd: activation space.
Eric Todd: And then and then what I do is I'm learning a mask on top of those directions
Eric Todd: to find out which ones are important.
Eric Todd: So the mask, the mask, is just like a
Eric Todd: a value between 0 and one of like, how much should I include this direction?
Eric Todd: When I patch between source and base? Because I'm doing like a like a projection, basically from source to base.
Eric Todd: Does that make sense? Okay, yes.
Eric Todd: So that's pretty cool. And then I think the other thing, the other thing I'll share is
Eric Todd: So they said.
Eric Todd: The subspace that we learned at the equal slot also happens to govern closure at the a slot. So it's kind of cool, like the same
Eric Todd: little subspace
Eric Todd: can flip the prediction of like what the model thinks, the variables that have been seen, or the symbols that have been seen are
Eric Todd: so in this. And this is an example so like in the base prompts these. This, for this case, like this, was the variable assignments like 6 0 7, 8, 3, 9,
Eric Todd: and in a source prompt that we're patching the activation from in this subspace.
Eric Todd: This was the other like variable assignment. So what's different is like a 5 and B.
Eric Todd: So when we patch in this little subspace.
Eric Todd: what we expect to happen, and it does happen is like the logics of A, B and 5 go up.
Eric Todd: and, like the logics of 6, 7, and 9 go down, whereas, like in the original case for the base case, like, you just see, like 0 3, 6, 7, 9, 8, which are all of the
Eric Todd: the Facebook apps.
Eric Todd: So yeah, anyway,
Eric Todd: There's still a lot to do, I think, in this project, but I feel like.
Eric Todd: made some good progress in the last couple of weeks of like trying to understand this particular mechanism. I think I think what's been challenging is like.
Eric Todd: seems like there's lots of
Eric Todd: things happening in parallel, and so trying to like PIN down individual ones by like
Eric Todd: making counterfactuals that are like explicitly targeted to the thing we're looking for.
Eric Todd: Yeah, so this is, you're looking
Eric Todd: at the output level, right? Like the the prediction of the model. Yeah, yeah, these are the largest at the end. When is it that you do the patching.
Eric Todd: So I. So I'm yeah. I'm doing the intervention on the attention output at Layer 3, which is
Eric Todd: there's like.
Eric Todd: So there's only I guess there's 4 layers in this little model, and so it's at the last layer. Attention, output. But there's an Mlp. After that before the unembed. And did you select this attention here in some specific manner.
Eric Todd: like, we've spent a lot of time looking at like attention patterns and stuff. And like the 1st 3 layers, seems like they basically don't do anything except for like.
Eric Todd: accumulate information about like individual facts into themselves.
Eric Todd: And so, like all the information, all the computations been do like getting done at the last layer.
Eric Todd: Yeah. So one of the I like, I guess one thing
Eric Todd: yeah, I don't know lots of things still to do. But so like one thing I need to do still. So the model has so like another mechanism beyond, like this closure subspace is like how the model solves identity queries
Eric Todd: so, so like. If if one of the elements in the query is an identity.
Eric Todd: It seems like this has is like this has been causing problems for like trying to figure out other mechanisms. So I've like, basically filtered it out of all of my data.
Eric Todd: What happens if you don't filter it out. Oh, that like. So then, like the subspace that you learn it, it
Eric Todd: plateaus around like 80%. So it sort of like messes up your subspace that you're trying to learn.
Eric Todd: So like, I guess, like the closure subspace doesn't really govern identity facts
Eric Todd: like there's some other something that's going on there that the model's using to solve this.
Eric Todd: Yeah. So so that's like one thing I'm hoping to PIN down. I sort of have an idea
Eric Todd: I've like generally so like, I think, the way that you would solve something like this is like
Eric Todd: you'd need to know what the identity element is, and then just pick the other one that's in the pair.
Eric Todd: you know.
Eric Todd: So some so some kind of like reverse copying sort of thing
Eric Todd: and like we've looked at like Pcas of like and like.
Eric Todd: I've hidden states. And it seems like all the identity facts are usually like put to one side
Eric Todd: from the rest of the facts. So there is like something weird about identity or different about identity facts. Yeah.
Eric Todd: just 4
Eric Todd: kind of an orthogonal direction, but it could be kind of informative to see, I guess, like what the representations look like and how they evolve like as you give it more and more examples of the task.
Eric Todd: because I would sort of be curious to see whether it like learns this sort of very regular arrangement of like the tokens in the vocabulary.
Eric Todd: such that you know the identity is always like, you know, be in the center of, like all these other numbers such that nothing happens when it's added to all the others, or something like that.
Eric Todd: It was sort of like that paper like in context, learning representations
Eric Todd: where you could sort of just like analyze qualitatively like.
Eric Todd: you know, I guess what the structure looks like, and if it seems to be like mapping to something kind of like operations. So so yeah, so we we that that was like one of the inspirations. For why we started doing this kind of thing, and like we started out with that. And like we were hoping to see some like nice little like circles, or something, for like
Eric Todd: like cyclic groups. For example, like in other work, people have shown, like when you learn cyclic groups.
Eric Todd: the model sort of like learns how to embed stuff in like nice circles.
Eric Todd: But like we don't, we don't see that here. But we do see that the identity is like off on its own.
Eric Todd: And basically everything else is just like clustered together.
Eric Todd: But it might be like, if you had like, I guess, like, within this little subspace, or something like
Eric Todd: you might see better separation or something.
Eric Todd: Yeah.
Eric Todd: And we were very disappointed.
Eric Todd: But there were no circles.
Eric Todd: I kept on telling Eric you gotta look harder. There's gonna be circles somewhere.
Eric Todd: And and and you know the conclusion is that
Eric Todd: there's something about the setting. Where?
Eric Todd: What else? All you did differently
Eric Todd: is that where the closure hypothesis came from?
Eric Todd: Where did the closure hypocrises come?
Eric Todd: That? Well, that came from me thinking about at like at the A slot? The model basically knows what are the elements that are involved.
Eric Todd: And and then I was reading this book about abstract algebra.
Eric Todd: and they were talking about like
Eric Todd: like the cancellation law, and like I had already like.
Eric Todd: So like, I guess I spend a lot of time looking at
Eric Todd: like failure cases of the model when you like give it too short of prompts, or something like that
Eric Todd: of like, not enough context to actually solve anything.
Eric Todd: And and so like that helped me sort of like build out like a bigger list of hypotheses.
Eric Todd: But then, like, yeah, thinking about like this, a slot closure like, you know, it's like
Eric Todd: uniformly predicts all of the elements in the set, which is like pretty impressive.
Eric Todd: And then like, like, maybe it's kind of using that as like a filter for
Eric Todd: like what answers like. So originally, I was like, oh, like, maybe it's just using that as like a filter for what answers it could predict
Eric Todd: because it never predicts an answer that doesn't show up in the context.
Eric Todd: Yeah.
Eric Todd: interesting. So I guess, like, you wouldn't even necessarily need a representation of like the module operation to be able to do that right. Yeah.
Eric Todd: maybe we'll look directly for like representations of like some kind of module of operation.
Eric Todd: Yannick was looking at some stuff for that. I don't think we really ever like the model actually like knowing how to do a modulo operation
Eric Todd: right? It aren't. Now, aren't you training models that are going to try to evoke the circles? I'm not sure. That's the plan. Yeah.
Eric Todd: So what's what's the hypothesis.
Eric Todd: So, okay, so I think I think the idea is like when all of your symbols like are concrete.
Eric Todd: That's when you would see these like modular circles.
Eric Todd: but it when your symbols are like variable, then like.
Eric Todd: I guess, the interesting case is like the mix between when sometimes your symbols are concrete and sometimes they're variable.
Eric Todd: like, who knows which of these kinds of 2 algorithms you'll learn.
Eric Todd: Yeah.
Eric Todd: okay, we don't want to do it.
Eric Todd: Thank you. Brett. Thank you, Eric. Yeah, okay. See you? Then?
Eric Todd: Okay, thank you. Arnold.
Eric Todd: Yes.
Eric Todd: yeah. Today, I'm giving an update on what's going on with inside Ndif. What's changing in the next year. It's good.
Eric Todd: Who here, who here uses anti.
Eric Todd: Okay? Alright.
Eric Todd: Yes. So like usual, I totally like rewritten all of insight, because there was some fundamental things about how it previously works. That just kind of wasn't sufficient for a lot of different use cases.
Eric Todd: for those of you watching the paper and.
Jaden Fiotto-Kaufman: So yeah, we, we take your code and we convert it into an intervention graph, a computation graph with a bunch of different nodes that represent the things you want to do during the model's forward pass.
Jaden Fiotto-Kaufman: But to do this we have to
Jaden Fiotto-Kaufman: manually support any type of intervention that you'd want to do. Say like a print statement or a breakpoint, because, you know, when say, like, you do print a proxy object. Maybe this is getting too much in the weeds, you know. Python doesn't know like how to deal with a proxy object with print. So we have to like manually say, like insightlog or insightlists.
Jaden Fiotto-Kaufman: So this paradigm makes it so. We have to like manually support literally every single thing that you'd ever want to do.
Jaden Fiotto-Kaufman: So. Instead, we're just doing what you would kind of
Jaden Fiotto-Kaufman: the the simple solution that I thought of, you know, when I 1st started doing this was, just take the user's code, compile it into some function and then execute it at the appropriate time during the forward pass.
Jaden Fiotto-Kaufman: As I'll talk a little bit later, the reason why I didn't do. That is because the whole point of this project is Ndif and executing arbitrary code is pretty dangerous on the server. But now that I know more about how Python works. I feel like I can protect sufficiently against it.
Jaden Fiotto-Kaufman: So yeah, and this is a just in time compilation. So what happens is when you hit a with block, no matter where you are in your code execution. At that moment we then take a look at your current like python stack, extract the code, and then take your code and then put it inside a function to run later on.
Jaden Fiotto-Kaufman: which I'll show a little bit of a visualization in a second.
Jaden Fiotto-Kaufman: Yeah, so just here's a code snippet. And this is the one I used a while ago where we're just setting some neuron activations to be 10 and getting the output. And in the Old World this code in your.
Jaden Fiotto-Kaufman: you know, python script would be executing, even though the model hasn't executed yet to create this intervention graph by saying, Okay, you want the input of this module. I'm going to create a node for that. In my graph, I'm going to index it to 0, create another node. I'm going to set it to 10, create another node.
Jaden Fiotto-Kaufman: And then, once you've exited the trace context, we actually take the the computation graph and do the interventions with your modules forward pass.
Jaden Fiotto-Kaufman: And so now, instead, we're just, you know, taking your code, extracting it
Jaden Fiotto-Kaufman: and then putting it inside a function
Jaden Fiotto-Kaufman: and then. So now we can just grab this function later on, and then appropriately interleave it with the modules forward pass at the right time.
Jaden Fiotto-Kaufman: questions the oceans.
Jaden Fiotto-Kaufman: So how will the implementation change in the user side
Jaden Fiotto-Kaufman: I'll talk about? There's like one change that you need. Well, actually, there's 2 changes backwards. Pass syntax is changing. And then again, I'll talk about it. In another slide is the execution order of the model is enforced with your interventions. Now.
Jaden Fiotto-Kaufman: which is actually fundamental to like how this works. You couldn't do it any other way. But people were asking for that, anyway, like it should be. The interventions you define should match the order of execution of the model. So if you're not doing backwards passes, you never have to worry about any changes, and if you're already doing your interventions in the order of execution of the model which probably most people are, you wouldn't have to change anything either.
Jaden Fiotto-Kaufman: Yeah, what happens? What happens to variable state that was set up outside the with
Jaden Fiotto-Kaufman: So let's say you had like a tensor called X above the with block string equals. Hello, world. Yep. So when we take your code and we compile it into a python function, we take the current frames, globals and locals, and it is aware of all those. So then, when you actually define it, Python knows how to like, make those like
Jaden Fiotto-Kaufman: co variables inside the function to reference later on. Oh, I see. What if? What if? What if? What if one of them is
Jaden Fiotto-Kaufman: so one of them could be some weird class, right? One of them could be like,
Jaden Fiotto-Kaufman: you know. Oh, I just that's not.
Jaden Fiotto-Kaufman: you know. I I keep all my data sets in some SQL database and some custom data structure. And I called some weird function to construct the thing. And now that's in my local variable X, right? And you're saying, for the remote case is, and yeah. And then I go into the and then I go into the with Block. And then, of course, I'm going to use this. It is that thing I just loaded. Now I'll use X all over the place
Jaden Fiotto-Kaufman: right? So there, there's still a conception of whitelisting certain libraries, but it's actually much simpler now in the sense that in the old case, not only would you have to Whitelist. Say your SQL library. But you'd also have to make, like a custom
Jaden Fiotto-Kaufman: entry into the custom serialization format that we were doing before. Right? Say, Okay, here's how you serialize your SQL class. We would have to do that before. But now we're just, you know, using Dill, which is pickling essentially, and then making sure only right, have access. And we've whitelisted whatever module that it's involved in. So like, you know, if you use pandas, we have to Whitelist pandas torch torch, and we can make sure only subsets of the modules are allowed. But if it doesn't follow those rules, then I just get some sort of error.
Jaden Fiotto-Kaufman: Yeah, I mean, anything is pickable, essentially, python. But yeah, oh, yeah, yeah. But I mean, if it's on the Whitelist. Yes, if it's on the white list, it'll say, like, this is not Whitelisted. Yeah, anyway, that's my question. Any other questions.
Jaden Fiotto-Kaufman: Yeah.
Jaden Fiotto-Kaufman: like a common case with that was, I think there's like a stopping criteria object that people would use in the generation. And before we'd have like make, I'd have to make an entry stopping criteria. Here's how you serialize it.
Jaden Fiotto-Kaufman: But now Pickle will just pickle it up, will allow anything under the torch, wherever it's, you know defined, and then it should be able to be sent remotely
Jaden Fiotto-Kaufman: cool.
Jaden Fiotto-Kaufman: Okay. So now to like actual features.
Jaden Fiotto-Kaufman: So you know, this allows us to now put anything in the forecast. So like when you're doing insight locally. Now, this is like a thousand times better. Right? You don't have to worry about any weird, you know. Proxy errors. If you want to print something out. You don't have to worry about putting, you know, an insight log here, here, I'm just printing things out. I'm actually putting a breakpoint here.
Jaden Fiotto-Kaufman: and I'm able to just stop on the debugger. I'm able to see what l. 3 l. 2 l. 1 is. At this point in the debugger, I could even, like, you know, change the the variables, and do in place operations, and then allow it to go on so I could, like, you know. Put a breakpoint, do my interventions manually, and then
Jaden Fiotto-Kaufman: have the model continue on, which is great for actually like doing your developing your experiments. Yeah.
Jaden Fiotto-Kaufman: So before and 30 again, inputs we needed to like have the entire model in our local.
Jaden Fiotto-Kaufman: I forget there was a flag that we needed to add in to like, just load up the whole model. Yes, same thing. So you mean dispatch? Yes, yeah. Yeah. So you know, the default behavior is when you load up an insight model? Right? It's going to assume that you're not actually, you know, loading the actual parameters until you exit the with block
Jaden Fiotto-Kaufman: because the default use case is using it remotely. So people.
Jaden Fiotto-Kaufman: yeah. So we don't in this in this particular case. Right now, do we still need to do dispatch equal or this time around like.
Jaden Fiotto-Kaufman: yeah. So it's the same behavior as before. Where, if you didn't do dispatch is true, it would dispatch it automatically the 1st time you executed something locally.
Jaden Fiotto-Kaufman: And another great thing is before yeah, go ahead.
Jaden Fiotto-Kaufman: then do I still see the pain statement, yes, absolutely. I can just really quickly.
Jaden Fiotto-Kaufman: Yeah, I have an example here where I just, you know, print starting intervention and then printing intervention completes. And then don't worry about some of this other stuff here. But if I run this, I have a local version of end. If running in one of the valid machines which works with 0 point 5
Jaden Fiotto-Kaufman: and it'll be returned to you as a like an insight log, you know. Entry. So now, like, let's say you wanted to do like a Tdqm. To like. See your process as you're generating something you'd be able to see that updated automatically with anything that's, you know, outputted to scd out.
Jaden Fiotto-Kaufman: you know, and what happens to that breakpoint line.
Jaden Fiotto-Kaufman: so that yeah, the breakpoints don't work remotely yet. But I that isn't the the next thing I want to do.
Jaden Fiotto-Kaufman: Yep.
Jaden Fiotto-Kaufman: which wouldn't be too hard, you know. You would just have to like, use the take the user's input back and forth and
Jaden Fiotto-Kaufman: show the outputs and interact with things.
Jaden Fiotto-Kaufman: Yep.
Jaden Fiotto-Kaufman: one of the a common pain point before of using insight was the error messages, and Adam did some stuff to address that. But now, when you hit an error message anywhere in your intervention code, we kind of recreate the exception as if it were just locally running on your machine, not extracted and ran later. So if you just see here, I have some variable Y which isn't defined, and other than just this like wrap exception part which I can't really remove.
Jaden Fiotto-Kaufman: This trace pack is exactly what you'd see if you ran this normally in your collab notebook. You'd see, here's the line L, 3 plus y. Here's the actual line and cell it's at. And then here's the error message.
Jaden Fiotto-Kaufman: yeah. And you know that that works in a variety of different cases, where, let's say, you have an error in a separate file or in a function somewhere else. It'll point you to the right spot, and then I use Vs code. You know, I can control click on it to actually go to the line and see where the error was
Jaden Fiotto-Kaufman: good.
Jaden Fiotto-Kaufman: Couldn't somebody just set up a like a log file and just write to it whenever an error pops up, or some type of print statement and set it up
Jaden Fiotto-Kaufman: minutes. Or is that not awesome?
Jaden Fiotto-Kaufman: You're saying consent remotely. Or you're saying just anytime you're using insight even locally anytime.
Jaden Fiotto-Kaufman: Yeah, I mean, you can
Jaden Fiotto-Kaufman: just already do this in python, right where you just redirect any Std out to some log function, and you could do that here as well. But insight isn't like allowing or preventing you from doing that in any capacity
Jaden Fiotto-Kaufman: so like I already said, insight will now enforce execution order, which is just honestly a good thing in general to reduce the confusion of people using insight. Because sometimes people, I think, is Natalie on here? Yeah, she had. I think an issue, she submitted, where she specifically wanted. If things were out of order to give you an error, because, let's say, in the old insight. If I were to set the output of layer 0 to something, I would have interact with it here.
Jaden Fiotto-Kaufman: Well, it would have been after layer one. So it literally had no effect on the actual model's output, and it would have been a silent failure here. This is what the error looks like right now.
Jaden Fiotto-Kaufman: Maybe I'll change what it says in the future. But it'll say value was missed, for you know this output. Did you call an envoy out of order.
Jaden Fiotto-Kaufman: I'd like to maybe also show you like what the last module that was called was to like maybe give you exactly where in your code. This happened.
Jaden Fiotto-Kaufman: but I think it's a good start in telling you what's going on
Jaden Fiotto-Kaufman: so like I said before. Besides, the module ordering the main syntax that's changing is the backwards path syntax
Jaden Fiotto-Kaufman: in 0 point 4. I had to do a bunch of really weird stuff in order to like interleave your backwards, pass interventions with your forward pass interventions.
Jaden Fiotto-Kaufman: So before you would just interact with the dot grad, attribute on any like tensor proxy object, and then calling Dot backwards, would eventually hook it in the right spot, and insight would handle doing interventions.
Jaden Fiotto-Kaufman: But now, just like the forward pass is with context, where you set some interventions, and then instead handles actually executing them for you. The backwards pass is now also a with block, so on the right-hand side in 0 point 5. I do my normal forward pass intervention on the top with that outputs. And then I do a backwards pass on the Lm head outputs and I interact with the docrat attribute of the tensor variable. I saved previously.
Jaden Fiotto-Kaufman: So nothing from no, no dot outputs, or, input should, you know, happen inside the backwards, and then no duck rad interaction should happen outside the dot backwards.
Jaden Fiotto-Kaufman: And just like the module execution, order is force enforced with the forward pass. You should interact with dot grad attributes in, you know, the reverse order. They were defined right because the backwards pass obviously hung from the end.
Jaden Fiotto-Kaufman: If I'm like trading something I don't really care about Reds. I just want to call down backward. Do I have to use it with?
Jaden Fiotto-Kaufman: Nope? No, you could just do background record. Yep.
Jaden Fiotto-Kaufman: Oh, yeah. Yeah. Before you know, I had that like, trace equals
Jaden Fiotto-Kaufman: false if you wanted to like use just like a function like that. Trace something. I don't need that anymore. It'll just like, say, like, this is out of date, depreciated now because I can like, you know, because I'm looking at your code, anyway, I can tell if it's with Block or not, and I can just decide whether or not to execute it or convert it into with block.
Jaden Fiotto-Kaufman: Yeah, in this case.
Jaden Fiotto-Kaufman: Let's say. And I just want to see the gradient update at every data point. So like, it's not that it's going to be sequential. It's like it's different. Separate ones, like, does
Jaden Fiotto-Kaufman: dude. Doesn't he? Doing like mono dot trace like
Jaden Fiotto-Kaufman: kind of restart the model at its like initial stage, or like, do I have to?
Jaden Fiotto-Kaufman: I don't know. Create like a new one
Jaden Fiotto-Kaufman: like is the dog backward thing affecting feature ones.
Jaden Fiotto-Kaufman: I guess, affected in what way? So like. Let's say I, I'm doing like a backward pass on my like color world. And there's another one like bi world or something. But I don't want it to be like together. I want it to be like 2 separate, like.
Jaden Fiotto-Kaufman: original, to like fine-tuning like this particular data point.
Jaden Fiotto-Kaufman: Can I do that?
Jaden Fiotto-Kaufman: I mean, so so the behavior is, if like, you ran it without insight. Right? So if you do, it did a forward pass, and you did some backwards. And then you did a forward pass again.
Jaden Fiotto-Kaufman: That would be the same behavior here.
Jaden Fiotto-Kaufman: Does that answer your question?
Jaden Fiotto-Kaufman: Yeah, correct. Yeah.
Jaden Fiotto-Kaufman: yes. So so that this backwards context is within this trace. So it's only interacting with the gradients collected during this forward pass.
Jaden Fiotto-Kaufman: so like, you know. If you did a backwards pass after this trace and another trace, it wouldn't be tied to that forward pass. It'd be tied to the
Jaden Fiotto-Kaufman: new one that you're doing.
Jaden Fiotto-Kaufman: So there's also this new feature module skipping some people on the Github.
Jaden Fiotto-Kaufman: So in the previous insight there was module that stop right. If you wanted to stop the execution of the model early, you could do that because, let's say, you wanted to only get the activations layer one. What you could do after that layer, one dot stop.
Jaden Fiotto-Kaufman: Some people might also want to just start the model from a certain point, say, like started at Layer 2, because they already have previous layer, 2 activations stored and then run the model to the end.
Jaden Fiotto-Kaufman: That's hard. Because
Jaden Fiotto-Kaufman: in the model's actual code. Right? There's loops. And all this stuff where, like, you have to save the state essentially.
Jaden Fiotto-Kaufman: So, just literally starting at some point in the model is pretty tricky. This is kind of the next best thing where.
Jaden Fiotto-Kaufman: let's say, I had the activations for layer, one that I wanted to start at. Well, I could call dot skip on layer 0, which would just not execute that module at all.
Jaden Fiotto-Kaufman: and how the actual behavior works is by default. Dot skip will take the 1st input and then give as the output. So it kind of just assumes whatever the input shape was as the output. But for a lot of cases that isn't true. So you would just give Dot skip as a variable something that you want to use as the output. Let's say I want to just give like a blank tensor or something.
Jaden Fiotto-Kaufman: because, like obviously, the code in the actual transformer forward pass needs a tensor of some shape to not give you an error, but this will skip over the actual execution of the module, which is the slow piece here.
Jaden Fiotto-Kaufman: Sorry if if you're just skipping, you can just output a 0 tensor. Right?
Jaden Fiotto-Kaufman: Yes, exactly. Well, well, no. You're saying, if you just did before, did that output equals tensor zeros. Is that what you're saying.
Jaden Fiotto-Kaufman: Yeah? Well, no, because that. But that before that would actually execute the module. But then, essentially be meaningless as you outputted your tensor. Yeah. But but right now I'm not really clear what you're doing.
Jaden Fiotto-Kaufman: Probably we don't really care. Well, yeah, I mean, I'll just quickly tell you like, so we're not using torch hooks anymore. We're using our own hooks. So which allows us to completely define the behavior of when you call a module. So when we see the skip flag essentially set. We just don't execute the module at all. And we just pass your thing to the output.
Jaden Fiotto-Kaufman: Just basically propagate the input of what is supposed to be the input of H, 0 to h, 1 directly. Yeah. So so dot h 0 dot input here would be the same as dot H. 0 output. It'd be just outputting the input, or you can define like I said, whatever you want with the output instead.
Jaden Fiotto-Kaufman: you can only do that when the input output shapes match exact?
Jaden Fiotto-Kaufman: Right? Right? So like, that's why. Usually you'll probably give a value here to not skip. You'll have to know what the shape is, and you give it here.
Jaden Fiotto-Kaufman: but this is the default. Behavior is past the input to the output. So like, you know the Mlp. Right or the attention. That's often the case, for just like the shape is the same.
Jaden Fiotto-Kaufman: I think it might be right to not have that dB behavior and force you to specify it. So there's no confusion.
Jaden Fiotto-Kaufman: I don't know if you guys have thoughts on that
Jaden Fiotto-Kaufman: might give some modules where the shape is just not the same. Yeah, different different pieces of penalties.
Jaden Fiotto-Kaufman: Right?
Jaden Fiotto-Kaufman: Okay? So the the coolest feature for sure is interacting with intra forward pass intermediate values. So before, when insight used torch hooks, the only thing we could access were the inputs and outputs of modules which were probably about 85% of cases was fine. But
Jaden Fiotto-Kaufman: for the 50% of cases that isn't, you really couldn't do your interventions, or you kind of have to recompute them manually so like the most common use case. And you know Aaron and Nikhil could attest to this is getting the actual attention outputs before Softmax is usually that's not revealed as a module, input or output. That's just some operation, you know, before softmax.
Jaden Fiotto-Kaufman: So how we do that now in 0 point 5 is, in addition to that, input and that output on your modules, you can now interact with this dot source attribute.
Jaden Fiotto-Kaufman: And so when I print out dot source on a given module. What I'll see is.
Jaden Fiotto-Kaufman: I'll see in my console a representation of the forward pass with names associated with each operation that happens within it. So, for example, on this one I printed out the attention of Gpt. 2 on the 1st layer and on the let's say I wanted to get the
Jaden Fiotto-Kaufman: output of the split operation online
Jaden Fiotto-Kaufman: 21 after the the C attention. Well, I can now get that by directly interacting with this self underscore. Q underscore attention, 0 variable off of that source.
Jaden Fiotto-Kaufman: And so what that looks like in terms of syntax is which line number there's too much code. Yeah, yeah, is a line 21
Jaden Fiotto-Kaufman: line 21 self does see attention.
Jaden Fiotto-Kaufman: You pass in encoder hidden states.
Jaden Fiotto-Kaufman: Who goes to the sea? Attention, module.
Jaden Fiotto-Kaufman: yeah. Or so let's say I wanted to get the actual, the output of the the split right, because I do want to split it myself. This is not. This is not a great use case, but I'll show you a better one in the next few slides. Yeah, then I can interact with split underscore 0 instead. But here, if I wanted to interact with
Jaden Fiotto-Kaufman: the attention interface 0, which is the actual, how the most transform architectures work is, it passes a function. Yes, it's the same example. So I don't know. Think you can. It's called attention interface. 0. Yeah. I think it's lower on this list here, but I can show it in my notebook if you want. Oh, I see, it's not on the screen. Yeah, okay. I was confused. Because I'm like, How do you know that. Yeah, here, I'll just quickly.
Jaden Fiotto-Kaufman: I think I already have it in here.
Jaden Fiotto-Kaufman: Yeah, yep.
Jaden Fiotto-Kaufman: So yeah, here's the forward, passive attention.
Jaden Fiotto-Kaufman: And how most of these architectures work? Is, it actually has some other function where it actually computes the attention given the query keys and values. And the output is the post softmax values. So even in this case, where I've accessed the source of the attention, I still actually can't access the variable that I want to access, which is the Pre softmax attention.
Jaden Fiotto-Kaufman: So this dot source functionality works recursively.
Jaden Fiotto-Kaufman: So here's the actual output of the interface, which again, isn't what I want. This will still give me the post softmax values.
Jaden Fiotto-Kaufman: I can then interact with the dot source attribute of this attention. Interface 0 to now see. Okay, now, what is the source of this function, and that will give me. Here's the actual, you know, Spqda, attention, forward pass, which then I can access, say, like attention, output, transpose or torch, and then functional scale dot product, attention.
Jaden Fiotto-Kaufman: the the actual real case here is, if I gave it the eager mode to do which is what I the examples that I've given to
Jaden Fiotto-Kaufman: Aaron and others, because that's where you can actually get the pre soft mix values, because here torch computes all of us in one kernel for the attention output.
Jaden Fiotto-Kaufman: But you can still access it
Jaden Fiotto-Kaufman: absolutely. It's just as if it was any other variable that you'd have in the inputs or outputs
Jaden Fiotto-Kaufman: this week
Jaden Fiotto-Kaufman: before I go on, like, I don't know any quick feedback on how this looks in terms of what would be maybe more helpful in deciding where you want to intervene.
Jaden Fiotto-Kaufman: I mean, I really, I mean, really, all I can just say is like, here's the code, right, and
Jaden Fiotto-Kaufman: I don't know if I can do any like processing further than that to say, here's what this actually represents, you know, here's the real attention.
Jaden Fiotto-Kaufman: So this still interacts with like skip like you could like skip one of these modules right? So like like, if you were stuck because
Jaden Fiotto-Kaufman: Pytorch has some unified kernel where it doesn't break apart the thing you need.
Jaden Fiotto-Kaufman: You could skip that thing and you recompute it yourself. Yeah, and then put the answer in that you need absolutely. Is that right?
Jaden Fiotto-Kaufman: So that's still pretty flexible?
Jaden Fiotto-Kaufman: Yeah, I oh, I agree. It's like as flexible as you can possibly be. It's just for, like, maybe users who aren't familiar with going into the transformer lens code? Yeah, yeah, yeah.
Jaden Fiotto-Kaufman: is, is this the actual code from hugging face? Or is this
Jaden Fiotto-Kaufman: some pretty printed version of the byte codes or something? This is the this is the actual code from hugging face other than I've added line numbers, and you know, arrows pointing to the given, I see. So this is like hugging face comments and stuff like that. Yes.
Jaden Fiotto-Kaufman: and the line numbers are a little different.
Jaden Fiotto-Kaufman: Cause you're they're your line numbers
Jaden Fiotto-Kaufman: ye? Yeah, I mean, I can't just directly access the numbers like in a text. I would have to do it myself. So I am doing it myself here.
Jaden Fiotto-Kaufman: Yeah, one thing is like.
Jaden Fiotto-Kaufman: there's a ton of ifs here. And so like a lot of these variables might be dead. One actually exists.
Jaden Fiotto-Kaufman: Yeah.
Jaden Fiotto-Kaufman: And so like, you'll have to really know, like what's in your config, I guess, to know? Like, Oh.
Jaden Fiotto-Kaufman: what? Which of these like, I guess. Do you have like a way to know like, this one's dead
Jaden Fiotto-Kaufman: so what will happen is, let's say I wanted to access.
Jaden Fiotto-Kaufman: the is causal item at line 33, which is in an if statement, what would happen if that actually didn't? The if statement wasn't true is after your models are executing. It'll say, Hey, I'm still waiting for a value. This is probably an error, and it'll say I was still waiting for this thing. Did you call something out of order? Or was something not called
Jaden Fiotto-Kaufman: so? So that's that's that's that's right. That's a nice what thing.
Jaden Fiotto-Kaufman: So you find the error message. But it's a reasonable question, right? It's like.
Jaden Fiotto-Kaufman: wonder if there's some clever way of you asking for the source.
Jaden Fiotto-Kaufman: Maybe even after it's executed, since you're in the with block, anyway, and then, or or asking for the source in some way that the printout is deferred
Jaden Fiotto-Kaufman: so that you can highlight the variables that actually ended up existing
Jaden Fiotto-Kaufman: in the source. You can be like, huh!
Jaden Fiotto-Kaufman: You know, these blue ones are the ones that actually were, live in this pass. Okay, you're saying people could do like a test run or a run before. And then I would know which ones happened or not. The funny thing is you're inside this with, anyway. Like to to even get this horse. You're like doing a run right, aren't you right for sure that's what I'm saying, like any previous run I could have already be catching the stuff. But like, if
Jaden Fiotto-Kaufman: you know, in the very 1st time they invoke this. They wanted to interact with this variable, and then also know if it was live or not right. By the time it's I know what it is. It's well, like, yeah, when you run your experiment, you're not going to be like
Jaden Fiotto-Kaufman: saying that source right? This is for this is like a developer tool.
Jaden Fiotto-Kaufman: Right? No. People in their experiment will like so like in the code that every mib and Michael Hanna they'll like if they want to interact with the attention and like, do interventions on it. This is how they would interact with the attention. Oh, okay. So I'm confused. Yeah. So so like, you would say, Blah blah, you know you print blah blah.
Jaden Fiotto-Kaufman: and you need to do that in order to get access to
Jaden Fiotto-Kaufman: you know. Torch under, you know, or, you know, is underscore causal item, underscore 0, right like. If you don't say that source, then you then that other variable doesn't exist exactly exactly.
Jaden Fiotto-Kaufman: If you say, dot source, you can't access it, because that's how insights providing access to it. Otherwise, you're stuck with the inputs and outputs of modules. Oh, okay, because what I imagined was, you print out dot source to find out this secret, variable name, both both. So you print it out, and just be that
Jaden Fiotto-Kaufman: you could use that variable name without
Jaden Fiotto-Kaufman: saying dot source again. But maybe I missed this intent. What is it? So? Yeah, you can print it out to see this. But then you also take these variables off of the dot source attribute to actually interact with them. Dot source dot this. Yes.
Jaden Fiotto-Kaufman: but
Jaden Fiotto-Kaufman: like the the full chain to grab like attention output here.
Jaden Fiotto-Kaufman: Do you have something?
Jaden Fiotto-Kaufman: Yes, the full chain would be something like this. I wanted
Jaden Fiotto-Kaufman: I would go to the module dot source, the variable that I want, which is a function, and then dot source. And then, after this, it would be dot. Whatever other variable name, Dot, save if I wanted to access that variable, it would be like dot torch and ends
Jaden Fiotto-Kaufman: attention up. But yes, not saying yes.
Jaden Fiotto-Kaufman: I think I have the actual.
Jaden Fiotto-Kaufman: So if you said something like instead of just say printing dot source, if you had something like
Jaden Fiotto-Kaufman: print dot source dot, code, then that code could have the semantics of oh, it's actually
Jaden Fiotto-Kaufman: collects the code after the whole thing has been run so that you can color the things that actually were variables that were used in blue or something
Jaden Fiotto-Kaufman: right?
Jaden Fiotto-Kaufman: And so like code wouldn't exist until you've ran it one time. Yeah, something like that, like, because people
Jaden Fiotto-Kaufman: people don't care when the code is run, when they're just looking at the actual code. Right? Yeah. Yeah.
Jaden Fiotto-Kaufman: And then that could, that could can be. So I like, I agree with you like, it's it's like, you see, this complex code
Jaden Fiotto-Kaufman: and a lot of it is dead right in a typical run
Jaden Fiotto-Kaufman: that could that could make it a lot easier to navigate.
Jaden Fiotto-Kaufman: That's a good idea, right? Like like. What they really care about is like attention output. But you have to know whether using eager is true and self reorder dot guest. Attention is true, and if not, then I have to have a different
Jaden Fiotto-Kaufman: right?
Jaden Fiotto-Kaufman: It depends on which what the what all those bullions are
Jaden Fiotto-Kaufman: which, like, you can figure it out. But people won't want to. Yeah, it'll take, it'll take a long time trial and error.
Jaden Fiotto-Kaufman: Right?
Jaden Fiotto-Kaufman: Yeah.
Jaden Fiotto-Kaufman: yeah, I think that's a good idea.
Jaden Fiotto-Kaufman: And then just a quick end. If update. So so this. Really? So that's really neat. I actually haven't. You haven't seen that haven't seen that feature before. It's really cool, is that is, that is is that gonna interact with. So
Jaden Fiotto-Kaufman: you've had this longstanding thing of like.
Jaden Fiotto-Kaufman: you know, making attention easier, more uniform between stuff like this sort of uniform naming. Yeah. So if you've been thinking about like how the uniform naming idea will like interact with this. Yeah, exactly. So you could do renaming like instead of people having to access this like long thing. I could just say.
Jaden Fiotto-Kaufman: whatever you want to call this like the real attention or something you could just do like dot attention dot real attention. And I would have remapped that path to just be okay, dot source or whatever. So you think. So remap would interact with this. Yes, I see. Yeah, yeah. So I don't talk about it here, but one of the other, probably the biggest pain point I don't really address is in transformer lens, right? They totally recreate all the models based on their architectures, so that they're all consistent across architectures. And that's probably a little too
Jaden Fiotto-Kaufman: intensive for us so. But what we can do is we can at least have a common set of like renamings. So if you want to interact with, say, you know the output of attention. It's always called blah blah about attention.
Jaden Fiotto-Kaufman: which will work in a lot of cases. But obviously some models are so weird, right where that doesn't even make sense, to begin with.
Jaden Fiotto-Kaufman: So the model models are use existing, naming. So you care about that?
Jaden Fiotto-Kaufman: Oh, oh, you
Jaden Fiotto-Kaufman: Most of the modern models. They have the same like naming structure. Well, I know the llama. I mostly interact with llamas these days. It's like mostly dot model dot whatever. But I don't know. Do the other ones also do models?
Jaden Fiotto-Kaufman: They all do that model that layers essentially yeah, like sometimes they're just
Jaden Fiotto-Kaufman: the norm name is a little bit, but but it's still the module name is not maybe not
Jaden Fiotto-Kaufman: like. Sometimes they have like oak, Raj or outrage. Oh, hmm!
Jaden Fiotto-Kaufman: There's like, there's like slight differences, which is just yeah, it's annoying.
Jaden Fiotto-Kaufman: I see it so so yes, for mapping or no for mapping.
Jaden Fiotto-Kaufman: if you I mean, if you want if you want it, like as individual
Jaden Fiotto-Kaufman: yeah, I think, maybe most of us end up building
Jaden Fiotto-Kaufman: mapping on top, oh, you build a mapping thing.
Jaden Fiotto-Kaufman: Yeah, what using get adder or something?
Jaden Fiotto-Kaufman: No, I just like, I just call. I just like, build some like dictionaries of like module, common module names. And then
Jaden Fiotto-Kaufman: and then use bracket. 2.
Jaden Fiotto-Kaufman: Yeah. But one thing that exists in 0 point 4 that I'll reemphasize in 0 point 5 is you can pass this Dictionary of Rename to any language, model, or envoy, and that will just like rename anything you know, with this name to some other name, so I could like access it with either of them.
Jaden Fiotto-Kaufman: I see. So they so so that already exists that already exists. And so the only thing that we're really missing is like somebody to create standard Renames for everything. Yeah, is that right? Yeah.
Jaden Fiotto-Kaufman: interesting.
Jaden Fiotto-Kaufman: And so just a quick end of update. If you guys are interested. I can talk more about any of these things. So we've moved all of our non-compute services to aws. So that's like everything that isn't on Delta, and isn't executing the actual model.
Jaden Fiotto-Kaufman: that saves a lot of headache for us, because aws comes with a lot of these services like backups like telemetry.
Jaden Fiotto-Kaufman: and then it also moves away from this. You know, Mgpcc cluster that we were on, which
Jaden Fiotto-Kaufman: isn't super reliable in some ways and means. If one thing goes down, everything goes down
Jaden Fiotto-Kaufman: we've decided to move our entire backend service to delta from Delta AI.
Jaden Fiotto-Kaufman: So the difference is, Delta AI is this new cluster they had, and which was billed to us, as you know, the way to run AI models in the future. But
Jaden Fiotto-Kaufman: and I can talk more about this. There's this kind of this critical problem with this architecture, because this very new architecture.
Jaden Fiotto-Kaufman: And it comes with all these problems which prevent us from doing more interesting features
Jaden Fiotto-Kaufman: with Ndif. So now we're moving back to Delta, which has a bunch of a 40 s. And some h. 200 s. And that will allow us to do so many more cool things and also runs faster. I don't know. That's Michael's experience has been so far.
Jaden Fiotto-Kaufman: Yeah.
Jaden Fiotto-Kaufman: The low times on Delta now are like, it was like an hour for 4 or 5 people. It's like 15 min now. Oh, that's great. Yeah. What? The heck, yeah, yeah.
Jaden Fiotto-Kaufman: it really really screwed up with the grace. Yeah. Well, again, we also have to like again, copy all of the shards from the thing to the node and then back. But yeah.
Jaden Fiotto-Kaufman: yeah.
Jaden Fiotto-Kaufman: hopefully, in like 6 months they have. Does the large does the 16 bit 4 or 5 B fit? Yes, if it's if it's a 1 node. Yeah.
Jaden Fiotto-Kaufman: I think it has, like 30 GB left over. I don't know if that's enough we'll have to do some, I guess. Testing.
Jaden Fiotto-Kaufman: yeah, that's great.
Jaden Fiotto-Kaufman: That's great.
Jaden Fiotto-Kaufman: I have a question for you guys, do you guys have any sense of when you run
Jaden Fiotto-Kaufman: just your general experiments, like what percentage of the Gpu is being used by the activations versus the model like, is it like the models like 85%? And the activations are 15. I'm looking for some heuristic where I can say, like, Okay, make sure, this percentage is reserved for activations on any given Gpu.
Jaden Fiotto-Kaufman: that's exactly yeah. Less than 4,
Jaden Fiotto-Kaufman: right for the matching experiments. At least, I see.
Jaden Fiotto-Kaufman: Yeah, source base and intervention like 3.
Jaden Fiotto-Kaufman: Right? Interesting. And that's why I was talking to Delta. Right, they say, like, Oh, the Gracehoppers are so much faster than a 100 s. What are you talking about? And it seems like that is true for batch size greater than like 20, you know, if anything under 20, the A. 100 s. And a 40 s. Are still faster. So
Jaden Fiotto-Kaufman: yep.
Jaden Fiotto-Kaufman: So before Nemi, we have to have a lot of this done, what do you think? Well, I mean, I'm playing with tiny models that my batch size is like a thousand. So yeah.
Jaden Fiotto-Kaufman: but like, I know, you know, you use 70 B sometimes like anything you've done with. 70. B is probably like 50, 50,
Jaden Fiotto-Kaufman: best size of 50 would usually fit.
Jaden Fiotto-Kaufman: Okay, I should see what percentage by size has 50 is of what was it going to be
Jaden Fiotto-Kaufman: interesting.
Jaden Fiotto-Kaufman: So when the user code comes along.
Jaden Fiotto-Kaufman: is, are you able to look at it before you start running on the Gpu and estimate
Jaden Fiotto-Kaufman: like what its batch sizes are going to be, or how much memory it's gonna take. Or.
Jaden Fiotto-Kaufman: yeah, we could do that.
Jaden Fiotto-Kaufman: Yeah.
Jaden Fiotto-Kaufman: you know, before we execute, we can know we have the batch size we could. We could look at it, and then
Jaden Fiotto-Kaufman: again have some, at least with language models. Right? Have some easy heuristic of what it'll take cool.
Jaden Fiotto-Kaufman: So before Nemi, we want to have a lot of this done, so 0 point 5 definitely will be released by Nemi.
Jaden Fiotto-Kaufman: We're gonna have our own queuing service, and not to get too much in the weeds here. But how one of our dependencies Ray, is set up. We can't really get the queue size, which would be really valuable for users right to see where they are in the queue.
Jaden Fiotto-Kaufman: Surprisingly, that's difficult with what we have now. So we're gonna make our own queuing service
Jaden Fiotto-Kaufman: the the big end. If feature set coming in year 2 is involved around having a huge list of models that are available, and so they might not all be loaded into Gpu memory right? But they'll be available
Jaden Fiotto-Kaufman: to be loaded where you will have different tiers right where, like the the highest tier will be, it's in the Gp. Ready to serve warm will be. It's in the CPU memory ready to be swapped in, which is pretty fast and then cold will be. It's on, you know. The hard drive needs to be loaded in, and so you'll ask for a model that isn't available yet. And this will just happen automatically. So you know, our available models will go from like 7 to like, you know, 400
Jaden Fiotto-Kaufman: overnight. Once we have this
Jaden Fiotto-Kaufman: and that's probably like the the big feature that we're gonna be working on the next few months.
Jaden Fiotto-Kaufman: And on top of all this I don't know if you guys
Jaden Fiotto-Kaufman: no, but you know, one of our main offerings that we're having with Ndif is that, you know, we have insight which allows you to interact with models and end. If a backend actually do this on large models, and then on top of all this. We want to have all these interfaces for you to do. You know preliminary interp research that use, and if and insight as a backend in the classroom as preliminary experiments, so you can understand some behavior and then maybe go on your own, then do the real experiments.
Jaden Fiotto-Kaufman: And so by Nemi Caden is working on a logic lens version of this. And this is kind of our 1st Ui offering for Ndif.
Jaden Fiotto-Kaufman: And then I just have a little like this is the
Jaden Fiotto-Kaufman: the screenshot he has right now.
Jaden Fiotto-Kaufman: You know, you'll be able to select all of our like end of models as a dropdown
Jaden Fiotto-Kaufman: have little workspaces where you have, like your previous experiments.
Jaden Fiotto-Kaufman: run different prompts. See the you know the logits of the different probabilities via logit lens?
Jaden Fiotto-Kaufman: yeah. And you know, we see this obviously being much more useful in the future when we have more techniques like doing like circuit, discovery would be one of the coolest ones.
Jaden Fiotto-Kaufman: In this benchmark. Yeah.
Jaden Fiotto-Kaufman: did you guys see the the anthropic Ui thing for the circuit fit? That's cool.
Jaden Fiotto-Kaufman: Yeah, I was thinking like something like that for end, if would be like.
Jaden Fiotto-Kaufman: end goal for me for the Us.
Jaden Fiotto-Kaufman: Yeah, it's cool, you know, something like Nikhil's doing like, use that here? Yeah, somehow. Automate that.
Jaden Fiotto-Kaufman: Okay?
Jaden Fiotto-Kaufman: Alright. Any other questions for Jaden?
Jaden Fiotto-Kaufman: Thanks a lot, Jaden, thank you.
Jaden Fiotto-Kaufman: Hey?