177 Huntington: Oh, all right.
177 Huntington: right.
177 Huntington: See here, good! I'll get this all set up.
177 Huntington: I'll try it anyways.
177 Huntington: Well, you have a little card to get yourself up right? Yeah.
177 Huntington: Take a look.
177 Huntington: Thanks.
177 Huntington: Do we need more?
177 Huntington: Maybe a couple more.
177 Huntington: Okay, yeah.
177 Huntington: Website.
177 Huntington: Really. Only managed without one.
177 Huntington: put it out.
177 Huntington: Yes.
177 Huntington: alright, and then there's a there's red bean paste inside.
177 Huntington: Feel free to grab any extra chair that you find outside. Oh, oh, so there is one over here actually. Oh, Hi, sir.
177 Huntington: it's great! Here, we'll just throw this chair, the mix here.
177 Huntington: Do you know what to do myself? Oh, got the mouse.
177 Huntington: Right? Just leave it. Just leave it the whole time.
177 Huntington: So yeah.
177 Huntington: thought, that was something else.
177 Huntington: So that's your website now.
177 Huntington: So today we have Sheridan
177 Huntington: presenting you can just use this
177 Huntington: and and so she's going to be talking about research in progress. So
177 Huntington: you know, we very much welcome any
177 Huntington: comments and suggestions and ideas. She's got some very cool some cool observations and hypotheses that she's pushing forward. So with that? Sheridan. Yeah, go ahead. Oh, that's great. Yeah, that's great. Yeah. So yeah, like, David said, this is basically,
177 Huntington: yeah, this is basically some work that is like in progress. And he kind of suggested to try and structure this talk as if, like the project is done. Paper. I have the paper all done. But I I think I just didn't have enough time to really do that very well. So it's kind of going to be basically just like a bunch of results, and I tried to string them. It's like a chain, but it's not like a nice cohesive, like interlocking thing. If that makes sense.
177 Huntington: so I'll kind of be like starting with some like
177 Huntington: observations. And then like going into like an experiment that I did that. Yeah.
177 Huntington: So
177 Huntington: yeah, so first, st I'll just start with like, kind of a interactive activity. So you don't have to actually literally do this. But just think about doing this like if I asked you to like copy down like what is in like this little piece of paper on the slide? Just like, think about, you know. Like, if you have to like, write it down on a piece of paper something
177 Huntington: what that task kind of feels like to do?
177 Huntington: versus. You know, it's kind of like annoying right versus like if you rearrange all the tokens into like a meaningful sentence.
177 Huntington: it makes it like a lot easier to do the same task. Because you have this like layer of meaning that makes it like a lot easier for you to like, read and understand what it's saying, and then be able to like, repeat back, if you're trying to like, write it down on a piece of paper.
177 Huntington: So yeah, so
177 Huntington: there's this kind of observation that yeah, it's like easier to copy meaningful text than it is to copy scramble text, because, like with meaningful text, you have this like layer of meaning that like may let you basically are able to like, actually like, read and understand what's happening, and then be able to repeat it with the language skills that you already have
177 Huntington: versus like scrambled tokens in which you have kind of have no choice but to like labor over every single token or every single character, or whatever depending on what layer it's scrambled at, or what level it's scrambled at.
177 Huntington: So basically what I'm trying to do right now in this project is figure out like whether Llms copy meaningful text in like a smart way, where they like, read the meaning in from the surface level symbols, and then like, say that meaning back out, or whether they just like blindly index through tokens and spit them back out undigested. And so this isn't like in context, like copying tasks.
177 Huntington: And so I personally think this is interesting enough. But I think, if you need like, more of a grand justification for this. You can kind of think of it this way. So like, in order to model language. Llms have to have like representations of meaning that are
177 Huntington: like pretty like independent from tokens, especially because a lot of tokenization is again is kind of like unreliable, but also because, like, even if you had like a perfect tokenizer that had like one word for every like one, token for every like word, you would still get cases like names and stuff where, like the meaning of the token doesn't actually correspond. The meanings of the individual tokens don't actually correspond to the meaning of those tokens like together in that particular combination.
177 Huntington: So like previously, what I was working on was like show kind of showing evidence of like models like erasing token, level information at the end of multi token words. So, for example, like in this example, like Clifford Brown, that's like 4 tokens
177 Huntington: And so like. If if you imagine that you're like a language model that's like autoregressive auto regressively reading in those tokens you would see like cl, IFFO. RD. And then brown, and then you would recognize like Oh, Clifford Brown, like the trumpet player.
177 Huntington: and so like the actual like meaning of the actual tokens like Cl and Iff aren't really relevant to like representing, like the concept of that trumpet player. And so, yeah, and so like, presumably this erasure, or like what I was hypothesizing is that this erasure is to create concept representations that are independent from token representations.
177 Huntington: but yeah, so like. The reason why this is interesting is like, in order to like, do any sort of like language modeling task, you have to have, like the meaning representations. So, for example, if you want to translate a concept into another language, you need to have, like, you need to like, have the representation of the concepts that you can then, like, spit out like, or convert that into a different sort of token representation in like a different language, for example, and so basically, the reason why I'm looking at this copying setting is like, maybe
177 Huntington: maybe we can try to separate
177 Huntington: this like put token level representations from like the meeting level representations. Even if the model could just be doing like token level stuff, if that makes sense.
177 Huntington: So yeah, does anyone have any questions about about that.
177 Huntington: Have you seen? There's these 19 seventies studies on chess by like these learning psychologists, where, if you take a chess board. You can scramble the chess board. You can take grandmaster, chess grandmasters, and they'll they'll it's like chunking research, so they'll like they can. You can scramble the chess board, and they won't be able to reconstruct it. But if it's not scrambled, then they can reconstruct the whole board from memory. The non grandmasters, if a non grandmaster sees a scrambled board, they're better at chunking it than a grandmaster.
177 Huntington: I don't think they're better, but they're like equal. No, I think they're better
177 Huntington: like the non masters are worse because they're so like their priority is so fixed on like.
177 Huntington: are you sure? I I remember reading equal, anyway, this kind of like, really relative. Yeah, also, I'm going to be writing down a bunch of stuff.
177 Huntington: Yeah, we have board game people in the room. Right? Is, is John here? Yeah. Oh, that's a nice little board game representation thing to to reproduce that that other one exactly, just in chess
177 Huntington: with a chess model to see like an auto encoder with chess or something like that.
177 Huntington: Yeah, fine. So looking at se features or like peaches stuff
177 Huntington: good at reconstructing the board. It's often the
177 Huntington: as you say. It takes chunks, and it lands for both Othello and chess. It's like we have an sae feature that's both like high precision and recall. It mostly corresponds to like
177 Huntington: the bottom left, being, like in a specific configuration, with like 6, 7 individual pieces.
177 Huntington: the constellation of 6, 7 individual pieces.
177 Huntington: So yeah, right that that aligns on.
177 Huntington: That's cool.
177 Huntington: Yeah, the others. Oh, yeah, the other thing is
177 Huntington: so yeah, I I just kind of included this like philosophy thing. So there's this essay called The Task of the Translator by Walter Benjamin.
177 Huntington: and I think it's just kind of interesting, because in like sort of from like
177 Huntington: like literary theory and stuff, there's like this idea that, like for a text to be a text. It must, like already, bear the property of being translatable.
177 Huntington: So the translation does not come later. But is there from the start as a proper possibility of a text that defines that text. Essentially. So. I just thought that that was kind of an interesting, like theoretical, like justification, or like theoretical, like idea. That kind of aligns well with this project of like
177 Huntington: the Llm. Representations of text, should like kind of bear, the property of being translatable from the start, as in like the Llm. Should be like taken like when it's processing text, it's like
177 Huntington: converting it into like a meaning representation that is just like inherently, like translatable. But yeah, that that was just like a yeah. Do you care about the trans, like the model itself being able to translate that text, or like as long as other models are also able to translate that text.
177 Huntington: I guess I guess what I'm trying. What I'm thinking of is more like the models, internal representations, as in like. If if a model like reading like a piece of like English text, it's internal representations should be something that like
177 Huntington: is sort of like language, independent if that makes or not, should or like it would be cool if it was that that's just kind of what I was thinking. I just thought it was a cool little
177 Huntington: like idea.
177 Huntington: But yeah, the noon.
177 Huntington: So yeah. So yeah, first, st I'll just talk on some background on like copying in general. So I'm sure, like, most of you guys
177 Huntington: all know about induction heads. So yeah, they're like described in some like of the classic, like anthropic blog posts. And so yeah, like one. So basically, they discovered these heads in like 2 layer toy models.
177 Huntington: and they kind of
177 Huntington: like. So yeah, they find these circuits that are like important for copying random strings of tokens, but also like lots of other different types of tokens. So, for example, if you have like a sequence like the these random tokens here like en high fly, and later the goal, like the idea is, if you see high if you see that sequence repeated, you would want to like copy over like
177 Huntington: the next token. So like you would want to copy over fly so like the way that they find those models implement. This is with like a 2 stage process. So first, st there's like previous token heads that store like information about the previous token in the current hidden state. So I have that like drawn like here.
177 Huntington: So like, for example, the representation for fly represents that high came right before it. And then later, you have, like the actual induction heads where, if you're at high, you would basically like, search over these
177 Huntington: previous tokens and see like, find one that has a little flag here that says like Hi came before me, and then that that's how you can know that you can. You want to like promote same fly next that makes sense.
177 Huntington: But yeah, but like what they also claim is that induction heads can be used for like fuzzy copying tasks so like, for example, like translation?
177 Huntington: so. But then, if you think about it, you kind of start to run into some problems with tokenization. When you try to like
177 Huntington: you know, do the same process again. So, for example, like, if you're you have the sentence, German I eat potatoes. Pomme de terre is like 3 words. It's actually 4 tokens and you'd have to be able to map those like 4 tokens, or like those 3 words into like just the word potatoes in English, and so
177 Huntington: basically, so yeah, basically like the token, indexing thing that was like here wouldn't actually like work exactly in this setting anymore. And so you can kind of imagine, like a different type of induction process that operates over like these chunks, instead of operating over like the actual individual tokens. So
177 Huntington: for example, in this this diagram that I have here.
177 Huntington: yeah. So if you have, like, home deterre you would imagine that, like, maybe
177 Huntington: this the the end of this chunk store says that like each came before me.
177 Huntington: And then later, when you see the English word eat you can like, then copy over this entire concept of potatoes so that you can output
177 Huntington: the English version of that concept.
177 Huntington: Does that make sense? Is it.
177 Huntington: Yeah. So this is kind of like this idea that I have in my head. For, like the rest of this, of what I'm trying to like, figure out. See if we can show that this is happening. Yeah. Do you think that information is lost with respect to translation, like as an example in mandarin
177 Huntington: lots of fruits, basically are 2 words. The 1st one is basically describing what type of fruit is, and the second is the word for fruit. It's the same with berries. So in a sense, there's like a categorization that's sort of done like, just just
177 Huntington: by the how the words are defined. And if you translate to English, I mean strawberry, that's, you know, blackberry, that that holds. But for, like you know, apple, banana, orange like you kind of lose information in that respect. So yeah, I'm wondering whether you think there's like a like a 1 to one like kind of invert inversion between the 2. Or if there's like a loss.
177 Huntington: Yeah, that's interesting. Yeah. Cause cause to me, like,
177 Huntington: yeah, cause like, apple is like ping pour right? Like, it has the little yeah has has the little like fruit word in it. But I never thought of that like, I wonder.
177 Huntington: like, I wonder how much like the actual like word.
177 Huntington: like the actual like fruit word, if that's like extra information or yeah, I haven't really, I don't know cause
177 Huntington: like to me, this feels like part of the word like it's just how you write the word, if that makes sense. But I'm also not like a native Chinese speaker. So maybe when you speak like, if you are a native Chinese speaker, when you say apple, you actually are thinking fruit but like, for example, like in English, you could say like jackfruit, or something like I don't know if I really think like Oh, jackfruit like the word fruit is in there, and that's how I know it's a fruit like. If it didn't have the word fruit I would still know it's like
177 Huntington: a fruit. If that makes sense. English has inconsistency, though where some of the there's some
177 Huntington: forgetting the example. But sometimes the name, it seems like it's a category, but actually isn't
177 Huntington: so. Sheridan's hypothesis were true, then what is richer in your case? Chinese representation would just like have this
177 Huntington: maximum rich rep internal representation of of the concept that's happening. And then.
177 Huntington: whatever you have to query data, maybe you can like, do specific group queries or other queries. It would. It would be able to access any part of the information. So
177 Huntington: from from yourself. I don't suspect that will be
177 Huntington: much loss of the language models. Good at modeling.
177 Huntington: Yeah.
177 Huntington: Oh.
177 Huntington: so here are you also assuming that the relative position of the words, they will be kind of same across all languages.
177 Huntington: Yeah, that's that's that's a hard thing. I haven't really thought of that yet. So far I've just been doing word to word translations. But then there's the entire problem of like, if word order changes what you're copying between
177 Huntington: all languages. So yeah, need more layers.
177 Huntington: Yeah.
177 Huntington: Yeah. Then, yeah. So that that's a that's a whole other thing.
177 Huntington: Yeah, like, John has some projects right where you, where the things are.
177 Huntington: they bind together in the like the wrong way.
177 Huntington: where you cannot just do induction
177 Huntington: when you put things into boxes
177 Huntington: right? And then you ask, what's in the box?
177 Huntington: So it's actually it's flipped.
177 Huntington: I put an apple into box one. I put an apple, a banana into box 2.
177 Huntington: What is inbox? One.
177 Huntington: It's not an induction head, but the model still can do it right. So
177 Huntington: maybe it does something like that so? What was Anam's question?
177 Huntington: Oh, the I mean here the word order is convenient, such that
177 Huntington: eating or manche is before potato
177 Huntington: right? And then it's just like an induction problem.
177 Huntington: But other there might be a different language where the the word order flips.
177 Huntington: Like, like, I, I'm kind of like in Bengal this week. Yeah.
177 Huntington: there, usually the word eat. It comes after different.
177 Huntington: So you need a different mechanism
177 Huntington: like the putting things into boxes.
177 Huntington: Maybe you need a representation of the entire sentence.
177 Huntington: And then, Oh, yeah, which is what? What? That's what Karen was working on.
177 Huntington: Yeah.
177 Huntington: But yeah, it's it's hard. What are the I feel like? I don't run into that good ways to get sentence representations like there's there's like sentence embedding models like vert ones. But I feel like they're like.
177 Huntington: that's like sort of something that popped up.
177 Huntington: So one thing that I have observed in some of my research is sometimes model, do encode like the high level meaning of the sentence at the at the period token, or the full stop token, or one or 2 tokens before that.
177 Huntington: But I'm not sure how generalizable that is.
177 Huntington: Yeah, that's what that's what Ken is trying to look at. Oh, right now. But it's it's
177 Huntington: can I say something. I think, that even if, like, it's not certain about the order of the words
177 Huntington: steel induction heads will want to take like a higher responsibility in this type of tasks. So
177 Huntington: last semester I was doing a project with Professor Bao
177 Huntington: about like I was testing. If it can complete some cycles like
177 Huntington: a sign sinusoid. Then Professor Bao asked me, do one thing, instead of having the frequencies having having rational frequencies, use irrational frequencies, and see if
177 Huntington: if if induction heads are still there, and I literally saw, like one induction head that lit up in these
177 Huntington: case of rational frequencies.
177 Huntington: Also, later, in the case of irrational frequency where it was not certain about the exact position of the token to copy from. But yeah, the its responsibility actually went down in in the case of irrational one, but it was still the most significant. There was a most significant attention head
177 Huntington: that was responsible for sequence, completion in both of the tasks. So maybe
177 Huntington: in this case, as he was saying, that the order might not be the same. But still induction head is doing the heavy lifting in this type of tasks.
177 Huntington: Hmm, I might actually ask you about that later. Because, yeah, I'm not sure the setting exactly. Oh, yeah.
177 Huntington: yeah. I mean. You can imagine that like.
177 Huntington: but there can be a layer below this layer, but this is the layer that does the copying.
177 Huntington: As long as the tokens are nicely annotated, and you can just copy the right one, right?
177 Huntington: Just need to nicely annotate your tokens in the previous layers.
177 Huntington: and that can go all directions
177 Huntington: like some sort of entity type. In, yeah.
177 Huntington: right? Cause you do have like the entire sentence.
177 Huntington: yeah, before you have to translate it, I think, yeah.
177 Huntington: what do you say? It can go both directions, like all that. Well, I'm not saying it can go against the causal masking thing. But
177 Huntington: been the model can do. Apple is in box one, what is in box one. So
177 Huntington: this is kind of it's like A, it's a BA pattern instead of a B oh, okay.
177 Huntington: yeah. So yeah, this is another slide for say, the same thing. So basically yeah, i'm kind of hypothesizing like, 2 types of induction. So there would be like token induction, which is kind of like the classic induction head setup over individual tokens you have to store, like the previous token, and it would work for like any text like, it doesn't matter if the text is like random tokens or whatever. So it's kind of like the models like backup plan for one like, if it just sees a bunch of random tokens, they can at least like look for patterns within, like the token level stuff.
177 Huntington: Whereas, like you could also think of like concept induction, which would be like over word, like chunks. And you have to store like a kind of like pointers like the previous chunk. And it only works for meaning. It would only work for meaningful text, because you can't make chunks out of like
177 Huntington: random tokens, and the idea is this would maybe be used for a lot of different tasks.
177 Huntington: So sorry. I don't know what happened to?
177 Huntington: Yeah. So yeah, so next, I'm just gonna kind of show some qualitative like head behavior to give intuition that I think these heads actually do exist.
177 Huntington: so
177 Huntington: yeah. So for the individual token induction. You can look at like the prefix matching score from previous papers, which is just how much attention
177 Huntington: is paid to like the next token that's like about to be copied by induction. So, for example, in this setting, if you add the token like how much attention is being paid to top star, which is like the next token versus like a chunk score, which is
177 Huntington: what I just made to try and look at these heads, which is like, How much, if you're at the token like, how much does that token pay attention to the end of the next concept. And so to actually do this, I just took a bunch of counterfax subjects like I did for the previous stuff that I worked on, took a bunch of counterfac subjects and stuck them in induction, random induction sequences, and then just measured how much
177 Huntington: the like ahead attempts like the end of that in like an induction context?
177 Huntington: and so just to clarify. So you have just like one sentence or you have a
177 Huntington: giant products. Is it like, I like Star wars full? Stop. I like. And then you check.
177 Huntington: Or is it more than that? Yeah. So the the actual, the real seat. Yeah, that's a good question. The real sequences that I'm using. I didn't put in here. I should probably include an example. But it's a length 30 sequence of.
177 Huntington: So so I sample some text from like the pile. I think I use it either the pile or Wikipedia, and then tokenize it and shuffle all those tokens. So you have like a random sequence, and then take like 30 of those, and then added like a new line separator, and then, like the same 30 again, and then measure at the very end whether it is able to predict, able to complete that sequence.
177 Huntington: Does that make sense. Okay, and so when you shuffle it and do a new line character.
177 Huntington: after that, you you you take some like chunk of those previous tokens, and then paste it, and then it let it let it generate. Yeah, I just take the entire. Repeat the entire thing
177 Huntington: again. And what do you check then? Yeah. And so then, for the prefix scores, I check how much attention the last token pays for each of the each of the token that you pasted in the second line. Yeah, actually, only for the very last one, because.
177 Huntington: like in this particular setting, I have, like a bunch of random tokens, and then those tokens again. But if you started at the beginning, the model might not like realize yet that it needs to be copying. So I just do it at the end of, like the second chunk, just to make sure like
177 Huntington: that model has seen like this like copy happening. If that makes sense. Okay?
177 Huntington: Oh, oh, so the thing which is not clear is so.
177 Huntington: But are you making sure that the end token there is like the was in that in that concept
177 Huntington: thing. Yeah, this is a question. Oh, yeah. Yeah. So for yeah, for this score. I just like insert the Counterfax subject at the end of the yeah. Okay.
177 Huntington: And how do you check the chunking? Then, if there are all random bookings?
177 Huntington: well, for like, for the like. I have like a random sequence. But then I like, have the counterfac subject that's not scrapped. And I just like stick it in
177 Huntington: yeah,
177 Huntington: And so, yeah, when when I do this, I'm able to get like.
177 Huntington: you know, for each head there's like a score of like how much it attends the next token, and how much it tends to the next chunk. And so we can use this to kind of pick out heads that have certain behaviors. So, for example, like this, oh, for sorry. Yeah, I didn't mention 2 llama, 270.
177 Huntington: How many layers and how many? So there's 32 layers and 32 heads per layer.
177 Huntington: Okay, yeah.
177 Huntington: So, for example, this is, this is an ex. Oh, my, things are a little messed up. But this is an example of like a regular like token induction head. So I know the the text is hard to read. So I just put like the actual like a bigger version down here. So at this position. So this is showing attention from this position. Like the Ave token at the end of microwave
177 Huntington: and it's attending to. So if it's doing token induction, or since it's doing token induction, it's attending to the o of like the very next token. But then, if you look at like a different attention head that has
177 Huntington: a higher chunk score, and a lower induction prefix masking score, then this one. This Abe attends to like, then it attends to like.
177 Huntington: not not the next token, but like the end of the next word that makes sense.
177 Huntington: so that's just like an exam. This is just like one example. But you know, I was kind of like looking at these attention heads, behaviors to see like there actually are heads that attend to like the end of the next like word, instead of attending to like the next token.
177 Huntington: This, for specific example, do you? Does it? Does this distribution look similar? Average of a bunch of examples? Yeah, interesting. So or like, at least when you look at. So the score is the way I calculate. The score is again like how much attention is at the end of the chunk, and so like you can find heads that have like a high
177 Huntington: like chunks for. But I I haven't actually averaged like this exact attention thing over a bunch of examples which I could do. And I think it would show.
177 Huntington: yeah, smoother. That's cool. Yeah. So your hypothesis that this chunk scores is kind of like a proxy for how this attention head is modeling the concept instead of one specific token. Yeah, like, yeah, head, that would have a high trunk score is a head that is like the hypothesized like concept induction head. That's like attend attending to the next concept that it wants to copy.
177 Huntington: Yeah. I'm also wondering if the token come at last, move it.
177 Huntington: It's the same token, but it's it does not refer to the previous concept that does it also. Attend to them. For example, if the work here is not microwave, but it's like avenue where it's
177 Huntington: it still be the same, or it will be different, like if the next word was actually like, if it wasn't actually coffee. Yeah. So I guess in that case the model would maybe predict it would still be predicting oven, because it's trying to predict what's coming next. And so
177 Huntington: I guess it would predict oven. But then get it wrong if if the next word is like actually avenue.
177 Huntington: if that makes sense.
177 Huntington: And if we're not compound tokens, does this head not fire for like the token? 2 after? Yeah. So that's a good question. So this this particular head fires. If there's not compound tokens, it does regular induction. But then there's also heads that only fire for concept induction like, yeah.
177 Huntington: have you tried checking this for us? I don't know what a synonym for microbe, but something similar setting. And does it fire on those tokens as well. That's it. Yeah, I haven't checked that. But yeah, that's
177 Huntington: yeah. I haven't looked at that.
177 Huntington: Which model is this llama? 2 7. B.
177 Huntington: Thanks was like that was the token
177 Huntington: attention at an earlier layer, or like, because this one is at the layer at 11.
177 Huntington: Oh, yeah. So actually, yeah, they're not like the the token ones is at 16, like, they're actually not.
177 Huntington: Yeah. And if you look at like, Yeah, go ahead. Matthew, yeah.
177 Huntington: yeah, it's it's weird. Yeah, like, they're pretty mixed like, if you look at the top, 50 token heads and top 50 chunk heads, they're like all over the place.
177 Huntington: Definitely layer like 1516. There should be some guy copying stuff around.
177 Huntington: Look, wait! What do you mean? Based on my prompts.
177 Huntington: based on my multilingual prompts like, somewhere in the middle.
177 Huntington: There is some crazy concept copying going on.
177 Huntington: Okay, yeah. And I think, yeah, cause all the all the heads are like are in the middle layers.
177 Huntington: oh, yeah. So even the token one was at 16. Right? Yeah, the token ones at 16. The concept one
177 Huntington: is that 11, this other concept one. This is a concept, one that only that only fires on concepts, but it also like fires
177 Huntington: based on my tiny slice. 100. Wait, I just I. Just from that conversation I actually just realized that I didn't catch which model we're using here. 14 is a good vintage
177 Huntington: for your concept infection heads.
177 Huntington: If you go back one side, I'm kind of surprised that it's not just looking at the full concept of microwave oven, but also
177 Huntington: leaks a bit into the start of the next sentence. You think it's more like an area like position encoding thing where it's like, Oh, I'm or is it all the
177 Huntington: yeah question, are the concept induction heads only looking at concepts? Or are they just looking at an area around the token? Yeah, that's that's a great question. So that's what I still have to take like separate, because I think they're also probably are heads that just. They're just copying like the next few tokens instead of yeah, somewhere around there. And and so I think there, there definitely are heads that do that
177 Huntington: And so I need to make sure to separate like kids that are just like, Oh, copy kind of forward from here versus, like, yeah, both is interesting. Right? Yeah. Checking with similar words or synonyms would be a great way to verify this. Yeah.
177 Huntington: or broad level translation. Yeah, yeah. Yeah.
177 Huntington: Which I might have missed. How do you compute the chunk score? Oh, yeah. So the chunk score is if so, I have, like the these like random token sequences that are like a bunch of random tokens, and then a counterfax subject, then a bunch of random tokens and
177 Huntington: like, right right before the Counterfax subject. Yeah. And how much does that last token attend to the end of the counterfax subject instead of the 1st token of the so okay, so so that number that 0 point 0 6 is like
177 Huntington: the average across some bunch, some number of counterfactuals. Yeah, for this particular head average, on the last token, which which is, looks really low, but it is in like the top 20 of
177 Huntington: the chunk scores, because the Max is like 12.2 6. Yeah. So I think you what you can do is instead of using just the normal activation score, you can use that value weighted attention score.
177 Huntington: It's like you take the
177 Huntington: You not only take the K, the multiplication of Kv. You also multiply it with the like l. 2 or l. 1 norm of the value vectors that might give you a bit more clear result, because if you just do like attention, score a lot of it's just attended. Attend a lot at the 1st 1st token.
177 Huntington: But do not do not really bring in a lot of information from the 1st token, so which gets
177 Huntington: kind of ruled out or cleared out. If you use the evaluated one. That's that's awesome. Actually, yeah, that's a good point. Because also a lot of the time like the heads are attending a lot to beginning a sentence and new line. And you can see a little bit of induction stuff happening. But it wouldn't be captured. Yeah, yeah, that's a good suggestion.
177 Huntington: Oh, yeah, any other.
177 Huntington: Yes, yeah. So that's like, sort of this.
177 Huntington: this like intuition from looking at these heads that like, Oh, there are actually like these 2 processes going on. But of course, this is just like looking at attention patterns. So we need to actually like, get some causal stuff happening.
177 Huntington: so oh, yeah, so. But before we do that I'm going to also add this like experiment that we tried, and I told Quinn about it and it. And then I realized that it actually wasn't as cool as like. It doesn't work as well as what I showed you, because I kind of made a little oversight. But it's still like kind of works. So it's kind of interesting.
177 Huntington: so yeah, so yeah, so we know that there are heads that are attending to the next chunk instead of the next token. And so the hypothesis is that these heads are copying over the entire concept of like Star wars, even if it's just doing simple induction. It's like copying over the entire thing in Star Wars and and like saying Star Wars out again.
177 Huntington: So if that's the case, then the hidden State, right before the model repeats, Star Wars should like.
177 Huntington: should contain the entire concept of Star Wars, so like this hidden state once it's copied over here, this should have
177 Huntington: Star Wars. Not just star right?
177 Huntington: So
177 Huntington: yeah. So to try to like, see whether this is the case. Whether this hidden state actually does have the whole concept. We have like 2 prompts.
177 Huntington: doesn't. I don't know if this is the clearest diagram. But
177 Huntington: so the 1st prompt is like an induction prompt. So again, like just a bunch of random tokens. And then you have an uppercase version of a counterfac entity.
177 Huntington: And then again, a bunch of tokens and you know, and and the model has to like output.
177 Huntington: you know, copy like that entity.
177 Huntington: but then the second prompt that I'm working with is just like a Wikipedia excerpt that talks about that entity and, like eventually outputs Star Wars. So like
177 Huntington: this, I don't know. This is kind of a contrived one. But like, you know, some some Wikipedia article about Star Wars.
177 Huntington: And so
177 Huntington: yeah, so assuming that the model is copying over the whole concept of Star Wars. Then this. So at this, in this induction setting, if we're assuming that, like this, entire chunk has been copied into here, then if we
177 Huntington: take that hidden state and put it into another context, the hope. Or the idea is that if you then get them all to out model, should be able to output
177 Huntington: Star Wars even like, if you like, change the tokenization. So if you like, if this one is uppercase, but it's converted into like a meaning representation, then this should just have the meaning Star wars, and it should be able to output the lowercase or the regular title case version of that entity. Does that make sense
177 Huntington: as a are you doing direct patching from main to main
177 Huntington: from, yeah. Okay, yeah. So this is like, just like, kind of a preliminary thing to see see what happens when you do this. If if you have access to the heads with higher chunk scores.
177 Huntington: you kind of have access to the specific subspace in the residual stream, that this attention has a writing tool. Right? Yeah. So I, yeah, I'm gonna do that after this. But I I didn't end up being able to do it for like cause. I just did this patching thing just to like as a sanity check kind of thing.
177 Huntington: even though it doesn't didn't really check my sanity because it didn't exactly work.
177 Huntington: But that's what I'm gonna do next is try it with specific head outputs, and it will might actually work better. What do you mean it didn't work.
177 Huntington: So, yeah, so this is what
177 Huntington: what the results look like. And
177 Huntington: again, this, yeah, I did this pretty recently. And so the plots aren't like super clear, but had the way that you read this plot is so yeah, the X-axis is the layer that I patch the hidden state. Y-axis is the probability of outputting, like the lowercase version of the token of the lowercase token version, and then this, like orange, is yeah. The probability of, like the lowercase version, so like the one that we're trying to get it to output
177 Huntington: And so which one is the uppercase. And so uppercase is the blue.
177 Huntington: Oh.
177 Huntington: and so the uppercase. Yeah. So the model doesn't really have like any probability about of putting the uppercase one in like the Wikipedia article but then, when you patch in
177 Huntington: but it does have like a little bit of probability already about putting like the lowercase version, because the context is like relevant to that concept. Which is again, another thing. That kind of bothers me about this like it would be better if it we were patching into like a totally random context. And so I kind of want to fiddle around with that more
177 Huntington: but you can see that like patching like the.
177 Huntington: So the representation that's about to copy the uppercase version also does promote the token for the lowercase version
177 Huntington: in the Wikipedia context.
177 Huntington: If that makes sense, it it is. And so the reason why I said it doesn't really work that well is because it doesn't work for every example. And it's like.
177 Huntington: sometimes the probability changes. Only like tiny, like this was like a cherry picked version. So like
177 Huntington: this is another example that kind of worked, but it's also at an early layer as well, which is not what I was expecting. I was expecting it to be mid to late.
177 Huntington: and so I kind of like, don't really trust
177 Huntington: this experiment. And I feel like.
177 Huntington: yeah, I feel like, I need to work on it a little bit. Yeah.
177 Huntington: also, is it expected that the model has no probability on the capitalized tokens?
177 Huntington: Yeah, yeah, it's it's kind of weird
177 Huntington: cause cause that cause. Well, cause the context is all lowercase, and so like.
177 Huntington: why would it be trying to output like an uppercase
177 Huntington: next token? But it should be like a little. I don't know. It feels like it should be a little because of the induction example.
177 Huntington: You you should add another line, because, like.
177 Huntington: there's a lot of probability mass missing here in this plot.
177 Huntington: So I'm sure the model output something at Layer 30 right? And that probably is uppercase in some way
177 Huntington: it's there if everything was uppercase in their source prompts.
177 Huntington: Right? So yeah, so I
177 Huntington: right? So at least, when I was looking at the outputs of what the model is like actually outputting.
177 Huntington: It seems like it's kind of just ignoring my patch and just outputting at some sort of likely next token in that context. Are you overwriting the residual stream?
177 Huntington: Fantastic?
177 Huntington: Right? You cannot ignore the patch at layer 30. Right? Yeah, no, that that makes sense.
177 Huntington: So the 32 you should feedback in your source prompts.
177 Huntington: Right? Yeah, yeah, I should check that. Maybe it's maybe it's just a mistake.
177 Huntington: Yeah, that's a good check.
177 Huntington: Sharon. Is this a joint probability over all the tokens? Or are you looking at the
177 Huntington: the 1st talk about the laughing?
177 Huntington: Oh, it's just yeah. So it's looking at like.
177 Huntington: So yeah, we patch like,
177 Huntington: it's looking at the second token.
177 Huntington: So for wars, for wars. Yeah. So again, the probability of wars, right? So it makes sense that the capitalized one is 0. Because the 1st token is.
177 Huntington: yeah, right? Yeah, because the 1st token is lowercase. Yeah.
177 Huntington: Okay?
177 Huntington: A second question is, I see that
177 Huntington: the chunk index token the the chunk head that you have
177 Huntington: Is it an assumption that the location from which it's getting the information from the Rs. Token there
177 Huntington: that probably the language model already figured out that it's a Concept Star Wars, and it has the information already present. There
177 Huntington: is it assumed? Do you have an evidence in your previous work?
177 Huntington: Yeah. So that's assumed. That's what I'm trying to see. Do you think it's possible that maybe it isn't there at that layer
177 Huntington: or no, maybe not. I don't know. Yeah. So you mean, you mean at Rs or at yeah yeah at the Rs where it's copying the chunk index.
177 Huntington: Yeah, definitely, it could be well, I mean, no, but the thing is.
177 Huntington: if it's if it is doing the chunk copying. Then it would have to like it would have in order to make the chunk. It would have to know what Star Wars is right so.
177 Huntington: but maybe, like again, since it's an induction setting.
177 Huntington: there's maybe there's no need to actually represent the meaning, or, like, have a really rich concept representation. Maybe it's like a I don't know like a week
177 Huntington: chunk or something. But I yeah,
177 Huntington: yeah, actually, after seeing this that you patch like this, I take my comment back, I revert the comment.
177 Huntington: okay, fetching at the last layers shouldn't do much here.
177 Huntington: Yeah. Oh, yeah, because we're measuring the next next. Not then. Yeah.
177 Huntington: I forgot, it shouldn't do much because it's not going to attend to that anymore.
177 Huntington: Gonna copy whatever it needs to copy. Big st, right? Yeah, if you, just yeah, right? So that's why it's a forcing forcing experiment. So yeah, because here, this would definitely just output St, sure. But the idea is like, maybe if you then tell the model. Oh, it's actually we're actually outputting the lowercase version.
177 Huntington: Then it would attend to the concept here and output lowercase.
177 Huntington: Yeah, do you
177 Huntington: think the attendee and entity is actually the entity in the sense of like? You know, since the representation you're using is like from like random tokens beforehand, and then just capitalize tokens of it, then that maybe it doesn't have that star wars
177 Huntington: like entity meaning representation that we actually want. Yeah. So like, maybe if it's like from an from, let's say, a blurb of like Star Wars text that for sure, we know that entities should be talking about Star Wars is that by any chance make any improvement? Yeah, I want to try that. Because, yeah, maybe
177 Huntington: yeah, maybe again, like the model, like, if it's in an induction setting like it doesn't
177 Huntington: concept as much. If that makes sense. Yeah, yeah.
177 Huntington: how can you distinguish between the model retrieving some knowledge from its pre-training versus the model reading from from the context. So, for example, in the induction has thing, I think they do just names
177 Huntington: random names that they defined in context, and then look at induction here for the words Star and Wars, although I may have.
177 Huntington: don't use induction at all.
177 Huntington: I could still like predict walls from memory. Right?
177 Huntington: Yeah, maybe maybe it's
177 Huntington: it's more clean and set up to just use ambiguous concepts that are not like made up entities made up entities. Or, yeah, but you want you want entities that has a concept representation of somehow.
177 Huntington: Yeah, how would you know the ambiguous one has an actual concept with that
177 Huntington: cause. I guess the Ambi cause. I guess the ambiguous one would be the token level copying, but
177 Huntington: a known one would be like
177 Huntington: the meaning level, like a concept level copy. Right?
177 Huntington: I guess it's like a nice counterfactual. But it's a good. Yeah.
177 Huntington: Maybe the cleanest thing would be a surprising entity.
177 Huntington: like if the model was going to naturally predict New Jersey.
177 Huntington: and then you put New York there, or something like that.
177 Huntington: and then it. Then there's this other concept. But at that token level of one or 2, something else. Would that would that
177 Huntington: yeah, mitigate your your concern?
177 Huntington: Yeah.
177 Huntington: So with this passing, did you try to localize it to heads?
177 Huntington: Not yet. But I want to do that
177 Huntington: next. Yeah. Then you can look at their attention patterns as well. Yeah, I think that would be the nicest thing if you could show that specific heads will promote like this behavior. Yeah. And those heads are the ones that are attending to that. Yeah.
177 Huntington: I'm not sure what? What time?
177 Huntington: Oh, my gosh, I haven't even gone to the cool part. Okay, maybe. Okay, how much time? Okay, let's see. So I have.
177 Huntington: you're almost 1 h. So you can. Okay, okay, yeah. So
177 Huntington: right? So the next thing is like actually trying to do causal experiments to find heads that output chunks.
177 Huntington: So
177 Huntington: yeah. So this is gonna be the setup. So we know that there are heads that attend to the ends.
177 Huntington: And maybe it seems like, you know, the whole thing is being copied over.
177 Huntington: so yeah, so let's just directly identify a bunch of heads that actually promote chunks. So the setup is, I'll have like a clean prompt. That's like again, random tokens, and then like an entity like Graceland, and then random tokens again.
177 Huntington: and then grace and then a crop version is just the same thing, but it has, like some other like sequence at the beginning. So it's not copying. It's just like.
177 Huntington: you know, like the end of the
177 Huntington: you just replace the beginning with some other stuff. So
177 Huntington: I also have a diagram for this, but I'll also see it in words as well. So what I'll do is patch the outputs for every attention head from this token in the clean run to
177 Huntington: this token in the corrupt rung.
177 Huntington: and then measure the impact on outputting
177 Huntington: like so weather patching that attention head makes it output grace here. But then also, whether patching that attention head makes it output land later on, after it sees grace.
177 Huntington: So another here's like a diagram version.
177 Huntington: So again, this is like the hypothesis of what's happening that there's some sort of chunk induction head at at QUX,
177 Huntington: that is like attending to the end of Graceland and copying over the concept that entire concept into this representation.
177 Huntington: and so basically, we're just patching over every individual head. And then hopefully, once we'll once we find the chunk induction head when we patch this into here that will like inject the concept of Graceland into here. And so then we can see that by seeing whether, first, st whether this outputs grace, but then also whether this next position outputs land.
177 Huntington: Does that make sense?
177 Huntington: Sorry there is no graceland in the yeah. So there's no graceland here. Yeah. So it has to come from up there. It has to come from up there. Yeah.
177 Huntington: yeah, clean.
177 Huntington: Right?
177 Huntington: Right? It's clean. Right? This one's clean. Right? Okay. But this is this, I mean, this is the same one that you had before. But now but now you're doing for a head.
177 Huntington: It seems like.
177 Huntington: yeah. And I'm also not doing like the capitalization thing. Yeah. So the other one is, yeah. Again, I agree, it's a little weird. I don't trust it. This one, I trust.
177 Huntington: So
177 Huntington: right? So yeah, that's the setup. And so we basically just measure the logic impact here on grace and the logic impact here of land.
177 Huntington: So that's the result that this is like
177 Huntington: the heat map, for all the heads. So like on the left is the impact for grace, and on the right is impact for land. So this is the one that we're really interested in, because this would just be like a regular induction head. And so it wouldn't differentiate between a token induction and like a chunk induction head. But this would be this would only happen if it's like copying like the entire concept. And so you can see I also made it different colors, because they're different scales. So like this is like a lot less
177 Huntington: a a logic change than like this one which makes sense because it's you know, it's
177 Huntington: like a future position.
177 Huntington: Oh.
177 Huntington: but yeah, you can see, like, right away, like this, head sticks out a lot as like a head that has like a really high and well relatively high impact on outputting like land.
177 Huntington: And so if you look at that head. It's actually also the highest chunk scoring head. So like it's a head that has a lot of attention on the end of chunks.
177 Huntington: And so if you do like a correlation between, like the chunk score and
177 Huntington: like this logic difference? You can see like this head is like way up here. It has a really high chunk score, and also like, has an impact on
177 Huntington: promoting like land. And this correlation is kind of like it's I think it's just mostly because of this guy. But you know, you can see, like these other heads that have pretty high trunk scores. They also have like a positive impact on on land.
177 Huntington: sorry. I think I missed that. You talked about logic difference, which exact difference are you taking? Oh, yeah, right. So yeah, the difference is
177 Huntington: between, like the corrupt prompt and the patch prompt
177 Huntington: and like the the left matrix was at the
177 Huntington: token position, T, and the right matrix on T, plus one, yeah, left matrix and right matrix.
177 Huntington: Yeah. And it's just how much patching that head promotes land more than it already was.
177 Huntington: More than a lot of value was before. But just looking at the right plot, we're seeing
177 Huntington: raw logic values and not a logic difference.
177 Huntington: Oh, it is logic, Val. It is a difference. Yeah. I didn't say that it was
177 Huntington: between 2 specific tokens. Sorry media. Yeah, probably it has like 0 0 lot, like, the logic is like 0, or like negative something, because it has no reason to without patching without patching. It had some probability for land, right? Right? Yeah. And then with patching, it has a different one. Yeah. So like this dark green would mean
177 Huntington: that the corrupt prompt.
177 Huntington: Didn't you know like that after you patch in the corrupt prompt. Now, as a like the patch one now has a higher logic value than the corrupt prompted.
177 Huntington: That's that's nice. Yeah, have you tried looking at the exact logic values like, how much 0 point 0 6 signifies?
177 Huntington: It's not much. But I think we are patching one single edge suite. We're not expected.
177 Huntington: And also does it kind of like generalize across examples?
177 Huntington: Or is this had have something very specific for land really likes land. Oh, yeah, also. Yeah. So this is averaged over a bunch of it. I think it was like.
177 Huntington: I don't remember if I did 200 or 1,000 for this. So but I'm just saying Graceland, to make it easy to explain. But it's actually an average. Okay, I should specify it, though it's an average.
177 Huntington: Because that's misleading. What is your interpretation of like on the left matrix?
177 Huntington: There's a lot of green going on at the very late layers would be just
177 Huntington: so. Yeah, I guess in the left matrix. That would just be like regular induction.
177 Huntington: Well, it it would be both. It would be like both types of induction, I guess.
177 Huntington: But mostly like regular or like more like token induction.
177 Huntington: But I guess this does show that it's like later.
177 Huntington: Yeah, the later heads are like copying over, like the next token more.
177 Huntington: Which kind of is inconsistent with what I said before, that like they're kind of all mixed together.
177 Huntington: Yeah. But I mean the the I mean at the end. You need to output a token. Right? Yeah. Yeah. So I think clean one. Those heads might already have the grace value. So when you just patch it, it works
177 Huntington: right? Yeah, that's why you see it at the later later layers. Right?
177 Huntington: So if you go back to your patching, spend yeah here. If you see in the clean one
177 Huntington: the crot at the last at the later layers on the cracks token the model will predict grace.
177 Huntington: It's a great grace.
177 Huntington: sure. So probably the heads at the later layers are encoding grace. So when we just pay paste those heads from that clean to the correct one, the the logic of the grace increases.
177 Huntington: Oh, I I would think they're like the like mover head style was like literally copying the thing.
177 Huntington: Probably.
177 Huntington: Yeah. But one thing go to the next slide.
177 Huntington: Why do you think there's only one
177 Huntington: such a chunk I would expect. There are a lot more. Yeah, that's
177 Huntington: I don't know. Yeah. And I'm wondering, like, so, yeah, I don't know.
177 Huntington: I don't know. I think like the changes are like, really subtle. And so like, maybe these light green ones are actually also important, because there, I don't know if you can see very well, but there are quite a few like light green ones, and
177 Huntington: I don't. Yeah, yeah. How significant do you think the budget. Difference of point O, 8 is since like
177 Huntington: doing your patching experiments on the grace token. Which is
177 Huntington: it? Has it goes up to like point 6.
177 Huntington: Yeah. Yeah, I'm I'm I'm I'm not sure how to interpret the
177 Huntington: differences in scales here. And another sanity check.
177 Huntington: It's like, if you only look at the logic difference of a single token.
177 Huntington: since the probabilities, calculate. Look, looking at the full distribution. If the model is uncertain, it could just like, ramp up all the logic values, and then
177 Huntington: you might, you might.
177 Huntington: It might be better to look at logic differences between like a reference token and this specific land token that you're looking at. So to like catch
177 Huntington: like global shifts.
177 Huntington: yeah, I'd have to think about that. And but I think I think you're.
177 Huntington: I do standard next token that they would
177 Huntington: predict when just like something like, I'm going to or something.
177 Huntington: Yeah, just just have like a reference.
177 Huntington: You you are you talking about the
177 Huntington: like? Lots of different variations for? Or
177 Huntington: I'm not sure about like variation score, but like in general, like, just like looking at a single logic change. I thought, that's what we're looking at.
177 Huntington: Okay, yeah, wait. Yeah. So I might ask you about this after, because I I'm I'm not super like. I also didn't know whether to use logics or probabilities which are like they look kind of the same. But
177 Huntington: yeah, I don't know there's there's that kind of stuff that I'm not sure if it's I'm doing it this way. So
177 Huntington: yeah, but it might. Yeah, that might be better to make it
177 Huntington: like discourse, make a little more sense.
177 Huntington: alright, yeah. But then, also as a baseline, I also just did so. Instead of having an entity like Graceland. Here I just did like. The entire thing is all random tokens
177 Huntington: and did like the same patching experiment.
177 Huntington: And so that's like what the results look like.
177 Huntington: You can ignore. I don't know if that note makes much sense. But yeah, like, I think, so yeah, this is just to get like heads that like always promote like the next next token. And you can see that, like the logic, difference is smaller, like the highest. Here is like
177 Huntington: point 1 point, oh, one, whereas like on the right here, it's like point O, 8
177 Huntington: so basically, then I don't know if this makes any sense. So this is already difference. But then I also did like a difference of differences as well. So we have, like all of the
177 Huntington: like at the amount that the head increases land for counterfac entities, minus the average amount that it promotes the next next token like in general, and so then that gives like a ranking of like heads that especially like to promote the next next token for counterfac entities. And so this is, I'm going to use this set of heads to do something in the next experiment. That I think is kind of interesting.
177 Huntington: but I don't know if this is the most sound thing. But at least, when I do this I can do something cool.
177 Huntington: because both of these averages are taken on different data sets. Right? Yes, yeah, they have to be
177 Huntington: so.
177 Huntington: So 1 1 set of heads is just looking ahead, regardless of meaning. Yeah. And the other set of heads is looking ahead, especially when it's meaningful. Yeah, especially for counterfaced entities.
177 Huntington: Yeah, wait. So the last plot it's likely.
177 Huntington: what is that, citizen? Those heads which we see there. Those are promoting the next next token, or is it the ones which are
177 Huntington: like the character, the the chunk heads that you're looking for?
177 Huntington: So those are so. These are what I think are the chunk heads, because they
177 Huntington: seemingly seem to promote the next next token, just for counterfac entities
177 Huntington: like or like, more for counterfac entities than they do. For like regular, just regular, like, random tokens.
177 Huntington: Yeah, okay, that makes sense. So. But but you are doing subtraction here, for this is
177 Huntington: the one, the the left hand side one is the one that you want.
177 Huntington: The middle one is what you don't want. And then when you get the final result, and most of them are below 0. Yeah. So a lot of these heads, I guess
177 Huntington: yeah, below 0. But there are some heads that are like green. And so I'm gonna take those
177 Huntington: and try to use. Okay, I don't see much of a green there. Yeah, you can't see it. But they are there. Okay, there's a couple.
177 Huntington: Yeah, okay.
177 Huntington: looking at the scales of this block. It's like point O, 8.0 1. And then in the difference, it goes up to
177 Huntington: point 4.
177 Huntington: Yeah. I was wondering if I've like made some kind of mistake with this. I'm not sure.
177 Huntington: And the other thing is, I, you do like patching across your data set for each head at a time. Right?
177 Huntington: So that would be 32 times 32
177 Huntington: times the number of examples that you have been data set.
177 Huntington: That's very time consuming. Yeah. But we can use Dcm thing. Learn a mask over the attention heads that would be way more quicker, and
177 Huntington: you make so one of the problems with patching is one head is if there are group of heads which are working together to do something. It's hard to find that with just a single when you're just patching in
177 Huntington: one head at a time. But when you learn a differential mask over all potential heads.
177 Huntington: the chances of getting the those group of heads together is higher.
177 Huntington: Yeah. So that's also advantage.
177 Huntington: Yeah, your your previously super important head is gone.
177 Huntington: Didn't notice before. Oh, no, it's still there. It's green. It's 1 of the few green heads. Okay, so it survived. Yeah.
177 Huntington: it says that. 14 one, yeah. Just go to your web. Browser type. Dcm.
177 Huntington: I literally did that.bowap.info. Okay, dcm.bowap.info. These guys.
177 Huntington: we need a we need a web page that lists all of the project pages. And maybe we should create that thing?
177 Huntington: I haven't changed.
177 Huntington: Did it work issue? What do you compare that like with a simple, like greedy algorithm of like
177 Huntington: taking the most impactful head. And now condition that this head is there, what is the next most impactful head?
177 Huntington: Adding that what is the next most impactful had condition that those 2 heads are there like, didn't
177 Huntington: epsilon video algorithm from like submodel asset function optimization?
177 Huntington: I'm not sure we can discuss about that later, I guess. Oh, there you're done.
177 Huntington: I have one more thing that I I kind of buried the lead, because this is the experiment that I was the most excited about, but I wanted to do all the other stuff to explain why I was.
177 Huntington: yeah, what it? What I'm doing?
177 Huntington: So yeah, okay. So now, I did this kind of jank thing, which I'm still not sure if
177 Huntington: makes sense. But we have like a bunch of heads that I'm going to call like the counterfac promoter heads, which promote, like the next next token especially like to promote the next next token for counterfaced entities. And so my hypothesis is that these are actually concept induction heads. And so if if that's the case, then they should be important, for, like other tasks that involve copying concepts like like translation. So there's like 3 types of icl tasks that I wanted to test.
177 Huntington: So
177 Huntington: basically, I set up 3 tasks. So there's a translation task for words. So from French words to English words.
177 Huntington: and then
177 Huntington: the same prompt setup, but copying English words to English words, and then the same like template, but copying nonsense to random tokens, to random tokens.
177 Huntington: And so
177 Huntington: yeah, so the translation wouldn't use token induction heads. But it would use the concept induction heads and then the nonsense would use the tokens. But it wouldn't use the concept induction. And so then copying English words would use like both of them, because you can copy it like with the meaning, or you can copy it with like the individual tokens, because it's the same thing. So here's like examples of the prompts.
177 Huntington: Yeah, so like translation. I just use the same data set that Eric used with his function vectors, stuff of just a bunch of words from in English and French. So I like, this is all one prompt. I just like looped it, because otherwise it's too long.
177 Huntington: So yeah, you have, like a list of in French words. And then a list of the English translations. And the model has to complete, like the translation for this word
177 Huntington: But then copying is the same template, but it's just the exact same. Oh.
177 Huntington: sorry this is this. This is, I didn't actually change this. This should be, yeah.
177 Huntington: But you know, you know what I mean. Yeah, there should be this.
177 Huntington: Yeah, that's what copying looks like.
177 Huntington: Cool.
177 Huntington: instantly fixed so. But now to be clear, this is not the icl setup right. This is the copying setup. So you go down the 1st column. Yeah, and then go up to down the second. Yeah. So what it really looks like, what it really looks like is this.
177 Huntington: But I just couldn't fit it on the slide. So I just put it like this.
177 Huntington: Right? And then nonsense is just like a bunch of randomly sampled like tokens.
177 Huntington: And so yeah, so I'm gonna have, like a bunch of
177 Huntington: 2 sets of heads they'll ablate. So one set of heads is these counterfac promoter heads. And then another set of heads is just the heads that
177 Huntington: these guys, the law on the left plot that promote or actually not these the heads that promote
177 Huntington: the next token for random tokens. So just like classic like token induction heads. So that's like the 2 sets of heads that I'm ablating
177 Huntington: for all of these tasks.
177 Huntington: And so this is what it looks like.
177 Huntington: So for this, this plot is so on. The x-axis is the top K like counterfac promoter heads, which I'm hypothesizing are like the concept induction heads. And so you can see. And then, yeah, you can see that if I have, like the top 128 concept. Counterfac promoter heads that, like destroys the translation, accuracy relative to what it was originally a lot more than it destroys the
177 Huntington: accuracy. For, like copying in English words and accuracy for copying nonsense,
177 Huntington: does that make sense as yeah. The blue line goes down faster. Basically, what are those numbers? Yeah. And so this number. And so like, nonsense accuracy.
177 Huntington: Yeah, so this this plot is just to make it easier to read, I'm put percent of the original performance percent of the model's original performance. So the model only has, like a 53% accuracy for this translation task. But I just normalized it. So that.
177 Huntington: like this is the percent of its original of its original accuracy. So you can compare it. Obviously. So this. This is just showing what the original accuracies were.
177 Huntington: And so yeah, you can see that like ablating these like counter frag, promoter heads
177 Huntington: makes the blue line go down faster. But then on for the exact same tasks. If you ablate the top k regular like token induction heads, then you can see that the model stops being able to copy the nonsense tokens as well. But it's still able to do the translation task and the
177 Huntington: English copying task. And so
177 Huntington: yeah. And so to me that that pretty much shows that like
177 Huntington: these like counterfac promoter heads, at least, whatever is in the top 128 of them. Some set in the top 128 of them are like, actually also important for translation.
177 Huntington: and the induction heads are important, for, like the verbatim copying but that the English task, if you're copying English word to the exact same English word that actually uses both of them.
177 Huntington: Yeah, does that make sense?
177 Huntington: That's a cool experiment.
177 Huntington: Looks like you have some signal here.
177 Huntington: somewhat signal. Yeah, it's yeah. It does. It is actually a signal, I think.
177 Huntington: Oh, yeah, that this is just me saying the same thing again.
177 Huntington: yeah. So if you ablate the concept induction heads. It stops being able to do translation, but it can still do these guys.
177 Huntington: Oh, I didn't fix it. But then, if you ablate the token duck and heads. It stops being able to copy nonsense. But it can still do those guys.
177 Huntington: So that's that's my last thing.
177 Huntington: Be
177 Huntington: so in in your last slide.
177 Huntington: When you destroy the induction heads.
177 Huntington: It is still able to copy English words.
177 Huntington: yeah. Or if you destroy the concept induction heads, it can still copy English words, because the tokens are the exact same
177 Huntington: still. Yeah, that's fine. But when you destroy induction heads, when you destroy the token, induction heads.
177 Huntington: The model can still copy the English words because it can copy the meaning of those words
177 Huntington: right? And knows what? Those words are. Okay. Yeah, yep.
177 Huntington: But yeah, so this is what I have. And I would love if people have, you know, if they want to talk about like feedback and ideas. So you said, is there an overlap between token induction heads and concept induction heads?
177 Huntington: yeah. So from the attention patterns like that, I was showing like when I was looking at individual heads, there were heads that did both.
177 Huntington: but also the fact that you can do this ablation means
177 Huntington: that I guess there is like separate enough.
177 Huntington: But I'm kind of curious about that. Yeah, up.
177 Huntington: So a hundred heads.
177 Huntington: So it's a signal. It might be pretty distributed among heads. It's kind of interesting. Yeah, but that is a very naive way of taking 100 heads definitely. Yes, you just sort them according to the score. It was nice.
177 Huntington: So you don't take synergies in between the heads into account.
177 Huntington: You've picked the 100 individually most impactful heads.
177 Huntington: Yeah. So that taking more of the synergies into account.
177 Huntington: Might. So yeah, change how fast these lines dropped.
177 Huntington: Like, yeah, I agree. I think there's probably it would be cool to find a better way to find exactly which heads like are
177 Huntington: cause. Yeah, right now, when it is imprecise to just take like the top.
177 Huntington: Maybe you could
177 Huntington: take Jack Maruto's method and see which heads are talking with each other and ablate them like, or or Dcm. Would work right.
177 Huntington: But do you have like a hundred green heads green. So when you showed those this manage, this equals to this.
177 Huntington: And I think that's how the resultant figure that you there, you're selecting the greens. One. There you have hundreds.
177 Huntington: Yeah. So I'm I'm guessing
177 Huntington: that the top 100 is going to be the ones that are green and white right?
177 Huntington: Because I guess white means that it's it also likes to
177 Huntington: output the next next token in general.
177 Huntington: right? So it's still like a next next token outputter. But it's not specifically for counteract entities. Yeah.
177 Huntington: So maybe if you block the top 10, there are others appearing.
177 Huntington: Oh, yeah, that's true, like the ones which you wouldn't find your 1st one.
177 Huntington: If you block the top 10 and then do the same experiment again. Maybe you'll see like new ones popping up.
177 Huntington: Yeah, to be more complex.
177 Huntington: I don't like this slide. So I'm gonna go over here.
177 Huntington: So good.
177 Huntington: So do you. Do you think that the
177 Huntington: you know you mentioned this? You know synergies between heads. Do you think that this single do we do a lot of experiments where we do single head scans
177 Huntington: to, you know, to find things in logistic media sets.
177 Huntington: But do you think that it's possible that the reason is scenarios week is because
177 Huntington: maybe the way meetings brought over is
177 Huntington: from heads that are only triggered when a previous layer head has noticed that there's meaning like maybe one head at a layer, comes over and says,
177 Huntington: there's some meaning here you have to pay attention to. And then at an at at a later layer that says, well, in that case
177 Huntington: I better go back and look carefully at what the meeting was.
177 Huntington: whereas if you just test that second layer by itself.
177 Huntington: it's never triggered because nobody told it
177 Huntington: that there there was a candidate meaning thing to look for, right? So I feel like there's.
177 Huntington: you know, there's this possibility that because we like. When we look at these other circuits and
177 Huntington: huge detail, we see these, you know, secondary triggers
177 Huntington: that are happening, that that would that that you might not see if you're looking at the heads on their own right.
177 Huntington: It! I would guess there's some kind of like
177 Huntington: for entities. There's probably some like binding happening right? That's like a signal of like. All these tokens are related to like to one entity, whereas, like nonsense tokens like you don't get the same kind of binding like the model wouldn't pair them together. Naturally, right? Simpler. Yeah. And I would guess that probably happens earlier than these
177 Huntington: right? Right?
177 Huntington: Right?
177 Huntington: Maybe one thing we can also do is the it's true
177 Huntington: experiment on the data sets for discovering this template had, like concept induction versus organization. Let us have our
177 Huntington: discovered based on like some data, says, maybe some synthetic data. But but if you use another data set, would you like also
177 Huntington: discover the same hat, or it's totally different
177 Huntington: kind of like robustness chat. And I think this would be useful.
177 Huntington: Yeah, it would be useful to do this thing because I use counterfax entities for this. I think it'd be useful to do it with like multi token words and stuff like that as well.
177 Huntington: Yeah, let's called, you use the downstream data sets as well the translation copy translation.
177 Huntington: English copy and nonsense copy.
177 Huntington: If you could also use those 2 icons and loss function for their
177 Huntington: yeah, that's what I was, I think, like using other other tasks as well could be like basically trying to score this in another way.
177 Huntington: Oh, yeah.
177 Huntington: Do you know how many totals are? Total heads? Are there? A 1,000, 24,024,
177 Huntington: yeah.
177 Huntington: And there has to be like some overlap between these heads, probably in your top? 20 to 26.
177 Huntington: Okay?
177 Huntington: And do you think like, where do you think this overlap would start between induction and your concept actually
177 Huntington: overlap between heads.
177 Huntington: like, I'm guessing, maybe not. Top 50, not top 100 right? At some point. It would start that. Yeah. Well, I guess
177 Huntington: like, it would start, yeah, I guess it starts to destroy the other types like around top 256,
177 Huntington: like if you're for the left plot.
177 Huntington: But I also again wonder if the same like one single head can do both things.
177 Huntington: Which I haven't really like this kind of assumes that each there's each head is either one or the other right? But, like, if you're going to talk to 56, you said you might go into the white territory. Oh, yeah. So there could be
177 Huntington: overlaps. Yeah, right? Right like, especially if you're blocking the top 10,
177 Huntington: maybe the new ones that become token inductions are the previously concept induction, which is also taking this additional charge of doing right because it's so interesting that after, like 200 heads or so, everything drops to 0, and the model is like, I don't know what to do now.
177 Huntington: So I wonder like, if it's because of concept ones, or like the token ones, or maybe mix of both.
177 Huntington: yeah, so so you're saying, maybe the heads that do both start to do chunks for what the the main chunk heads are.
177 Huntington: Yeah, like, all of this drop is probably happening because you removed all participants. They don't exist anymore.
177 Huntington: It's just a different dimensions.
177 Huntington: See? Why is the drop going to 0?
177 Huntington: Yes.
177 Huntington: I see. So maybe there are concept heads hidden in this induction head set.
177 Huntington: And either one of those are like, really important, for the board would work.
177 Huntington: I mean, it'll be cool if
177 Huntington: you can say, yeah, it's just the concentrates. If you remove all of them. There's no way the model works.
177 Huntington: But if you remove all the token hands screen type of books.
177 Huntington: One thing, too, is, I wonder? Maybe, instead of looking at heads
177 Huntington: like cause, I kind of imagined that there's like
177 Huntington: maybe 3 or 2 previous token subspaces if that makes sense.
177 Huntington: And so, maybe, if you intervene on the the subspace that the heads reading from, instead of ablating the head, you know what I mean.
177 Huntington: yeah, that's what Jayden does for our stable decision. Once messes with the input software.
177 Huntington: yeah, that should be the best way to like target ahead to like affect it.
177 Huntington: So you're you're not looking when you ablate your heads right now. So this is an ablation experiment. It's not a patching experiment. Right? Yeah, ablation. Yeah. And and but you're not. You don't care what that head is attending to. You're just sort of going
177 Huntington: overall has, regardless of where it's getting information.
177 Huntington: Do you think that?
177 Huntington: so I I yeah, I like that, because, you know you, you have this hypothesis that should be attending to the a certain token, but
177 Huntington: you know it would make sense for it to have alternate paths, to be able to attend to other tokens.
177 Huntington: And maybe it shouldn't matter.
177 Huntington: Do do you think that this experiment also
177 Huntington: works for identifying heads as a patching experiment like you'd be able to sort them
177 Huntington: through a similar patching setup instead of an ablation setup like, use the translation stuff.
177 Huntington: or or some sort of patch, right? So where? So here, with the ablation you're always looking for.
177 Huntington: Oh, I've mixed up all your experiments. So when you identify the heads, is it from ablation, or is it from patching? Oh, identifying the heads is patching. Right? Okay? So that's okay. So that's actually, I was asking about the thing you already did. Okay.
177 Huntington: good. Yeah.
177 Huntington: Be. This, is it a regular practice to compare larger differences, or is it?
177 Huntington: I don't know if it makes sense to compare the logic. Yeah. Probabilities. That's what Byron is complaining about as well. And I agree like, I don't really know.
177 Huntington: It just makes sense hoping more signal on the logic, you know. Yeah, that's true.
177 Huntington: But then you're comparing against different runs. I wonder how much of a sense it makes
177 Huntington: like. I think that's what John was saying in the beginning, where originally the model is not so confident across all the tokens. Maybe it'll boost one of the tokens really high
177 Huntington: or like it knows the answer for sure that it might not boost it as much.
177 Huntington: Oh.
177 Huntington: yeah, to get a more stable baseline over different runs. Maybe you can choose a specific reference tokens, token like, so maybe it makes sense in this like super paired settings.
177 Huntington: You know where this is, where this is, where this comes from.
177 Huntington: Yeah. And maybe like in this setting. When you have grace and land, you can just like, look at the highest token probability when you just prompt the model on grace, and take whatever token comes next, and then you always like
177 Huntington: kind of like, have the difference of differences. It's always like your reference versus land
177 Huntington: in the patch and clean setting.
177 Huntington: Oh, I see what you're saying, so it's a difference of differences before I do the other difference just scale differently. Your probability is fine, too, because then you don't need to worry about
177 Huntington: like it's already normal. Normal. I said that
177 Huntington: like, if you're worried about the logic is not being normalized to some baseline. Just take the probability, and then easy to read on a plot, anyway.
177 Huntington: if we could like. You don't know what? 6. What does the logic of 6 mean? We don't know.
177 Huntington: you know, and the probabilities do actually like, I didn't cherry pick like the probabilities do actually look kind of yeah to this. I just use logic, because people say, to use logic, but I yeah, I don't know.
177 Huntington: Push it better. April. Everything will be clear on the use case.
177 Huntington: Yeah.
177 Huntington: Oh.
177 Huntington: so in your cleaner setup, when you patch the the Graceland thing into the one which doesn't mention Graceland ever.
177 Huntington: it's almost as if the model is fighting between its factual recall and its
177 Huntington: and the patching that you do.
177 Huntington: Hmm.
177 Huntington: fighting between factual recall. Yeah, you say, can you go to the example. Yeah, when I worked at something, Grace, right? Oh, yeah. So it's probably like looking at. Oh, I know this answer
177 Huntington: like some fake fact which doesn't exist. But the model thinks it knows it. Yeah.
177 Huntington: I think actually, sorry. Again, this is a problem with, my, yeah, this is random. This is a problem with my slides.
177 Huntington: Yeah, I should. I should clean up my slides. But I did do this a little bit.
177 Huntington: Okay, yeah, thank you. Thank you guys.
177 Huntington: it's perfect.
177 Huntington: Alright we have. We're hit next at 12. So.