177 Huntington: Convinced my son is having this project week, where he's supposed to be hanging out in my lab and seeing what it's like, and helping out with a project, and he's he's studiously avoiding coming to this talk. So I'm gonna see if I can have him roll in and listen to your talk. So give me a second. It's just personal issue.
177 Huntington: And we need to roll in a chair, too. Yeah, yeah, we'll roll in a chair.
177 Huntington: If not, this chair is technically still moving.
177 Huntington: If you guys have more chance. Just good.
177 Huntington: Okay, thanks. Thanks.
177 Huntington: So, okay, so we're recording here. And I want you should go ahead and project your slides, Patrick, and you can turn around to the camera and say, Hello.
177 Huntington: yeah. The camera is right behind you.
177 Huntington: Hello! Hello! Anybody who might be remote. I think that we have you, sue remote.
177 Huntington: So so welcome to Patrick second year. Phd. Student
177 Huntington: who I know that, you know. So I'm not sure what work you're gonna talk about. But you know, you you had this very interesting work
177 Huntington: this sort of skeptical. You know, quantification of
177 Huntington: whether essays are getting interesting things or not. And so that might be what you're talking about today. Is that right? Excellent, excellent. So we're we're looking forward to it. I think some interesting insights here.
177 Huntington: and so take it away, Patrick cool. I think if I maximize this. It's going to vanish from the screen. So oh, wait, will it release what happened?
177 Huntington: Oh, wait. Oh, amazing. Okay, that's great. Okay. So I guess the the overarching theme of my work has been finding the units of analysis for mechanistic and touchability.
177 Huntington: And I guess quickly. I mean, you all know what back into this. So I just get the slide. But basically, how do you figure out what Llms are doing internally?
177 Huntington: and one of the key problems of this is finding the the proper units of adaptability to use for this
177 Huntington: So originally, people were looking at neurons which turned out to be quite coarse and polyspeatic, and we had the same problem with heads, language models,
177 Huntington: and they seem like quite complex complex objects to actually interpret.
177 Huntington: So then, and one of the reasons for this is that superposition may lead to polysponicity where your neurons represent multiple unrelated concepts
177 Huntington: and sparse autoencoders were proposed as
177 Huntington: a way of kind of undoing that superposition potentially undoing the polysamaticity of activations.
177 Huntington: I went through that pretty quickly. So so yeah, so anybody want to discuss that before we go on
177 Huntington: this this whole agenda.
177 Huntington: Boom.
177 Huntington: Yeah, the polysematic
177 Huntington: hypothesis that the hypothesis, as far as auto inputers might be a way of reversing this.
177 Huntington: we can discuss it in more detail. Yeah. So sparse autoencoders is from dictionary learning and dictionary learning is like, Oh, let's try to get signals from some.
177 Huntington: Let's decompose inputs that we have. Suppose auto encoders like one specific
177 Huntington: approach. You often have, like linear encoders and decoders.
177 Huntington: What do you think of the linear representation hypothesis are really linear representations. The things that we expect in
177 Huntington: it's followed.
177 Huntington: yeah, I guess that transformers are like, kind of linear seems pretty plausible that all, I guess representations are structured in a linear manner, and linearity is the best way of interpreting the structures is less clear to me.
177 Huntington: But you can clearly build nonlinear structure in the activations of language models. Given that they, this has been observed.
177 Huntington: So it's possible that, like essays are not the best way of interpreting this kind of activation space.
177 Huntington: Does that make sense?
177 Huntington: So maybe based on like experiments? We found many
177 Huntington: yeah directions that makes sense that are semantically meaningful in essays or like, yeah, yeah, so this needs to like semantically meaningful nonlinear structure in language models. But then there's also like, obviously linear stuff, too.
177 Huntington: So myself would be like
177 Huntington: lava seems to use helices to do arithmetic which is like a nonlinear structure. But then other things are like included as binary variables, which I think makes sense to a model as linear.
177 Huntington: Be sure.
177 Huntington: do we think that sparsity loss is a good enough proxy for interpretability. No, it seems pretty bad, doesn't it? I think it looks like it kind of works for some things, and it's really cheap. So
177 Huntington: also, like, for the people who have worked with essays, how many of the features are like a percentage? How many of the features are actually interpretable, and most of them are like
177 Huntington: little bit noisy, are very Nigel
177 Huntington: interpretable. How I think, like like you, pass some activation, and you see that
177 Huntington: it it has like one more in the distance. Yeah.
177 Huntington: And I'd say, like most of them look pretty good when you just eyeball them on activations. I think John probably can say more about this in the Seoc bench results, and most of them look pretty good most of the time.
177 Huntington: So we have one
177 Huntington: benchmark which is like automated interpretability. And then we take like a subset ses have like a width of like, maybe 64 K.
177 Huntington: You want want to have, like at least like 5 to 10 x from
177 Huntington: pure model, dense model activation space.
177 Huntington: And then we just like, take a representative sample of like a thousand features.
177 Huntington: And
177 Huntington: and you right out of these are, there's like 70% that we get at least some interpretability signal when asking Claude Haiku. Is this some concept?
177 Huntington: He's not talking about interventions now. Yeah, we had a sae project, and we did it on stable diffusion. Turbo.
177 Huntington: So we
177 Huntington: I think we could take the time to do it a bit more slowly, because nobody did anything like that there.
177 Huntington: And like we observed, if we have just a large number of se features, they still are interpretable, like they latch on to interpretable things.
177 Huntington: but we cannot like turn the bar feature on or off
177 Huntington: to get a picture of a borrower.
177 Huntington: But when we reduce the expansion factor to like 2
177 Huntington: like from, we started with language model settings where they use 64 when we chose 2.
177 Huntington: Then we suddenly could turn on and off a bar feature, and it would do something.
177 Huntington: So when you have very cause features level, I guess. Like,
177 Huntington: yeah, it's a trade off. Basically, if you have, like many features, you obviously get feature splitting and you cannot do anything about it.
177 Huntington: and then maybe they are not causally affecting the forward pass
177 Huntington: anymore, as you would expect it.
177 Huntington: Yeah, I think like so for me, the surprising thing is like I did a Phd in like, basically
177 Huntington: computing sparse representations with respect to fixed basis changes.
177 Huntington: So in the sae, this is kind of learned of, and my impression was like.
177 Huntington: these methods are really bad. Dictionary learning is really really bad.
177 Huntington: And so it's kind of surprising that, like for this use case? It seems to be fine.
177 Huntington: Does it seem to be fine? I think sparsity is more like a wishful thinking than anything else.
177 Huntington: It's just one simple simplifying assumption that we actually can implement.
177 Huntington: That doesn't mean that this is how networks represent things.
177 Huntington: Yeah, that is just important to keep in mind.
177 Huntington: Oh, what do they want?
177 Huntington: That's good.
177 Huntington: Wanted some, yes, but actually wanted to make a different point. But it will come later. Okay? One question I had is a naive question, is
177 Huntington: the idea of like superposition is like, you have.
177 Huntington: like orthogonal. This is great. Why don't we so because we have a mixed audience? Right? So we have some people who have, you know, done a bunch of published work in essayes.
177 Huntington: And then we have some people who are visitors who have not done any interpretability research at all. What? Yeah, can you go back and just explain what is this superposition business at all.
177 Huntington: Yeah, so, superposition is basically when you have sparsely activating features you can compress more features into fewer dimensions. Basically. So you have, like 5 features that are sparsely active and represent them using 2
177 Huntington: neurons in a like single little layer in port encoder.
177 Huntington: So yeah, my question with that was, if it's it's sort of like you have a vector which has multi directions like it's, it's like a sum of directions. In a sense like the angles, shift as little sub components.
177 Huntington: Nudget is an sae sort of like decompose direction, or is that a poor abstraction, as
177 Huntington: like giving principal directions for sub components of the representation that is being blown up.
177 Huntington: Yeah, yeah, that's how I think about it.
177 Huntington: So I haven't seen like, so that this toy models of superposition. Blog.
177 Huntington: Is there a paper that explicitly applies
177 Huntington: as some kind of smart autoencoder on this toy setting to explicitly reverse it?
177 Huntington: There's a whole tutorial on arena that walks you through this. They do some serious intuition building and an amazing work in teaching this stuff
177 Huntington: and and does the. And so I guess my my technical question is.
177 Huntington: you know, the setup that everybody uses with this sparsity loss
177 Huntington: like, does it work well in this toy setting? Does it recover? The
177 Huntington: does it require? Does it recover the hidden latents that you know
177 Huntington: that have? Does it actually work? In that case? Yeah, yeah, yeah.
177 Huntington: I don't know if it works every time, but
177 Huntington: I think like 2 things that are really good about the sae or the superposition story are like, first, st like this combinatorial argument with.
177 Huntington: there are simply more features in the world than coordinates in your D model vector like in your intermediate states.
177 Huntington: So certainly a single component cannot correspond to a single feature.
177 Huntington: That is a good argument.
177 Huntington: And then the other argument that I or
177 Huntington: why I think linearity makes sense in transformers. Is that like
177 Huntington: how you compute keys, queries, values is all linear.
177 Huntington: So you keep having this linear what we make this in your computation.
177 Huntington: So if you want to get good key queries values, it's better. This stuff needs to be read out linearly
177 Huntington: right at this, from the perspective of the attention layer.
177 Huntington: I think that should somehow promote linearity.
177 Huntington: That's all. This is also more like a wishy-washy situation. Yeah, yeah.
177 Huntington: Does that sort of align with your intuition, Michael, or is there a different question that you had about this? It's like, when I think about a vector, and I think about like
177 Huntington: directions. Directions could be subsets of the like, it could be a subspace that has a direction. And when you blow it up into a very large vector space, and you have a now subspace of that large vector space
177 Huntington: which has a more like broken apart direction. It feels unclear to me how that could then be tied back to the original smaller, vector, which is probably where my mental model is incorrect with it. So I imagine the smaller case, the superposition part is like things are imposed on each other in a way that you can't actually like.
177 Huntington: Piece them apart. And then in the large dimension space, you're probably primarily working from there, but I was wondering if there was like a back chaining thing you could do from the large dimension space to modify the low dimension space in any way.
177 Huntington: I don't really understand. What do you mean? So like so but does, does. So is it is what you're just discussing like in contradiction to this idea that
177 Huntington: you might have 3 independent features that you're trying to encode in 2 dimensions.
177 Huntington: It's not a contradiction, I see. I guess it's like, can you modify? New? Okay? Now, I think I'm you need to think on this bit more.
177 Huntington: Yeah, I think it makes sense in that regards. Yeah, it.
177 Huntington: Can you remove? Then, if there's 2? Say, there's 2 features in a low dimension space or 3 and 2 dimensions, let's say, could you smush it down, reduce it to the 2 principal ones, and get rid of the 3.rd
177 Huntington: I suppose it depends on the spacity
177 Huntington: at a certain level. It's supposed to yeah, be good, but it's counterintuitive. So if there is no sparsity.
177 Huntington: and these are 3 dense features.
177 Huntington: then the optimal solution is to do the Pca solution
177 Huntington: and and keep the keep the most significant ones, because you only have 2 dimensions you can represent. And so and you have capacity to do 2 independent things, and that's what you're going to do. So the counterintuitive thing is, if you have really high sparsity
177 Huntington: like these 3 things are all really rare.
177 Huntington: Right, then it's the wrong solution to throw away the rarest one.
177 Huntington: Right? The right solution is to overload the rarest one. You know, in this vector space.
177 Huntington: because it's really unlikely that what you have 2 of these things at the same time.
177 Huntington: And and so if you see the overloaded direction.
177 Huntington: then it's much more likely that it's that 3rd thing than it is like some
177 Huntington: bizarre coincidence of the 2 unlikely things. So like that's like it turns out that's like the efficient coding. Does that make sense? Yeah, right? Yeah.
177 Huntington: right? Say, huh, absolutely. Great example. Professional teacher, right? Okay.
177 Huntington: yeah. So so just just thinking about that like do that? Do people think about like using essays to learn about co-occurrence of data, or something like that like like these, things would like never appear together, so we should impose them on each other, or something like that.
177 Huntington: I don't know if you think about this.
177 Huntington: Thanks.
177 Huntington: Okay, cool, cool. Yeah. Let's go on.
177 Huntington: One second.
177 Huntington: Yeah. So essays apply to Llms, seemingly give you these nice, interpretable features. For example, this one activates on DNA sequences, and the more it looks like DNA sequence, the more strongly this essay feature activates.
177 Huntington: There's a type here in that 3rd bullet point. It should say nice and monosemantic but it seems like a response, like what kind of human logical feature the data?
177 Huntington: Now, whether you can like steer on that or program that or whatever
177 Huntington: it's a different question. But for now they just they look good.
177 Huntington: And then one thing that seems to happen as you, you increase the size of the sparse autoencoder. Is that your your more course, features, and smallest sparse auto encoders split into more features in larger ones, so you probably can't read this. But basically, this is like a high level feature that responds to mathematical notation in the text
177 Huntington: in a small essay, and then in larger essays. This seems to split into multiple features that respond in subsets of mathematical notation in text, so sub features of mathematical notation.
177 Huntington: And then this one is like the in mathematical prose that splits into like more specific uses of the maths related text.
177 Huntington: Was this like just manually like found, and like you made it in this empirical or like. How did I mean, how was this
177 Huntington: splitting feature found? So these were found using co-activation, all features in the 2 essays.
177 Huntington: This was found using the oh activation by how frequently they go activation.
177 Huntington: Okay, so could you describe that a little bit? Yeah. So 2 features like these 2 top, 2 features. If this one activating implies that this one activates, then it's likely that it's a sub feature or feature. So you have like 3 columns right? And these are sparse autoencoders with different expansion factors. So the one with the small expansion factor
177 Huntington: has you. Only we only look at 2 features, and now we go like one expansion factor up.
177 Huntington: since for these 2 features that you learned previously, now you learn like I don't know. 7.
177 Huntington: Oh, shoot, did I record?
177 Huntington: Yeah, yeah, it's a question. So you look for this feature. And with the higher dimensions, instead of finding like just 2 features you find like 8 or 7, or whatever features there is. Yeah. So when this feature is active,
177 Huntington: some of these features will be out of in the larger model.
177 Huntington: and when one of these features is active, this one will be active for the smaller model
177 Huntington: that makes sense. How did you find. Sorry.
177 Huntington: I just want to ask how you control for the locality, because, like, I mean in these contexts, you see, it flare up very specifically. But it's possible that there's another superposition where the in the machine learning also shows up in about like synthetic biology or something, I guess. Is there any like methods, for like probing in the external neighborhood of the what's hypothesized to be to just verify that it doesn't go out? Or really more, how do you actually like determine? Oh, it's very specifically this dog versus that versus yeah.
177 Huntington: so she's using autumn tub or human turb. So collecting new examples in which it activates, then.
177 Huntington: coming up with a thesis or for the future have sense. Nice?
177 Huntington: That would be any more questions on the trip.
177 Huntington: So yeah, the 1st thing we kind of wanted to evaluate was
177 Huntington: The hope was an issue with essays, and I think Chris would have been quite hard on this. So my essays find the features that you exactly need to, or mechanistic interpretability work,
177 Huntington: and the like economical atomic set of features.
177 Huntington: So we wanted to look at this more because obviously, you have feature splitting, which kind of like
177 Huntington: is evidence against this idea that at different dictionary sizes you get different types of feature, different granularities, feature.
177 Huntington: And so we just started by looking at what happens when you scale the dictionary of your sponsor encoder to train like 10 different esses of different expansion factors.
177 Huntington: And we had a theory that these feature splits happened so you could decrease the L. 0. These are value saes that was past you lost as opposed to top K ses.
177 Huntington: So when you split your features.
177 Huntington: you get features that are less frequently active and course, and respond to more specific
177 Huntington: features. So you don't need like if you want to do something niche like
177 Huntington: One of these in the this one in the smallest essa, you might need 2 features to represent that single feature in the larger essay, whereas in this one you only need a single feature. So your sparsity, you have a higher sparsity and the larger essay for this feature.
177 Huntington: and then alongside that
177 Huntington: you seem to get new features in larger, sparse auto encoders. So, for example, in this one, as you scale the Se, you start seeing latent sort of features that respond to specific elements of the periodic table in the text. So in a small essay, you might get like oxygen and nitrogen being very common as you scale the sparse order encoder, you get a feature that responds strictly to plutonium, something a bit more rare.
177 Huntington: And how we started looking at this was using what we called sae switching, which is where you switch features between essays of different sizes.
177 Huntington: So what that would mean is, for example,
177 Huntington: if you switch all of these features for that feature. What happens to the the 0 and sparsity and the
177 Huntington: reconstruction performance, or sparse encoder? Does it go up? Does it go down?
177 Huntington: So we thought that when you take split features from a larger se, so one of the right hand columns of this diagram and replace a unsplit feature in a smaller Se.
177 Huntington: This would not affect the Mse because they're strictly reducing the 0 loss.
177 Huntington: And it would reduce the of the small model. So this small model is now like small model, minus one latent small model one link, one feature from the small model plus 5 features from the bigger model that just been inserted in place that one
177 Huntington: feature.
177 Huntington: and then on the flip side, if you add what we're calling novel features which are not present in smaller Se. If you add a novel feature from the large Se. To the smaller Se. Your Mc. Goes down, but your L 0 isn't affected, or maybe goes up a little bit.
177 Huntington: So that makes sense what we're what we're testing here.
177 Huntington: So when you say switching, which way. Which way do you mean? Do you mean
177 Huntington: reducing? So when you okay, so like, you say, it doesn't affect the Msc. But reduces L, 0,
177 Huntington: is that from moving
177 Huntington: the split features into this unsplit ones, or they're moving the unsplit ones to replace this, but which one? It's replacing this feature in the small essay so featuring small se, with features from a bigger essay.
177 Huntington: I see. So that doesn't hurt Mse, and then it helps L. 0.
177 Huntington: This is currently just a hypothesis. Oh, that's your hypothesis. Oh, I'm sorry. So oh, not something. I should memorize. It's what you're guessing. It's what we're guessing.
177 Huntington: Just because I'm not super familiar. Msc is the difference between the the activations before and after the sparse. All encoder. And then L, 0 is the sparsity, basically like how basically, the norm of the actual sparse part. Exactly. So the other piece. And then the reconstruction. So when you say doesn't affect Msc, you're saying doesn't change the activations and then reduces L 0
177 Huntington: or increases is based on whether you're adding more features or removing features from the sparse part of it.
177 Huntington: So like. So the fact that you're switching it for
177 Huntington: okay, not show. So you're adding, like.
177 Huntington: you're switching the from like the ones that are more like
177 Huntington: broken up like a bigger essays into the smaller one.
177 Huntington: What's the intuition behind that like? Is it like? So? Maybe because of the diagram. I assume that, like the smaller essays, would still have those capabilities that the bigger essays have is just that there's just more like separate features that you can define bigger ones.
177 Huntington: Yeah. So the small ones could have the capabilities. But they might require multiple latents to reconstruct the activation, whereas the big ones might be able to reconstruct it with few latency.
177 Huntington: So I'm using features and latency interchangeably. Features are like I guess.
177 Huntington: features of the data, and then later into the like the actual learned features of the essay and like in the literature.
177 Huntington: And to Mike's question, sorry, would that L. 0? Specifically.
177 Huntington: the number of se latents that are active or like that are non 0
177 Huntington: when doing a specific construction. So it's not the number of all patents that we have in the set. Not sure if that works. Oh, so it's actually a little when you run them. It's an index.
177 Huntington: I'm sorry. Is it? Like a literally just a count of how many? Non sparse? Absolutely.
177 Huntington: How many are active, like, literally in practice, you take web text and try to 100 tokens, put it in spas autoencode, a map back down and count the number of non-zero sae. Oh, l. 0. I was thinking it was l. 1, for some reason, call time. Okay, that makes sense. So l, 1 is used for the regularization. But then we actually look at the L 0, because that's what we really care about spouse coding
177 Huntington: cool? Just quick question. So when you say novel feature from the large Se to the small se, could you give an example. Yeah. So that'd be like, I don't know. I can't read these things, but one of the rare elements being represented in large sae like, I don't know. Uranium. Sorry. So a bigger, bigger dictionary size.
177 Huntington: So yeah, we started looking at this like swapping setup
177 Huntington: And basically.
177 Huntington: yeah, okay, so on the the plot here is inserting latency from a larger sae into a smaller sae
177 Huntington: and this is basically testing whether they're novel or they're not novel.
177 Huntington: So with that novel, the we say that the reconstruction is improved.
177 Huntington: basically add information that is not reconstructed by the smaller city.
177 Huntington: They are novel. We say that. Sorry they aren't novel.
177 Huntington: We say that the Msc. Goes up because the duplicating information already contained in small essay.
177 Huntington: and that means it's like it worsens the reconstruction because it over reconstruct like one duration.
177 Huntington: Why would it be? Would it worsen the Msc. It could also be like, Oh, I put a redundant feature there.
177 Huntington: Yeah. So because they're linearly, linearly active, they activate linearly. So if you have something that's like in your small essay, you have a feature for mammal, and then in your larger se you have a feature for cat. Then you add the cat feature into the smaller Se. You have a mammal feature, and you have a cat feature that both fire when they see a cat, so it kind of like adds it twice.
177 Huntington: Does that make sense? This help possibly wouldn't be an issue with top Ess, that is an issue with value essays.
177 Huntington: So it's literally whether it increases or decreases the expression performance. So this is very creative chart. I wonder if you take a minute here to explain the X and the Y axes to me? Yeah, so on this chart. So basically, the issue we had was that combinatorially swapping and evaluating all possible swaps is really expensive.
177 Huntington: So we wanted to use cosine similarity to evaluate whether latents are novel or what we call reconstruction latents.
177 Huntington: So the y-axis is the effect on the reconstruction. When you add latent from a large essay into a small essay. So each point here
177 Huntington: corresponds to the large assay relationship.
177 Huntington: Oh, just one large, sae latent. Yeah, no, but you're swapping it with this small, so there's no swaps going on. Oh, it's just straight insertions. So just inserting a large sae latent into a small sae, yes.
177 Huntington: got it.
177 Huntington: And then the x-axis is cosine similarity the maximum cosine similarity between that latent and any latent in the
177 Huntington: smaller essay. And that's decoder, I see. So moving to the right is a measure of more redundancy. Every point that's way to the right means that I've I've just inserted a large
177 Huntington: feature that's very similar to a feature that was already present in the small thing, and then moving to the left means, oh, this is something that is very dissimilar to everything. Yes, got it.
177 Huntington: And there's like some sort of relationship between these 2 metrics is not absolutely perfect. Obviously,
177 Huntington: but and so there's this. So wait. So there's a reasonable question so like, how so?
177 Huntington: I guess you're not retraining or retuning the Se. So it so that's why Mse can get worse. So in principle, if you like, increase the size of the essay and you rebalance it, then Msc. Should never increase. But you're not retuning.
177 Huntington: Yeah, I see. Have you done this graph with for the natural retuning.
177 Huntington: We have not done with retuning, I think. Like you say, we'll just.
177 Huntington: I don't. Yeah, I guess if you retune. Having inserted this later. That's like very similar. Then it's like, be moved to similar ways to similar.
177 Huntington: How do you choose where to put the feature? If you'd swap the feature in like, literally in the
177 Huntington: vector I guess it just where the old one was. Are they the same dimensionality? Or yeah. So the decoder. Vectors. All the features are the same
177 Huntington: size, so it doesn't matter where you put them
177 Huntington: doesn't matter. So I could literally like randomly choose it, and it wouldn't affect performance at all. Yeah, it was dependent. You can shuffle the rows of decode when encoded without affecting performance essays.
177 Huntington: So yeah. So the point of this was that basically you can relate the codes and similarity to the effect of on the reconstruction of adding in a latent from a larger se into small Se.
177 Huntington: And we arbitrarily chose no point 7, just like here as our threshold, for do we think it's going to increase or decrease.
177 Huntington: the reconstruction performance of smaller city. When we add a new data, obviously, that's not
177 Huntington: perfect. But it's what we went with. You can move it like 0 point 1 in either direction. So we're getting some results
177 Huntington: all right.
177 Huntington: So what we looked then was like, not just inserting.
177 Huntington: So the novel features here, but also swapping
177 Huntington: the reconstruction features, or split features. So what I mean by that is like you have a parent in your smaller say.
177 Huntington: and then you say, all of these features are children of that parent feature?
177 Huntington: So you construct a subgraph that includes this feature and those, and then you just swap it
177 Huntington: so in this case, between these 2 esses, you construct subgraph that includes this one, this one and all of these.
177 Huntington: and then the swap would be. You remove these 2 features from the smaller C, and insert all of these features from the larger C,
177 Huntington: and what we see in this plot is that we 1st add the novel features, so
177 Huntington: the ones that low lie below this line well.
177 Huntington: less than 0 point 7, but all of these green ones in the bottom left. Here.
177 Huntington: We 1st add those in to the smaller Se. And we see that Msc goes down
177 Huntington: and L. 0 goes up, which is kind of what we predicted before.
177 Huntington: and then when you add the split features, those subgraphs or blatants
177 Huntington: the L. 0 goes down, which is great because it's what we predicted. But the Msc also goes down, which is not actually what we predicted.
177 Huntington: yeah, does this plot make sense. The different colors are different sizes of spots encoder. So we start from a 1 x expansion factor
177 Huntington: and then add all of the novel latents from a 2 x expansion pack. To Sa here
177 Huntington: add those into the one expansion factor.
177 Huntington: then do all the swaps, and that gives us the 2 x extension factor sae.
177 Huntington: and we do that between the 2 s. And 4 x 4 x and the 8 x so like roughly interpolating between the essays, using this like ordered approach to swapping the novel latence, and then the reconstruction latency
177 Huntington: for the split license.
177 Huntington: So doesn't it make sense that the Ms. Will go down for the split as well.
177 Huntington: I mean, because now you just have more information to reconstruct reconstruct that same information in the in the
177 Huntington: like. What was the intention?
177 Huntington: when you say that, Ms, you will not go down. Yeah. So the intuition was that,
177 Huntington: say, you have something that's like this feature, plus this feature, like
177 Huntington: a cat that is ginger, and in the smallest a can represent that by saying
177 Huntington: cat featureative ginger feature, active
177 Huntington: it can be structured, but it uses 2 latent to reconstruct, whereas in the larger se you might have a ginger cap feature. So you get a single latent
177 Huntington: that can be constructed, but only uses one actively. Isn't delivery instruction.
177 Huntington: So both of the Esses can represent the same
177 Huntington: thing. We should change accounts, but
177 Huntington: they use different numbers of latency to reconstruct it.
177 Huntington: So that was our thinking initially that, like, maybe larger S are basically just doing like this feature, plus this feature equals new feature, and as it goes down, but they can still represent the same information
177 Huntington: by interplacing the features you are using. You are assuming feature absorption in the larger essays. So yeah, you are, you are assuming feature absorption in the larger se ratings. Right? Yeah.
177 Huntington: Yup, yeah, as the spot shows, the Msc also goes down when you do the swaps.
177 Huntington: Which I guess I kind of knew probably what happened.
177 Huntington: it kind of like also disproves that large se is just doing this perfect
177 Huntington: intersection between features and small essay.
177 Huntington: Are there any questions about any of this? This is like
177 Huntington: all participants. Sorry. So so the adding novel step is, you're not. You're not removing all the small essay features. You're just adding the bunch of
177 Huntington: split ones exactly, and then sorry. That's new ones, the new ones, the unrelated one. Okay, so then, what's adding split.
177 Huntington: So splitting is where we're like, well sorry. This is
177 Huntington: badly written. This is like swapping.
177 Huntington: So when you replace the small ones with features with.
177 Huntington: okay, so it's like, let's add the new ones that you didn't have the information about, and then we'll also remove the ones that are more coarse and just replace them with, okay, yeah.
177 Huntington: So on the other side like
177 Huntington: you could also just like, take the larger. Sae because you're like they can
177 Huntington: the new, the new essay features from another.
177 Huntington: And you're also replacing se features contained in the larger and smaller. So in the end, you kind of like, take the larger. Sae, yeah, every time you've been every time the line changes color. That's when you've arrived at an identical se to the larger se, so at this point, this is just the the 2 X expansion factor. Se.
177 Huntington: okay, thanks.
177 Huntington: So kind of looking more into this idea that larger Esses don't like
177 Huntington: feature a plus feature. B is the split feature.
177 Huntington: We figured that we might be able to like undo this process, so linearly decompose the large essay features into
177 Huntington: Meta features so features that, like form, parts of multiple features in the larger se.
177 Huntington: So an example of this is like
177 Huntington: a small essay might have features for shape and features for color.
177 Huntington: and then these get composed into like other chips in the large Se.
177 Huntington: Which I guess similar to the ginger cat. Example
177 Huntington: and what we came up with was Metro essays. Which is where you take an essay.
177 Huntington: essays decoder. And then you train another sae on that decoder. So instead of using activations as the training data, you're using the decoder of another sae as your training data.
177 Huntington: And the idea here is that like, if essays can recover
177 Huntington: features from activation data, then they can probably also recover meta features from features if that makes sense.
177 Huntington: So what's the input to this?
177 Huntington: So it's the decoder of another sae.
177 Huntington: But like the decoder is a matrix of vectors. Right? Yeah, so.
177 Huntington: But congratulations.
177 Huntington: So the activations that get multiplied with this matrix, so the essentially is trained on, like.
177 Huntington: all the activations of your data set or a model at a certain point. Yeah, then you learn a decoder, an se decoder based on these activations.
177 Huntington: And then you basically apply the same process. But you, instead of using the activation data sets, you're using just the decoder of another essay.
177 Huntington: But the decoder is a fixed matrix.
177 Huntington: bizarre activations decoded decoder, you mean, like the output of the encoder when you feed through the no, no, I mean, like the fixed decoder matrix.
177 Huntington: What do you do? Is the fixed decoder matrix. The train is also to encode part on each roller.
177 Huntington: On this, I think, yeah, on each room.
177 Huntington: Okay? Oh, yeah. So each row is like a an example training example for
177 Huntington: the bachelor se got it? Yeah.
177 Huntington: thanks. I'm sorry. I still didn't get. Isn't this kind of like a small data set? If you'd like to usually put on the rows? Or do you like also train on activations that are like activating those rows. It's pretty small data set if you get it right, like, if you have an expansion factor of 2,
177 Huntington: then the data set size for the meter. Sae is 2 times the model. Yeah, it's like 1,500. So we did this on knowledge once we normally use like 30 patches. But I mean, it seems to train pretty well on stage.
177 Huntington: So essays are one way to decompose
177 Huntington: that input data. And you could do Sv. Or Pca.
177 Huntington: Why did you choose like?
177 Huntington: Do the sae training again instead of
177 Huntington: yeah. Because you have this hypothesis that'd be like,
177 Huntington: large Esses are doing like feature addition on the decoder directions of smaller s, so it's doing like a times feature. So it's doing like.
177 Huntington: added the large se decoder directions would be like linear sums of small se decoder directions. Yeah, so like this, feature times x plus this feature times y equals the new
177 Huntington: The feature in the lodge, I see.
177 Huntington: and also just empirically, Pca. Directions on decoder or sees didn't give us anything interesting in the same way. That's like you don't assume them to be a functional
177 Huntington: Pca gives you orthogonal directions. You don't really want to have. You don't want orthogonal directions. Yeah.
177 Huntington: In the same way that Pca on model activations also doesn't give you intelligible features, those essays, maybe Duper
177 Huntington: So applying a metro se to decoder or a big Se.
177 Huntington: We looked at an Einstein feature in the larger Se. Which is our debate on Einstein things
177 Huntington: and the metal latents responded to
177 Huntington: this is going to be quite complex to explain. The metal agents responded to
177 Huntington: latents in the larger city that responded to
177 Huntington: things that started with E. If that makes sense. So in this feature response to like
177 Huntington: latents corresponding to Einstein and to Estonia, and to eagles, and to education, and to the name Eleanor.
177 Huntington: So if that makes sense.
177 Huntington: Yep, where? Where does the Einstein latent live?
177 Huntington: So this lives in the large essay that we're decomposing. Okay? And then the green ones also live in the large Se. That we're decomposing.
177 Huntington: And then the blue ones are these meta latency that we're looking at?
177 Huntington: Why do they reset my thought? That's also not. I just see.
177 Huntington: So the metal latents are in the decoder of the mesher sae, which is the essay that we trained on the decoder weights of the larger, sae?
177 Huntington: So there's like 2 tiers of se you've got like your activations. Then you've got your large se. Then you've got your measuring C, and they're all decomposing the one below it.
177 Huntington: So is the hypothesis here that there are like more themes, almost like in the. As you break things apart, you reach like the bottom of the the decomposition, or is, when I look at this, I'm abstracted from the actual like setup, but it feels like there would be cycles like you get to start with. E. You get eagle, you go. You just keep going off far away from Einstein, and maybe branch back somewhere.
177 Huntington: So is it? Just? Yeah. Yeah. So I guess this is the confusing diagram in that. Einstein is in the same set of things that all of the green ones are. So
177 Huntington: it's kind of like the wrong way around in that.
177 Huntington: All of these belong to that. But this doesn't like this only connects to that through Einstein.
177 Huntington: Oh, so yes, you could get cycles,
177 Huntington: and the breakup is more semantic rather than like syntactic or anything like that.
177 Huntington: These things are like, well, I guess starts with E is a syntax thing. Yeah. But like Germany is semantically similar to Einstein. But it's not like syntactically similar. There's not like a structure of the pieces of the the letters in Einstein. Yeah.
177 Huntington: so it does both. And I'm not sure there's like a meaningful difference between
177 Huntington: semantic features and syntatic features.
177 Huntington: They just make you like pictures. Is this a real example? Yeah, it looks pretty good.
177 Huntington: It would increase increase the font size, though
177 Huntington: I can't zoom. I apologize. This is all screenshotted from the paper. I put this together like yesterday. This is layer 8 of Gpt. 2, which is 12 layers. Gpt too small.
177 Huntington: So like somewhere in the middle ish.
177 Huntington: So
177 Huntington: what we found is that you basically by doing training mesh, s, you get the same thing as smaller Sses.
177 Huntington: So when you compare a smaller C of like expansion factor 2
177 Huntington: to a metro se of expansion patch 2 that's been trained on like an expansion patch. 32 sae
177 Huntington: you get very similar latency.
177 Huntington: and you can also just do the same thing that Metro ses do by feeding in the decoder of a large Se. Into a smaller Se.
177 Huntington: And here the same, by the way, add regions clustering.
177 Huntington: Oh, so I think this is like
177 Huntington: some kind of evidence to the fact that
177 Huntington: larger sees seem to like some composition of small se features, and you can also do.
177 Huntington: I think, like what all of this means is that
177 Huntington: essays don't give you these nice atomic, canonical features out of the box like you get feature splitting, which means, as you scale your essay.
177 Huntington: you get more and more narrow features, so it can be like hard to find the right level to steer on
177 Huntington: waterproof but also you got new features that aren't included in smaller. S, your Msc goes up.
177 Huntington: so yeah, basically, we're arguing that if you just like, scale the essay infinitely, you do, and end up with the optimum set of units for any intelligent work.
177 Huntington: why don't people want orthonality in the decoder basis.
177 Huntington: It seems like, because then you only can have the model. Sorry? What?
177 Huntington: Yes, but in principle, if you add some sort of authority loss. It's not like it's gonna
177 Huntington: like overrides all of the other losses
177 Huntington: you could have near orthogonal. And then you can fit D squared in. No problem. Yeah, I see. I mean, yeah, I mean, it can kind of like encourage that. These features are as far as possible, and then it will solve the feature splitting problem.
177 Huntington: and then it will encourage finding new features. Is that what Llms do? Is that what Llms do to the encode features is like nearly orthogonal.
177 Huntington: We don't know. We don't know. Yeah, I feel like, this is a a major assumption.
177 Huntington: Well, that's what the superposition hypothesis is suggesting. Right? So that's just 3 blue, one brown videos explaining how great it is to.
177 Huntington: Okay. Even if you have only 2 dimensions. But you want to squeeze in 5 features. You kind of want those 5 features to be as far
177 Huntington: from each other as possible. Right? And if you use the authority laws, maybe
177 Huntington: so I'm I'm not assuming that you'll you'll kind of like find something that is completely orthogonal, but it will probably encourage the decoder. Be factors
177 Huntington: to be more diverse from each other, maybe capture new features. So, and then how do you choose the features?
177 Huntington: How do you choose the features? You can't do that with a series as well.
177 Huntington: are you just proposing a loss term encouraging or token less? Yes, approximate. Yes, yes, literally something like Qqt is equal to. I, yeah.
177 Huntington: chance, what did this at maths? Right? Do you remember this? I'm sorry someone tried this at maths where they input an orthogonality regularizer to see fractions. I can't remember what the outcome was.
177 Huntington: Who was that? I don't know.
177 Huntington: I feel like this very much. Didn't work, but I can't. All these sae people. Okay? More sae questions, because I, you know, don't do I say work. So so you know, one of the one of the so. And I just finished teaching this in my Cs 71 50 class.
177 Huntington: you know, auto encoders this old ancient area. Neural networks
177 Huntington: and one of the dissatisfying things about it is, there's no information theoretic, you know, basis for you know what you're doing
177 Huntington: just by changing dimension, dimensionality, bottleneck, I mean, like a mathematician, a pure mathematician would tell you. If you have a single real number, you encode infinite amount of information in a single real number, because you have as many digits as you want
177 Huntington: and so so you know the
177 Huntington: the, the modern solution to it is you do what the Va people do like. You introduce noise, and then you have, like, you know what your bit rate is. Right? So it's like a like a a variational autoencoder.
177 Huntington: Right? Would introduce noise. And so I haven't. So like like.
177 Huntington: I don't follow the sae literature. Has somebody tried a fan out Vae, where.
177 Huntington: you know you're you're controlling the information content, using like sort of the noise formulation that statisticians have
177 Huntington: proposed as opposed to using like sparsity loss
177 Huntington: to reduce in the clip. Because then you actually know the bit rate right? Then you actually know the information that's going through rather than just counting dimensions. And right.
177 Huntington: what do you say?
177 Huntington: You've tried it. You've tried it, but you've you've you've physically tried it yourself.
177 Huntington: Just great. So have you tried training.
177 Huntington: Our goal was to get a binary essay with like not features having
177 Huntington: having like digital like running positive weights, but just like have binary on off activations. And the way we try to train. It is like
177 Huntington: giving giving a probability of its activity.
177 Huntington: assigning appropriately, yeah, that's similar. Yeah, that's great. Yes, that's pretty similar. And
177 Huntington: just out of experience like in in our experience that didn't really
177 Huntington: work in twice settings. But we should.
177 Huntington: This was just an experiment, and we should look into the theory of like. Why, it doesn't work. Yes.
177 Huntington: and then there's an early, like anthropic note in one of the big blog posts that says like, Oh, we also tried better, like, yeah. And it didn't really work. But like.
177 Huntington: there hasn't been a rigorous analysis, right to my knowledge, like, How would you interpret the Vas bottleneck features.
177 Huntington: Oh, I mean, you know, I'm not sure like it's kind of like the way you'd interpret this right? So that, like in in typical vas like. There's this whole community around Beta vas
177 Huntington: right where they interpret them the same way that
177 Huntington: this community interprets as a as they just look at the 1st dimension.
177 Huntington: Right? They look at the 1st neuron. And they're like, Oh, that's pretty meaningful. Yeah. But like you cannot get this entanglement from that, like, if the this combinatorial argument is true, like in language, if you applied that to a language model.
177 Huntington: Oh, well, so I don't know. Like, so Beta Va is disentangled. That's sort of one of the arguments of Beta. Va is right. But but I just that's that's that's my question is, has has somebody tried to adapt it to the setting?
177 Huntington: I chatted about this a lot with Nish.
177 Huntington: Because, like in this traditional disentanglement, literature, people found kind of does it that it's basically impossible from unsupervised data alone.
177 Huntington: And so there's sort of a mismatch in between this possibility results and essays seemingly disentangling right.
177 Huntington: It's a good direction.
177 Huntington: Seems like it's actually got it.
177 Huntington: Yeah, there's old stuff right? There's people have been looking at it for a long time, and it's just like there's a little bit of disconnect with the literature is not connected. Yeah, yeah.
177 Huntington: setting is slightly different, though, because, like like.
177 Huntington: they didn't train on the whole Internet. Right? That's right. They have like the toyed balls. Yeah, the colored balls, the ovals of different angles, or that kind of thing.
177 Huntington: Maybe they might have if anthropic says they try to.
177 Huntington: They mentioned this stuff in their blog posts. Yeah, so they at least are aware
177 Huntington: of it. We don't know how much they.
177 Huntington: But maybe that's the secret sauce.
177 Huntington: Okay, cool.
177 Huntington: Thanks a lot.
177 Huntington: Oh, there's more. If you'll hold. Yeah, yeah, that's great.
177 Huntington: yeah. Let's let's go. Let's do. Can you squeeze? Well, let's keep on going. Okay, okay,
177 Huntington: So basically off this project, I was kind of like, dissatisfied with what's possible encoders were learning.
177 Huntington: it seemed like these decoder directions were like the nice units that we that we thought they would be
177 Huntington: and then.
177 Huntington: yeah, kind of inspired by relative representation, metrics and kind of inspired by some notes Gdm. Had, on applying orthogonal matching pursuit to the decoder matrices of essays.
177 Huntington: I started looking at whether you could basically just take activations and use optimization
177 Huntington: to deconstruct activations into a set of anchor activations.
177 Huntington: If that kind of makes sense.
177 Huntington: So
177 Huntington: yeah, basically, the hope here was that instead of having these many, these messy dictionaries that you get as well sort of encoders
177 Huntington: that don't transfer between models. You would have this not so messy collection of activations that you transfer between models you can evaluate.
177 Huntington: You do like potentially model dipping this kind of stuff with it.
177 Huntington: and then you're just using infrastructure optimization to do a decomposition rather than learning your encoder.
177 Huntington: So so I think I asked this question in past meetings, but I guess I don't really have a
177 Huntington: yeah answer for it. Yeah. So whenever I hear model dipping, the 1st thing I hear think about is Cross Coder. Is that
177 Huntington: so similar? Like, what's up right now?
177 Huntington: Yes, like, yeah. So I guess
177 Huntington: both this and cross coding are
177 Huntington: decomposing model activations into a fixed bice at matrix.
177 Huntington: So is there any particular difference? Yes, so cross coders learn encoders and decoders for all different models that you're analyzing and then
177 Huntington: they characterize on like the the same data point to the same
177 Huntington: later, it should be active spells good, whereas in this view.
177 Huntington: I'll explain more. But basically, you iteratively, iteratively construct your set of anchor points on 2 different models, and then you just
177 Huntington: satisfactors directly, so that they're pretty different approaches and that there's no optimization. Grant time optimization. Yeah.
177 Huntington: how are you?
177 Huntington: But a very naive and simple way. Just train an essay on the smaller batch that you get.
177 Huntington: Say, again, put a very
177 Huntington: intuitive and maybe boring way. Be like train an essay on the small incoming batch that you optimize your
177 Huntington: the suit on, or is it? Is it like it so like infantry training essays?
177 Huntington: W. Would that fit in your matching pursuit framework? Or is that a different it?
177 Huntington: Yeah. I mean, you could do that. You wouldn't train the decoder. Then I think you train. You'd like minimize. You'd optimize this pass code.
177 Huntington: But is the whole decoser decoder frozen
177 Huntington: in this? Yes, it's just a collection of activations. Okay.
177 Huntington: this is this thing where you have a fixed matrix. And you're like, Oh, I have this magical matrix. And on the other side things are sparse.
177 Huntington: If I just cross my fingers.
177 Huntington: And now the the matching pursuit computes these non-zero values on the other side.
177 Huntington: Okay, so yeah, so basically what this method does very naively.
177 Huntington: it starts with an empty dictionary. And you try to deconstruct incoming activations into the dictionary and you set a threshold. And if the reconstruction loss exceeds the threshold, just add that activation to a dictionary, and then you continue.
177 Huntington: So you iteratively build up a dictionary like this, and then you can stop when it reaches the specified size or not.
177 Huntington: And the idea here is just like it's very naive way, or
177 Huntington: building something that behaves somewhat like a spot sort encoder, without actually doing all the training, which is like a major issue with spot sort encoders is that they often have more parameters than the Llm. They're decomposing the activations for which seems like pretty inefficient.
177 Huntington: But you've got to get the decoder directions in the 1st place, and that's
177 Huntington: it requires a lot of training. 1st of all, silver encoders. It just connected them. But are they just taking them into a matrix? You're just getting them, really. Yeah, you just say, take the 1st one. Yeah, it's pretty good, is it?
177 Huntington: And then the answer is, no, not good enough. Take the second one, yeah. And the 3rd one is good enough for the 3rd one. Yeah, maybe it's good enough. It's a linear combination of the 1st 2. Drop it
177 Huntington: exactly. Yeah. Yeah.
177 Huntington: So there's no optimization of decoding.
177 Huntington: And this kind of works. Sometimes some like, in terms of reconstruction works in some models. Okay, like, on 50 70. M seems to. Okay.
177 Huntington: does bit worse than 1 60. And then it's terrible on Gemma, 2 TV and then
177 Huntington: interpretability results are mixed. So the auto interprets like, kind of Okayish. Whereas linear probing is pretty dreadful.
177 Huntington: But the advantage to this approach is that you can train on 0 point 2% of the data that you do for ses roughly, the same performance
177 Huntington: to the Max performance. So, for example, I trained
177 Huntington: these. It does. I'm putting them or laboratory might be
177 Huntington: for all layers in like one light on a single portion. 90
177 Huntington: which I think would be quite hard to do for ses
177 Huntington: and then this is kind of like propped off. But you need, like a hundred 1 million tokens before essays, achieve the same reconstruction performance as it does
177 Huntington: achieve with 1 million tokens.
177 Huntington: So obviously a greedy approach.
177 Huntington: It's gonna depend on the order of your presentation of your data. Yeah.
177 Huntington: So now, saes are also unstable. If you retrain them. You get a different sae right? So like, how does stability compare between your greedy approach and saes how much in terms of like, yeah, well, I'm just curious. I don't know how you quantify stability of sae lanes like how different you know it tends to be if you train in flights.
177 Huntington: but also to be honest like, how would you?
177 Huntington: How do people quantify? You know stability of these 2 things?
177 Huntington: Do people want to buy know they should right like, because, you know, there's sort of like.
177 Huntington: should I should I trust an ite? Well, I guess it depends on whether, if I trained it again randomly.
177 Huntington: if it gave me the same thing, or if I'm just getting my understanding is very susceptible to initialization, I think you tend to get like a core set of features that reproduce cross runs, and then the less frequent basis will tend to vary has that been quantified. So the main of essays are the sparsity penalty and the resulting sparsity, and
177 Huntington: you can.
177 Huntington: The only thing that I can say is like in comparison to changing the sparsity of the Sae.
177 Huntington: The stability is much smaller. So if we have a plot of
177 Huntington: sparsity on the X-axis and the reconstruction performance on the Y-axis then change over sparsity induces a much bigger change than like the error that you would plot
177 Huntington: for like retraining essays with the same philosophy.
177 Huntington: And yeah, so I guess a meaningful metric would be like
177 Huntington: is is a change in just like retraining on the on a different random seed.
177 Huntington: much bigger than a change when varying the sparsity, and from the essay event results.
177 Huntington: Essays are pretty robust when compared to the sparsity scale or changes in sparsity.
177 Huntington: There's there's a paper that looks like they do across training runs, and only 30% of the features are shared across different seats. This is from.
177 Huntington: I lose your AI people, 2025, yeah.
177 Huntington: But this is great. Do you have a paper
177 Huntington: all of these things hopefully by the end of the month? So I missed how? How you initialize the vectors, you're choosing
177 Huntington: in your in your pursuit matching pursuit. Yeah, so it's random in it.
177 Huntington: So you you start with an empty dictionary.
177 Huntington: And then you basically say, each incoming activation? Can we decompose it into the practice in the dictionary? What's the residual?
177 Huntington: If the research yeah. And if the reconstruction error exceeds the threshold, you add it to the dictionary. There's no you add what to the actual. I think the thing to note like to make this really easier to digest is like this matching pursuit algorithms.
177 Huntington: They assume that the dictionary is there already.
177 Huntington: Okay, the the matching pursuit itself doesn't learn a dictionary.
177 Huntington: Yeah. Yeah, but I'm what's the fixed dictionary that's getting? He creates a dictionary on which he later can run matching. So how do you make the dictionary? You really construct it by iteration of the training data. So it's like, it's like, approximate gram-schmidt. So you know, the algorithm, right? Oh, right?
177 Huntington: And and so it's like, oh, yeah, can I decompose into the
177 Huntington: you know the rows that I already have? Right? Oh, I can't.
177 Huntington: That's a great point, right? Instead of oh, so now in graph, you would just take the residual as the next row here, instead of taking the residual. You're taking the whole vector, right? Yeah. But I think the residual is like a bit. Yeah, yeah, of course, it's just it's just error. Yeah. Yeah. Whereas attribution is going to like.
177 Huntington: I see cool. So then I looked at using the the goodly constructed dictionaries as measure of model similarity.
177 Huntington: So one experiment that in the Ck papers is like, can you match layers between different initiative, different seeds
177 Huntington: all the same model.
177 Huntington: So this is like 5 instances of Gpt. 2. Sport.
177 Huntington: They have 12 layers. Can you match the layers up using your representation similarity metric
177 Huntington: and it turns out that the intersection of a union or the Googlely constructed it. The dictionaries is like.
177 Huntington: considerably outperforms other methods for doing this layer matching task.
177 Huntington: which seems like it could be pretty useful for models, I think, which I've not explored. Really, that's why playing around with. But
177 Huntington: like, have I exceeded? Can you explain a bit slower how this matching works?
177 Huntington: Okay, so you have
177 Huntington: 2 instances of Gpt. 2 small, which have 12 layers? And they're trained on different initializations.
177 Huntington: Oh, different seeds.
177 Huntington: And the question is like, Can you look at 2 layers of the model.
177 Huntington: and basically like assigns a metric to that similarity.
177 Huntington: You do this for all pairs of the 12 layers. Is it fine if I imagine, like the residual stream after some layer like, let's say, residual stream after layer 5, exactly. Yeah. Okay. So it could be that, or it could be like any other.
177 Huntington: or it could also be Mlp. Output at Layer 5.
177 Huntington: So this is done on the residual stream in these experiments, but it could be anything. Okay? And the question is, how well can you match the layers? So how can? How well can you see, like model a layer 5 is model B layer 5. They're the most similar.
177 Huntington: because the idea is that, like
177 Huntington: each layer in the model, should be very similar to the same layer in the other model.
177 Huntington: Think so? Yeah.
177 Huntington: And on this layer matching task, like linear regression, does really badly. Ck and Svcca do slightly better. And then just the intersection of a union, the jackal similarity between the greedly constructed dictionaries. Of these it does that are trained on the different layers of these models.
177 Huntington: Gets like much higher. So you you are able to do that, you here because, like every row in your greedily constructed matrix, you can say, okay, this was the token blah in sequence. Blah, yeah. Yeah. So the the dictionary is just like pointers to tokens. And this is how you compute the I/O, exactly. Okay, that's yeah. That's
177 Huntington: and then the nice thing about this is that
177 Huntington: you can do it in like a couple of hours on a single Gpu versus essays where it's like hundreds of hours of training. And then you put like, match the se decoders. And it's even faster than just doing like linear regression or Ck on these models.
177 Huntington: I think this is maybe a really nice message
177 Huntington: on a model diff. Yeah, I think it's like not super exciting for touchability essay. It's just like, seems to do better. And essay is a whack, anyway. But the model dipping and for
177 Huntington: representation similarity, it seems potentially useful or just have adverse idea.
177 Huntington: Something. Also to just get a, you know, get to some intuition building on the new model. Right? Yeah, very quick readout. How can you have it built in?
177 Huntington: Show me what? Show me what the model diff is? Yeah, absolutely. You can train it in like 2 min a large model, whereas that's easy. Dozens
177 Huntington: make sure. I understand you basically just feeding in data into it. And with respect to a particular say, residual. You have this dictionary. And you're almost just finding a, you're trying to build a vector space that
177 Huntington: the vector is similar or dissimilar to all the previously collected ones, basically Grant Smith style. And at the end you just have a bunch of varied vectors that serve as your sort of like basis of what it is
177 Huntington: cool.
177 Huntington: Oh, and that's kind of the end. But I can keep going. All those cool pictures skip to the pictures. The pictures are not my work. Don't worry. My work never looks as good as this.
177 Huntington: So yeah, this is just like a toy model I'm playing with at the moment where you get this really interesting structure for like days of the week, once the year doing addition in llama seems to like use helical structures.
177 Huntington: And basically, when you train says on this kind of structure, where you have fewer latency dimensions and structure, they completely break
177 Huntington: And I've just been like starting to look at how this can be fixed. So one really dumb approach I've used on this problem is just like taking the fario basis of the activations and training an essay on that set, and it seems to work pretty nicely.
177 Huntington: But yeah, maybe maybe should. Yeah, maybe it should be done for all essays. Maybe all essays should be done right? Exactly. It doesn't work. But yeah, yeah.
177 Huntington: which we could do.
177 Huntington: Yeah, that's kind of what I'm starting to look at. Great.
177 Huntington: Yep, thank you, Patrick.
177 Huntington: Yeah. Back to the pre deep learning, feature engineering.
177 Huntington: But it's great. Oh, so who who do? Who yuxu Chan?
177 Huntington: Are you? Are you here for a visit day, or have we? Oh, yeah, that's great. Okay, cool. You're welcome.