David Bau: What do you expect?
David Bau: That's great. And okay, here.
David Bau: And let me do this over here. Okay, great. So, so today
David Bau: we're lucky to have Josh Josh Engels visiting from way way across the river at Mit, practically a different country, and even in a different type of department he's in. He's in Max Tegmark's lab
David Bau: and
David Bau: And so Josh has been making a name for himself
David Bau: in the last year or so by taking a really interesting and skeptical look
David Bau: at sparse autoencoders he had. He had a I don't know if it was a blog post or preprint that had a very interesting title
David Bau: that that said, you know. Sparse auto encoders, you know, are are actually good for something.
David Bau: And and then and then there was a title change that got some attention where he became more skeptical of them. And so it looks like it looks like you're going to talk about that adventure a bit. Is that right, Josh? I will talk about a little bit towards the end. Okay, I'll talk.
David Bau: Yeah. I'll basically kind of give a tour of some of my work on talk a lot about the multi dimensional features work which I still think is quite cool. And then, yeah, also talk a little bit about the work. We did like studying the dark matters like what they're missing. And then also about this probing work. Okay, excellent, excellent. So, looking forward to all of it. Welcome, Josh, thanks.
David Bau: Right? So, yeah, I think this talk is actually pretty timely now. Because
David Bau: I don't know how you've been following the Twitter sphere recently. But there's been a lot of interest of Esses on Twitter in the last week. Kind of for both directions Deepmind recently put out a a new post, saying that they were probably gonna stop working on spouse autoencoders. And then anthropic put out like a you know, a really big post from the whole interp team that that looked at like a bunch of really interesting cases.
David Bau: And so, yeah, I think there is. There's some kind of like cognitive dissonance and cross wires right now in the in the community of of just how useful sports autoencoders. And then what's going on with the technology. So I'm going to give kind of a an intro that talks about sparse auto encoders, talks about some of my work on them, and maybe I'll speculate about where I think the field should go.
David Bau: Yep.
David Bau: okay, yeah. Talk about the background of se, and and then, yeah, some talk about some limitations that my work has found. So maybe some progress and then some closing thoughts.
David Bau: Okay. So also, just want to say, like, feel free to interrupt or like, ask questions you know, can be a little. It can be dynamic. So yeah, I guess in this this talk I'm mostly focusing on the representations that models use.
David Bau: I think this is maybe not the only thing you need to understand. Models like you might also need to understand the computations that models do. But understanding the representation seems like a really good starting point. And one nice thing, I think, to point out is that there's this Platonic representation hypothesis, which is that as this did not come from us, it comes from who at all like this is Phil by Solo's group at Mit. Which is this hypothesis that
David Bau: as models get bigger
David Bau: the representations that they learn kind of converge for like a bunch of different reasons, maybe because they, because in order to predict many things well, it helps to have, like a small set of the same representations of like the true world, or maybe a bunch of other kind of learning theory reasons. But but just empirically, it seems like this is like human perception, and these are these are other models. And and this is like color, and basically in color, it seems like
David Bau: all these models are kind of learning, similar maps of color. And then this is kind of a bunch of representations and and kind of the the reason this is nice, because we probably in the study a couple models might be able to draw conclusions about the representations of a lot of models.
David Bau: So yeah, in this work, we're just gonna study a couple of models. So okay.
David Bau: quick refresher. You guys probably all know this. But I'll just be just getting some terminology how the transformers work.
David Bau: Basically a bunch of tokens. Those go to Benning vectors, and then you repeatedly do 2 things over and over. You like, mix them together with attention, and then you do processing on top of them. That's the Mlp. And then just repeat that. And then I'm going to call this
David Bau: these vectors that are between the attention the Mlp. That are getting added to like the residual stream, which is kind of like a mechanterpy way of describing, like the models, hidden states. And the original stream is really nice, because it's the only way to communicate between layers. So if you study just all the residual stream, a certain layer, it's like a nice bottleneck. It has all the representations and models thinking about.
David Bau: Right? So now that we have this bottleneck, that kind of is all like. At this, we have this, we can look here and say everything the model is thinking about that it uses to predict the next token is stored in the original stream. The natural question to ask is.
David Bau: how representation is actually stored in the result stream. Like, what? What does the information look like? It's just a bunch of random noise? Or is there kind of like more structure there? And so the really simple explanation that'd be really nice
David Bau: would be each dimension, individual stream store a single concept. So you have, you know, a 10,000 dimensional visual stream. You have one dimension, for you know, cats and one dimension for dogs, and one dimension for cars like say, it's an image model. Unfortunately, as like this awesome thing from Chris Ola. 2020, I think it says, like zooming in studying circuits.
David Bau: But this is like in the inception of v 1 model. It's usually not the case that neurons are monosemantic. Usually each neuron represents many things. And in fact, this is kind of necessary, because if you think about it, a language model knows way more than you know, 5,000 or 10,000 concepts, or whatever the dimension of the visual stream is. And so
David Bau: it's not really plausible to just have one neuron for each concept. Maybe you could imagine training a super, super, wide, sparse model. But as of now, those are not what our models look like. I think that's pretty interesting direction, though.
David Bau: Okay? So so how are 1, 1 hypothesis to how these representations are stored in the digital stream is the linear representation hypothesis, which is that basically, most concepts in the model are stored as one dimensional vectors, and they're all kind of stored almost orthogonally.
David Bau: And if the model wants to and as long as the model wants to represent. Like only a sparse number of these. Then it can rep, then it can kind of just
David Bau: store the sum of all these vectors as the activation vector and then it can actually decode these vectors relatively losslessly. This is like basically like some really classical, like sparse coding literature, where, if you have an over complete basis, but they're only a sparse amount of the basis is active. Then you can actually basically decompose you can decode which things were active.
David Bau: yeah. So this is an example. We have the sentence. The orange cat jumped. Then maybe the model has one feature for orange, one for feeling, one for action, and the representation is just the sum of those things.
David Bau: And this is also a hypothesis like, why linear probing works so well, because if you have, you know, say, you're trying to probe out orange right then you can have a probing direction that's like very similar to Orange. Maybe it won't be exactly orange, because you want to also be orthogonal to as many other concepts as possible, but it's kind of most in the direction of orange. And then, whenever orange fires, the probe will fire.
David Bau: Okay?
David Bau: Sorry later. Okay,
David Bau: right? So now, we have this model of of the activation space. How do we actually, you know, get all of these concepts? How can we get like a listing enumeration of all the concepts and models thinking about and this key idea that I guess I talk is sparse autoencoders. So the idea is that we can train a basically one layer, one hidden layer mop
David Bau: to basically have it have a super wide and sparse hidden layer. So you have a super wide hidden layer, and then you project back down and try to reconstruct the input. But you additionally require that this hidden layer is only a sparse number of the neurons that are actually active.
David Bau: and the hope is that you have one neuron here that's activating on feline and one neuron here that's activating on orange. And the idea is that hopefully the best minimum loss solution for this model for this network
David Bau: is to learn
David Bau: basically one actually learn one neuron per concept. And you just train it on like a bunch of hidden activations from the model. And in practice we do find that these hidden neurons are a lot more monosemantic than the original neurons
David Bau: of of the activations.
David Bau: And yeah, I guess this is just this is from like, I think, scaling monosticity. Or maybe the dictionary, original dictionary learning, anthropic work where they basically find, like a bunch of different, some pretty monosement features that they that they label.
David Bau: Yeah.
David Bau: but there's some limitations here, I guess, before I talk about limitations and lead into the market any questions so far about the background.
David Bau: I don't know. Probably it's review for lots of you guys just to like.
David Bau: I think I understand the general premise. But, like
David Bau: how good at reconstructing the hidden state are these models? It seems like that's right. Pretty good, I I would say. Usually you get
David Bau: you can get depending on the model, and how wide and how how much you're willing to allow sources. But I would say usually greater than 90 or 95% of the variance explain explain depends again, depends on the model. I think, bigger models. It's harder to explain as much. Another way. The thing you can look at is when you insert the reconstruction back into the model. How much of the loss, do you? Or how much? What's the cross entropy loss difference downstream? Yeah.
David Bau: and typically for like reasonable essayes, it's
David Bau: it's something like, I think Leo Gao and they did training on Gpt. 4. And they found that it's like equivalent to a model that had been trained with like 10% of the training compute even though. So the cross entropy lift difference is like pretty small but because
David Bau: like small cross entropy, loss differences are like pretty big training gap training training. So it's like, it's like, maybe if your model has a cross entropy of like 1.6, it'll now have a cross entropy, loss of like 1.7 or something on, like some, some like pile distribution or something, but like they actually turn out to be pretty big difference in terms of like effective training computes. So so missing out on 90% of training seems like a big deal. Yeah, like, so so maybe they're not that good.
David Bau: Well, it depends. Maybe they're still capturing, like, a lot of the representations. It's it's yeah. It's not really clear like, what? What is it missing right like like? Is it missing things that we don't care too much about? Or is it kind of missing like the actual things that make. Gpt 4 different. Gpt, 3, yeah, sort of. And we don't really have a nice way of
David Bau: yeah measuring that. Yes, yeah. Okay.
David Bau: I guess. Have you like played with a model that's just like full essays and subjectively just
David Bau: like you're saying, replace every layer with missing
David Bau: or even just one layer or just one layer. Yeah, I I think it feels similar.
David Bau: I don't know. I guess I haven't really played with it too much. One thing you can do also is like, say, you want to steer a model like 1. 1 thing like, for example, good fire is done is like trained ses on models, and then used the the sae vectors for steering. So then you don't even need to instruct the sa reconstruction in, and it's like Esses might be useful, even if they don't perfectly reconstruct the output. Oh, yeah, yeah, it's just sort of an interesting question. Yeah, like, you know it.
David Bau: You had a paper. You're going to talk about dark matter. Yeah. So maybe that's relevant. Yeah, definitely like, yeah, maybe like you were saying, there's this like these pointers. Maybe maybe the reason that you get worse loss because you're just missing all the pointers or something. Right? Right? Yeah.
David Bau: yeah.
David Bau: Oh, another technical question. Yeah, because you know, without that much experience in training. So what what's the level of sparsity that we typically get like? So if you
David Bau: you know, you put it an ordinary.
David Bau: You know, representation vector through. Yeah. Then, how much sparsity is the se like, you can kind of choose which what you want. There's like a trade off between sparsity and reconstruction. I think pretty normal is to have like an L 0, which is the number of non non 0 hidden neurons of around like
David Bau: 50 to 150 or so. Oh, how how big is the sae? Usually like 50 to 150 out of out of maybe like a order of like
David Bau: 20,000 to a few 100,000. Yeah, yeah. Yeah. So so the usually the the blow up factor from the activation dimension is probably on the order, like 8 or 16 or something, or maybe maybe bigger. 32. But then, yeah, usually you have, like 10 or 20 or more times fewer active concepts.
David Bau: Okay, so
David Bau: there's some limitations of the linear reputation hypothesis. Lrh, for short, if you're in the know. And I guess one thing is that if you look at this plot here right like the this is like a you map of all the features, and they're clearly like, not randomly, almost orthogonal, like, like semantically similar feature vectors that the se finds are actually like closer together in the activation space?
David Bau: And can we say anything more than almost orthogonal, or even say anything more than that? Like similar things, similar vectors are close together. And are really, I guess, a kind of a a
David Bau: question off that that. Yeah. You know, David. And people, you guys touched on this like, what's the correct mediator is that our one dimensional vectors, even the correct model of like, what an atomic feature is, and if not like what types of features might be inherently multidimensional. And how might we find them? What would they look like? So
David Bau: here's the Lrh activation of sparse linear combinations of meaningful feature vectors. And what we hypothesize is the multi-d representation hypothesis, activation of sparse linear combination of meaningful feature manifolds Aka multi d features. Also I realize I changed the slide. It's supposed to have all the collaborators I worked on this paper with. So those are
David Bau: Eric Michaud, Isaac Lau Wes Gurney, and Max Tegmark. Yes. So this paper, called Modeling. All features are one dimensionally linear, used to be called modeling features are linear. But we added the one dimensionally linear, because I think that more clearly articulates the aspect of linearity, that this paper is kind of finding evidence against
David Bau: but yeah, so basically, we find in this paper. We kind of argue that many features might be inherently multidimensional. It doesn't make sense to think of them. These kind of like one dimensional sparse auto encoder latents.
David Bau: And so here are some examples we find. I'll talk more about this next few slides, but we just find like for the days of the week and months of the year. Kind of the clearest examples where these kind of representations lie along a circle.
David Bau: Okay, so what do I mean by a multi d feature like, what does this even mean? So we kind of find a feature as a function that maps some subset of like the input distribution, the tokens to a k dimensional point cloud. And we say, this point cloud is reducible. If it is basically composed of 2 lower dimensional features. So if there's just 2 independent features that make up this this like this like blob, then we say it's probably not a multi. It's probably not like a real multi feature
David Bau: or if it's composed of 2 features that basically never occur together, because lots of times you might have things like, you know, submarine. And like, you know, Mars, that are like basically never occur in the same sentence. And so they're like, kind of anticorrelated. But we don't want to say that this is like some multi feature. There has to be some more interesting relationship between these concepts like a very. I don't know if this is a perfect definition. But it's kind of like a
David Bau: kind of a 1st pass statistical test that you might run, and then a circle, for example, is neither of these things. There's no way to decompose this into 2 kind of like to project this onto a basis such that the basis is independent, or that the basis like never co-occurs together.
David Bau: And so we can actually run these tests on a bunch of clusters. So so why do we want to run this on a bunch of clusters of sae features. So the idea is that
David Bau: there's some feature space in the model.
David Bau: And
David Bau: If there are truly some some feature manifolds kind of in this space. Then we might expect many sae features to be learned that lie along these feature manifolds like, say, there's some circle, you might expect a bunch of features to kind of get learned that lie like spokes along a wheel for that, for that manifold, basically, any places that there are, that there are density, this density in this. In this feature space we might expect to be many, sae we might expect there to be many se features that are learned
David Bau: except sometimes when there's density. You might expect this somewhat interesting concept. Sometimes it'll just be density, because there's like a lot of
David Bau: a lot of features there, but sometimes it might be more interesting. So we can do is that we can cluster the sae features. Kind of finding these areas of density in in the feature space. And then for each of these clusters, we can run these historical tests from the last slide. To kind of figure out, is this like a real multi d feature? Or is this just like an area of density? And when we do this we find that that, like some of the features that are most irreducible by these definitions, kind of like with these these things, these nice multi D features just kind of pop out.
David Bau: And so that's really cool. It's kind of like an unsupervised way to maybe find multidimensional structure in in models.
David Bau: What's the distance metric for clustering? We use a couple, I think, mostly cosine similarity between the decoder vectors. Yeah. And then the clustering. We try a couple of clustering techniques in the paper. But yeah, I think that's it's not totally clear what would be the best way to do it.
David Bau: Yeah. And also I think that there's some argument that maybe clustering wouldn't have worked to find all types, multi features, because it kind of assumes that you kind of learn this. These like this, like a bunch of se features that lie along this multi d feature. But maybe that doesn't always happen.
David Bau: Have you all seen this line of works about feature, absorption and feature splitting. Yeah. So when you are clustering these concepts you do. Do you also find that they can be like queries together to a single concept. Sometimes. Yeah, I think so, I think totally, you get that. Yeah.
David Bau: yeah.
David Bau: Yeah. In fact, in in the days of the week thing in the months of the year we actually see some future splitting. I forget, if you can. It's on here, but like there's also like a direction. For like I think it's maybe I, it's on a different model, the mistral model. But there's like a direction for like spring and fall and winter, and you know, weekend or something. And so you might imagine those are kind of like splitting off from like days of the week or the months directions. Or maybe those are just atomic concepts. Yeah, I'm not.
David Bau: I don't think we look too much at clusters of like
David Bau: non multi features like, we don't look at them too much.
David Bau: But I probably other people might have looked at this. So so why? I was asking this question, because now the difference between actual multi concepts and some kind of like an atomic concept that got splitted. Yeah, it. The difference between them becomes a little bit shady. Yeah, definitely, I think. Yeah, like, like, the difference we argue is that, well, one thing is that you just look at them right? And you're like, Oh, like like this is
David Bau: can most just be represented as like a 1 d feature or something. And that's what we kind of try to formalize with these statistical tests. But I agree, it's not totally
David Bau: clear. It's a little bit. Also, we're also appealing a little bit to like
David Bau: human intuition. Maybe right? Like at the end of the day, we're kind of saying, like, like, we're like trying to find areas of dense structure in these features, and some of them like, really pop out as like this interesting multidimensional structure. And like this is pretty clearly not just like a 1 you can't just reduce like this is important structure. And then we try to like, formalize it a little bit. But you might not really need the formalization. Awesome. Thanks.
David Bau: So so is the idea you can only use like, you need 2 parameters to parameterize a circle. Yeah. And so it's not 1 1 dimensional, I guess is that? Yeah, yeah. And also, I guess another way, think it just like the relationship between these features is like very structured and
David Bau: it's not just like random multi stuff. So like, did you look at these other ones that are, or in the corner?
David Bau: let me see, do I have it here? Yeah. So here are some. Here's some like bonus, like other. What other features does it find? I find a lot of like
David Bau: common tokens and letters which which might be this feature. Splitting thing so like here, like is N, and like this is this is the end token, and I try to. I, we kind of
David Bau: split it out by like capital. NN, with a space. Another capital. N, and like.
David Bau: yeah, this, this, maybe, is some interest like, there's this really cool set of work from Adam Shy and Paul reaches about and some other people about belief, states and language models which is like, you know, assuming that the model is like assuming models like trying to predict like like
David Bau: next tokens, and like a hidden Markov model like like they might have like, really interesting, like fractal structures. And like, maybe these, like common tokens, are like also similar of like where the model is like has this like kind of dense fractal structure. For like when it's doing thing that's very like a common token or something. But I'm not. I'm not really sure exactly what's going on in these examples.
David Bau: but they're interesting to look at.
David Bau: Can I ask a very preliminary question, having never worked in essays. Yeah, what is the
David Bau: like? Why are we trying to find only linear
David Bau: essays? Why not nonlinear? Why not 2 layer essay? You could do that, like a 2 layer, encoder or 2, layer, decoder or something or 2, layer encoder, non-linearity in between like, why is the linearity so desirable here? Right.
David Bau: So
David Bau: one thing you totally could do, I think that would doesn't hurt interpretability that much. And people have shown this is to use a more powerful encoder. So the idea is you're trying to come with a bunch of latents that reconstruct the space doesn't really matter how you get them
David Bau: as long as but I think if you make a nonlinear decoder, it's a little less clear how your latents relate to the actual model activations that you're trying to reconstruct, because now you have a bunch of latents, but they're being combined before they reconstruct the activation. So I think a linear decoder is probably what you want. You could have more powerful encoder.
David Bau: Is that
David Bau: answer your question right? So if if I were to have a dual layer encoder with nonlinear piece, that's fine, I think then it wouldn't like this wouldn't show up. In the 1st place, I think it would. I think it still. Probably I suspect it still would.
David Bau: because because if the decoder is still linear, you have to, you have to reconstruct the circle somehow in the decoder. And so right? So you're saying, linearity and decoder is preferable because you want to interpret it. Yeah, like, how are these being? Exactly
David Bau: exactly.
David Bau: Yeah, can we not interpret nonlinearities? Like, if we have a simple value or something.
David Bau: then you have to. So then you have to say, like, Okay, you have these features that are active in the hidden States. But now they're being like combined in like this dense way, like. So now it's like a lot harder to figure out, because the nice thing about having a linear decoder is, you have your latents, and the reconstruction is just a sum of the weights. A weighted sum of the awaited. Some of the weights right, like each activation times, the the decoder. And now we have these combined things that are combining them. And it's a lot less clear, like what the activation actually is.
David Bau: yeah.
David Bau: okay, so these, so so like qualitatively, let's take a look at these clusters that you put under right. So you describe the cluster as such, and another cluster as.
David Bau: and the letter n and and so but now
David Bau: but you've got you've got this multidimensional so like, what is it using the other dimensions for like, so for for the, for the, for the your previous paper, right, it was pretty clear that you use the other dimension, for you know the days of the week. Yeah, you know, you know, which day is it
David Bau: so? Is this? Is this using the other dimension to figure out what kind of an it is, or something, or what like. This is what I'm trying to understand. Yeah, I think yes, I think
David Bau: exactly. I think it's using. I mean, it's a little hard. Maybe Pca is not the right.
David Bau: Wait, look at this.
David Bau: but yeah, my, so it's colored by the different type of N, and what are the different types in there? There's like a lowercase, NA. Capital N, and N, with a space capital N, with a space, and then a few other like N, weird N tokens, maybe just a space token. Yeah, and what about the such as such as we have a lowercase, such capital, such.
David Bau: And then those are the biggest ones. And there's also these, a as such. So including there's like the top 10 total, like similar similar words, that, yeah, that play the role of such. Yes,
David Bau: But yeah, I don't exactly know. It does seem like there's some interesting structure here.
David Bau: I'm not really sure what? Exactly what? It? What it? Okay, yeah. Okay, cool.
David Bau: Yeah. We have in the paper we have in the appendix. We have, like the top, 20 or so clusters.
David Bau: okay. Now, another interesting question is, do models actually use these circles for anything? And so the task we look at is like this natural modular edition where we have questions like, let's do some day of the week math 2 days from Monday is blank. Let's do calendar math. 4 months from January is blank.
David Bau: Interestingly, models actually can't do normal, modular arithmetic. If you just like like language models. I guess you're you're or who is training. Yeah, you were training. You were training like, on actual or you're just training a toy model. Do modular arithmetic. It can probably do that in that case. But but language models can't
David Bau: and but they can do this kind of natural stuff, which I think in of itself is kind of interesting
David Bau: imperfect generalization. But so models can do this task. And and we can do a Pca of the hidden States and find the circle in the months and the days of the week. But we can also intervene in the circle, and not just along kind of the outside edge. We can also intervene in the center. And we get these really nice plots where this shows like, if you take a task like 2 days from Monday is blank, but you change representation on Monday to like anywhere in the circle then
David Bau: it kind of is predictable, and that, like the output changes in the way you'd expect. And if you change the 3 days, and you can kind of the circle rotates a little bit. So basically, it seems like the model probably is using this circle. Now, caveat for you guys, you do a lot of like patching and stuff in order to get this to like work. Well, we have to ablate the other dimensions. In in the Pca. Because
David Bau: there might be like some other pathways, the model is using to compute this. And if you don't ablate those, it gets more confused.
David Bau: Yeah.
David Bau: okay, some recent follow ups. So this is kind of why we changed the title. So Chris, with this cool blog post that basically kind of makes the Lh 2 parts. One is that like features are inherently one dimensional, and the other is that features. Magnitude correlates with intensity, and you can add them together, and so we only show evidence against part one casortus et Al. With Stanford, Nlp group show some evidence against Part 2. By finding these kind of like onion like features in this really cool paper.
David Bau: Our evidence is only against part one this part 2 is in a toy model. So maybe it doesn't actually happen in language models.
David Bau: So we'll see.
David Bau: And then, yeah, this is the belief state stuff I talked about. I think that's really cool. And these are multi-dimensional features alone, toy model and Livegorton in this really neat result finds like the curve detectors in vision models also form a circle.
David Bau: Okay, so that's the end of that section. Any any questions before I kind of go on. Talk about some other papers.
David Bau: Okay.
David Bau: yeah. So hangs out here sometimes. And Max, yeah. So this this was kind of the the the next kind of
David Bau: piece of work they're working on, as far as auto goes. Also somewhat kind of investigating what, as far as autoencoders like not learn
David Bau: and we find is that the the we can look at we kind of like study the se error object as kind of a
David Bau: as kind of like a 1st class object to study. And what we find some interesting things. One is that, like the norm of this of this error term is very linearly predictable from from the input activations. And moreover, the actual vector itself is like kind of just a linear transformation of the activations many times. And to me, what this is kind of roughly saying is that there's
David Bau: possibly some subspaces in the impactivations that the models isn't learning and this seems kind of problematic.
David Bau: And then one kind of neat thing is that like, it seems that the amount of linear error in many cases decreases at the same kind of rate. That. The that the reconstruction error goes down. And so it kind of seems like this nonlinear error component is
David Bau: kind of like we can kind of like project out the features. We'll learn. And if I say, like, as the essay is really wide. There's probably going to be some chunk of linear error still left and some chunk of nonlinear error. So how do you decompose these 3 components of error. Right? Okay, so the here the linear error is just the fraction of the error vector that we can predict with a linear transformation of the inputs. And the nonlinear is like, whatever we can't.
David Bau: yeah, fraction of the error vector that you can predict with a linear transformation of the model activations.
David Bau: Yeah.
David Bau: So so if, if, if, like, you had just learned, like some
David Bau: some random subspace of the input, then you could predict 100% of the you predict, whatever the orthogonal subspace was as a linear transformation of the input. Yeah. Whereas if if you had just kind of learned like
David Bau: some some randoms set. Then I think it would be a lot harder to pick anything
David Bau: of of the reconstruction of the error.
David Bau: Does it make sense to to you guys?
David Bau: So I like the words you're saying. You know, they seem reasonable. I just I it's just I haven't been thinking about this. Yeah, yes, yeah. I think it is a little
David Bau: plausibly, a little confusing like, exactly what I think. We're just trying to get a handle on like, what is the error vector made out of? Is it mostly kind of like unlearned features? Or is it maybe something weird? The essay is doing, or is there some? There's some like consistent linear error correspond to. It's just unlearned features, or which one is that I think linear error is mostly unlearn features. I see so like linear errors like oh, if you sort of had
David Bau: infinite dimensional sae.
David Bau: then you would recover all the rest of the linear error. You wouldn't necessarily. That's that's what this is. Kind of showing that there might even be linear error, at which maybe is saying that some of these, like maybe some of these features, are like not linear like like, if linear error could be maybe modeled as like some, some like dense random noise that's really hard for A sparse representation to learn
David Bau: may be impossible unless you have like an exponential number of directions or something.
David Bau: yeah, yeah, that's what I'm trying to understand. So like, why? So like absent features. That's that's your. That's that's that's your curve of. That's the curve of just like increasing the size. Yeah, that's just like Barron explained to increase size. I see, yeah, I see.
David Bau: So what? What the heck is? Linear error.
David Bau: So it's just the error that can be predicted. Yeah.
David Bau: from the activations. So what do you mean by can be predicted like, what's what's a what's an error prediction just a linear transformation of the to go to a scalar or to go to a vector to go to a vector so you're
David Bau: right.
David Bau: So it's just a simple linear model
David Bau: of the residual. It's kind of trying to say like, are there subspaces of the residual that Esa is not learning?
David Bau: And so your hypothesis is these are the things that are not one dimensional. Or these are the things that are not sparse, sparse. Exactly. Yeah.
David Bau: maybe they're not important. They're a not important. So some things that are, yeah, they're they're not like sparse or or one dimensional in the right way. Yeah.
David Bau: they're and they're inherently dense. Yeah. And also the essays could learn dense things as long as they're low, dimensional. But if you have some high, dimensional, dense subspace.
David Bau: you know, like, say, say, you just say, I mean, one thing is, this could be some random Gaussian noise like, say, you just have some some literal, random Gaussian noise that you just add to your activations right? Then you'll be able to pick that
David Bau: It's not totally perfect, maybe, but I kind of obviously there's no, there's no noise in a language in a transformer. So but it might need it right, it might be. Oh, I need right? Right? I might. I need to make some stochastic decisions.
David Bau: I have my own internal pseudo random number generator to allow me to do this could be yeah.
David Bau: Anyways, I should. I should keep moving. So yeah, yeah,
David Bau: so another other kind of another couple of interesting points that we find in this paper is that sae errors can be pretty consistent across scales. So this is like Per token error, and you kind of see that that you could predict a lot of
David Bau: how good you'll do on a much bigger sae by how much you draw smaller. S. How well you draw smaller. Sae, this is kind of more. This is like air norms, I think, but this is is kind of more evidence that, like
David Bau: some tokens or some things on some tokens, might just be like fundamentally hard for saes to to predict
David Bau: And then this is a little more analysis kind of breaking down this like linear and nonlinear components and finding that they're kind of fundamentally different in a few different ways. This is a few examples that we train se on the linear error component.
David Bau: You do much. You do better in terms of like how interpretable the resulting features are and also, like the the loss of the sae is lower. Also, you can predict the norm better of the linear error than nonlinear error. Which we kind of argue in the paper I won't get into too much. Here is like an evidence that there's more like unlearned features in it as opposed to like weird stuff from the sae.
David Bau: Yeah, okay, any other questions here. And then I'll talk about
David Bau: Rses useful, which is the recent stuff on Twitter. I'm sorry. Can you recap the like? The takeaways from this work? Okay? Sure. I think the takeaways from this work are saes seem to be missing
David Bau: consistently missing some some subspaces of features as they get bigger like the errors are like, seem to be correlated across scales. And it's like seems tractable to study se error. And maybe we can find some interesting stuff about
David Bau: about like like the non one, dimensional features and language models by studying what essays make mistakes on? Can you actually find that subspace like in the language models like what that subspace is that essays are missing?
David Bau: Then probably we should. If we are only interested in like understanding linear features, maybe we should take the other subspace and
David Bau: don't don't really bother ourselves with that sort of space at all. You could try doing that. Yeah, I mean, but I feel like that sometimes has interesting stuff in it.
David Bau: yeah, I'm not. I'm not totally. I think it will be interesting to try to look. Also, it's not totally clear. So like
David Bau: this subspace that's linearly predictable. Also includes features you haven't learned yet, probably, so you don't just want to throw away the whole thing. There's like. Seems to be some part of it like, like at the asymptote. Yeah, you could throw away the linear subspace. But but like, until you get there, it feels like you can't quite just throw it away, because, there's also like unlearned things that like as you make your essay bigger, you want to get
David Bau: like there are like, it's a combination of like one true one d features you haven't learned yet, and also, maybe, like some like some like nonlinear stuff. You haven't learned yet. Some like some like noise or something.
David Bau: Also, I had a question about this figure. So so this the the one in the bottom. Yeah. So you. So so to draw this figure, you had to boil your vector down to a single number.
David Bau: Yeah, the norm of the. So this is the fraction of variance unexplained. And so so this is like the if you just do the sae, and then you can. You have it. You can ask the asymptote, and then you can do so. That's this line, this bottom line, the asymptote of the top line, and this line is, if you is, if you take the fraction of variance unexplained. If you have your se, and then you also do a linear prediction of the error, and you add those 2, and then you look at the fraction of variance unexplained. Okay, yeah. Okay, awesome. Thanks.
David Bau: Okay, so, but how many are on time? 1246. Okay?
David Bau: So yeah. This was alluded to by David at the start of the talk. We have this paper. Rsa. Is useful. A case study in sparse probing. This was subash Sen Max and Neil,
David Bau: And yeah, like, like, you're very perceptive that you we had. We had this blog post at 1st that said something like saes. What are they good for? Absolutely something. Which is the play on the on the old song like war? What is it good for? Absolutely nothing.
David Bau: and then we kind of had this update where we we had a prepaid. We kind of. We. We spent a few couple more months on it. We looked kind of more, tried to make the baselines more rigorous, kind of looked at more case studies. And we found that
David Bau: there really weren't that many positive results for saes in the end. So it was kind of a negative update for me on saes overall
David Bau: and also for deep mind, because Neil's on this, too, Neil's the the head of the the deepmind inter team, and they also had some internal results that were negative on saes, so yeah, I'll talk a little bit about
David Bau: It's about perfect now.
David Bau: So I guess. First, st
David Bau: why do we expect saes to be good at probing. I think this is an important question, because if we don't expect them to be good at probing, then the negative results of our paper are a lot less interesting.
David Bau: One reason is that other people have found positive results with sa probing. So there at least should be some prior that like this is a reasonable thing to try, because other people found preliminary positive results. So so Trent, bricken from anthropic, has this cool blog post where they find that if you pool sa activations together across context, sometimes you win out of probing. Oh, by the way.
David Bau: sure, everyone knows linear program. But just I'll recap. It's where you have a data set of prompts and targets, and you train a linear logistic regression probe on hidden states of the model. Try to predict the targets
David Bau: right? I say, I said, that here, too nice. There's lots of nice applications in AI safety detect deception checked models being honest malls like self aware of the benchmark. And okay. But so
David Bau: maybe there's some positive results here, some other, maybe mildly positive, results in literature. But intuitively, I think the the argument for why saes might be useful is that they have this really strong kind of inductive bias for interpretability. Saes are basically built
David Bau: to learn a bunch of one dimensional vectors that are good for reconstructing model activations and probing like logistic regression, is also trying to find one dimensional vectors. So you would think that in case where it's pretty hard for probing to find a good one dimensional, vector
David Bau: then kind of relying on this inductive bias in the sae the sae that's already found a bunch of like one d vectors that you think are like pretty good you should be able to kind of like use that as a nice deductive bias crutch to kind of do probing better.
David Bau: That's the intuition. So so we look at?
David Bau: Well, my shirts inside out.
David Bau: anyways, it's intentional. It's a new style. It's a new style, new style. So kind of fall in the situation of
David Bau: of
David Bau: evaluating on difficult task for probing where normal probing is a hard time. Right? We look at like some pretty difficult probing tasks. So so this is this, this condition, this thing on the left here is standard conditions. So it's just normal probing of a lot of data. And we look at like 113 data sets. And this is like the average testing you see. And then on the right here we look at some other settings. So we look at data scarcity class imbalance with noise in the labels. And when the test set is out of distribution.
David Bau: and you can see here that all the settings like, indeed, the average performance goes down so like these settings are harder for linear probes.
David Bau: Unfortunately, we also find it's harder for sae probes, and basically across the board, like sae probes do no better than baseline probes
David Bau: And there's this quick thing here about interpretability like like, maybe you might argue that saes probes are really just good, because we can look at the latents that get identified by the probe, and we indeed find this does help. But we also find that that
David Bau: baseline methods can surprisingly also help in this setting as well like in an interpretability setting.
David Bau: Yeah. So by sae probe, you just mean you take a latents. And oh, I can explain here. Yeah. So so yeah, what I mean by sae probe great question, basically, you take your your class, your activation vectors and your class, one activation vectors, you pass them through your sae that gives you for each for each. Vector, you now have, like a latent representation.
David Bau: And then you choose K of these latents like, where K is pretty small. The intuition is that, like you, you want
David Bau: some like sparse set of se latents to make your pro
David Bau: Here we choose them by the like. The average the average difference between the class labels on each latent. But you can imagine other ways of choosing them. And then you train all this regression probe just kind of on these, like lower, dimensional, latent vectors.
David Bau: and another question, I guess kind of is that this quiver thing quiver. Errors I have here is you might hope that. Okay, well, essays might help on all data sets, but maybe they at least help on some data sets right? Like, maybe the reason this is lower is because it does twice as well sometimes, but but half as well other times. And so we actually
David Bau: test this by taking like a bunch of different methods, both with and without sae probes, and choosing the best method based on the validation. Auc, and then reporting the test auc that method. And we basically find that this doesn't really say that that the essays do not have a consistent advantage even using this technique. So so like this. This is not enough to like recommend sae probes as like a tool when a practitioner's kit
David Bau: quiver in their arrow. Yeah. So so the the way you choose. The latency is you pass a bunch of 0 class and one class, and you see what is the major difference? And those are the top. K, you choose so intuitively, you're saying.
David Bau: if you club a bunch of
David Bau: these parts latents, you could probably form concept representations for the class you're trying to
David Bau: sorry for the classification task.
David Bau: Right? Like the top K latency, you club them together. They are the concept representations for the classification task. Yeah. Only then would it work?
David Bau: I mean, only then would the probe give you like a very good accuracy. If these K representations work really well together to give you the class. Yeah, yeah, and you found it didn't work. So what if you use all of them? You can. So we have a. I don't think I have it here, but we we look at
David Bau: like K, which is how many you use versus the Auc, and one we in the paper like for these plots, we use 16 to 1, 28. And we basically find that beyond 1 28 doesn't help too much. So it kind of it kind of like flattens out at some point
David Bau: past, like the 1st 120 at 200, which is not so surprising like you took a Pca. Of the activations. It would probably also be 128, 2, 56, beyond which didn't help that much to to look at, more dimensions. And
David Bau: yeah. So basically like, it doesn't seem to help too much to go to use more more more latency. So that's surprising. So out of
David Bau: 10 k. You have 1, 28
David Bau: is the maximum that you can club together for you to represent upon. Good.
David Bau: Yep.
David Bau: But then the if you don't pass it through an encoder, and you just use the latency itself. It's probably much higher, not much higher, like like it's. It is a little higher, like here. It's, you know, but it's higher. Yeah, it's higher. So in saes, here is this like sort of the 1, 28 k, the 16 case. I I think it's the
David Bau: not sure. I think it's probably the 1 28.
David Bau: But yeah, okay. So earlier you were. I thought you were saying that
David Bau: compared to other methods, it didn't work as well. So I I thought you meant like compared to the latency sense.
David Bau: No, the compared. It does worse than just probing on the original activations. Because you lose some information within the se, I think, yeah,
David Bau: yeah, I actually don't know. It's a good question, David. If this is 1, 28 or 16, we should check that.
David Bau: Yeah.
David Bau: But we add both 16 and 1 20 to the quiver. So if if 120 to really help, then it should do better.
David Bau: The other thing we find is that the multi token results from anthropic didn't use like a great baseline they were exploratory, so it's not. I don't. I don't think it's like totally their fault. But if you use like an attention, probe baseline instead of just like Max pooling of digital stream as a baseline, then then even the multi token probing helps helps a lot less than than it would otherwise. So I think this is kind of a lesson that like, if you're gonna
David Bau: spend a lot of time optimizing your your new techniques, you spend a lot of time optimizing baselines as well.
David Bau: And then, the interesting thing here is probe interpretability.
David Bau: so certainly we find that saes can help interpretability like essays can identify spurious correlations. You can look at the the top latents that activates in that are like in a probe. And you say, Oh, like this is a punctuation feature, which is what we find in one case. This is like
David Bau: this is like a spurious correlation, because you can see like, Oh, you know, most of the positive class. And in question marks well the negative class, and in spaces, so ses can help you identify spurious correlations in this way. But it also turns out that baseline methods can identify this sort of spurious correlation. If you just look at like on the pile, the tokens that are most similar to your baseline classifier turns out they're also punctuation. And so this is like, it turns out, I guess what this says is that like, maybe
David Bau: like interpretability is in some ways like
David Bau: easier. If you're a little creative, you don't need to do use like high powered se tools. You can just like, take your lips, regression, probe and.it against the the pile.
David Bau: Okay, that's the end of the probing session. Any any questions there?
David Bau: So let's say you had an essay that actually was good at reconstructing the hidden state. You can keep find better latency for probing. Like, I guess, yeah, like, I feel like
David Bau: 90% of the training
David Bau: plus, it's a lot. It's a lot. Yeah, you know. Although I think that many of the concepts
David Bau: or that the pros look for or maybe not in that 90%. I don't know they're like nouns, which I feel like essays are pretty good at getting like is. But maybe maybe maybe
David Bau: yeah, I guess I think
David Bau: my my hunch is that a lot of the problems that make se program not work as well are
David Bau: not the fact that it's not representing things, but the fact that all the other problems that he's like feature, absorption where you know, it comes to work together, or or feature, splitting or feature hedging. Which is this new thing from David Shannon, where
David Bau: it features like kind of move towards things that aren't in the sae. So it's kind of all these problems with with
David Bau: with sae latence. And
David Bau: I think that's 1 of the like. I think that the kind of the takeaway is that I don't think essays are finding like true concepts. I was excited@firstst They were a long time ago, but now I just think that maybe there's some way of characterizing like the models, like feature manifold. And there's
David Bau: if you don't have, like some like true characterization, manifold. It's kind of not much of reason to expect that, like this characterization will like help. You do better probing. I think that's I think that's kind of my intuition nowadays, if they're not like true model concepts.
David Bau: It's sort of a tiling of the manifold. Exactly. Exactly. Yeah. Yeah. And so they're not like super crisp atomic concepts. They're just like some imperfect tiling. And
David Bau: at that point the inductive bias doesn't really seem to just help over logistic progression like L, 2 inductive bias like L, 2 regulation, which is just like really good. It turns out like linear probes are just like, really good. But you need data. Right? So that's the maybe the you need data. Yeah. So this, this is unsupervised. That's true.
David Bau: but many cases you would have data, I guess. Yeah, I think, saes in terms of like an unsupervised discovery method. For like
David Bau: learning things about your model like
David Bau: that that they can do. It's hard to make a benchmark for that to see if they're actually helping over something else. Like I don't know, asking the model about like what is thinking about, or or like guessing things and checking them.
David Bau: but I do think that's the most promising avenue for essays.
David Bau: Okay. And I guess one kind of last thing I'll talk about is a recent like kind of more interesting methods. Paper like that we found to maybe improve essays a little bit. It turns out, that if you take a frozen essay, but then fine tune the rest of the model
David Bau: to have low kl, with with the model output you can actually get like a pretty big jump in performance like you can recover like up to 60% of loss gap to no sae. So I don't know if this will solve any of the problems that we talked about before. But I just think it does solve the kind of this problem of like inserting the sae, you know, is like 10% training computes. Now, maybe it's like 20 or 30%
David Bau: so we what this is, you have your trained sae. And then you just fine tune, kind of using Laura the model, the rest of the model around the sae to adapt it to have low to low. Kl, so like, I think it's kind of clever, because.
David Bau: you know, like changing the models and just changing the sae one of their reviews. They don't like that very much he was like make to interpret the original model. But I think it's kind of cool to to say like, Well, what if we relax that? What if we just want to have a combined sae model system that's more interpretable? You might have this analysis right? I'm interested. Do you think the neuron characters are learning how to make the model number or how to
David Bau: covered the gap that se did cover. Yeah, you don't have any like evals on the, on the like mixed models. How much worse they get? Or, yeah, yeah, so we do. We have it here, like, I don't think I have it here. We we test on like the classic evals, like we test like Mmo and a couple other things, and like it does get smarter like. Then the model just the Se.
David Bau: I I think it's mostly fixing the fixing like places the se messes up. Actually, there was a recent work from Adam carbonin that finds you can get a lot of the benefit here by just like taking a trained sae, and then just finding the sae on kl so so I don't know how much like a lot of this might just be like fixing things in the sae I use my intuition.
David Bau: Yeah.
David Bau: I do think overall. I I know Adam. So Adam found that you could maybe a lot of the benefit here without training the rest of the model.
David Bau: I'm just kind of still excited about methods that like, take the model and like, try to improve it to make it more interpretable.
David Bau: Yeah, yeah. Helps other metrics, I guess. Okay, some. I'm gonna end with some speculation. I think.
David Bau: What is the control for? Ses. I wrote this actually, before the recent stuff from Twitter. So maybe I'll just kind of go off book a little bit, too. But I think one kind of important question is, are sees like useful for real downstream tasks like like this recent se work from anthropic from Sam Marks like using it for auditing. They find you can do it without Ess. But maybe if you maybe Ess will help you do it faster? And then I guess I'm also interested in like.
David Bau: how can what auxiliary losses can we? Can we add to, get like better late in some way, like like, if this is just some like, you know, tiling, as David said of the of the essays, feature, space. How can we get like better tilings, like more interpretable tilings? Because I think I'm kind of. I've kind of given up on like the fact. Essays are like a true, like true atomic features. Maybe we can say, Okay, like.
David Bau: what? What's the actual, useful set of features, you know, like what what gives us a lot of control, or what gives us a lot of understanding? I think, adding that to the se loss function somehow be really useful.
David Bau: And then I guess one question that also is kind of brought up from the anthropic work is, maybe we maybe like transcoders across layer approaches help in some way, or maybe like the Apollo. Or maybe you just kind of need. The Apollo parameter based decompositionally, maybe activation space is just kind of doomed I guess I'm also just pretty excited now about like
David Bau: supervised techniques, like, like, just like trying to make probing really good, or just like classic, you know, David bow style, interp like do like do patching or or like, just like, just like, look at the model and figure out what's going on instead of trying to like. Come with tools to better, General. But I don't know.
David Bau: Anyways, that's the talk cool.
David Bau: Can you go back once. Yeah, let's see. So do we have people really not try it, just like adding the kale loss to.
David Bau: Oh, okay, they. They did try that. So we do compare against that. That's the red line here. It trains a lot slower, and people haven't been using it because the train is slower. We didn't try the like. Take your top, jc, and just like just train on count the very end which turned out to like work pretty well, which is why Adam tried. I think it gets like most of the way to our thing. Yeah.
David Bau: So here, here's a silly question. So like Sa features seem to have like nice interpretable properties.
David Bau: Can you like take the data that they activate, activate on as like
David Bau: to train probes and to the probes that you train end up with the same latent, so are they better than the let's say, features that you started with.
David Bau: That's a great question.
David Bau: I don't know. I don't think anyone's tried that
David Bau: of like, oh, like, maybe this is a punctuation thing. So then you like, have some data set where it's like, okay.
David Bau: punctuation versus, not punctuation or something. Oh, so you wouldn't use the actual. So like, would you do like the label to like, generate new data or something. Yeah, I don't think anyone's tried that.
David Bau: Yeah, it could be cool also. Wonder if
David Bau: if you so say, say you train a linear probe on a base model on like a concept that you think is like definitely represented with just one feature in the sae like you look at the sae features. You know, like this concept that concept that you train linear probe on the base model. And then you take all of the positive activations that you find in the base model and run them through the sae calculate the sparsity of that, and then compare that to the sparsity of like
David Bau: just random text that you run through right? Has anyone done that? And is the sparsity like? Is that ratio kind of high where there's like much more sparsity in these things that you think should be sparse compared to just like random stuff that you run through. I don't think anyone's tried that.
David Bau: I kind of suspect that it would be the same similar, because
David Bau: even in cases where, like one like, there's not really many cases where, like one future fires like. Certainly there's like times when, like fewer future fires, more future fires. Here's fire.
David Bau: But
David Bau: yeah, maybe maybe you could kind of identify like tokens or con, I'm sure, like things like the the beginning of sentence token, for example, like bos token that usually has like if you train it. Well, it's just like one feature. I mean, most people don't even train on it because it's just like a constant
David Bau: But a lot of times I think there's like lots of kind of like, subtle, like other things hanging around in activations that are like hard to disentangle from like a single dominant concept
David Bau: I still like, you know, on this, on this theme I still like Eric's old paper, his quantization paper, where he made this attempt to identify these monosemantic predictions. Where, if you look at me, you know his idea was, if you look at enough text.
David Bau: then once in a while there's like a word prediction that the model makes for one reason.
David Bau: Right? You know. Usually there's, you know, usually word predictions like have lots and lots of different.
David Bau: you know, reinforcing, you know, redundant circuits or lines of logic that simultaneously let them predict that same word. But once in a while
David Bau: there's just one dominant thing that it's doing. And and so he like tries to distill those things out. I think it's I think it's kind of interesting idea. Yeah, I do think that methods that take like Eric does? Is he clusters gradients right? I do think if you did something like.
David Bau: did some sort of like sparse auto record or gradients, or maybe this attribution parameter decomposition stuff. Then it's kind of then you don't have to have it by itself. You can like kind of pull out the mechanism when there's other mechanisms going on at the same time.
David Bau: little fancier clustering yeah, a little fancy exactly. I think ses are basically fancy clustering where things can belong to multiple clusters at the same time. That's what dictionary learning, basically.
David Bau: So I like that. You started the motivation with a poly semantic hypothesis. And then essays are probably a way to
David Bau: yeah, get these individual features.
David Bau: Yeah, so can I assume that the say features that are popping out are the true
David Bau: features that the model is encoding, or is it
David Bau: probably some weird mixture of? I think it's the latter, but it's a weird mixture also, I'm not even sure what it means. I'm not. I don't know. Yeah, what's a true concept? Exactly. Yeah. Like, like, I kind of, yeah, like, this is the mediator paper, right? Like I I think that
David Bau: it's I would probably best model it as like some sort of hierarchical thing where you can. You always break down kind of concepts more if you want, but but I don't know what it means for it to be like a true level like I think I think ses are finding
David Bau: like a tiling of the feature space really like that that wording, the tiling of the feature space.
David Bau: that works to reconstruct it. But I don't think there's any sense in which, like they're the true level. Right? So the the reason I asked this most of the essay features that were popular. Yeah, are the ones that we can interpret. Yeah.
David Bau: And there's this
David Bau: like, there's this, not always in the paper that says, Oh, we found like 20,000 features out of them. 100 are like, really cool. Here are these 100, I think most. So. There's a lot of auto interp stuff research about like automatically interpreting latents and and sae latence, and trying to
David Bau: figure out. And and I think so many of them are interpretable, I think, in practice not just like 120,000. Yeah, I would say, it sounds about right, maybe a little more, maybe 60, 70. But but even the ones that are interpretable. It's like
David Bau: interpreting like you're like, Oh, this fires on periods, but like not all periods. And why doesn't it fire on all periods like, I don't know right? And so there's like.
David Bau: there's like a specificity. And then like sensitivity, part where that it's, it's also hard to measure certain things, because if like a lot of metrics like detection, like you say, like does the feature fire in this place? But but it doesn't say like, does it not fire in like some similar situation. So so like, how do you hard find hard negatives right? How do you find like like places where
David Bau: you like? The explanation, thinks it should fire, but doesn't actually fire. So I think, automated Interp, is like.
David Bau: I have that here as like, I think it would be good if we made it way better. But yeah, because it's kind of not entirely clear how good the features are beyond the fact. They're just like better than neurons. Right? So that's my question. Right? Like, so if you're saying, not, all features are good or yeah, like some sense, not so clean. Yeah.
David Bau: right? Now, I say, maybe there is. I don't not aware of it. We don't rank the features like what is the feature that the model we don't rank them. We don't rank them so if you were to rank them, what's your hypothesis like? Do you think it's the interpretable, like human, interpretable features that will surface on the top, or these noise sort of features rank them in sorry what way?
David Bau: In terms of like what the model like best for loss best for loss. Do you mind as something like this, where they they like? Do a weighted auto interpret score by frequency?
David Bau: I guess my intuition. Well, one thing is that there is a bunch of high frequency features. Where they seem somewhat interpretable, actually work I didn't talk about here is that where we look into, we find, like, maybe some of the high frequency, interpretable features maybe, are actually interpretable. Right?
David Bau: But yeah, it seems like so many, many of like very high frequency stuff which is responsible for long model loss is uninterpretable. And then.
David Bau: yeah, I don't know kind of how, what the rest of distribution looks like too much.
David Bau: Yeah, I think I think, yeah.
David Bau: you know, for this scoring things. And I you know you, not an essay person. So you can tell me if all people have been doing this. But I but I really like
David Bau: these causal tests. So like, I really like what Chris did.
David Bau: It was in his his application of essays like last week.
David Bau: where he's like, oh, you know the way we can tell the difference between a good essay feature and a bad one is that when we transplant it it has clearly causal effects, and a lot of them don't
David Bau: right. But then some of them do, and that's how you can tell the difference between a good one and a bad one. You know, I feel like there's there's this higher order of causality. I think that for these representation factors I like in Lisa.
David Bau: That's paper
David Bau: about how they're like, Hey, can you transfer this to a different model? Can you like distill this
David Bau: this latent into another model and actually have it learn this behavior. Just kind of.
David Bau: you know, this meta hyper version of causality is kind of interesting.
David Bau: And so I kind of like, I like these causal tests. I feel like a lot of the sae work is right now very worried about reconstruction and about some correlative things. Yeah. And I feel like, it's not necessarily it doesn't smell to me like the the most reliable compass. I would agree. Yeah.
David Bau: I there is some like, se bench has some stuff on
David Bau: downstream effects. Yeah, like steer. I don't know if they have a steering one that they should right? John isn't here right now. Right?
David Bau: Yeah, cool. Yeah. Yeah. I agree. I think Evals is big problem. That seems
David Bau: right? Like, you know, when when essay, the essay community is onto something good?
David Bau: Would they know it or not? Right? Are they looking at the right thing right? Because there's some structure in here. There's some structure. Yeah. Like, one thing is, maybe like, I think maybe we have some decent metrics. Se bench is good, like Matryoshka. Essays do seem to help on some of the metrics on se bench, I see. Yes, that seems like maybe we have some some hill to climb. But it's not a very clean hill. Yeah.
David Bau: thank you, Josh, I really appreciate it.
David Bau: Thank you.
David Bau: I think that's