David Bau: Cool.
David Bau: Alright.
David Bau: Could you do the okay there and minimize it?
David Bau: Yeah, that with that mouse. Yeah. Yeah. Okay.
David Bau: okay. Try it again.
David Bau: -Oh. If I move the mouse, and it undoes the alt.
David Bau: So sensitive.
David Bau: No, no, stop. Okay.
David Bau: Okay to restart. Yes. Go ahead. Okay, so Hello, everyone. I'm Arnav.
David Bau: So well, the broad theme of my research for the last few years has been trying to understand how language models do factual associations. Basically, I want to understand how language models like, what facts about the real world these language models know where it stores this knowledge in the parameters, and how the information flows to the network during inference. And with this understanding.
David Bau: how can we update the knowledge? This knowledge in the language model?
David Bau: But okay.
David Bau: today's research talk is, gonna be it's gonna be very different. Actually.
David Bau: yeah, it has really nothing to do with what I usually do.
David Bau: So some of. You know that I've been doing maths, but last
David Bau: few weeks I think 5 or 6 weeks. I'm doing maths within London, and one of my reasons for getting into this program was.
David Bau: I wanted to know more about sparse autoencoders. There has been a lot of excitement around it. I well, a lot of people seem to like that like this thing a lot, including Neil Nanda.
David Bau: If you just go to his Google scholars, all of his recent works are based on spots or encoders. I just wanted to like, learn from him. And yeah, and people he, he's actually collaborating with.
David Bau: So
David Bau: So during the last 2 weeks of this program you end up doing a research sprint. Well, interestingly, my research sprint had nothing to do with spots, auto encoders. I thought that
David Bau: I thought it was gonna do have most definitely have something to do with fast forward encoders. But I ended up doing something completely different. So I can talk about that a little bit.
David Bau: And yeah.
David Bau: and maybe at the very last I'll tie something back to what I usually do.
David Bau: And you might have seen that the the title of this talk is just research update. It was.
David Bau: yeah, I was trying to. When I put I I thought that I'll put everything in the slide and I'll find a
David Bau: find, a title that will basically tie everything together. You might have guessed I was not really able to find one. But yeah, if you have any suggestions for me after this talk.
David Bau: did I try Chat Gpt? Actually, no, that that might have been a good idea.
David Bau: Okay.
David Bau: So first, st I'll talk a little bit about spot encoders some basic preliminaries, the base, the overall idea and some of my underbaked thoughts about that idea. And some questions, I have some experiments I did to investigate these questions. And yeah, we'll see.
David Bau: Okay, so I'll I'll try to go over this slides a little bit quickly, because, everyone here or most of you here kind of already knows about them.
David Bau: So yeah, what do you feature? It's
David Bau: it's super hard to find a satisfying definition of a feature so I stole this definition from the coy models of superposition, like the properties of input, that neuron or direction, responds to
David Bau: with the same. With a slight modification I added our output.
David Bau: Well, many of you attended when Lisa presented her work on Alpha 0.
David Bau: Well, sometimes you can find concepts about some strategy. The model is going to apply sometime in the future. So it can also be something about output. And I think this idea it kind of like generalizes across all generative models as well.
David Bau: that was kind of the idea of future lens, and also, I think,
David Bau: in diffusion model as well. The model is going to generate a car. So it has to have some concept in latent.
David Bau: So there are some desirable properties
David Bau: of a feature. One is your deductible, basically something like a red circle
David Bau: should not be a feature in this in this property, because a red circle you can basically
David Bau: you, you can divide it into different concepts. Like it is a circle shape and the color is red.
David Bau: Another thing is interpretable. Basically, we humans understand what the model is looking at or looking for, or basically what it's trying to do.
David Bau: And okay, at the very last sentence
David Bau: I was a little bit uneasy putting it there. Well, sometimes maybe we want an uninterpretable features as well, special specifically for the things
David Bau: humans are not traditionally very good at
David Bau: well, something like alphafold, like protein structure. Modeling is not something humans are very good at. But there is an effort in this lab to actually change that fact. Maybe you guys will find something super cool and interesting, and you'll you'll then teach
David Bau: humans how to do protein structure modeling. That will be super cool.
David Bau: Okay.
David Bau: So here I am drawing these concepts as single neurons or single channels in the embedding dimension. But life is always not that easy because of something called poly semanticity.
David Bau: Sometimes individual neurons, they activate or respond to completely different concepts. Here you see that in a single neuron is activating for different engrams that really has nothing to do with each other like researchgate prime factors, nanograms, Eurovision stuff like that. So it's probably more reasonable to think of features as directions.
David Bau: And then you will see that. Okay? The axis is here you can think of like corresponding to one neuron, basically, X-axis correspond to one neuron. Y-axis correspond to another neuron. And if 2 features they are
David Bau: represented by orthogonal directions that really has nothing to do with each other. But they can still end up exciting the same neuron.
David Bau: There's another thing called superposition. Basically the model learning more features or directions than it has dimensions of
David Bau: basically, it's a, it's a 2 dimensional plane. It can only technically learn 2 2 directions, but it can start squeezing in more features. And with. And these different features can also start interfering each other.
David Bau: But why the model can get away with this because of something called feature sparsity. Not all the features are present at the single data point like, think of an example like how many times uranium and jazz appear together in a sentence.
David Bau: unless there is a jazz band called uranium, which I am not really sure.
David Bau: Okay.
David Bau: and what this experiment is. Basically, if you train a data with different level of feature sparsity, you'll start seeing that the model can start learning more features than it has dimensions of.
David Bau: and also in very high dimensions. Things actually start acting a little bit differently. It has more room for flexibility. You can. You can do a very silly experiment. Actually, you can. You can start with a very fat matrix for individual columns or individual basis. And you can try to just make Qtq is equal to identity.
David Bau: And basically, what is supposed to do is that if the columns are different. Then their dot product is 0, basically make them orthogonal. And you'll find that you can squeeze in a lot of columns. Basically. Yeah, you can probably start with a
David Bau: 1,000 by 10,000 matrix, and you'll find that the different columns if you do gradient descent to tune that, you will find that. Probably after you're done tuning the columns
David Bau: the angle between the columns. They are roughly orthogonal, maybe between 88 to 92.
David Bau: But yeah.
David Bau: So this is the promise of spots or tangleders.
David Bau: you break the representation into monosemantic and interpretable directions.
David Bau: Basically, there is an encoder here. Okay, you take one of the latents from the model's representation, you encode it into a higher projection where? Yeah? And then you, you basically decode it back to to basically reconstruct the original matrix.
David Bau: and you back propagate 2 kinds of losses. One is the construction loss, and one is the sparsity loss.
David Bau: What the sparsity loss is supposed to achieve is that it's supposed to make
David Bau: most of these these things. 0 and only a few of these things are on.
David Bau: So basically, what it's supposed to give you. Is that okay? So the columns of the decoder matrix
David Bau: think of them. Think of them as basis of the implementation which is supposed to capture these individual concepts. And this is a mixture. This is kind of like a. This is kind of like a distribution. How this this latent is distributed across different features. And this mixture is supposed to mix all of these concepts together to give you back the implementation.
David Bau: Does that make sense.
David Bau: Okay?
David Bau: All right.
David Bau: So yeah, there has been a lot of different ways of getting the mixture.
David Bau: I think there is very little room for creativity in the decoder part it has to be. It has to be a down projection or something. But there are actually different ways of getting the mixture, and people have been innovative about the structure of the encoder playing with the laws.
David Bau: Yeah.
David Bau: like, these are just some of the figures. But if you just go to Google scholars, you will find a lot of different ways of doing that people have been doing. I think top case something. And course, variance. So there is a.
David Bau: it's kind of like an engineering subfield in sparse auto encoders.
David Bau: And yeah, people have also been training this
David Bau: mixture of experts far sort of encoders and training this router to refer back to, to decide basically which parts of encoder to refer back to now. And there has also been some proposals to basically try to encode a bottleneck interpretability. Sorry interpretability bottleneck inside these transformers. And
David Bau: okay, why not? Then kind of like, replace all the layers implementations with this bottleneck
David Bau: with this, with these interpretability bottlenecks.
David Bau: Yeah, I think Gemmascope was kind of like an attempt at this.
David Bau: Yeah, sure, if we can
David Bau: discuss details with this one.
David Bau: Which one I mean, there's those 2 things, these 2 things. Yeah, yes, they really have nothing to do with each other. Actually. So here you'll take a latent. And you, instead of just training one sparse encoder, you'll train
David Bau: 3 different spots of encoder sort of in kind of like a mixture of things. And you'll train a router here that will basically give you a decision which sparks of encoder. Now, you want to refer back to
David Bau: basically, which sparse or encoder is best given this representation.
David Bau: If that makes sense, okay, okay? And then what? What's the benefit?
David Bau: This is kind of like a mixture of experts benefits, I guess.
David Bau: Well, here you are not really mixing the knowledge between expert. You are just
David Bau: trying to like.
David Bau: yeah, we're just trying to decide which which is the best part sort. Even this data center.
David Bau: Okay, that's it.
David Bau: And the right one is that you are.
David Bau: you are basically trying to encode an interpretability bottleneck inside these transformers. Basically, this is the so that
David Bau: the figure here is.
David Bau: it's kind of literally the well, it okay the the whole idea to be honest is a little bit different. You frame this, you kind of like do a another layer of fine tuning on top of this saes. Basically, you don't really get your presentation just from the reconstruction loss, but also like how it overall affects the model.
David Bau: Okay, this is just normal language model. This is normal language model. Okay? Yeah.
David Bau: And
David Bau: kind of like you you'd introduce an interpretability bottleneck when you train when you do further fine tuning the model with essays. And then you do that, after all the modules. And eventually
David Bau: you kind of like end up replacing the the model to a decompiled networks.
David Bau: Okay.
David Bau: yeah. So it's something like a vision. I'm not really sure. I I actually, I'll rephrase. I don't think Gemma Scopes was an attempt at this. Actually, okay, yeah.
David Bau: Gmo was kind of the 1st step towards this right? Yeah, kind of the 1st step. Yeah. Procedure. We want to try to find the features, the sparse features. Yeah. And then.
David Bau: if we have 2 essays. But Jamascope was also not trained like this, right? It was kind of like trained as
David Bau: like this normal essay losses. Right? You're right. So that's why I I was kind of like changing what I just said, Yeah, and are these like the final.
David Bau: the compile network available on? No, no, it is. It's just a vision paper. Yeah, it's like a I think it came from Lee Sharkey. If I'm not wrong. Oh, okay, this is applicable to language models as well. So the idea is to replace layers. Yeah. And we don't understand what an Mlp. Is doing, really. But
David Bau: we want to
David Bau: force the model to be in this last auto encoder basis. Yeah. So we had an Mlp before. But we train
David Bau: to reconstruct sparse autoencoder features from an earlier layer to a later layer skipping the Mrp, yeah. And in the end we have like a totally different network. Yeah, that's just composed. Yes, of these train skip connections
David Bau: there was actually a paper at new reps which I reviewed, and
David Bau: which kind of did like that? They tried to replace the mlps with auto encoders, and they they they mentioned some.
David Bau: They came up with some new architecture which they use
David Bau: to replace the Ml. Piece. That was the high level idea, I think, did it work?
David Bau: I think so. That's why I think I accepted it. I would need to check the details. But I that's what I could record.
David Bau: We covered. Yeah.
David Bau: wait, what's the advantage of doing that over? Just training a full model like a really.
David Bau: the sparse stuff is probably a lot more easily understood. Like training training is like what they did in the snaps paper. Oh, yeah, yeah, like, like, you read the
David Bau: toy models paper. And they're like, what's actually going on is it's a simulation of this like giant, sparse network. Yeah, in this double descent anthropic paper. I think they mentioned that they tried just like enforcing a scarcity penalty throughout network training. Yeah, it just doesn't work that well.
David Bau: But like training the full model towards like full performance, and then only afterwards trying to approximate.
David Bau: is what they are trying to do now.
David Bau: And you think it's just hard to optimize to
David Bau: this phone number.
David Bau: Yeah.
David Bau: Hmm.
David Bau: I don't know why.
David Bau: Yeah, one reason I can think of is that maybe when you're training this far, certain encoders on top of something already trained.
David Bau: Well, it kind of already knows the features. So it has a very strong.
David Bau: prior, very strong initialization. But I
David Bau: I actually don't have a good answer to that. It's kind of it should be beyond. Yeah.
David Bau: okay? And yeah.
David Bau: people have been doing tons of work with sparse encoders, finding different features for entities, like directions for sycopantic phrases, harmful behaviors in language models.
David Bau: Yeah. But I
David Bau: I honestly think that the most interesting paper it actually came from this lab. Probably I'm a little bit biased. Well, some of you are were authors of that paper.
David Bau: because this is my reason is that this is one of the very few papers that actually tries to do something causal that actually looks for causal evidences.
David Bau: Most of the papers I have seen. They kind of
David Bau: look for. Okay, this. So this for this kind of pattern, which can be
David Bau: correlated by this
David Bau: anyways. So this is the basic idea of sparse of encoders.
David Bau: But let's kind of step take a step back for a moment.
David Bau: So many of us are are familiar with sparse auto encoders not really sparse auto encoders
David Bau: from images. Right? You take an image, you you basically smash it down in a lower presentation. And then you try to back. Right?
David Bau: Okay.
David Bau: here in the text domain, what are we actually doing?
David Bau: So in the text domain, a text is comprised of a bunch of tokens
David Bau: for each of the tokens. You have a presentation.
David Bau: and what you are literally doing here is taking one of the columns of this representation and trying to do sparse auto encoder on top of that
David Bau: like.
David Bau: does that? Does that seem weird to anyone like, how does it even make sense?
David Bau: I mean, if anyone has any ideas. I have some ideas, but
David Bau: weird, in what sense? Like. Okay.
David Bau: So here in the image, I guess you have an eye
David Bau: which is kind of like this concept is kind of like spanned across
David Bau: multiple pixels, multiple columns
David Bau: in textual domain. Well, what about the features that appears
David Bau: for like a span of tokens.
David Bau: There is cell potential, right?
David Bau: Not on sparse of encoders, but the activations are formed based on attention. Mechanism.
David Bau: Yeah, I think the idea is you hope, that gets captured in the hidden state that you're trying to
David Bau: to decompose? I think that's the reasonable hope. Yeah, because things are moved. One token at a time.
David Bau: like maybe because it's autoregressive, it makes more sense than if it was just like
David Bau: that number. That was kind of also my thinking as well. Well, in auto, aggressive language models.
David Bau: Well, there is some incentive to capture concepts in a single token, especially
David Bau: if the model thinks that this concept is going to be useful multiple times in the future.
David Bau: like you can think of it like something like a
David Bau: something analogous to some eager processing. We do in automotive type of dynamic programming.
David Bau: But yeah, I mean.
David Bau: am I super convinced by this argument myself? Well.
David Bau: probably. No, I don't really know.
David Bau: And also well, this argument, it does not really.
David Bau: I'm not really sure if if it generalizes to masked language models.
David Bau: But maybe people don't really care about mass language models. Because we, I think we have verified that autoregressive language model. It just works. It just works better.
David Bau: And also, yeah.
David Bau: I want to investigate these 2 questions, how good are this essays? Are they capturing the data characteristics instead of model characteristics?
David Bau: And do they reliably find monosemantic features?
David Bau: And there are some other questions I add, but I'll not really get into. Okay.
David Bau: yeah.
David Bau: So 1st of all,
David Bau: we want to understand, does the essays, captures, data, characteristics versus model characteristics.
David Bau: So I have this simple setup. Assume that you have 2 models, m, 1 and M. 2. And you have 2 data sets. Ta and Tb.
David Bau: you'll basically train 4 essays. Be on every model and data set characteristics. And you'll try to measure how aligned those representations are.
David Bau: Basically here, each of the nodes you can think of as like representing one of the learned representations one, a is model one, and sae trained on model one on date with data set a
David Bau: and and you can think of the
David Bau: the length is basically
David Bau: is a proxy for a proxy, for how distant they are, basically. How the more aligned they are, the closer these nodes will be.
David Bau: And you want your essays to kind of look like this.
David Bau: If it is more aligned with the model, it should be like this, and if it is more aligned with the data set. It will be kind of look like this. But the data sets are kind of like, aligned.
David Bau: does that setup make sense?
David Bau: Okay.
David Bau: okay, so I I basically did that. I chose 2 models the llama, 3.2 1 billion, and the Oma 1 billion. One of the reasons I chose these models is that they are.
David Bau: well, if you just look at their model cards, they are almost exactly the same. They have the same number of players, same representation, size, same number of attention, heads and everything. These are my 2 data sets. And this is my evaluation data sets. I
David Bau: see if they see at different check different checkpoints.
David Bau: Yeah, that's how I
David Bau: get the essayes.
David Bau: And how do I measure the alignment between learned spaces?
David Bau: I collect a bunch of se latent mixtures. Basically, this is the, this is the Se representation, the M thing I showed. Okay.
David Bau: for N documents. It is roughly yeah. N, times. Rd, I build a kernel of cosine similarity of these representations, and then.
David Bau: sure sorry is d the sae number of hidden dimensions. Because D is usually the original. Oh, sorry. Yeah. D, is the C number of hidden dimensions? Yeah, okay, could you explain what? What is late dimension.
David Bau: Okay? So
David Bau: the lading mixture is basically this.
David Bau: oh.
David Bau: this
David Bau: is that correct? Yeah, okay.
David Bau: And there's a like, the columns of the decoder matrix they are called se latents.
David Bau: Okay? I'm sorry. Why are the columns called? Wouldn't it be better to call them like sae basis or something. Because why would they be the latent? It's just it's just something people choose to call them.
David Bau: Okay. There was a way the
David Bau: but I'm familiar with the literature is that you like call individual dimensions of the hidden vector latence. And then you you'd call like
David Bau: the
David Bau: column of the decoder matrix corresponding to a latent, that decoder. Vector can I also add one another thing to this question?
David Bau: if you call them basis, you kind of
David Bau: expect that they'll be orthogonal.
David Bau: But how you train the
David Bau: decoder matrix. At least there is no automatic constraint.
David Bau: I mean, you can think of the
David Bau: that's the fact that you necessarily think that a basis has to be worked on. But you'd you'd assume that they come up at most. E, yeah, at most. E, linearly independent.
David Bau: Okay? Yeah. Oh, oh, and they might not be linearly independent. Yeah, they're most definitely often, not. Linearly independent. Sorry for stopping you. I did not let you finish. I feel like I already finished when I hear, like sae latence. I always think of individual entries of the of the activation, the hidden activations
David Bau: oh, not the latent directions.
David Bau: No, these. These would be like the decoder vectors.
David Bau: Oh, okay, okay, okay.
David Bau: But it's like some. Yeah, they're closely related.
David Bau: Okay.
David Bau: okay.
David Bau: okay? Hmm.
David Bau: okay, okay, moving on once you have this kernel.
David Bau: I
David Bau: want to use an alignment metric to basically understand
David Bau: how these 2 kernels are similar to each other.
David Bau: What I ended up choosing is Svcca. Basically, the idea is that, given that 2 documents are mapped similarly or differently in the same in the sa latent space. I want to understand if they are learning the same decomposition of concepts for the same document.
David Bau: Does that make sense.
David Bau: Okay.
David Bau: okay?
David Bau: So I'm choosing Svcca. But there are different ways of getting that. I think there is one called Cka and Mnn. I mean, we'll not get into that.
David Bau: Okay, this is just one of the techniques to do that. You have like a high level, like, what is Svcca? And how does it work like, just from abroad. I actually forgot about that. But yeah, but this basically what they literally do is that given to kernels? They're supposed to just give you a score like how aligned these 2 kernels are right like. I can add another thing. Think of them as post and similarity. But for matrices.
David Bau: Sure.
David Bau: Yeah, I just, I just know the full form of this term, single and vector.
David Bau: canonical correlation analysis.
David Bau: But I don't really know. Again, I forgot about it.
David Bau: Yeah.
David Bau: okay, so yeah, I,
David Bau: I basically did that. And huh.
David Bau: and these are the results. Sorry the X-axis is training steps.
David Bau: Next steps is checkpoint steps. Yeah. The training steps
David Bau: and the y-axis is the alignment score.
David Bau: The again the more aligned they are, the smaller the distance between the edges are.
David Bau: and you can see that well.
David Bau: it seems like it is always more aligned with the model.
David Bau: It starts. It starts dipping a little bit. But as you increase the data set sizes, it starts getting more and more learning models to presentations. But but yeah, and the data set, it does not really improve.
David Bau: So I'm sorry. I just have a couple of questions about this. So is this.
David Bau: you're starting with pre-trained right? Yeah, both of them, and then this is, and this is full fine tuning, or this is like Laura.
David Bau: This is just sae.
David Bau: Oh, oh, I'm just learning A/C, but right? But but you you trained it on tiny stories and on Wikipedia. Right? The 2 models.
David Bau: Just the sa latency. Oh, just the sa. Okay, okay, sorry. Yeah.
David Bau: can you? I mean, I I don't think it makes sense to do Laura and essays, yeah.
David Bau: okay.
David Bau: yeah.
David Bau: Sorry. I already forgot which is lower like lower, better, and then also, which sides are model. So 1 1 stands for the the 1st term stands for the model, and the second stands for the data set. Yeah, I'll I'll stay a little bit on this slide. Okay? Yeah. Because because I was a little bit surprised with this experiment. Actually, so, yeah.
David Bau: so
David Bau: these 2 are. These 2 are where the model is the same. But the data set is the different.
David Bau: And these 2 are where the model is different. But the data set is the same. Okay, and
David Bau: the higher score here means that
David Bau: the distance between them is actually lower
David Bau: and the lower score between the 8 is means that the edge between them is actually larger. So what I'm trying to say here is that
David Bau: the essay is actually learning.
David Bau: or at least this experiment suggests
David Bau: this setup suggests is that the essay is actually learning the model representations.
David Bau: which is nice, which is what we guess I, we kind of want.
David Bau: And we also, yes.
David Bau: maybe you can complete that. Yeah. And we also kind of see that
David Bau: it, it starts dipping a little bit. Well, it starts with random. So this is kind of like similar to whatever the models, models cosine similarities. It is similar to the cosine similarity learned in the models. Representation space.
David Bau: Okay, it starts dipping.
David Bau: So what what I'm trying to say is that
David Bau: with initial steps, it starts learning. It starts to pick up like more schoolious correlations in the data set.
David Bau: But as you train it longer as the it it starts picking up the models representations.
David Bau: That's what I'm trying to say.
David Bau: yeah.
David Bau: Yeah. And why? I
David Bau: why, I told that I was a little bit surprised with this experiment. Is that?
David Bau: Well, we start our investigation with the question, and we design experiments.
David Bau: Try to be as fair as possible. But we have some hunch what you're looking for right?
David Bau: I was actually expecting for a negative result here.
David Bau: And well, I've
David Bau: found a positive result, then, which is which is great, I guess.
David Bau: Okay, yeah. Do you plan to repeat this with different models and different different data sets, just to verify that this is a general phenomenon? I don't know. Do you think it'll be interesting.
David Bau: I mean, it would at least
David Bau: give additional evidence to the idea that this, you know, the sparse auto recorders are actually learning
David Bau: a sparse, monosemantic representation across different sort of models that have similar characteristics. I I did not say anything about monosemantic yet. Yeah. So I still have some doubts. But I am just telling that. Okay, what is the representation being learned?
David Bau: Okay? So I I like the setup. So you know, there's this fundamental question that's underneath all interpretability methods, which is, are you?
David Bau: Is your method, especially all these methods that have a lot of training.
David Bau: Right? So are you learning, are you? Are you revealing something that the model already knows?
David Bau: Or are you learning something new
David Bau: that the model didn't know? And and you know, you're just developing, developing some new model to solve some new problem that might be unrelated to the underlying thing that you're trying to study
David Bau: right? And so, especially if you spend.
David Bau: you know, thousands of dollars of electricity training something
David Bau: you start to worry that maybe you're, you know, learn, you know, like creating some new
David Bau: thing instead of studying the old thing
David Bau: and and so so here you created the new thing.
David Bau: you know, with 2 different pieces of data.
David Bau: And and they ended up being kind of the same right which suggests that, like the task that you set out
David Bau: to train these saes on.
David Bau: you know, you couldn't really
David Bau: get the essays to do something different by changing the task a lot. Yes, right? Yes, that they. They were like stable. They were like giving you a window into the model, even if you're training them very differently. Yes.
David Bau: right?
David Bau: Yeah, which is kind of interesting. Yeah, this is very interesting. Actually, yeah.
David Bau: And yeah, that is kind of why, I was a little bit surprised. And
David Bau: yeah.
David Bau: yeah, so people, people actually.
David Bau: okay, before you go to the next.
David Bau: So what what is your explanation for? Why the the alignment or the supply drops for the the top one and the bottom one.
David Bau: because, when so I made sure to initialize all the essays like, similarly like this is basically a linear, a random milia projection
David Bau: and this is steps, or this is trained on a 1 k steps. So the models is still kind of random. You can.
David Bau: You can literally see that they are kind of kind of the same
David Bau: right?
David Bau: So my explanation, why it starts dropping is that this is kind of equivalent to
David Bau: what the model has, what the language model has actually learned. This is kind of equivalent to taking the embedding treating what? Treating the last token or cheating?
David Bau: yeah, treating basically, the average of the document as the embedding of the document.
David Bau: And this this has not really learned anything. But as the essay starts learning, this starts learning characteristics of the data set. But then they starts going. They start going up
David Bau: because they are again like this is kind of like a generalization phenomenon like they start learning characteristics of the model. Yeah, I'm asking about the top and the bottom one.
David Bau: Oh, the top and the bottom. The data set is same in those cases, the architect, the model is different.
David Bau: The model is different. Yeah, they're they are. They're they're learning. They're trying to like.
David Bau: become different, based on yeah, different representation based on the model.
David Bau: What, what layer is you comparing here? So for both of them, I take the 8th layer, which is, which is exactly at the middle. They have 16 layers.
David Bau: both of these products.
David Bau: Yeah.
David Bau: I mean.
David Bau: I'm wondering why
David Bau: that
David Bau: your
David Bau: switching between model experiments at the top and the bottom
David Bau: are going
David Bau: down. If you think that
David Bau: in the 1st phase.
David Bau: especially in the 1st place, it's mostly picking up on data specific
David Bau: properties.
David Bau: This kind of depends on what the model has actually learned. Right? If the model has learned different concepts, you are going to end up with different kinds of mixtures.
David Bau: Although
David Bau: does that make sense? This is kind of my hacky reasoning, I, yeah, yeah.
David Bau: now, these are both
David Bau: natural language models, yeah, so you didn't. You didn't like it would have been
David Bau: yeah, you you might expect this to go further down if one of them was like a code Lm or something like that. Yes.
David Bau: yes, right.
David Bau: But I
David Bau: yes.
David Bau: yes, because the concepts in the code space will be something completely different.
David Bau: maybe. But I guess
David Bau: even in the left, in, even in the data mixture for language models, there is a there is a substantial amount of vote.
David Bau: sure, often, sure. But you'd expect, like the
David Bau: the balance, the number of features are to be very different in balance. Right? Yeah.
David Bau: yeah, I'm a bit naive to this. But if you had 2 models
David Bau: that were trained for different tasks, but they end up somewhat aligned. Would that mean that you could possibly use
David Bau: one sort of in a transfer learning sense for the other task, or
David Bau: like, if if the models were trained to do different things, but they end up having sort of a a very close distance. If they're very aligned.
David Bau: would it be
David Bau: okay, or would it? They perform similarly on each other's tasks?
David Bau: I I don't know, like there has been a weird line of work which I'm not really following. The models teaching type of work which learns, which tries to learn or connect between representations of different models with app and transformations. You can try to look at them a little. But I I don't have a
David Bau: good answer for you.
David Bau: And but yeah, this is.
David Bau: yeah. Sorry for saying that weird. I mean, I just this is my just my personal opinion. Yeah, I mean, we look into that and find that it feels like a blue sky sort of situation of like, hey, this is very interesting results. What could this possibly? Yeah, yeah.
David Bau: yeah.
David Bau: It's okay for you to call it weird. It's weird and cool. Yeah, okay.
David Bau: so yeah, I
David Bau: what I'm trying to show is this slide is that I start with kind of similar similar
David Bau: similar initializations for the for the auto encoders, and they kind of start at the same thing. But as I train them more
David Bau: twin, like kernel space for the models. This starts looking similar. These are for different data. This is for the same model, but for different data set. And this is for the for different model, but for the same data set. And you can see that the kernel space it starts looking similar for the model which I was trying to show it with us. How does this visualization work?
David Bau: I literally just visualize the kernel of of you know with what on the X-axis and what on the Y-axis? Both document indices.
David Bau: Oh, I see. So how? So this is.
David Bau: How many documents do you have on. Okay, I had 8,000 documents. But yeah, if you just project 8,000, you don't really see anything. So I'm selecting the top 1,000. 0, that's that's what I'm saying
David Bau: so each scalar value is that score that you had before. Oh, it's just cosine similarity between the sa representations between the sa representations of the documents.
David Bau: Okay, did you have any question, why is it that
David Bau: this may be a stupid question, why does the cosine similarity of these 2 representations mean that they're like similar, semantically
David Bau: like, if you, you basically try to see if the model is trying to cluster
David Bau: things with the same feature, I guess.
David Bau: What if they they learn the same
David Bau: like? Couldn't you have 2 saes that both learn the same
David Bau: kernel? But they're rotated versions of each other. So you learn you learn Kernel X, you run it through some rotation matrix. And then wouldn't the cosine similarity? No.
David Bau: because when you finish your question so that everybody understands. Oh, yeah, yeah, sure. So, so, yeah, so like, so like you, you still learn the same neighborhood relationships. Or at least close to the same. It should be the same kernel right? Right? If you rotate it, it should be the same kernel. But if you take cosine similarity between the 1st kernel and the second kernel. Then they won't be. They won't be somewhere. Okay, I'm not taking the okay. They will be similar.
David Bau: because
David Bau: here each of these representations, they are actually cosine similarity of the same se basis. Right? So if it has learned the same concepts.
David Bau: it will cluster same documents as high cosine similarity
David Bau: even that actually stands, even if the kernel is rotated or the other setting
David Bau: for the other sake.
David Bau: Does that make sense.
David Bau: Are you talking about cosine similarity for each entry or comparing the kernels? Yeah, I thought you were comparing the kernels here. Yeah, yeah, we are comparing kernels. But how we get the kernel is each of the cells in this kernel is a cosine similarity
David Bau: between 2 different documents. Yeah, does that? Okay? Oh, so we're just visualizing the kernels. Then we're not visualizing the cosine similarity between pairs of
David Bau: of CC staff. That's the same thing. It's the same thing.
David Bau: So okay, so you have model one, you have. Sa, one, it learns kernel one, you have. Sa, 2. It learns kernel 2. Yeah, you have document one in the 1st thing and document one in the second thing is this the cosine similarity between document one and the 1st thing and document one in the second or document one and one in the 1st one and 2 in the 1st thing. Oh, okay, yeah. So I thought I thought it was. It was between the 2 kernels. Not not between. Yeah, yeah, I probably should have spent some time on the setting to explain that a little bit more, I see. But does that make sense to everyone now?
David Bau: Yeah, yeah, thank you for this question. I mean, I guess I guess it's possible that they're clustering them the same, but for different reasons.
David Bau: But for so many documents it'd be unlikely it'd be unlikely. Okay, yes.
David Bau: yeah. Okay.
David Bau: So yeah. So is this. So this experiment is clear to your phone, right?
David Bau: Okay, cool. And so so on the right.
David Bau: Let's see here.
David Bau: So so these are the 4.
David Bau: These are the 4 that you trained? Yeah.
David Bau: And you can. You can see this, the clear.
David Bau: the clear invariance to the data?
David Bau: Yes.
David Bau: yes, you can see very similar banding patterns
David Bau: in the same model.
David Bau: I'm not sure if I understand this comparison, because each column or each row in your kernel corresponds to a document from your data set right?
David Bau: No.
David Bau: In my kernel, each of the cells correspond to 2 different documents. Oh, yeah, yeah. Yeah. So the individual documents
David Bau: are just different.
David Bau: oh, but it's the same. 2,000 documents. Oh, your evaluation set is the same. That's right. So the matrix on the right
David Bau: uses 2,000 documents and the matrix to the left uses the same 2,000. Where did you draw your 2,000 documents from? Are they from Wikipedia, from territory? There's some completely different data sets, movie reviews. Yeah.
David Bau: yeah. So evaluate the Wikipedia and the tiny stories essays
David Bau: on the same data set the movie movies. Yes. Okay.
David Bau: okay.
David Bau: yeah.
David Bau: okay.
David Bau: can I move on? So do you think that they should have
David Bau: done this as a mass project?
David Bau: Do you think this is interesting enough?
David Bau: That's a great stop. Because, like, you know, like Neil's feedback was, this is not interesting as a math project. Wait, I think it's very interesting. Yes.
David Bau: yeah, I know exactly. That's what I think. I think it's a great setup. Maybe he wants to go write his own.
David Bau: I don't keep working on this man. I mean, one of the reasons I actually presented this work is to get this feedback. Do you guys
David Bau: think this is interesting? Yeah? Yeah? And and like, so there's a couple questions I would have. So back up to your your graph.
David Bau: so thick
David Bau: does this keep on going down lower? You stop at 2 202 million, whatever iterations. 200 million. Yeah, 200 million. Sorry. Sorry. Yeah. 200 million documents. These things are so expensive to train. Yeah. So now we don't have access to the case cluster. I mean, we're gonna have access. We're gonna lose access after. Oh, what did they extend? Oh, like.
David Bau: hey, Janon.
David Bau: reduce the restrictions. I'm running jobs there now. It's it's weird. But it works. Yeah. So they they had some weird restrictions. Only 20. So at some time in like like a couple of weeks ago. Only 2 20 nodes are operational. But you know this is like the thing that's killing the case cluster is these saes like everybody wants to train like if we just did regular like
David Bau: inference based interpretability when there'd be plenty of gpu for everybody. Instead, everybody wants to be training these things. And then you have these advisors like me saying, what happens if you train more than yeah, because I was training, a lot of essays and a lot of different data sets. Yeah, so you didn't save more checkpoints than this, did you?
David Bau: I have more checkpoints. But another. Really, time. Consuming thing is to get this representations. Also
David Bau: 8,000 yeah.
David Bau: Yeah. Well, you don't have to do 8, because you only grab
David Bau: 2. But
David Bau: sorry. What? Oh, yeah, I only draft 2, but I'm and your smcca. You could do it on, too.
David Bau: I mean, you can see visually. So so this score I'm presented here is based on 8,000 8,000. No, no, but you could do it based on 2, and it'd probably be the same number?
David Bau: Right?
David Bau: Probably. Yeah, and be 4 times cheaper.
David Bau: Yeah, probably. Yeah, I know you can have different events.
David Bau: Yeah.
David Bau: yeah, what if they compare these representations like between pairs of kernels.
David Bau: Cause I don't think you could do this cosine similarity thing anymore. No, but
David Bau: like what some some stuff that I've seen uses like a neighborhood similarity. Where? Yeah, yeah, where? Like, you look at each point, and you look at which points are in its k nearest neighbors. Yeah. And then you look at how similar you look at the union over. Yeah, I actually already have that. Okay, great. And I also looked at the scores. But I did not present them. It's consistent with this story. Yeah, it is consistent with this story. Yeah, that's nice.
David Bau: Okay?
David Bau: Maybe maybe essays work. After all, yeah, people, people probably know what they're doing. Actually, yeah, maybe they didn't. And now you're giving them the right fundament.
David Bau: Okay? Someone had a question. I had a question.
David Bau: who?
David Bau: Yeah, yeah, sure. So like, we are using essays to learn the concepts that each of the layers learn right?
David Bau: Yes, and also like do a monosemantic analysis like if there are monosemanticity in each of the layers, I did not talk about monosemantic. The question is, the question is, we do all of this analysis using essays. So yeah,
David Bau: if we have one model.
David Bau: and we are confident that essays are learning what we think, it is learning. Then we take a model, and it's a genetic model.
David Bau: and we create an artificial layer set from that model.
David Bau: And we use the data to train this exact same model. And then we perform this analysis. Then we perform another set of essays on that new model that is trained on the data that has been generated by
David Bau: the 1st model? Do we expect the same features to be learned, and the same directions to be in the essays?
David Bau: What's your setup again. So you have 2 different models trained on the same data. We have this, we have. The. The architecture is the same. There is one model. There is the 1st model trained on data set A,
David Bau: and we, we train essays, learn the directions and features, and then we use the model A
David Bau: to generate artificial artificial data. We give some context. And so you want to, you want to. You want to distill the model into another model. Yeah? And do we expect the same features to be in both of the models?
David Bau: My
David Bau: hunt
David Bau: would be
David Bau: my hunch is I? I think this is a reasonable research question. So distillation is known to be really effective at allowing you to shrink the model parameters.
David Bau: In a way that just continuing training with the existing model parameters cannot
David Bau: and so that would suggest that distillation leads to different solutions that are not accessible. But here he's not suggesting distillation. It is not really video, not a smaller model. So there's like one Gpt, for example, one Gpt, that was, that is, but you start with a random initialized one for the second go right?
David Bau: Yeah. So yeah, that is not framed. The second model is not, I guess, in distillation. You want the student model to be often smaller than the bigger model. Yeah. So it doesn't matter. But like, either way, if you if you train on like distilled data
David Bau: like it can be smaller.
David Bau: and which suggests that the distillation training
David Bau: it's a different solution.
David Bau: Yeah, it can lead to different solutions. At least, at least that's what the literature suggests to me, yeah, probably, like,
David Bau: learning a representation from the input and actually generating the output are 2 different tasks.
David Bau: And here you are only
David Bau: learning the representation
David Bau: from what other model has generated.
David Bau: Yeah, if your model A has really understood the concepts.
David Bau: then it should generate right. But I I haven't seen any interpretability research on this setting, it would, I think that would be an interesting one to study right?
David Bau: yeah. Or maybe even how far you could generalize like, if you could do this over and over and over again, see how far you can distill.
David Bau: Yeah, exactly. So that was the limit. So I was about to go there. So yeah, that was the limit. If the model collapses. So that's my ultimate question.
David Bau: If like, eventually, if we just train on and on with artificial data over and over again would the model collapse and fail to recognize any of the features that was learned in the 1st model you should read Jb. Grill 2020 Bootstrap, your own latent.
David Bau: which uses this procedure
David Bau: and and they have, you know, guidelines on how to make the call. Well.
David Bau: bootstrapping your own Lane Bootstrap, your own late in 2020.
David Bau: Yeah. Should I go on? It's 11 for? Oh, yeah. What should I do? Go on.
David Bau: add more snacks. They'll take a snack break. Okay? Nice.
David Bau: Okay.
David Bau: okay. So
David Bau: this is another question, well, why do we care about saes?
David Bau: Well, the promise of essay is
David Bau: is to actually break the representation space into monosemantic and interpretable neurons.
David Bau: Does it actually do? That is l, 1. Sparsity loss enough
David Bau: to basically bake in interpretability. And basically another way of asking this question is, Rsa is the right unit for feature analysis.
David Bau: Okay?
David Bau: So there is a line of work investigating this one is feature splitting.
David Bau: So there are here like 2 scenarios for the mixtures.
David Bau: Let's say it has learned one direction, for something like a feature starts with a
David Bau: and it has learned 2 directions.
David Bau: starts with a without the word avocado, and only activate for avocado.
David Bau: which one of the settings. Do you think sparse? It will favor
David Bau: the second setting right.
David Bau: because it will allow you to get to more. Get to a reduced sparsity, more zeros, more zeros, literally.
David Bau: So this is called feature splitting.
David Bau: Sometimes we want the features to split basically starts with an L, okay, we've pro. Why was the second setting. Better
David Bau: when you said second setting, you mean in the animation, it's not clear that it's better, though right it it is.
David Bau: It is not better. But this is something sparsity laws will actually favor. It's more sparse. It's just not necessarily better from a performance perspective. So I still don't understand why it would be more, why, the right thing would be more sparse. You're like, it's the same sparsity. Yeah, right? Because
David Bau: for both avocado and a you have a single neuron.
David Bau: Right? So isn't it the same sparsity?
David Bau: So in the end, like the ultimate thing, this is just like a single variable. But as soon as you have multiple variables like red ball, and then green green ball and green square, whatever? Then.
David Bau: in absolutely, he has a slide for that. That's what I had like feature entangle. But yeah, even even in this setting so wait, wait. So let me say so. Yeah. So on the on the left thing. So you're you get a bunch of data. Things start with a so then so then, and let's say, like
David Bau: everything else, is always 0. There's no other features in the universe. Okay, so so then, 1% of the time that thing will be 0 0 1 and then the rest zeros. And so. So now
David Bau: now let's say of that, 1%, half of them are yeah. Half of them are are avocados, and the other half are other awards. So then in that case,
David Bau: you'll still have the 1% thing on the left. So that's yeah. And then on the right.
David Bau: point 5% of the time you'll have 0 0 1, and then the rest zeros and another point 5% of the time you'll have a 1 in that purple thing. Right? So
David Bau: yeah, I understand the argument you're making it up is still the same. l, 1, yeah. Yeah. You kind of end up with the same l. 1. You would go into a different setting. We have red and green balls and red and green squares, and you could like imagine learning a feature
David Bau: separately, for, like the square property, or you can go to the next slide here. So let's you can talk through this if you like.
David Bau: yeah.
David Bau: I'm just. I'm just trying to remember when I put this slide, what I was thinking.
David Bau: So if they're mutually exclusive, that it's the same sparsity right? If they're mutually exclusive, then it is kind of the same sparsity. Yes.
David Bau: so. But if you had like, just start the day.
David Bau: So you you got rid of the without avocado, and then also another one that only activates on avocado
David Bau: but then but then the left would be better. Yeah, then this is what you want. Yeah, exactly. Yeah. Yeah. Because then the one on the right would sometimes have 2 ones in it instead of a single one, maybe forget about the sparsity loss. But this is something
David Bau: that has been known to happen
David Bau: when you scale the se dimensions, basically if you and sometimes it splits in human interpretable directions, like starts with an L, it splits and starts with an uppercase L, and starts in lowercase. L. There has been some words. Training saes on se latents
David Bau: and something like it, kind of like splits.
David Bau: but
David Bau: sometimes often it does not really. It just kind of like splits randomly. It does not really give you.
David Bau: give you some. It does not really split in some interpretable direction. And that's kind of like a problem I see. So like, so you would count starts with an L, with exceptions, is not very interpretable, like your avocado example. Yeah, yeah, that will be a bad example. Yeah, I mean that that happens. That happens. But it's it's undesirable.
David Bau: I actually don't know. I mean.
David Bau: I think something desirable, like, if you go to a kindergartner. And you say, this is, these are words that start without. But yeah, but not laser.
David Bau: But the thing is that well, yeah, the sparsity loss. I I'm just trying to make an argument here is that is the sparsity loss enough
David Bau: to to basically take in these human, interpretable directions.
David Bau: basic and the sparsity laws, it has no reason to.
David Bau: So okay, wait. So what if, instead of the avocado thing, it was the color red. And then, on the other side. It was like a bunch of different things for each like type of red.
David Bau: like, right? That's more what I imagine feature splitting to be where where you have some general thing. But then, you know, when you have more space for a representation, yeah.
David Bau: the incentive to like have this spitting is to spitting is mostly observed when training auto encoders with different widths or number of letters, so you can retain better reconstruction. You can get better reconstruction loss if you have like a decoder, vector that's only corresponding to like one very specific setting, maybe. Starting from like all words to a and like just points to the general a cluster. But then, if you, if you are allowed.
David Bau: if you have enough latence, to encode all these like individual notions, then you just have better reconstruction loss. So I guess that's the main
David Bau: reason why you see feature splitting arise with like different sae widths.
David Bau: Hmm, yeah.
David Bau: you also have. You are also working with a benchmark. I know that right. Are you guys also investigating this
David Bau: like, do you? Do you have a score for this? I mean, if I just give you 2 essays.
David Bau: can you just give me a score like which one is
David Bau: so features building is like a very subjective thing of like, I want to have the avocados as separate. But what's the like? The right level of granularity? You can. You can imagine that like it's very basic. And then it's like interpretable. And then it's too like narrow concepts. So what we do is like coming up with a list of
David Bau: concepts that we would like to find encoded in a single
David Bau: latent, and that that should be like the right level of granularity. And then we just probe.
David Bau: We check whether we find individual latents that correspond to concepts we came up with.
David Bau: Can you give me a score like f 1 score class using the se non-zero activations
David Bau: for specific Sunday. I'm asking this question, okay, it'll it'll come a little bit later. Okay, yeah. Okay, okay, this is the question. Yeah, ken already kind of told. Let's
David Bau: you can also think of like feature entanglement
David Bau: like, let's say you have 2 different. You have 4 different things. You have blue circles, blue squares.
David Bau: red circle and red squares.
David Bau: So the thing on the right thing on the left is kind of like desirable. The model learns different concepts for different different
David Bau: different directions for different things.
David Bau: But the thing on the right is not desirable. Well, 1st of all this is not atomic or deductible, but
David Bau: the sparsity laws will
David Bau: actually favored the right one
David Bau: right? Because now, if you have a blue circle now, it has to turn on
David Bau: this neuron here and this neuron here. But now it can get away by turning just one neuron.
David Bau: But this is not really what you want.
David Bau: Okay.
David Bau: so so, but a 2 by 2 is a little bit of a degenerate case. Yeah, you can as soon as you have 3 by 3. Yeah, yeah. Now, now, you have an advantage to do it like this. Right? Yeah.
David Bau: But
David Bau: yeah, I mean, people have looked into this in in like actual trained essays, and they found this to happen. I see
David Bau: I'll see it
David Bau: so
David Bau: One solution to address this is that you proposed it. But I had a different way of putting it here. So trying to.
David Bau: yeah. Yeah. So in some sense, this sae is
David Bau: are kind of similar to how you signal how you process audio signals with frts, so ffts are supposed to give you like this spectral components in the frequency domain. And is, that is what you
David Bau: kind of wish, what saes are doing like
David Bau: like breaking a representation into its individual concept components. Right? And so
David Bau: say, what? Oh, I was just talking to myself, it's signal separation. Basically. Yeah.
David Bau: Yeah. So fft is, has this really nice properties? It kind of it is kind of linear.
David Bau: yeah. So can we just incorporate this? Wait. Sorry. Does it? Does it? Also have the the scaling that has the fot of a 1 plus a 2 is fot of a 1 plus fat of a 2, but if you scale it has the scaling, it has the scaling one too cool.
David Bau: So
David Bau: you can just add it with this kind of loss like that is just another term in the loss of loss function. Right?
David Bau: I already implemented it?
David Bau: That's why I was asking you these questions. I already have the essays with this loss added, trained with this loss. Yeah. Trained it. Just just another loss term, basically, yeah, compositionality loss. Just just I call it entanglement. Okay, yeah.
David Bau: yeah, so this is, you want to minimize the distance between the thing on the left and the thing on the right, basically just just assuming that the space, the if you just take 2 random documents, from your data set, they will have very. They will be very different, and we'll have nothing to do with each other. So you kind of like, do this. Oh, I'm also not adding another thing. Sorry
David Bau: so another disentanglement loss, I make sure, is that they're also orthogonal. This is not. This is, this is a part of the loss. This is not
David Bau: this is not the whole thing. Yeah.
David Bau: does that make sense? What is the second thing I do this is okay. Okay. This is one thing I do.
David Bau: And I also make sure that
David Bau: what actual metric, what? What? What are you minimizing here?
David Bau: So minimizing 2 things. One thing is the Msc. Between left and right.
David Bau: I also make sure that this representation and this representation they are thrown up.
David Bau: If you and and h. 1 and H. 2 are drawn out
David Bau: randomly. So if you have, if you have 2 random documents, they should tend to be orthogonal. Yes. But if you add the thing, they should tend to be the same. Yes.
David Bau: yes. Yes.
David Bau: Yeah.
David Bau: Someone.
David Bau: Okay.
David Bau: So yeah, I have another question from his boss.
David Bau: so it's half of a contrastive loss. So like, so you. So you've got this classical
David Bau: contrastive representation, learning.
David Bau: regime. And the thing that you're missing is.
David Bau: if you have 2 unrelated documents, they should be different. Right? This is a nice thing. But then, if you have
David Bau: 2 different things that are supposed to be related, then they should be more similar. So, for example.
David Bau: if you have the 1st half and the second half of the same document, or something like that. Then they should be more similar.
David Bau: Right? And then that would be a classic like, contrastive lots. Yeah, right? Yeah, yeah. Yes. And also another thing.
David Bau: well, it might not be okay. So so it goes back to one of the questions you submit it to Nurex or something. Then the reviewer asked, well, what happened to that term? Yeah. And and and if I was the reviewer, I'll I'll ask another question that if so, there are some concepts you kind of expect to be present. Let's say there are 2.
David Bau: There is an English feature like 2 documents. They should have the English feature, and I am actually penalizing against capturing that as well with this construct contrast laws right? But if you have the other term, it might neutralize that right? So, yeah, there is a there is a tension between different losses which you
David Bau: hopefully compensate for each other. Yes, yeah.
David Bau: Yeah.
David Bau: I'm unsure about how you get from the frequency premise to the simple, like editivity of those
David Bau: the hidden representations. Could you walk me through that once more which one you're motivated by
David Bau: really having the frequency of like 2 representations, a 1 and a 2, where a 1 and a 2 are single, sae ladens. Is that correct?
David Bau: these are actually, properties of Ss
David Bau: team
David Bau: so a 1 and a 2 these are just simple audio signals and fft of a 1 is basically
David Bau: basically the modes of the of that signal right? So in, when when mapping that to the sae domain would be.
David Bau: we would we would treat a 1 and a 2 as
David Bau: hopefully concepts. But, like the the thing we were measuring are activations of individual latence. Right? No. a 1 and a 2 are actually equivalent to the representations
David Bau: fft of a 1
David Bau: is actually developed to the m. 1 and M. 2.
David Bau: I'm actually just realizing that I am changing the notation here. I should not have done that.
David Bau: Yeah, basically, this is so your fast fuel transform will be like the analogy of like my essay. In the 1st place. Okay, yeah.
David Bau: does that mean
David Bau: easy cents? Probably.
David Bau: Okay, this is a very interesting idea. And I got the high pushback
David Bau: from colleagues that they say, Okay, adding to like very different representation. h, 1 and H. 2. Because I I asked him to evaluate this also. Yeah. Yeah. But what happened?
David Bau: And they were like adding representations. There will be so many interferences. Yeah, so we can't really trust
David Bau: this hidden representations like added together. But that's what the ft is for is to separate all the signal. But I guess that's all you can just verify. Right? I mean, we can. We can. Just we can just see which one which one works better.
David Bau: That's what whole this whole
David Bau: se, or this whole machine learning literature is, I guess. Sometimes sometimes principle thing don't really work. Sometimes random thing just ends up working. So
David Bau: yeah, yeah, so yeah, so you have something else to evaluate.
David Bau: like, yeah, I'll I'll yeah, can. I just give you 2 essays
David Bau: like trained on the same model, on the same data. Set one without this loss, one with this loss.
David Bau: And just you'll just
David Bau: tell me, give me a score or something.
David Bau: So the the thing that I looked into was
David Bau: 2 essays trained on different with different architectures, with different hyper parameters. Yeah, that both don't know about this loss. But
David Bau: if we evaluate with the on this property, does one do better than the other, and it does show big differences like top K, but in terms of what? And
David Bau: of
David Bau: disentanglement.
David Bau: Yeah, the composition challenge, yeah.
David Bau: in terms of just wanna make sure. So what better is in this context
David Bau: to be specific? I'm looking at precision. Recall of
David Bau: the
David Bau: features that we get when encoding the sum of both representations
David Bau: and the union of
David Bau: features or latent activations that we get from encoding both individual representations. I mentioned this loss mainly deals with the union of representation and seeing sort of the splitting. So I actually did not encode the other part of that loss as well. But it has the loss has a basically 2 terms.
David Bau: One is this, and the other is just checks for 2 and 3. Basically.
David Bau: the intersection, basically the multiplication of those 2
David Bau: should be 0, yeah. And seeing how close to 0 it is effectively.
David Bau: okay.
David Bau: interesting.
David Bau: Yeah.
David Bau: Yeah. So do you have any of your own evaluations of this?
David Bau: Not yet. Do you have any of your own
David Bau: by hand inspection of it? Oh, I was not expecting this slide. Okay, yeah. I mean, have you looked at them. Do they avoid splitting, and things like that?
David Bau: Not yet
David Bau: not yet. So this is, this is kind of very fresh, and I finished training the essays like, like yesterday, okay, yeah. So you don't know if you have LL. With exceptions, I I was just being a little bit lazy. I just, I just thought that I'll just give Ken this to Representative.
David Bau: Bring it on. Okay, great.
David Bau: So with this loss, you do not include the sparsity one right. The L. Word, no, everything is still there. Or it's just another additional time. Yeah, to disincentivize entanglement, to try to get towards monosemanticity. Right?
David Bau: That's the hope. Okay, that's the hope. Yeah.
David Bau: I guess. Just practically, how do you have any idea how to weight that?
David Bau: Oh, sorry. What? How do you know how to how to adjust this hyper parameter to section.
David Bau: How do I adjust that? Like you have a bunch of different loss terms like, how do you weight each one you you just you just look at this raw values
David Bau: just just for different training steps.
David Bau: And then you try to adjust the coefficient so that it's not a good answer. Yeah, yeah. All the. I don't think there is a good answer for any kind of hyperparameter tuning that you do. You just do it.
David Bau: The question was down to what do I want from my essay? Do I want it to like, be a perfect reconstructor above my initial like? Input, then you have this reconstruction loss. If you have sparsity, you have something more like
David Bau: you have, like an Llm. Looking at the features, interpreting each one and then giving like an interpretability score or something. You can probably also do something like linear scheduling like like you can. You can also linear schedule on the hyperparameters. Basically. First, st you try to minimize the reconstruction loss for a while, and then one. It has achieved a good reconstruction loss. You kind of switch to the other 2.
David Bau: Maybe
David Bau: maybe I mean, I you'll find out. So what you'll find out when you go about. Yeah, that's that's that's 1 of my question was that, do I want to find out what is research, if not finding out?
David Bau: But what is research, if not finding out, because there are multiple, interesting directions. And you are only human.
David Bau: You only have so much time and so much effort. That's fair. You have. You have limited amount of bandwidth, I guess.
David Bau: Okay.
David Bau: yeah.
David Bau: should I actually stop here? I mean.
David Bau: the other stuffs are actually a little bit different. And I think it's 1130, almost
David Bau: we we can. How does everybody feel? Should we wrap up
David Bau: or do you want to hear more? No, John is like, no, I want to stay. So why don't we say this so like people who, you know, have other things to do? We should. We can stand up and walk around.
David Bau: And if people, wanna you know, hear more of what is going with our can hang out here.
David Bau: But yeah, let's all stand up. Take a little walk, break, and then, if you want to talk to more, you can come back in a couple minutes.
David Bau: Okay, I guess you better walk around. Whoa, right? Yeah.
David Bau: Yeah. 12. Like. I just
David Bau: say that I should speak to somebody who is everything that came up. He was giving a talk. Oh, I'm not sure I'd have to high level.
David Bau: Oh, yeah. So I think it's why he's interested in a couple of things you're interested in. He's pretty senior researcher as in like, what would be the main application of this
David Bau: actually constructed. So that's
David Bau: would be constantly asking, do they ask?
David Bau: I can handle it on my own.
David Bau: So this, this claim that essays are
David Bau: more bottle based, not data set based type.
David Bau: I'm not sure about. Well, it's only in this particular instance. So that's right. So I think the thing that the main thing about that study is he's done on like one pair.
David Bau: That's right. I feel like, you know, that like, if you really want to do a paper. You want to do it on like a bunch of different models. There, yeah, that's something like that. And.
David Bau: you know, try to stress it as much as you can.
David Bau: I I guess the thing is like, if you were to train with a dip like a
David Bau: very different data set like a code data set, you would learn completely different features than like some other. That's the hope.
David Bau: right? And then the Se. Should pick it up. Or but if you took right so, or if he, if like, he took 2 different data, sets to train his saes on.
David Bau: and Wikipedia and poetry, or something like this. And they thought, oh, those are different enough.
David Bau: you know. Maybe he should make the data sets. He's training essays on even more different. Like one is Wikipedia and the other one's code like, would that be more convincing if, like the data sets he trained essays on were like as different as
David Bau: you know, code, or make the models very different like, look at the 1st layer versus the 8th layer, like, it's a it seems like a methodology thing. There's a interesting book where they try to train specialized essays by like taking specific data sets. So like one which is only biology related, one which is like only code related, and they do find that
David Bau: there are much more, many more like biology, related features. And they're seeing so.