David Bau: And so I want to
David Bau: see if we can host here.
David Bau: So we're lucky to lucky to have David Beck. Yeah, visiting here today.
David Bau: So there's there's a very interesting alternate alternative
David Bau: to the traditional cross entropy, you know, loss, way of looking at training.
David Bau: that that he's he's been proposing
David Bau: that seems to lead to some disentanglement, I think, is a very interesting
David Bau: interesting topic. So, David, you're you're a second year student. Is that right? That's right. And you're you're actually, you're with Max Techmark. Yeah. And does that mean that you're in the physics department? No. Actually, I'm in the Ecs department. You're in the Ecs, yeah. And then, like, I looked for Pfizer after I got admitted. So oh, I see. Yeah, yeah, I see. So yeah, so. But you're Ecs and mit, yeah, and and so, well, so thanks for coming. So welcome, David.
David Bau: Yeah, yeah. Thanks for the introduction. Yeah. I'm David Beck from Max Tech marks group. I'm a second year. Phd, student. And today I like to talk about harmonic loss for interpretability.
David Bau: Yeah. So if we like zoom out a little bit, and we think about what we do in mechanistic interpretability. There are several things we do. So, for instance, we do like circuit analysis. We try to find a localized circuit that does things like addition, or some numerical operations, or some in context, learning, knowledge, exception, all the other things.
David Bau: But oftentimes like it is very challenging to find a localized circuit that is really devoted to a specific task, because large language model is like a
David Bau: huge kind of object that is able to do like a whole bunch of things. And it's really really hard to just isolate a circuit that is just devoted to a single thing.
David Bau: But we also do other things like representation analysis. So we take the residual stream representations. We apply a Pca and try to look at the top few principal components and try to like manually interpret what they actually mean, or, more more like systematically, we can we also try train, sparse auto encoder, and representations to kind of like. Look at the features! What features are there, what features are like, unique to reasoning models or visual language models, and try to understand, like, how different models evolve, etc.
David Bau: But these representation analysis methods also suffer from a lot of limitations. So, for instance, oftentimes, when we do like Pca on representations. There is like a huge amount of unexplained variance that remains, which means that there is like a
David Bau: pretty much like non-trivial fraction of the representation that we don't actually understand. And the same thing goes for a sparse auto encoder. There is like non-trivial amounts of like reconstruction loss that we are not able to fill in under this like framework of like linear representation, hypothesis and linearly encoding representation and decoding it back. So like mechanistic interpretability, or
David Bau: maybe more specifically, like post hoc, interpretability is pretty challenging. And what we want to do ideally, what we want to do is we want to build a model that is interpretable by design.
David Bau: And then we can actually think about like, how can we build models that are interpretable by design? And
David Bau: what are the things that kind of like blocking us to make models that are interpretable by design.
David Bau: So I think there may be multiple reasons here. Maybe there is some kind of like inherent trade off between like model performance and interpretability. I'm not sure about that, but I think one like important reason is that we don't actually have a good metric for quantifying interpretability.
David Bau: What I mean by this is, we don't have a good measure that can measure like how interpretable a model is. So, for instance, if we had some kind of metric that can like, take, for instance, a model as an input and then like output, like how interpretable a model is, then we can just do some kind of like simple gradient descent or back propagation to like train a model to be just more interpretable.
David Bau: however, to the best of my knowledge, like, there's no such kind of metric.
David Bau: So there are many benchmarks for measuring, like normal models, performances like Mmlu or Imagenet glue and many other benchmarks. But we don't have a suitable benchmark or metric to measure the interpretability. I think the recent se bench work from Google. Deepmind is kind of work in this direction. But this work is kind of this work only applies to sparse autoencoder, and it cannot like measure the quality of like a general interpretability.
David Bau: So the question I want to pose here is, how can we actually define a global metric to evaluate the quality of an interpretability tool. And yeah, I I really want to like, hear, like everyone's opinion on this, like after the talk as well. Yes, it is like a very recent paper, this mechanistic interpretability bench? Oh, yeah, yeah, I think. But I have not really read it. Yeah, yeah, yeah, maybe that would be very relevant. Yeah.
David Bau: yeah. So the message here is that quantifying interpretability is challenging. And like, we kind of wanted to think about like, what would be the best way to quantify interpretability.
David Bau: And
David Bau: like one way, I like to think of intelligence is like intelligence equals compression. So when we try to understand very complicated or complicated concepts, we usually decompose them into like multiple smaller parts. And then we understand those smaller parts 1st and then kind of combine all of them together to like get a better understanding of an otherwise very complicated concept. So in this way, I think, like intelligence can be kind of translated to compression.
David Bau: And, for instance, like decomposing a problem into like 4 sub problems is probably much better than decomposing it to 100 problems, because, like, it's probably it'd be probably complicated to, like, you know, combine all the answers from 100 different problems into like understanding a larger one. So in mathematical terms
David Bau: this means that representations have relatively low dimensional structures, and then, or equivalently, if we apply, like principal component analysis on like residual stream representations, then top few principal components dominate or low. Yeah, so like, we'll get like 100% coverage of explained variance within like top 5 or 10 components.
David Bau: So and then another thing we want to think about is
David Bau: okay. Intelligence is compression. Then what is like the good unit of compression. So basically, going back to the example I gave. If we want to like, decompose the big concept into 4 different problems, then what are those 4 problems? What are those like unit things in large language models.
David Bau: But we actually think that those unit up compressions are parallelograms. So this is a figure from the group's function vector paper, which I like it a lot. So
David Bau: basically, in order for large language models to generalize, we believe that they should be able to have some clever representation, and they should be able to compress, like the huge amount of web web text, into some clever way, and we think
David Bau: those units are actually parallelograms. So this can be either like a form of function, vector where there is like a direction for English to French, English, to Spanish, or some country to capital. But this could also be a form of more like traditional word, embedding style, parallelograms, where we have, like a gender direction, and also have a royalty direction. And then men, women and king, Queen kind of forms like a parallelogram.
David Bau: So
David Bau: yeah, so we think intelligence is intelligence kind of translates to compression, and those units of compression is probably parallelograms.
David Bau: Then we actually want to think about how can we make large language models learn better parallelograms.
David Bau: So motivated by this, we actually propose this new loss function called harmonic loss.
David Bau: So here we have, like a comparison between the harmonic loss and the traditional cross entropy loss. So the major changes we make here is, we would basically modify how the unembedding layer computes the computation.
David Bau: So in order to compute the logic, instead of computing a dot product between the unembedding layer weights and the final layer representation. We instead compute L. 2 distance between unembedding layer weights and the final layer representations, and in order to obtain the probability. Instead of taking the weighted sum of the exponent of the logic, we take the weighted sum of these harmonic function of the logits, which is like one over D to the power of some number.
David Bau: So this harmonic loss, like by design, has like multiple benefits. So if we define a logic as the L. 2 distance between unembedding layer and representation by construction, all the unembedding layer weights become interpretable like each of each row of the unembedding layer weight simply becomes like the class center. And also we have a new, very good property. Which is this, our harmonic loss is scale invariant. Which? So
David Bau: that that means that so in cross entropy loss, you can increase the logic by simply like having, like a larger magnitude norm of the weights. But that. Yeah, can I stop you? Yeah, yeah. So okay.
David Bau: so this is a, this is a little bit of a silly question. Yeah, like, what? I. I understand the formula for harmonic mean, okay, but what does harmonic mean? Actually mean?
David Bau: If if it's a if it's a mean of of a bunch of points. I know that it is something somewhere in the center. Yeah, right? Yeah. But if I plot harmonic mean.
David Bau: what will this mean? Okay? So so harmonic can mean multiple things here. But basically the name harmonic here comes from like taking one over. D, and the logic, yeah, so like one over, yeah, that part. I understand. Yeah, yeah, what does it physically mean? And why you did?
David Bau: You said something like, if we
David Bau: take harmonic mean and use this use this L 2 distance between those things. Then by design, yeah.
David Bau: it will become interpretable. I'm just having trouble to make this connect. And does that. Does that make sense? Oh, okay, yeah. So yeah, so one thing I have to probably clarify is that here in cross entropy, like larger, higher logic, is better. But here, like lower logic, is better, technically, because we're taking the inverse of the logic and then taking the power of N,
David Bau: okay, so
David Bau: when we get when we have, like a perfectly trained model, then we'll have, like, Wi equals X,
David Bau: okay for X that belongs to class I that is kind of like cross entropy as well. Because when the vectors are same, you get one. Yeah. But yeah, yeah, but but the thing is like cross entropy. Laws cannot get like a perfect class mean? Because if you just simply
David Bau: like, increase the logic.
David Bau: increase the weight. Nor, yeah, increase the weight. Norm, you can simply like increase the logic further. Yeah. So there's this kind of like imperfection. That kind of prevents them to like.
David Bau: Make the like point place wi into like a perfect position. Now, now, my, another thing that I want to say is that before you apply this concentrate laws. Yes, there is a model norm.
David Bau: There is a layer which layer norm. There's a yeah layer norm, which kind of like makes sure that
David Bau: you can't gain that you can't. You can increase that by gaining the magnitude of the things. Right? Yeah. So
David Bau: and yeah, I think that that does help definitely. But
David Bau: okay, anyways, it has some things. But how does this harmonic main design encourage interpretability? You were
David Bau: to be like.
David Bau: yeah, so yeah, so harmonic loss encourages interpretability a bit because it like enables like Wi equals exactly x
David Bau: when it's when when it's like perfectly trained.
David Bau: But here, like so in cross entropy, loss wi doesn't necessarily is the class mean. It's like a so Wi will be the representation vector of like average representation vector of the entities that belong to class. I
David Bau: because that that will like maximize the top product right?
David Bau: Hmm.
David Bau: So like W. When WI. And X point to the same direction, the logic will probably let you continue, we can have this discussion later. Sure? Yeah, sure. Yeah. Okay.
David Bau: yeah.
David Bau: yeah. So harmonic loss has this like scale invariance, which prevents like the weight norm from diverging. And this can have, like multiple good properties, has, like better, faster convergence, and also like better generalization bound. And all the other things.
David Bau: Yeah.
David Bau: So you you said that this harmonic loss was
David Bau: inspired by so 2 questions first, st you said that you think that intelligence is
David Bau: the manifestation of having this kind of parallelograms. How we compose different concepts. Yeah. And then you said that
David Bau: this design is kind of inspired by those. Yeah. Are you? Are you about to make that connection, or did you want me to make that connection? I think I didn't make that connection explicitly. But so so why harmonic loss encourages like learning, more lower, dimensional, compressed representations. I think the key point is that.
David Bau: like L. 2 norm as a distance metric is much more favorable to like learning, lower, dimensional, like compressed representation, as opposed to like dot product, because, like so l. 2 distance. Norm is like kind of native to like Euclidean plane, like some lower dimensional thing, but cross product like by design. I think it kind of like makes the points diverse to like a like, a much higher dimensional space.
David Bau: Yeah, yeah, okay, yeah, yeah.
David Bau: Okay. Okay. And then, like, we we observed that experimentally and think it's cool. Yeah. So we claim that harmonic loss has 3 major advantages. First, st improved interpretability, which which, by by interpretability, we means better compression and like learning, more like compressed lower, lower dimensional representations. Second of all has
David Bau: better data, higher data efficiency. If there's a type over here higher data efficiency, which means that it can generalize with less data.
David Bau: And it also has less delayed generalization or brocking. So like, there have been like a multiple papers like when we train Mlp toy models to learn like model addition, there's like a gap between the rise of print accuracy and test accuracy. And this is not favorable, because ideally, we want them to like kind of rise together, so that, like, we have, like a better.
David Bau: like way of monitoring like the training process, but also, like this implicitly means that, like harmonic loss, kind of converges faster and generalizes faster.
David Bau: And another thing is, it doesn't lose performance like compared to see loss. We. So we we like kind of benchmark the performance across like the toy task, and then gpt 2 image net. And it actually doesn't lose performance. And it sometimes it's slightly better. So. Perhaps there may not be like a trade off between interpretability and performance. Perhaps that's not like something that we cannot fix.
David Bau: Yeah. So I want to show you the figures that went viral on Twitter a few months ago. So on the 1st row compares the standard Mlp. And harmonic Mlp. Train to learn a modular edition, so we can see the standard. Mlp. Learns a circle, but it's like a very imperfect circle, and also the explained variance explained by this 1st 2 principal components is far below 100,
David Bau: whereas for harmonic nlp, it is actually able to learn like a literally a hundred percent perfect, 2D circle, and they explain variance by the 1st 2 principal components is also 100%,
David Bau: and on the second row compares the standard model and harmonic model trained to learn mnist digit classification, and we visualize the weights.
David Bau: We can see that the standard model trained to learn mnist. It does kind of have this like shape of a 3 when we visualize the weights, but it's it's kind of like mushy and blurry. It doesn't like completely identify the edge. Whereas for harmonic loss, like, we can actually see that
David Bau: it is like there is like perfectly identifying all the edges of this number 3, which is pretty cool.
David Bau: Yeah. So I want to go through like each point of the harmonic loss one by one. So 1st of all, 1st of all, harmonic loss has better compression. So we evaluate this 1st on like a setup toy task. So we have, like in context, learning, model addition, permutation, composition, and some other task.
David Bau: So here I show results. For in context, learning a model addition, so in context, learning is actually like, very much relevant to like in context, learning and large language models so large language models. For instance, we give the words like, we give 3 words, man, woman, king, and the model has to predict like queen.
David Bau: or we give, like France, Paris, and Italy. And then the model has to predict rho, so here, how we designed this story task, we basically have this grid of points. And you're given 3 points on the grid, A, B and C, and you have to find another point D, when combined with A, B and C will form a parallelogram. So if the model is actually able to like
David Bau: perfectly digested structure, we expect the model to develop this perfect 2D representation. So when we compare, like standard Mlp and harmonic Mlp. Trained on this in context, learning task. We can actually see that standard Mlp
David Bau: can somewhat learn some kind of lattice, but we can see that there are some distortions here and there. And it's not perfect. And they explain variance by the 1st 2 principal components is not actually 100. It's like 90%. But when it comes to harmonic Mlp. We can see that, like, regardless of the training random seeds, but always learn like a perfect 2D. Lattice with 100 experience, explain variance by the 1st 2 principal components.
David Bau: and the same goes for modular addition. So here you're given 2 numbers A and B, and you have to predict, like a plus B modular 31 or something. You can see that standard Mlp. Learn some some kind of circle, but
David Bau: it's not perfect, and they explained variance by the 1st 2 principal components is like 50% ish 40% ish far below 100, whereas in harmonic Mlp, it can actually learn a perfect. 2D circle. In like most of the cases, like more than like 80% of the random seeds, produces like perfect circle. And even in those cases that failed like it's a pretty good circle. And they explained, variance is still higher than standard. Mlp.
David Bau: Yes, maybe I mean, you probably mentioned this earlier, but like, what aspect about the harmonic laws? Does it lead this to have like a better compression? Yeah. So I think so. I think the key thing is that number one by using like a l 2 distance instead of like a dot product like Euclidean, like L, 2. Distance matrix is like more native to like lower dimensional representations which can lead them to like learn like more compressed representation.
David Bau: And second thing is the scale invariance. That basically
David Bau: kind of bounce the weight norm of the unembedding like weight matrix on embedding like weight values. And that actually leads to like faster convergence. And
David Bau: like better generalization bound? Because, yeah, because, like the generalization, accuracy is bounded by some weight norms and other things. Yeah. So in your experience, like, when will be like, when would you like
David Bau: switch like, when would cross entropy loss be better than harmonic loss? Or is it generally the yeah. So we think that harmonic loss would be better and like learning, like more structured tasks like math or algorithms. When there is like more structure within the data, we think harmonic loss can encourage the model to learn more compressed structure representation, whereas, like, when it comes to like some like general language, or like writing a prose and stuff like that, I think.
David Bau: like cross entropy can like better, encourage some like diversity and like, have to kind of float around the high dimensional space.
David Bau: Yeah, okay? Also, another question. Since you're all getting this equation. Okay, you. You must be adding some kind of bias, just to make sure that you are not dividing. By, yeah, yeah, yeah, we're doing that in like code. Yeah, okay, yeah. And and that is not learned. That is kind of like, so it's just like Epsilon. Yeah, just like Epsilon, like E to the minus 6. Yeah, wouldn't it be more accurate to say that you're not? It's not. You still have cross entropy here, you're really substituting for softmax.
David Bau: That's what you're doing. Right? Oh, yeah, yeah, yeah. So it's really a soft Max versus harmonic.
David Bau: That's what you're doing.
David Bau: You know, because in the end you're taking a lot of probability.
David Bau: you're reacting to the undesirable properties of softmax. So yeah, the other question is.
David Bau: what is what is your N,
David Bau: oh, oh, yeah. So yeah, that that's a good question. Actually, so
David Bau: for our toy experiments, we just use N equals one
David Bau: but like. We have some like mathematical arguments in the paper that, like N has to be like a square root of like the intrinsic dimensionality of
David Bau: entrance dimensionality of the data.
David Bau: So, for instance, like in Gpt. 2, like, we take N to be like square roots of 768, which is the embedding dimension of Gpt. 2 and
David Bau: intrinsic dimensionality of the data.
David Bau: Yeah, this, this is not like a mathematically precise argument. But
David Bau: like, we can like, make some argument that one is like that and and is square root of some intrinsic dimensionality of the data kind of
David Bau: like matches. The date, like traditional data distribution. But to get the intrinsic dimensionality of the data, yeah, you 1st need to train the model and then do, yeah. But here we just approximate as the square roots of the embedding dimension. Yeah, yeah, yeah.
David Bau: Yeah. And also, like in Gpt 2, like, we found that if you use N equals one like the loss doesn't go down, down. The loss doesn't drop. So that that actually means like, the learning rate is too small, because, like, you have to use like a different learning rate scale for cross entropy and harmonic. Because, like.
David Bau: the logics are kind of on a different scale.
David Bau: Yeah.
David Bau: yep.
David Bau: yeah. And we also show that harmonic loss has better data efficiency and less rocking. So for on the top figure we show the test accuracy as a function of train fraction. We can see that our harmonic loss, our harmonic models, tend to like, generalize much better with less training data, and also on the second row. We show, like
David Bau: the number of fox to reach test train accuracy like larger than 90% versus the number of to reach test accuracy larger than 90%, and the points that lie very close to the Y-axis. Here, for instance, like in modern edition, we can see that standard Mlp. Almost always rocks very heavily, whereas when it comes to harmonic loss, those points like lie mostly on the diagonal line, which means that train accuracy and test accuracy kind of rise together.
David Bau: Yeah.
David Bau: So this is like a more concrete example of modern edition. So, and standard models
David Bau: they usually grog, sometimes fail to generalize. They learn some kind of circle, but it's not perfect. And they explain the variance by the 1st 2 principal components is far below 100%. And there are like multiple techniques that we actually use to circumvent this, which includes like layer norm and weight decay and some other regularization techniques. Even if we use regularization techniques.
David Bau: they? Yes, improve, but it's still not perfect. It's still like 58%.
David Bau: But when it comes to harmonic loss, even if you don't use weight decay, you can learn like a perfect circle with 100% explained variance.
David Bau: And there are some like other like loss functions like distance based loss functions that were proposed back in the days but we also do some like ablation experiments. And we we believe that, like both, like distance based distance based logic, and also like the scale and variance part, where we use harmonic like functions like, but we, we believe, like both of those comp components, are important and actually achieving this result.
David Bau: So, for instance, here we kind of compare Contrastive Loss. So like spherical loss and harmonic loss can see that
David Bau: like contrastive loss can, like somewhat learn a lattice, but like it doesn't generalize to transformer, whereas, like harmonic loss, can learn a perfect lattice with 100% explained variance for both Mlp transformer, regardless of the random seeds.
David Bau: Yeah. And then we we also experiment with like image net lms, and here, like we do some like preliminary experiments on applying harmonic loss to Gpt. 2. So we train Gpt-two with both standard software entropy loss and harmonic loss. And basically we look at the representations.
David Bau: And because we believe that the unit of compression is probably parallelograms, we take, like the We take pairs of word pairs from the group's function vector, data set.
David Bau: So, for instance, if you have like country to capital, we have France, Paris, Italy, Rome, and then we'll take the representations at any it can be at any layer
David Bau: so we take the representations from those like 4 words, and then we define what we call the parallelogram loss. So the parallelogram loss is defined by like France plus Paris, minus Italy plus Rome, and if it is a perfect parallelogram. This loss will be 0, and then what we do is we plot like a cumulative fraction of this parallelogram loss, for, like all all work pairs
David Bau: and
David Bau: the orange line. Here is the representations from harmonic harmonic models, and the blue line is from the representations from standard models, and we can see that across, like many different relations.
David Bau: and it is also independent of the number of principal components used. You are not optimizing it right? Are you optimizing it. No. So we're just training Gpt-two. For with the harmonic loss like normal expert prediction loss. Yeah. Yeah. And you are not optimizing. No, no, we're not optimizing for this loss. Yeah.
David Bau: yeah. So we just take the Gpt to train for trained with a harmonic loss. And then just use this as an evaluation. And what is this? Fractions like? It's basically accuracy, right? So fraction means so like.
David Bau: so you have like a total like data set right? And then
David Bau: it's like a cumulative like histogram.
David Bau: So, for instance, like here, you can see that.
David Bau: So, for instance, antonym, like
David Bau: person like 50% of the entire like work pairs have parallelogram loss below like 5 or something like that.
David Bau: Sorry? What? Oh, so here, like 50% of all the word pairs have parallelogram loss below like
David Bau: 5 or something.
David Bau: So this is okay.
David Bau: By some, by some.
David Bau: some constant, is it? Is that a constant? Oh, yeah, this is the normalized normalization. Constant. Okay, some constant and
David Bau: the highest value is one.
David Bau: No, no. So so
David Bau: the y-axis valued like doesn't really mean anything. So one means like 100% of the data. So it covered like 100% of the data.
David Bau: When you say covered, what what do you mean again? So for for each like pair of 4 words. Okay, we're going to compute the parallelogram loss.
David Bau: Okay? And then we're going to sort all of the parallelogram laws. Yes.
David Bau: And then, like, we're going to write down like, what is the accumulated count so far.
David Bau: beginning from like the lowest parallelogram loss?
David Bau: In what what do you get in that. What? What? What is the
David Bau: I'm just having trouble. Okay? Okay, yeah. So yeah, so you have parallelogram loss. And like, you'll have, like many different parallelogram loss values right? Yes, you sort them and you add them up. We don't directly add parallelogram loss.
David Bau: but we count as it's like a histogram like below, like certain parallelogram loss value, like, how many word pairs are there? So you you set a threshold and you yeah, you ignore some of these values. This is kind of like, how many of your samples are above those or below the threshold. So the x-axis is actually threshold.
David Bau: Yeah, okay, yeah. Yeah. Xx. This is the threshold. Oh, yeah, yeah. Okay. Okay.
David Bau: so in words, why? Why is it that you want this to be shifted to the left?
David Bau: Because that means, like each word, pairs are like a more perfect parallelogram, and
David Bau: if the unit of compression is parallelogram, we want them to learn like more perfect parallelogram.
David Bau: because your parallelogram, your parallelogram geometries, are closer to. Yeah.
David Bau: So when losses narrow, it means perfect parallelogram, and we want like this
David Bau: curve to be as close as possible to like a direct delta function. The step function.
David Bau: Okay, yeah.
David Bau: So which layer are you taking this implementation? This is from like the embedding layer representations. But we also do evaluations on
David Bau: like other layers, too. Yeah, if you're taking it from embedding layer, how can you do previous item or next item.
David Bau: No previous item. You mean, like, Monday, Tuesday. How does it look at other layers when you do this? I don't have the results right now. But
David Bau: I think in in general, like, the behavior is very similar to this.
David Bau: but generally in some layers like harmonic loss has
David Bau: like more high quality parallelograms, and
David Bau: sometimes also like a little bit more wars here, which will look like just drop.
David Bau: And in some layers we found something, this kind of behavior.
David Bau: Yeah.
David Bau: So it has, like more perfect parallelograms, but also like slightly more or worse parallelograms.
David Bau: But general general behavior is that, mono boss tends to have more per more perfect parallels. Yeah.
David Bau: yeah.
David Bau: Yeah. So this is like the end of the main part of the talk we still think there are like many things that we have still have to study. Like, for instance, like, how can we scale models with harmonic loss. For instance, how can we train, like ABC 8 billion parameter size models with harmonic loss? Will it give like even better interpretability? Or will it reach some plateau at some point
David Bau: but at a more high level. I think there are some other like interesting takeaways and questions that we all can think about. So, for instance, like, are there better ways to quantify interpretability? So here we kind of quantify interpretability as compression. But maybe there, there may be some better ways to quantify it, and we can optimize for the signal.
David Bau: Another thing is we can perhaps dive deeper into weight space. So a lot of the sae works. It's kind of like. Look at the representations, and you train like a sparse auto encoder to learn those features. But, for instance, like, from like a recent work, attribution based parameter decomposition for our Apollo research. They train an se on weights
David Bau: and in our work. It is also kind of like looking at the weight space, because we kind of modified the weight space of the unembedding layer to be like precisely become like the center of the center of each class. So perhaps if we look more into weight space like we can learn more things, and we can find some more interesting interpretability techniques?
David Bau: And also, how do we build scalable models that are interpretable by design? Or can we build models that are safe by design, by enforcing some geometric constraints, on which representations, because I think the core intuition for designing harmonic loss is that by using like l. 2 distance like you, kind of encourage, like lower dimensional representations. So if you like, think more about the geometry of weights or representations, we can learn some more interesting things, or come up with some interesting interpretability techniques.
David Bau: more details about the paper. You can look at the paper on Archive. Yeah, that's the end of my presentation. Thank you very much for listening
David Bau: wonderful.
David Bau: So you have trained some language models with this class. Yeah, so have you like, you have, you have a course in like, how does this language models perform like qualitatively like? Is it like like in terms of benchmarks? Not in terms of benchmark like, what's the vibe of them.
David Bau: They still generate kind of like legible text. Yeah, they they do generate, they do generate legible text. Yeah, I feel it's like.
David Bau: probably slightly better than the standard model. Oh, better than the standard model. Okay, but but
David Bau: not significantly better, but at least not worse, because, like like what is the like biggest model you have trained with this loss. Oh, this is just Gpt 2. It's the Max actually, yeah. So Gpt, 2 has different versions like Excel. Oh, like the smallest one, the smallest. Yeah, yeah. So 762, it's like, yeah 100. I think it's 160 million. 0, okay, okay, so this is Gpt too small. Yeah, Gpt, too small. Yeah. Gpt.
David Bau: okay, so I have some suggestions. So is this, is this still going through peer review? Or how is this? Oh, this is still going through peer review. Yeah, that's right. Yeah. So I have some suggestions. So
David Bau: I I wonder about.
David Bau: So I think I think you're you're framing.
David Bau: putting yourself up against cross entropy. Loss
David Bau: will not endear you to readers.
David Bau: I think that the thing to pick on is softmax. Okay, yeah. I think that you know cross entropy and Kl loss, you know. Kl, divergence. And these sort of statistical measures, you know, very principled. They have a lot of fans, and there's a deep well of support. Tell people that
David Bau: they shouldn't be looking at kale diversions anymore. Yeah, that you're going to have a lot of stuff to defend. Yeah. Yeah. But then Softmax is not very principled. And actually you have. You're you're preserving cross entropy. You're preserving the kale nature of what's going on. So instead of picking on cross entropy, I would pick on softmax, and so the other. So. But then there's there's a nice analogy to this. So there's you're familiar with
David Bau: the glorette work from 2,010
David Bau: from Bengio's lab. So. But you've heard of yeah. Xavier initialization.
David Bau: Yeah. So that so, Xavier Xavier Glara wrote several papers that I would I would recommend
David Bau: taking a look at and comparing to your your approach, and like the way that that Xavier wrote his papers.
David Bau: He picks on. It was very popular
David Bau: the sigmoid nonlinearity at the time. You know.
David Bau: he. He basically convinces everybody that the reason the deep learning is stuck is because everybody's using the seepoint dominarity. And I think that the consensus is that he was right back in the day. And so, you know, if you're making the argument that
David Bau: we're doing things wrong because everybody's using the stupid softmax for no good reason. Then I think that you have a very similar argument to him, maybe for different reasons. But I would recommend going back and seeing how he made the argument. And you know he might, he might consider doing similar things. I think that he had a lot of detailed look at.
David Bau: you know. You know, internal training dynamics and and stuff in in some of the things in great detail.
David Bau: And and so you know, I think that
David Bau: he, he ended up having a lot of impact in his work. And so I think that the challenge of your work
David Bau: is that
David Bau: you know, if if the performance of what you have is sort of about the same. So I've got other some other questions right? Then it's going to be very hard to have any impact, because people aren't familiar with your stuff. And it's it's costly to verify that it works well. So. But you can kind of see, like what he ended up doing to have impact
David Bau: that makes sense. Yeah, okay, cool. So then the other question I have for you is
David Bau: like, so I you know, I think it's kind of interesting.
David Bau: The impact you'll have on sort of the final layer embedding
David Bau: you know, because it's the end to end loss from the whole network. I
David Bau: do. You feel like there's something that we should be doing differently in the interior layers to have this sort of effect.
David Bau: or right, because I kind of feel like one of the things that we're finding is that
David Bau: you know, the final layer is fine, but a lot of the rich interpretability.
David Bau: you know, stories or things that would have to understand. Or, you know, really, in the middle layers of the network.
David Bau: Yeah, when in your loss is very far away from those middle layers. Yes, so
David Bau: I think like by modifying on embedding layer like it kind of like propagates into some middle layer representation. But
David Bau: think there should be something we should be able to do for the middle layer representations. But I don't think of anything like immediately. Yeah, yeah, I think that's I think that's reasonable. Yeah.
David Bau: You know, I think that if like, if you were if you if you had some proposal there. Yes.
David Bau: that works in the middle. And then I feel like, Oh, yeah.
David Bau: be like having. It'd be like Xavier glow rock.
David Bau: You do. You might be able to redefine a lot of things.
David Bau: Make sense. Yeah, yeah.
David Bau: I think those are my main 2 questions, hey? I guess. Like, if you want to do something at the like
David Bau: middle layer like, it's probably something about like changing the fundamental structure of transformers. But
David Bau: or I don't know, I don't know. Right? Yeah. But I mean, there's a lot of weird stuff we do. You know, we have all these layer norms that we keep on using. We have, we have all these, you know, nonlinearities that we've chosen.
David Bau: you know, done all sorts of like weird, arbitrary stuff
David Bau: that we do for no good reason. So if you come out swinging and say this arbitrary thing, nobody's questioned. But I'm gonna question it. I think that
David Bau: it's like the recent, like, neon optimizer work that that kind of.
David Bau: I guess. So it's a different kind of thing. Yeah. Yeah. But you're you're going after a different thing. Yeah, don't be envious of them.
David Bau: Yeah. Hold on.
David Bau: Yeah. So your work has, you know, is so the the other disadvantage of, or the disadvantage that you have
David Bau: is because because you're going after a phenomenal, you know, machine learning lots.
David Bau: then, in my my prediction is that you know in review.
David Bau: People will hold you to a very, very high bar
David Bau: of empirical evidence. Right? There'll be like.
David Bau: yeah, you only you. Only you only test it here. You only test there. I'm not sure what your comments are. Gonna be like, right? So they'll probably want a lot more.
David Bau: Yeah. So yeah, it's probably one of the one of the questions actually like under Icm right now. And
David Bau: I think, actually, like less reviewers, challenge like the fact that we didn't do experiments on even larger models. But
David Bau: a lot of the reviewers questioned like.
David Bau: if the claim is not performance but interpretability. And if you don't have a good metric for interpretability. Then how do you know that this is like objectively better? Yes, yeah, that's that's that's the concern. So on that note. So there's so you know, there's been attempts to modify interpretability. So yeah.
David Bau: my, my old like, network dissection work is like an attempt to do this. You're familiar with that work, I think I should take a look. Yeah, you can take a look. Yeah, it's actually, I think the title of the paper is actually called quantifying interpretability, or you know something or other.
David Bau: And but yeah, it's it's an old work from 2,017.
David Bau: And and the idea is, you know you train your model to do one thing.
David Bau: and then you have a ground truth of human, understandable concepts.
David Bau: like a large vocabulary of human understandable concepts, and then you.
David Bau: you know, you look for alignment with those with that large vocabulary of.
David Bau: you know, implicit concepts that weren't mentioned in the training data.
David Bau: And yeah. So it's it's just a brute force way of accounting. You could report things on that.
David Bau: Yeah, I guess. Ideally, like, we have like a bunch of different metrics, just just as we have, like different benchmarks for measuring performance. We have, like a bunch of different metrics. And then we can show that this interpretability met those like, achieve top. And like all metrics. But we don't have that much metrics. Yeah.
David Bau: yeah, yeah, interesting. Yeah, yeah.
David Bau: yeah, yeah, yeah. No. The interpretability. Metrics are very important.
David Bau: And then so then the other, so like like a really solid interpretability thing, would would take a few metrics.
David Bau: And then also we do like a human evaluation. Because within the end, that's right. Yeah, right? Yeah, yeah.
David Bau: yeah.
David Bau: So your parallelogram metric which you sort of reported here. Yeah, is that sort of your main, your main compiling evidence? Yeah, it is. It is. Yeah.
David Bau: Could you explain a little about what Uk meant? I think it was a slide back.
David Bau: Ek, yeah, oh, so this is. This can be like any representations from the model. So it can be like layer, one residual stream representations or but but here this plot is using like the embedding, embedding representation. Yeah, this is embedding. No, this is embedding. Yeah.
David Bau: But ek, what ek means. Here is the embedding of every vocabulary word, so that the sum is so? That's really an average over all the vocabulary. Oh, yeah, that's right. Does that answer your question. Okay? So so basically, when we construct the parallelogram, how far off were we relative to like the average just
David Bau: concept in here or average embedding the root mean, squared. Okay?
David Bau: Yeah, yeah. Yeah.
David Bau: So you normalize into the rubian squared embedding size.
David Bau: How does this compare? So there was this,
David Bau: there was this argument made by, was it Andy? Rdt about the entropy
David Bau: was an entropy neuron or the entropy? Do you? Do you know what I'm talking about? Oh, yeah, yeah, right. Where's where is the entropy token?
David Bau: Anyway, those or the entropy attention, head, or something like that, I'm not sure what. Which which abstraction this was.
David Bau: yeah, I remember something about entrepreneur. Yeah, yeah. So so there's this, there's this basic
David Bau: issue, which is, you've often got dimensions in these large representations, where they have a scaling effect.
David Bau: where, when the dimension rises, then, if
David Bau: has the effect of changing the temperature, you know, having an effect on softmax and causing it to be more or less confident.
David Bau: And so the network can, like, you know, pick a neuron or something like that
David Bau: and then use it to modulate its confidence. And so there is a you know. There's a paper, I think, from ready that I think the argument was made
David Bau: that this is
David Bau: this is what the networks are doing. So now, when you go to this like Euclidean loss, yeah, you you don't have softmax anymore. Yeah, and so you no longer have the ability to have like an entropy neuron or something like that. So I guess one another. So this is, you won't get this question from Icml, this is very interpretability inside baseball. Question, right? But like, do you? Do you lose like the entropy neuron?
David Bau: Do you need one? Is there? Is there a problem for having lots of right? Do you? Also, if if you use entropy, neuron, does that mean that your model has worse calibration.
David Bau: right? So like, when the model thinks that it's certain, or when the model thinks it's uncertain, is it? Is it reflective of its actual accuracy or not, or, you know, is it better or worse? That kind of
David Bau: cause? It seems like to me it seems like that's sort of the like, if you think conceptually what softmax is doing is this, you know, by removing softmax, you're you're potentially removing the ability to do that.
David Bau: That makes sense. As a question. So like, is, is it better or so? Have entropy or no right? Right? And you know, maybe maybe it makes no difference. Maybe your Cal, your models are just as well calibrated as before. Yeah, I think calibration is an interesting thing to test. Yeah, yeah, interesting thing. Yeah.
David Bau: yeah, yeah.
David Bau: Yeah. I'm sure you didn't get this paper. It's not in our edited paper. Who was it? Oh, okay, it's from Neil's group. Oh, it's a different Neil group paper.
David Bau: Pretty good, all right. I've mixed it up with a different thing.
David Bau: I also wanted to talk with you about you said something. Intelligence equal to compression. Yeah.
David Bau: And you say that. Okay. Dividing up tasks into
David Bau: 4 subtasks compared to 100 subtasks is better.
David Bau: It's not a strict analogy. I mean, of course. Yeah, yeah, of course. Yeah, of course. But yeah, I,
David Bau: okay, I can also make this argument, this intelligence about composing
David Bau: different style. The the greater this number of differences the more better, and also like, if those.
David Bau: if this 100 subtasks are more generalizable, more usable for other solving other tasks, maybe maybe you are supposed to break down in more subtasks. Yeah.
David Bau: I agree with you. The interpretability is not the same as compression.
David Bau: Oh, you said intelligence. Yeah, I said, intelligence is compression. Yeah. Oh, yeah, maybe generalization.
David Bau: But so. But you know, it's more like, there's an arrow.
David Bau: Yeah, right? Yeah, it's like, compression does not lead does not
David Bau: lead to intelligence. Maybe intelligence is compression like this, is this arrow? So this is the danger. If you say equals, then, yeah, yeah.
David Bau: I guess there is like a classic saying that like, if you understood something perfectly, that you should be able to explain it to like kindergarten, people
David Bau: be able to explain it very easily.
David Bau: Similar with the argument I was making. Yeah.
David Bau: also, I kind of like, got lost in the connection. So you made that okay. Intelligence equal to compression gave this analogy, and then you jumped to parallel ground. It was a little bit lost there, maybe.
David Bau: yeah, that's what I wanted to. Yeah, the connection I was trying to make is like we decompose the bigger task into smaller task. And like each of those units is like about like half like developing a better understanding of parallelograms.
David Bau: and like, when we combine like my parallelograms, why not? Something else
David Bau: like I know that I like, we like parallel accounts because he it came from kind of like our lab. But I mean, it's a it's a valid question. Yeah, yeah, it is a valid question.
David Bau: Bad?
David Bau: I don't have a very good answer for that, actually, but
David Bau: think the best way to compress like a huge text is probably, like you have, like different individual text, like individual, like feature directions, just like an se, and then, like you kind of compose them. So in that sense like, it's probably some parallelogram or hyper dimensional cube, and probably parallelograms, is the simplest setting to test how 2 things compose.
David Bau: Right? Yeah, we can test like cube and like 4 dimensionals. But yeah, that's what we I like this question. It's like, if you argue that the parallel ground geometry is better.
David Bau: And you also would expect that you might
David Bau: be able to detect better performance of a certain type of generalization. Right? So the parallelgram thing is about compositional generalization. That's right. Yeah. And so I feel like, you know, your your work would be really compelling
David Bau: if you showed, even in toy data or something like that. The compositional generalization worked better end to end, not just geometrically, but actually, just like, you know, in some sort of
David Bau: benchmark test. You know, traditional machine learning test. I think that that would be like a tour de force that you showed that you've got this lots that you can show through an interpretability method that you know why it's getting better geometrically and then end to end. It actually exhibits some better compositional generalization. Then I feel like
David Bau: could be a shoe into Icmo, like, in terms of like generalization performance. Right? That's right. Yeah, like. But it doesn't have to necessarily need to be classic, you know, broad generalization performance of of just randomly selected examples. You could you could curate it and say.
David Bau: our hypothesis is specific to a certain type of generalization like computational generalization. And then you could target your test to get exactly.
David Bau: look for that. And and then make the argument that that's because there's a lot of theoreticians who are basically arguing that
David Bau: the power of a lot of these large models comes from computational generalization. So you can, you can say, well, let's make that better. Yeah, sounds good. That makes sense. Yeah.
David Bau: yeah, just kidding.
David Bau: I'm gonna I'm gonna turn off the recording and thank David here.
David Bau: So but thanks very much, David, for for the presentation. It's great.
David Bau: very interesting, a very interesting investigation you're doing here.