David Bau: Okay.
David Bau: So Freya's visiting us for another week or so, and so she's been very
David Bau: interested in what's going on inside models. And she has a particular strategy that she uses, which is to
David Bau: analyze smaller models in great detail which I I like this approach. It's a i think, that Eric is online. He's been
David Bau: doing something of this flavor, and I wanted Eric to dial in and to hear what Freya Freya's been up to you.
David Bau: phrase look, she's about to graduate. Is that right end of the year end of the year? Yeah. And and sort of, you know, traveling the world looking for places that might be a good place to hang out as a postdoc, and so so so welcome prayer. And you know.
David Bau: chat and see if there's good research overlap.
David Bau: Yeah, okay, thanks a lot for having me welcome here. So let's see.
David Bau: Share.
David Bau: And then
David Bau: so yeah,
David Bau: yeah. So with the discussion you just had, I have some other work. I could have talked about that I'm not talking about today. But maybe we should talk about this later at some point, because it's kind of about this data set distillation and how the soft labels actually give you some information about memorized data that you didn't give to the student
David Bau: and kind of where the information comes from. So so that would be cool to talk about
David Bau: but the other parts that I want to talk about is looking at phase transitions between different mechanisms in very, very small models, so like one or 2 layers, and in particular, how different parts of training process or the architecture influence these models.
David Bau: And this work move.
David Bau: Yes, right? So kind of the motivation. Here comes a bit from the fact that kind of you see predictable
David Bau: performance gains on scale and different parameters right? But some
David Bau: some capabilities that you have are a bit harder to understand why they emerge like at a specific point in training right? And not before, and not after, or at a given scale. And at the same time there are these examples where you get a good behavior of one particular mechanism, but like a really bad one on another, and it's a bit harder to understand why why this happens
David Bau: so enough to understand these different failure modes
David Bau: and how they fail, and when they arise kind of the goal that we have, or that I find very interesting is understanding how the architecture, the training, and the data actually influence kind of these mechanisms
David Bau: and what people usually do. Not you here, I guess, but it's like analyzing, actually the performance of the model. See how that is influenced by the external factors and kind of what I want to do is actually understand
David Bau: these model internals. Which you know it's kind of what you're doing here. But in particular how this depends on external factors. And so, in order to do this, I really look at kind of smaller models on a smaller scale. To have
David Bau: kind of also theoretical bearing on this.
David Bau: So the the 2 works I want to show today, are considering ones like the sample complexity. So how many samples do I need to see? So that a specific mechanism emerges.
David Bau: and the other one is an analysis where I look really at different architectural choices, and how they influence
David Bau: the mechanism that's learned. And the Max size. Here is a 1 layer transformer just to get a prime you? Yeah. And if you have any questions on the way, or want to discuss something, just just stop me.
David Bau: Yeah. And for this 1st work.
David Bau: We really wanted to look if this kind of behavior of an algorithm emerges or there is a different algorithm that is or solution that's being learned over time of training if we can model this in like a
David Bau: synthetic data set and because I'm working with theoretical physicists, and we have, like a very special.
David Bau: a sense of phase transition. Maybe that's kind of worth saying here, because I think usually people think about it as okay. There's 1 behavior in one region and another behavior in the other one. But really, the way physicists think about it is that there's kind of the system many interacting small particles.
David Bau: And just by the fact that there are these many small particles that interact when you change kind of an external thing, there's really like a fundamental change in kind of the macroscopic behavior that occurs like water freezes under different temperatures or magnets become
David Bau: magnetic or not more than just like kind of a slight behavior from water becomes ice at some point, if it's like really something discrete.
David Bau: So we were kind of interested. If we can use these tools from physics to actually look at a small model in machine learning where you can actually see that the learned algorithm changes discretely under some sense. Here sample complexity from one version to the other. So are there like space transitions in the physical sense. And there's been like kind of a long history of this work, I guess, since the nineties
David Bau: but kind of what we wanted, maybe let me just skip this. This is, for people that don't see
David Bau: are not so familiar with interpretability, but like. And if the
David Bau: the idea is that we want to phase transition between different mechanisms and not just the model knows how to solve a problem or not, which is usually kind of the type of analysis, and to distinguish the types of mechanisms. We kind of do a very simplification here, where we use information encoded as the sentence, we distribute it in 2 parts, so the semantics, what is the meaning of the word or token. And what is their position in the sentinel?
David Bau: So kind of a very simple division.
David Bau: And then we want to know how a single layer of tension. So here we have, like
David Bau: length, L dimension D and some sort of attention metrics that mixes together the tokens learns to just predict the context. Vector, so something like super simple
David Bau: and this s of x, the S of the input sentence here.
David Bau: What usually would be your attention matrix can have be allowed to have different types of implementations because it can use
David Bau: kind of one kind of item in this matrix depends on kind of the vectors themselves and the positions of the vectors. If you think that there's a position of coding or something.
David Bau: and we want to understand whether this matrix actually depends on all of these 4 things, or whether it depends on purely the semantics or purely the positions.
David Bau: And then, yeah, kind of ask, when do you learn these different mechanisms.
David Bau: And actually, if you look at new models, there's
David Bau: kind of some ways where you can see that simple tasks. You're like a 1 layer transformer, learn
David Bau: attention matrices that depend on only one or the other one. So here for the histogram task I give you like an input sequence of letters and the output sequence should be, how often does the letter occur in in this input sequence?
David Bau: And actually, there are
David Bau: 2 minima, the loss landscape that reach as like 0, or like a local minima where you get 100 accuracy just kind of different loss.
David Bau: And they look a bit like this. So this is the positional minimum. Here, you see, like 3 different sequences, I put into the model the red squares highlight. What is the same
David Bau: and or like, where are these attention, heat maps, or what are these? Yes, yes.
David Bau: and just the red part is where the 2 tokens are semantically the same.
David Bau: So this is the positional solution. And this is kind of a semantic solution where you can see that in one case this function s, or attention. Matrix depends only on.
David Bau: or mostly where on where you are, whereas the other one depends mostly on
David Bau: what you are so, and all these compares with these, these get the same value? And if so, both these solutions like achieve high accuracy or 100. Yeah, I see they both can solve the task exactly. Do you have a sense for how these 2 are working? That will be the second part of the talk. Oh, so later on, you'll talk about that. So this is just kind of the motivation to see in this kind of easy thing that okay, something you can observe.
David Bau: But what we wanted with this work is, we need to have a solvable model. And by kind of the theory angle. What we mean is that for this given task. Given architecture, we can characterize these 2 different minima theoretically, that rely on this empirical landscape, and we know what happens to them as we add more samples to training or not. And then this allows us to kind of understand whether these different minima express like fundamentally different solutions in some sense.
David Bau: Okay, and it's static. So we don't consider dynamics in in this specific case. But in follow up work.
David Bau: So yeah, this was kind of the model that we were looking at. Now.
David Bau: input sentence, attention, matrix context, vectors, what we want to predict and what we usually do is that kind of in order to make the analysis feasible. We consider a data model where the inputs and the tokens are just like gosh and sampled. And in order to get the target to kind of establish correlations between the input, we kind of just define a teacher model.
David Bau: So kind of we sample these values cube also from a random matrix. And we just define the pairs. Xy to be like the target that we want to train our model on, and kind of the 2 parts that you see here in this blue box, which is kind of the function as of
David Bau: X is that kind of the green part is
David Bau: semantic, right? It kind of looks like a softmax dot product attention, but not quite like the Q. Star is low rank. It's matched right. But these are all simplification that help the analysis.
David Bau: And then this a is actually a fixed L by L metric. So there's no kind of variable length.
David Bau: setting going on, but kind of the nice thing is with that. With this Omega, we can just say whether our target is actually purely semantic or purely positionable.
David Bau: And what are you using this, this this objective for just trying to backing up for a second the like, the target object like this is how we define our input data distribution, right? For so this for the task, for that, you're training the neural network on or yes. Oh, okay, I see.
David Bau: because your original task was like this histogram task. Yeah. So that's kind of the motivation of why we want to. Actually, because, like that, one is hard to analyze, kind of in different limits, and to actually establish these phase transitions so, and the simplification that still exhibits this behavior, that there's something semantic and positional about it. That we came up with this and that we can attack with our tools. Oh, I see. So. So this is this is not.
David Bau: This is a task that looks like the histogram task. But really it's just like learning the attention part of the histogram task. Exactly, I see. And so you're just pulling that part out, and you're saying, This is your goal. You want. You want the attention to look like this. Yes, got it? Sorry. I'm still a bit confused. So is this like a lot like the loss for the attention essentially
David Bau: or the objective? I mean you, you could think well, let me just go ahead. One slide. So this is is like the this previous thing was the way we define our Xy pairs right? And then. Now we define. Okay, which what is the model that we actually want to learn?
David Bau: And what is the loss under which we optimize it? Right? And here, and if you see that this Fq is the model, and the Y of the X is how we define our targets.
David Bau: Yeah.
David Bau: and you can see here that if you kind of closely look at the function for the student.
David Bau: Now we have kind of a positioning positional encoding. So there's kind of a mismatch between the student and the teacher or model, and the teacher
David Bau: or target where? Which then later allows us to understand this parameter. Q. That we need to learn whether it kind of attends more to P, or X, yeah. And kind of the
David Bau: the way this analysis done is in the asymptotic limit. So we consider the case where, like the dimension and the number of samples go to infinity together, and by together we mean that kind of the ratio. Of how many more samples do you have over the data points is fixed?
David Bau: So that. And then kind of this becomes predictive also for fixed n and d, but in the limit. It will be like an exact approximation.
David Bau: Yeah, so the kind of very so. So
David Bau: maybe a bit more on the theory side is that kind of in this original model. You have these in this infinite number of parameters. Right now, if I wanted to run simulations, I can do it for fixed. N. But I want to know what behavior there is like at infinity, and in order to do so, the kind of
David Bau: theoretical goal is to
David Bau: describe the system, not with infinitely many parameters, but with like a fixed number like, say, 10, so that you have your theory of how this system behaves. And this is what you call observables in physics. So kind of the 2 observables, or may think of it as macroscopic summary values that we have for this model
David Bau: are the overlap between the
David Bau: this mu, which was the way we encoded positions
David Bau: and the vector that we learned and
David Bau: the overlap between the vector that we learned, and this original q star, that we sampled in our teacher. Right? So end of this.
David Bau: the screen Q star in the middle
David Bau: and the oops queue had to be learned from like
David Bau: that is the minimal solution of the loss.
David Bau: And then kind of these 2
David Bau: observables allow us to characterize 2 minima that we can identify in this last name. So I don't get what an observable is.
David Bau: So you have mu, and then Q, and then. QQ. Transpose or Q transpose. Q.
David Bau: Think if I had a finance system.
David Bau: I could kind of find, like the minima in my empirical loss. Landscape. And look at, how close is this queue? Had that I found to my original queue, start right?
David Bau: But your original Q star is the ideal, you mean. Yes, the original is like the ideal and the infinite dimensional. The the Q star is kind of the ideal semantic one. Yep, and the Mu is the ideal positional one.
David Bau: And I see, I see, I see, and then kind of the theory
David Bau: gives you predictions for these values for these observables without you having to actually do the dot product between your 2 infinite dimensional vectors.
David Bau: Chris says I'll bring this up again. I just I just closed. Chris, are observables, just statistics that help you write down conditions for distinguishing the 2 different minima. In some sense, yes. And the cool thing is that like we can predict these statistics?
David Bau: From the series, I guess.
David Bau: How do I get rid of this thing? So I can see it bigger.
David Bau: Yeah, I don't know nothing I can do to get rid of this.
David Bau: We've had this issue ever ever since I've been on video.
David Bau: Okay. Anyway, we've had this issue since, like zoom like upgraded itself. And we've had this issue ever since.
David Bau: Yeah, okay, hmm, so you have these 2 things. I guess in some sense you can.
David Bau: You can think of
David Bau: you finding an abstraction in for what's going on in the network that's independent of scale
David Bau: as as such an observable it. And
David Bau: so here. Wait. Q is Q. Hat. The ideal or Q is different. Q. Hat is the one that your model found, like the one in the lost landscape.
David Bau: And so M. And Theta are telling you how far you are or something. Yeah, they are telling you how close the solution that you found is to 2 versions of the ideal solution, one, that is the position and one that is the semantic. Okay? And kind of what you find. Is that right? You can do this comparison for any Q hat that you find. But then there are 2 Q hats in your lost landscape that are minima.
David Bau: and for those minima.
David Bau: You have 2 properties. Yeah, like they said the 2 like, and either they are positional. So either like they attend to
David Bau: or like have a high overlap similarity with the positional solution or with the semantic solution.
David Bau: And then kind of what you can do as you can
David Bau: predict, based on the sample complexity. Right? This, how many parameters do you have? How many samples do you have per parameter? And this original omega that we have right how semantic or
David Bau: position as your teacher. You can predict that
David Bau: the last landscape changes in a way so that, or for some of sample complexity, Omega
David Bau: combination, like the positional minimum, is the better one. So that's the blue part.
David Bau: And then for others, the the semantic one is the minimum and you can actually
David Bau: predict kind of this discontinuity like this green line analytically, where, like, you break
David Bau: like, the minima actually does the
David Bau: And then kind of what this looks like in
David Bau: so this year's training error, right? This is what the loss, the training loss looks like. But what we're
David Bau: usually interested in generalization error, right? So whether the the minimum that we found corresponds to something nice. And then kind of what you get out of it is that
David Bau: as you increase the number of samples for a fixed target, then 1st you learn this, yeah, the positional version of the mechanism, and then at some point you have a sudden drop, because then you would learn the semantic mechanism which in this case leads to better generalization. So you really have this strong face transition, or like discrete phase, transition, and the physics sense in between
David Bau: these 2 minima, which kind of was
David Bau: the the thing we really wanted to see in this model?
David Bau: Does this make sense so far? Do you have any questions? Is it so basically, the distinguishing factor is whether the positional vector or the semantic vector is similar to Q hat
David Bau: equal 0 means that they're orthogonal, I guess. And so this Q hat is sort of like determining. I I guess what I don't understand is like what Q. Hat is, or maybe I just forgot that notation.
David Bau: So so you had. So this is like the definition of your loss. Right
David Bau: and Q are the different points in your loss. So Q. Hat here be considered to be
David Bau: one of the minima in this loss. But really like this, maybe I shouldn't have put like the Q hat in the definition. It's just we can quantify a position and loss landscape according to these 2 parameters.
David Bau: And these 2 parameters are enough to define the the local minima that this last let's give us, yeah, makes sense. Yeah.
David Bau: no.
David Bau: Yeah, so yeah, this is kind of like the A, a small example where
David Bau: you actually have this this phase transition in like a physics sense. And you can kind of build up a little bit on it. You can look at the dynamics. And what happens if there's multiple layers and sort of this, phenomenology persists, but kind of what I was interested at with one of the collaborators. Kind of going back at this
David Bau: motivating example. The original one, where you kind of count stuff.
David Bau: Okay, let me read this.
David Bau: Observables are kind of like probes, but on the model weights.
David Bau: Now, that would be very hard to set for big neural networks, right?
David Bau: Unless they're fine tuned to the same model.
David Bau: then it might be easier. That's like what Elia does.
David Bau: So I mean, yeah, I guess you can still look at.
David Bau: for example, the attention mechanism, and understand whether, like the the queue
David Bau: K matrix, that you learn whether that admits the direction in kind of its Eigenspace. That is the positional one, right? And then it means it's sensitive to that. And if not, it's not so, I think there's maybe some ways. But
David Bau: right here the only parameters are the tension parameters. So there's
David Bau: a bit less complex than you love. Sounds like, well.
David Bau: yeah. So this like kind of the small model that then motivated us to actually look at
David Bau: the history room task again.
David Bau: Which is kind of nice, because somehow, you need to select stuff from the input and measure like the magnitude information. Right? It's kind of doing these 2 in
David Bau: operations that are kind of basic in a way.
David Bau: And when we formalize this a bit more, you kind of have size of the alphabet of tokens that you want to compare the sequence length.
David Bau: and of its size.
David Bau: And then kind of the question we looked at is actually, how can you solve this task with a with a single layer transformer. And I showed you the answer already. Right? There's these 2 versions. If you can find.
David Bau: But of course there's like
David Bau: many different ways of even in the single model, deciding how to do things and kind of the exercise we went through here is actually understand how.
David Bau: just in a single layer. The influence of these different architectures is on the history of task itself.
David Bau: And just to say what I mean by single
David Bau: layer transformer. So the the stuff that we we vary beyond, within the token and feature, mixing
David Bau: are beginning a sequence token
David Bau: we have it or not, the size of the embedding dimension.
David Bau: Whether we do kind of a dot product mixing, whether we use softmax or not, and whether, whether, like
David Bau: okay, good
David Bau: from a math point of view, a linear attention would be something that is constant, right? So where this value is a learned value, that only depends on the position. But I think what usually people refer to it as as kind of a dot product. Attention that doesn't have a softmax. So maybe just do the correct mapping in my mind, because here we actually mean a linear mapping, and we have a dot product mechanism with or without softmax
David Bau: and then finally, we
David Bau: kind of change around the size actually, of this, the same layer and the power there.
David Bau: and kind of the the other things
David Bau: to note is that kind of
David Bau: the learned parameters are the embeddings, the
David Bau: and the token mixing. So the Wqk, there's no value. Matrix.
David Bau: Yeah, and the hidden
David Bau: layer, and there's no positional encodings except that you, you can get positional information through the residual right? Like, you know, you can distinguish yourself versus others, which is fine for accounting?
David Bau: Yeah.
David Bau: So
David Bau: when can we actually learn these perfect solutions? How many parameters do you need depending on the different models?
David Bau: We kind of distinguish 6 settings
David Bau: where we either have this linear the dot product
David Bau: and adding the bos or not, with or without softmax. And I'm just showing you that with softmax. Here kind of the number of parameters actually influences at which point, like so hidden dimension
David Bau: and embedding dimension actually matter in order, we we saw in the past. We're giving
David Bau: as of length. L, and hopefully, we need to know.
David Bau: You've got dots and lines. So like every every.is the single experiment. We're very Dnp differently.
David Bau: And then the line is like the convex hall, right? Best possible model will give me. I don't know accuracy. 50% with
David Bau: one K parameters, or for the linear thing. So they need different perimeter sizes.
David Bau: So this includes, like the embedding count
David Bau: and what you see, if you kind of expand these little dots to kind of the squares they come from, right where I change is I changed P. And d
David Bau: then this starts to look something like this. So here you see just the accuracy of every experiment, and whether there is actually a solution that reaches 100%.
David Bau: And when you do this exercise for kind of different architectures, you see that the
David Bau: the regimes in which this task can be solved can be solved differ, depending on the architectural components.
David Bau: And we wanted to find out for this specific task
David Bau: what actually contributes to solving it or not.
David Bau: And I think.
David Bau: Yeah, it's like
David Bau: kind of specific to this counting. But maybe there are some broader intuitions that you can. So Chris says this is easier without Softmax, but I read it the opposite way. Isn't it easier with Softmax? So I love these plots, he says. Toy models are great, so do these plots mean it's easier without softmax. Or is it easier with Softmax? Right? So for this guy here, if you have this mechanism, it's easier with without softmax.
David Bau: But this guy actually has more solutions than this one, if you have the beginning of sequence token. So there's
David Bau: oh, so it's just the middle one where it's easier. So Chris was looking at the middle, and I might have been looking for that. Yeah, exactly. So. Here no, softmax is better. And here, actually, Softmax is better because you can get these solutions down here that you don't get here
David Bau: right, and then and then, for some reason.
David Bau: if you kind of compare along this line, just adding a beginning of sequence. Tokens. Kind of gives you this space here
David Bau: and like there's kind of maybe some unintuitive things going on like, and
David Bau: where you would say, Oh, like, why is this architecture part helping or not? We wanted to understand this a bit better.
David Bau: Does this help, Chris.
David Bau: Yep, I think Chris is totally is ahead of me. Okay, cool.
David Bau: yes, fast Internet on the train.
David Bau: I have a sort of theoretical question. Do you think if you zoomed in and like, made the squares more fine grain, it would just like get fractally at the edge of the boundary between where it seeds.
David Bau: I mean, I think, at the I mean. I've also seen these kind of plots where people change like learning rates and so on, right where you really see the fractal stuff? I think
David Bau: it will eventually happen
David Bau: if if you zoom in enough because of like the numerical stuff. But yeah, I mean, on this level. I don't know. I've not tried
David Bau: because also, the thing is that the embedding dimension and the hidden layer size they're linked discrete variables. Right? So you would need to get away to.
David Bau: maybe with precision or something actually interpolate between one or less more neuron.
David Bau: Yeah.
David Bau: So yeah, kind of the rest of this.
David Bau: This work is then about understanding what's going on here.
David Bau: and you don't see it here. Oh, I have to move Chris over. Yeah, just a little. I can also say that this is like length 10, and there are 32 samples, and I kind of colored the
David Bau: where my hidden layer size becomes larger than the number of things I want to distinguish. And when the dimension becomes larger than the number of things I distinguish.
David Bau: yeah, because, you can actually write down theoretical construction for the weights for these different areas.
David Bau: That tell you.
David Bau: Okay, there exists a solution. Right? This is an expressibility results. And they are like the kind of results we have are for General T and L.
David Bau: With kind of limits on the precision of the operations.
David Bau: But kind of what I want to do is just look at a few maybe more intuitive things that kind of come out of these. If you're interested in how like you would actually write these down, we can maybe chat later.
David Bau: Yeah.
David Bau: So if so, so, there's kind of 2 cases distinguished. That we're handling is when you have
David Bau: token embeddings that are orthogonal versus. There's some overlap, and for kind of the beginning we only consider the ones that are orthogonal because
David Bau: kind of makes sense. Right? You want to compare stuff, and you can tell it's the same or not by looking at the overlap
David Bau: and then understand the role of the attention versus the feature, mixing and kind of what you can understand is that
David Bau: also, if we look at the attention matrix. There are 2 different ways of using them. Where, in the one case which we kind of call relation based. Counting this matrix, A is really used to compare things to one another and to extract like, if you're a token, C, to find all the other token C's in the sequence.
David Bau: and the way this works is that.
David Bau: yeah? And then kind of once this is done after this aggregation is done, there's information in like a new subspace where you count things.
David Bau: That is aggravated. That then
David Bau: represents how many seeds there are, for example, and when you have this beginning of sequence token, it becomes really evident. But right, this is something people have seen before that
David Bau: this thing here becomes a proxy, for how often did B occur, or E occur, or d occur
David Bau: And the way this
David Bau: you can construct this is that you take everything almost orthogonal, except to like one small direction, which is counting subspace
David Bau: and then you apply this matrix a afterwards.
David Bau: This
David Bau: value of the view, as again the proxy for how much there is, and then your only task in the feature mixing is to read out this direction. Independent like, you can lose all information about
David Bau: the the tokens that you are like the token that you work in some sense. Just have the magnitude information that you read out.
David Bau: So this is more parameter efficient in that sense.
David Bau: And then there's the other version where you really just use the
David Bau: token mixing to aggregate information. So you say, okay, like, let me just get all of the tokens.
David Bau: let me sum them together. And now
David Bau: the direction in the sum together. Token
David Bau: so kind of after after your token mixing, you just have
David Bau: a representation of a set with a special element. Right? This diagonal thing that you are yourself, the token you want to extract.
David Bau: And then what happens in the multi in the Mlp is that with every neuron that you have, you can read out one
David Bau: of the elements in the alphabet.
David Bau: So you can say, okay, like,
David Bau: yeah, basically, what you need to do is you need to
David Bau: memorize the whole embedding matrix again here, so you can project onto it, and then you can extract the
David Bau: by some like regular threshold, and you can extract the final magnitude of
David Bau: the token that you care about right like somehow in here.
David Bau: Into entering the queue. Finally.
David Bau: so these kind of 2 mechanisms
David Bau: change how big your Mlp needs to be. So this is basically explaining why, some of these are kind of only happening up here, whereas these 3 are happening over there.
David Bau: And maybe to go to Chris's question, why.
David Bau: why do. I only use kind of the blue mechanism up here.
David Bau: and not like, and the better one down here, whereas here the soft Max, I can keep it. That kind of
David Bau: the the problem that you get with the soft mix without the bos is that it's sort of normalizing out the subdirection which is supposed to be with a magnitude so like the information that you want, that
David Bau: kind of gets normalized hours and disappears. And this is kind of the reason why.
David Bau: with the dot product and the soft mix, you have this problem, whereas you don't have the soft mix doesn't happen. So you find these sort of
David Bau: hmm?
David Bau: And then the other question is, okay, why do we still find solutions when
David Bau: the embeddings are overlapping? Right? So there's some sort of errors that are introduced in the comparisons.
David Bau: And why are they kind of at different magnitudes, or like different efficiencies?
David Bau: Right here I can go down up to here where it's here I go on the 3 rows.
David Bau: So those red, red, those red and orange boxes. What kind of solutions are those? So they're in order to find them, we use some
David Bau: guarantees on.
David Bau: So so for the the pink ones.
David Bau: you basically just rely on the fact that
David Bau: almost orthogonality only introduces small errors, and almost orthogonality is quite easy to get right. Like, if you sample random vectors
David Bau: in smaller dimensions, you can get a factor here.
David Bau: And the other thing is that when I only need to distinguish
David Bau: sequences of length or counts of length. L.
David Bau: Then I have some sort like sort of margin. So here you see, and of how I map this value in the Bos token to the final output class of how often did this item appear?
David Bau: And because of the softmax, there's some non-linearity in here, but there's still kind of some margin for error like you don't have to exactly be on 3
David Bau: in order to predict the 3 in the final classifier.
David Bau: There's a question long one.
David Bau: Oh, yeah, look, let's see here.
David Bau: That's Chris so involved you this window doesn't? Oh, I can grow it.
David Bau: But even though I can grow it.
David Bau: feel free to ignore this. Oh, okay, I just saw this, so you can read it. What does it say here? Want to read out loud for you?
David Bau: Relation based counting. I did not fully grog. It looks interesting.
David Bau: You have to attach this counting direction everywhere. So then, when you match the token A with a query, you will pull in 3 times the counting direction can be shared across token identities. I think that's a very, very nice way to say it that you
David Bau: kind of you pull in the tokens, and then
David Bau: you get this little contribution in the counting subspace by adding them. So every every token, when you're sitting at the location of a token a at the end of getting through this attention layer, or whatever it it knows there are 3 other A's.
David Bau: Is that right?
David Bau: Yes, yeah. Right? And then in the other solution. If you go to a random token. A what does it know, then, in the other solution? The token a is just.
David Bau: It contains the information that there are 3 A's. But it also contains the information that there are for these exactly. It contains that, and then kind of the task in Mlp. Becomes, you want to
David Bau: know know which
David Bau: to extract the A's in some sense, whereas in the other case, this information is already in a space that is just for content. Yeah, that. Yeah, it's kind of the same.
David Bau: Yeah, we're like, the solution is projected into a specific space where I know that
David Bau: the solution spaces for events.
David Bau: So
David Bau: In the inventory based approach is a At position one distinct from a At position 3 for the Mlp.
David Bau: But does the Mlp. Figure out we are pointing 2 times in direction.
David Bau: Oh, you understand?
David Bau: I think I understand. So is it? Is it? Is it just counting by by watching the magnitudes? Or is it counting by noticing that there's like a discrete
David Bau: like?
David Bau: Oh, you're you're you're an a at position 5, and you're an A at position? 7. 0, those are different things that you could have. You could imagine, like a set
David Bau: like a set representation that has different things for each of these and
David Bau: no, or they're just both A's. And we're just going to add up all the A's. Now that's like twice magnitude, 2 times magnitude at the point where you like. After the token mixing, you don't know anymore from which positions your A's came from. Right like that. Information is gone. It's just been added up to this is how many? A's.
David Bau: Okay? If that answers the question.
David Bau: yeah. And then kind of this magnitude is actually this gamma that you see in the plot right? So
David Bau: like you now have. If you use the softmax, you have a value between 0 and one
David Bau: on this specific subdirection, and you can translate it to
David Bau: account the count. Exactly. So. This kind of the readout layer of of the model.
David Bau: But because it's you're in such a physics area. You're making these little energy
David Bau: mines. It's like, we're orbital. Interesting.
David Bau: So funny. Yeah, it's great.
David Bau: yeah. And then kind of what you see here is that something else? Because the
David Bau: the way we acquire the this magnitude is by through the softmax. There's kind of this
David Bau: nonlinearity part here which makes this kind of thing very small and no unrobust noise in some sense.
David Bau: And kind of yeah, what's?
David Bau: And eventually, if you scale up to large sequences. This is what breaks your model like at some point, you're not able to count anymore because you can't or like count in
David Bau: high values anymore, because you can't distinguish numerically what's going on up there. So
David Bau: yeah, this is resolution. Yeah, it's still a resolution.
David Bau: And then kind of the other thing you can do.
David Bau: That influences robustness is kind of the temperature of the softmax right? And we can think of how how close is softmax to like just the Max by like changing the scale of the Wq. And the Wk.
David Bau: And kind of the closer you have it to the Max the better it is at, you know.
David Bau: pushing the contributions of tokens that are not you down, and the others up, which then allows you to be a bit more robust to kind of these settings.
David Bau: So that's actually where.
David Bau: And if you see that with the softmax you can kind of go down a bit lower here. So you require only smaller dimensions than when you don't have it for for the same case.
David Bau: Yeah, what questions on a second?
David Bau: Okay, cool.
David Bau: Yeah, so, and there are kind of these counterintuitive
David Bau: contributions from the architecture at play in the sense that here
David Bau: the softmax really helps you here. It doesn't. But yeah, overall kind of this
David Bau: dot product. Comparisons really help you with the robustness, right? Like something that is linear. Just doesn't see much improvement to like sort of error, correct out the embedding
David Bau: overlaps that you get from small dimensions.
David Bau: I think that's kind of all about
David Bau: the small, which is, I think, still very ambiguous, ambiguous even, for, like a very small case where you can write down the solution
David Bau: and kind of one of the things that I find find unsatisfying about this is that, like we give the expressions for the weights.
David Bau: and we can kind of
David Bau: try to find analogies. And, for example, measure to know the the rank of the the matrix and the Mlp. But actually understanding which solution is learned is hard, because, especially in some regimes, you can implement different ones. So you might implement them in parallel. So I think, generally, it's kind of what makes me curious about methods and interpretability that actually allow you to compare
David Bau: either kind of from learn solutions? Can you somehow map to the theoretical ones, or can you
David Bau: formulate theoretical ones in a way that's more amenable to kind of interpretability versions. But so yeah.
David Bau: I guess that's kind of off.
David Bau: And so how? So you so in. So you had this you blocked off. Oh, in some regimes you get this solution, some regimes. Again, that solution. But in the very last
David Bau: bullet point, yeah, you say, well, sometimes you get both in parallel. So how does that happen?
David Bau: Would I?
David Bau: The box is is that one specific construction can be implemented by models in this regime, which means that
David Bau: I now have understanding that this model can express it. But
David Bau: I can, and I could. Then I can test, you know, if does the performance go up here and like, I guess, due to optimizations. I don't find like
David Bau: things with just single neuron, right? Even though I can implement them in pytorge. No problem. But kind of
David Bau: this. Is it so? The inventory based solutions in principle? You could implement it in here as well.
David Bau: Page so. And then the question becomes, what
David Bau: what mechanism are you actually using? And I mean in in this work we tested a little bit like, I said, with like a low rankness like.
David Bau: do I actually use all of the the neurons that I have available or not? But it's a bit tough to understand correctly what's going on in the learning? Yeah.
David Bau: cool, excellent. Excellent. Yeah. Eric ran into a bunch of this when he's looking at some of his models, and maybe maybe you can get him to explain some of this stuff offline to you. But where he definitely has some
David Bau: situations where he's got 2 mechanisms that are being used at the same time, and it's really hard to
David Bau: clarify then that you, you know
David Bau: and he's got some different practical things he's doing to try to try to isolate them.
David Bau: Yeah, that would be cool, right? And and there's also, I mean, if you look at this, and you have sort of accessibility results here and here. But there is kind of this whole area that seems to be like doing well at approximating.
David Bau: So why does it look like this I don't know.
David Bau: Thank you.
David Bau: Thanks.
David Bau: As Chris asked all the questions. Any other questions.
David Bau: Right?
David Bau: There's 1 prompted.
David Bau: I can read Chris's question, but I'll let you guys ask questions first, st if you want.
David Bau: I'll ask Chris's question.
David Bau: So when when you run these toy tasks, you can imagine creating an automated mechanism, pipeline that.
David Bau: You know, figures out what the implementation mechanisms are like. So like, they're small enough. You could stick probes everywhere. Yeah, and see if any of these probes like correlate with any of the variables that you're hypothesizing your size. Yeah, I think probes
David Bau: would be nice, but I think it's harder
David Bau: if you want to interpret any of the features, because the model is already so minimal that if you add, if you stick up any component, it will like change the behavior right? Right like
David Bau: oh, so you you looked a little bit afterwards, right? And you right? And so
David Bau: and and you you can recognize the solutions that you're looking for in there.
David Bau: There are some indications that, for example, in this regime here
David Bau: I learn I can implement the blue and the green.
David Bau: But I learn low rank structures that correspond to. Okay, I actually don't need all of these neurons right? Whereas here you don't see this
David Bau: because you actually need all of them.
David Bau: And then
David Bau: what we also played around a bit is with manipulating this like identifying this counting direction. Right? So if you can actually
David Bau: play around. Can you read it directly?
David Bau: I mean, read it directly is a bit hard, because, like, you can read it.
David Bau: I mean, yeah, this one.
David Bau: You can read it off directly if you kind of push it through something manually constructed like this, and then you can have so like we have some
David Bau: bitch in the paper.
David Bau: Did I put the links on that?
David Bau: Jeez.
David Bau: yeah, no problem.
David Bau: Right? So kind of this is what I create. It's bigger.
David Bau: Okay, and then what you can kind of see that the model learns
David Bau: or so so kind of you can think of this grid of plots as embedding dimension, and again the
David Bau: in layer dimension. And you can see how
David Bau: what the model learned. So some of them don't do well, right
David Bau: but you can see that
David Bau: the ones that do actually have the similar like boundary of like classification boundaries to map the
David Bau: yeah, to map kind of from the single counting direction to something like this, the
David Bau: But this is a linear model. So you don't see this kind of soft Max going down.
David Bau: and then, and then one other thing. Okay, let's working the solutions.
David Bau: and this is, for example, looking at kind of the singular value decomposition of these different models where in the regime where they could implement 2 solutions, and and the more spiky they are the more it corresponds to. I guess the relation base, you don't need a lot of neurons.
David Bau: And the more we need exactly like the team
David Bau: directions you, you find us here. But then, again, there are some, we're like, okay, like, you have one that's prominent. But the others are small. So yeah, I think we were a bit shy to say or like, careful to say anything. Yeah, pretty strong books.
David Bau: So do you think that when?
David Bau: So, you have this phenomenon where
David Bau: you switch between different modes of the solution, and then and but then sometimes you have both them present.
David Bau: Yeah, and the same system. Do you think that when the models get bigger
David Bau: you tend to get more solutions where
David Bau: all the modes are present. They just like, oh, we'll just do it all the ways, I guess. I guess you know better. I have tried with these ones. Yeah, but you like, I wonder if that's something that you've explored by increasing the parameter sizes and then
David Bau: and then, and then just sort of observing that it would be like bigger, not in the sense of layers, but in the sense of parameters. Yeah, I think. Yeah, that's what I mean, because I've not made the parameters huge, huge, I see, because, like these are parallel systems, right? So there would be nothing stopping it from
David Bau: taking this subspace and using it for one of the solutions and taking another subspace and using it for another solution, and doing it all in parallel. Or maybe you would need more attention heads or something.
David Bau: Because there's a scalar stop scalar attention, channel.
David Bau: But but you but you sort of know what I mean, because I I feel like there's this, there's this issue. Where in practice, when we, you know, deploy these systems at industrial scale, of course, all these numbers are huge, right? And and then I guess
David Bau: the question is, you know, what
David Bau: what behavior does that encourage? Is that is that
David Bau: so? Like, I feel like there's there could be 2 different stories, right? It could be that
David Bau: one of these types of solutions is the kind that
David Bau: you would converge to if you had a lot of training data whatever. And you and it was just like, Go to one of these or the other possibility is, no, you have this permanent state of affairs where there's multiple solutions that continue to look good. And but then, maybe when you get a model that's huge, it would just tend to learn all of them at once.
David Bau: I don't know like what we would expect. I mean, it's also influenced by the data distribution. Right. So here we only want to solve a single task, and I think my intuition would be that I would still learn all those modes that are useful right that make solution robust, right? But as soon as I now want to use the same numbers to do sorting
David Bau: then maybe one of the solution becomes kind of more compatible. And like, yeah, compatible and like, implementable at the same time. And then maybe it just removes some other nodes. I got it? So
David Bau: yeah, I think.
David Bau: So you think there's an interaction with like the set of tasks. Okay, friends, yeah, yeah.
David Bau: Makes a lot of sense.
David Bau: Then I think
David Bau: thanks for asking. Thank you.
David Bau: Any other questions, any other questions.
David Bau: Otherwise. I was wondering whether the Observables were like where you think it's like an exhaustive set of them, or if there's like more that, just like
David Bau: theoretically, you just like are there? But you wouldn't have a lot of or
David Bau: but like in the the 1st part for the 1st part of the talk, you know, like these 2 are enough to describe the whole system, and
David Bau: we don't have like. It's enough in that.
David Bau: So in the 1st talk, in the 1st talk, to be clear like you, you solve, you solve for the what the optimal solutions are like in closed form, just right like you know exactly
David Bau: what the what the optimal solutions are. Oh, that's why. Oh, I thought you had like, if I was empirical or
David Bau: da-da-da go
David Bau: here.
David Bau: But this is why you had to make this this goal and everything so that you could solve it in close form. So like we, we know that there exist 2 local minima have exactly these properties. And the way. So
David Bau: you know, the way it's usually done in physics is actually not a proof, but like a
David Bau: kind of use existing methodologies to drive your results. And then the lines that you see here
David Bau: are computed just using these observables. And like the way they interact and solving like equations with these 10 parameters or something, there's some hidden ones that we don't care about for interpretation, and then if you solve the same, or if you
David Bau: look at the minima in the same system, for I don't know, I think here. It's like d equal to 100. And this are like theories already predictable kind of these things.
David Bau: and tells you that. And even on that level the
David Bau: the 2 pro, like the observables, are already describing. What's happening.
David Bau: Okay? Also true.
David Bau: What the fuck?
David Bau: There's maybe an only like a given number of equivalent classes.
David Bau: that's a bit like probing, I guess.
David Bau: So. One of the challenges, I think. And we we face this with with Eric's stuff, too. I'll ask a question on Eric's behalf, which is, you know, when we're looking at small models.
David Bau: Then there's this question
David Bau: of like, how much are these lessons transferable to the large ones. And so Eric is, is facing this here where he's finding a lot of, you know, interesting structure in the smaller models. And now we're
David Bau: we're facing this puzzle of like, oh.
David Bau: you know, are the large models trained on all the world's data?
David Bau: Are they? Are they learning similar solutions, or
David Bau: because of the nature of the small data. Are we pushing these transformers to learn something that they they don't don't do
David Bau: on on big models? So I wonder if you you have a perspective on that, or an opinion
David Bau: on the right way of generalizing.
David Bau: You know small model lessons like this
David Bau: to to you know things, you know, lessons for bigger bottoms.
David Bau: Bo, yeah. I mean.
David Bau: I guess the the way you can look at it a bit is like building up kind of empirical understanding like step by step, right like, do you want to explain all of the behavior
David Bau: at one given time? Then that's maybe kind of impossible, but I think
David Bau: kind of highlighting roles of different aspects is totally possible at like the small scale. And I mean, I think, if
David Bau: what you're hinting at with Eric's experiments is kind of looking at old. Is there some sort of convergence between like different representations, or something? Because I don't know. Like Eric? I I
David Bau: you know. I hope you're not disappointed, but I don't know, like you know, how we're going to connect your work, Eric, to the larger models. You know, we have to see right now.
David Bau: but yeah, it's it's kind of taking. Yeah, I don't. I don't think they're
David Bau: immediate connections. You need to be super clear in order to kind of motivate you, to kind of take that approach. Yes, to get there. Yes.
David Bau: yes, yeah. But that's a certain perspective. Yeah, that makes sense. And I, you know, I I share it. I think that makes sense.
David Bau: But some people don't.
David Bau: That's a huge
David Bau: okay, that's cool. I have some pointers for you. Actually, I'm gonna stop the recording here. But thank you very much.
David Bau: So so awesome.