j.fiotto-kaufman: Everywhere.
j.fiotto-kaufman: All right. So today we've got Eric talking about his current research.
j.fiotto-kaufman: And and so I'll I'll I'll give it over to Eric, okay, great.
j.fiotto-kaufman: So we still, we're still settling on the title. But this is our working title.
j.fiotto-kaufman: in context, algebra. The one that I came up with was learning group operations. In context, we'll see. But that's kind of what I'll be talking about today.
j.fiotto-kaufman: So I wanted to start out by talking about, what is a group
j.fiotto-kaufman: so oh, could we close the oh, I can't see all the slides here here.
j.fiotto-kaufman: So she's okay. And this okay, great. And then, oh, paper house, yeah. So
j.fiotto-kaufman: so in math, a group is a set
j.fiotto-kaufman: that is equipped with an a binary operation.
j.fiotto-kaufman: So it takes 2 elements in the set, and then maps back to the set
j.fiotto-kaufman: and and group satisfy 3 properties. So one is associativity. So like, if you have
j.fiotto-kaufman: like, basically, you stick these parentheses somewhere. So a times B is the same times. C is the same as a times B times, C,
j.fiotto-kaufman: and then the for something to be a group. The set has to have an identity element. So
j.fiotto-kaufman: we call this E and a times E should be A and E times A should be a.
j.fiotto-kaufman: and then also sets, have every element of a set has an inverse
j.fiotto-kaufman: which maps back to the identity.
j.fiotto-kaufman: So why, yeah, why, study, can I? Just I just want to. So I'm just like thinking about the symmetry group, for instance, here. So I'm just trying to map like that top thing to the symmetry so like for for one
j.fiotto-kaufman: or like a rotation group, so like one rotation would be, what are the 2 elements there that are that that output.
j.fiotto-kaufman: Yeah, I mean, we'll talk about some of those like specifically. But so we're like
j.fiotto-kaufman: for like a rotation group. For example, you could pick like
j.fiotto-kaufman: one permutation of the group, and then just use that to cycle through all the elements.
j.fiotto-kaufman: But yeah, and we'll walk through an example of that in a bit. Okay, yeah, okay.
j.fiotto-kaufman: cool. Yeah. So why? Why study groups? So yeah, so group theory is kind of the study of like symmetry and invariance under like a set of transformations, and I think these are like important properties in general, like in lots of domains. So the things that come to mind so one like
j.fiotto-kaufman: like
j.fiotto-kaufman: this kind of like symmetry is like natural both in text and in like vision. So in text, like, you can think of examples like this. So like when the when the going gets tough, the tough get going, you know. That's like a nice little symmetry.
j.fiotto-kaufman: That's not what you can, what your country can do for you. But we what you can do for your country. So the and then it's not the size of the dog in the fight, but it's the size of the fight, and the dog that counts, you know. So these are like, so these are called like chiasmus. So these like a B Va patterns.
j.fiotto-kaufman: and then like in the vision domain we use this all the time for like image augmentation, or like contrastive and other representation learning methods, we like leverage these symmetries and invariances to
j.fiotto-kaufman: to learn nice representations.
j.fiotto-kaufman: so so, yeah, so so, what are some of the examples of like mathematical groups here that that we'll be looking at? So one example is like the integers with addition.
j.fiotto-kaufman: And those. So yeah, we'll talk about this later. But like, order is sort of like the size of the group.
j.fiotto-kaufman: So like integers have. There's infinite number of integers.
j.fiotto-kaufman: There's like cyclic groups. So you can think of that as like modular arithmetic.
j.fiotto-kaufman: There's dihedral groups which are like the study of like regular end guns.
j.fiotto-kaufman: And we'll talk. We'll talk about examples of these in a bit.
j.fiotto-kaufman: And then there's symmetric groups which Alex had mentioned, which are sort of the set of all of the permutations
j.fiotto-kaufman: of the of the steps. So it's like
j.fiotto-kaufman: order, N Factorial, if you have a set of size. N,
j.fiotto-kaufman: and this is just an example of like the symmetric group of size 4. So there's like 24 l. 24 different elements, different ways that you can have like little cycles inside
j.fiotto-kaufman: up. Then
j.fiotto-kaufman: so is every is every arrow a group element here, I think. Every well, every set of like, every 4 number numbers is a, is a group element. Yeah.
j.fiotto-kaufman: yeah, sorry. What's the difference between the red and the blue areas.
j.fiotto-kaufman: I, this is from Wikipedia. I think it's just like the the kind of operation that you're performing. So like.
j.fiotto-kaufman: let's just pick like a little subgroup. So like
j.fiotto-kaufman: 4, 3, 1, 2 maps to 4, 2, 3, 1 let's see.
j.fiotto-kaufman: it looks like it's just moving like the 2 to the beginning. Right. If you take this little arrow, and then, like this, one would move the one to the front. Here the number of transpositions to get back to the identity element. That might be, it
j.fiotto-kaufman: could be. Yeah, I'm not entirely sure like, didn't we say we combine 2 elements? What's that? The group operation combines 2 elements right? Right? So I kind of forgot how to read this plot. So
j.fiotto-kaufman: th, this is not that this is not that important, but because I'll I'll have a better plot later to talk about it. Yeah.
j.fiotto-kaufman: so and then. So there's also all these other kinds of structures that you can stick on top of a set. And that's not necessarily a group.
j.fiotto-kaufman: So there's like, there's all these other kinds of things. So like magmas, quasi groups, semi-groups, and they all have different properties. So like a magma is sort of
j.fiotto-kaufman: like the least structured. And all you need is that
j.fiotto-kaufman: like the operation is closed. So you take 2 things. It just brings you back into the set, and it doesn't have any other properties that groups have, like
j.fiotto-kaufman: all that
j.fiotto-kaufman: great, and then just some other like preliminary definitions. So we a group is called Abelian, if it if it's operation, is commutative. So
j.fiotto-kaufman: this isn't always the case. But like
j.fiotto-kaufman: or like cyclic groups. For example, this sort of thing holds
j.fiotto-kaufman: and then, yeah, the order of a group is the is the number of elements that the group has.
j.fiotto-kaufman: and then, just to review, like a permutation, is a bijected mapping from a set to itself. So if you have, like a set of 3 elements you can think of like the out. The identity permutation would just like have an ordered tuple of 1, 2, 3, and if maybe there's like some other permutation that
j.fiotto-kaufman: flips 2 and 3, but keeps one the same.
j.fiotto-kaufman: and then there's another, and then there's lots of permutations. But like this one would keep. Move 2 to the front one, and then 3,
j.fiotto-kaufman: and then there's this nice theorem called Cayley's Theorem, which says, any finite group is isomorphic to a subgroup of the symmetric group.
j.fiotto-kaufman: So symmetric group, remember, is the set of all the permutations of a set.
j.fiotto-kaufman: And so, if you think of any finite group. You can just represent every element in the group
j.fiotto-kaufman: as a permutation of the set.
j.fiotto-kaufman: So so we'll walk through 2 examples just to kind of make that more concrete. Well, does everyone know what isomorphic means? I think so. Yeah, I guess. Yeah, is there any questions? Or up to that point.
j.fiotto-kaufman: It means bijective. Yes.
j.fiotto-kaufman: So here's A, here's a cyclic group
j.fiotto-kaufman: we'll call it C, 8. There's yeah. 8 elements you kind of think of as like
j.fiotto-kaufman: states on a on a ring. I guess
j.fiotto-kaufman: so like there's an identity element. And then the way that you move through the set is via rotation. So this is
j.fiotto-kaufman: so like
j.fiotto-kaufman: it. It would take 8 rotations to get back to the identity right? So if like, move if you like. Imagine you're at State one, you can add, you can like, apply a rotation. You'd get to state 2 applied a rotation. You get to state 3 etc.
j.fiotto-kaufman: So the way that you could like represent these things as permutations of the elements in the set
j.fiotto-kaufman: something like this so like the identity doesn't modify anything. So you just you just write that as E,
j.fiotto-kaufman: and then the way that you'd read this 1st one.
j.fiotto-kaufman: I don't know where my mouse went here. Yeah. The the way you'd read this 1st one is like
j.fiotto-kaufman: 0 gets mapped to 1. 1 gets mapped to 2, 2 gets mapped to 3, 3 gets mapped to 4. So that would be like one rotation.
j.fiotto-kaufman: A different permutation would be 0 gets mapped to 2, 2 gets mapped to 4. So it's like you do 2 rotations.
j.fiotto-kaufman: And so each of these kind of
j.fiotto-kaufman: rotations can be thought of as one of the the elements of the set or of the group. Sorry.
j.fiotto-kaufman: does that make sense like what this notation means?
j.fiotto-kaufman: Yeah, I'm just trying to connect this back to the original definition. Where a group is a function that maps 2 elements of the set to one element of the set. And I'm not. I guess it's it's what are the 2 elements here. So you can think of what's the one element, right? So like this one
j.fiotto-kaufman: you can think of like, basically, you apply this thing to the identity elements.
j.fiotto-kaufman: So so, so. But you're saying, like, where's the multiplication table here?
j.fiotto-kaufman: Yeah, I guess. Yeah. And I think that so the slight gap there is that
j.fiotto-kaufman: if these are the 8 elements.
j.fiotto-kaufman: then you can compose any of these 2
j.fiotto-kaufman: by doing one after the other.
j.fiotto-kaufman: So like like you demonstrated. If you just do r after R. Then you get r squared. And then you get this other group element which is just called R squared.
j.fiotto-kaufman: Right? He's saying you're on an element basis like what happens to one when you apply R to it. I guess. Yeah. So the group elements are not the 1, 2, 3, 4. The group elements are the permutations.
j.fiotto-kaufman: Does that make sense? I think both are valid, though. Yeah, I understand.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: But it's the permutations that you compose like you don't know how to compose one and 2, you know how to compose permutations. This is the cycle notation. So the rule of it is that every adjacent like, for example, in the top left 0 is one and one. The our operation takes 0 and maps it to one.
j.fiotto-kaufman: or it takes one and maps to 2. And when you get to the end at 7, it maps 7 back to 0. Okay, so you can think of it like that that's like more object level. But then, what David is talking about is like, you can generalize. And just think of the elements as the actual, like entire, like collection of numbers. And what happens to the entire. So like you could, you could take like, for example, if you take the this, this second element here, like 0, yeah, so like this guy.
j.fiotto-kaufman: and you like, compose it with this guy,
j.fiotto-kaufman: like. So this one would rotate once, this one would rotate twice. So this is like you'd you'd take this one. You apply it to this one, you'd get this one.
j.fiotto-kaufman: I see, and and sorry. What do the parentheses mean here, so that these are ordered tuples
j.fiotto-kaufman: so like.
j.fiotto-kaufman: So when you have a split, they're like little sub cycles, I guess so like here like 0 2, 4 and 6 like 6, Max, path to 0. And it's it's independent of what happens to 1, 3, 5 and 7. Okay, I see? Yeah.
j.fiotto-kaufman: cool. So so there's another set of groups called dihedral groups, which are like, the yeah, these are the study of like polygons.
j.fiotto-kaufman: So here, so like d. 4 is like symmetries of a square
j.fiotto-kaufman: so so simple, so and here it's not just rotations, but there's also like flips.
j.fiotto-kaufman: So you can think of like flips over any of these lettered axes of of symmetry.
j.fiotto-kaufman: or or you can also do rotations of the corners. So there's like
j.fiotto-kaufman: you could do like 4 rotations and get back to the identity element.
j.fiotto-kaufman: Or you can like, compose those with with flips.
j.fiotto-kaufman: Any any questions about that?
j.fiotto-kaufman: Yeah, here, can you do? R, and then sc.
j.fiotto-kaufman: sure, yeah, you could do that. So what would be the role of what's R. Times? Sc.
j.fiotto-kaufman: that element also needs to be an element of the set. Right? Right? So like, if yeah. So let's say you, you rotated it. And then you flipped over. C,
j.fiotto-kaufman: okay, so like 0. So 1, st 0 would move to 1, 1 would move to 2, 2 would move to 3. Okay, yeah, yeah. You can do that. I'm just making sure that this list is not exhaustive. Oh, these are exhaustive. So anything like whatever like composition of operations you apply, you'd get back to one of these permutations. Oh, you. Oh, okay, okay.
j.fiotto-kaufman: okay, cool.
j.fiotto-kaufman: Just be clear, because this notation is different than I'm used to. So each of the A BCD is like which axis it flips over
j.fiotto-kaufman: right? So you like, you could even just treat this like there's only 1 1 possible flip, and then like, compose
j.fiotto-kaufman: rotations and one flip, because you can get to everything with a flip and rotations. Yeah, I'm used to seeing it as R, and then R square, etc. And then just T, and then so that so that's fine, too. Yeah.
j.fiotto-kaufman: So so just make sure I understand. So the second row there would be the operation that takes that square, and then maps the top, left to the bottom, right and the top right to the bottom, left. Yeah, so it would kind of like, do this. And then this.
j.fiotto-kaufman: which is equivalent to 180 degrees. Yeah, yeah, makes sense.
j.fiotto-kaufman: which? Yeah, which is cool. Right? Like, you can do sort of these, these flips.
j.fiotto-kaufman: these double flips where you can like, do this rotation and ends up being the same thing makes sense.
j.fiotto-kaufman: Okay?
j.fiotto-kaufman: Yeah, any other questions.
j.fiotto-kaufman: Okay, cool. So, we're currently working on like studying
j.fiotto-kaufman: transformers and training them to like, learn these sorts of
j.fiotto-kaufman: group operations in context. So we're just using a standard transformer architecture. Pretty small models. They're like 4 layers and have
j.fiotto-kaufman: around 8 to 16 heads. And just a pretty small vocab size.
j.fiotto-kaufman: what is what is standard these days like? What is? What are you basing it off of? What is that model. So this is like.
j.fiotto-kaufman: Carpathy's mint. Gpt, okay, cool? Yeah.
j.fiotto-kaufman: But then I think, like, we're using rope and stuff like that.
j.fiotto-kaufman: Yeah,
j.fiotto-kaufman: And like, gel, you and I don't know.
j.fiotto-kaufman: I think that one uses J. Lube by default. Yeah.
j.fiotto-kaufman: So yeah. And then and then we just train it via, like autoregressive cross entry loss. I'll show you what the data looks like in a second. But basically, the way that we generate data is you, you take a group of order. N,
j.fiotto-kaufman: and you sample. So you generate a sequence by sampling a bunch of demonstrations of these, A B equals C pairs.
j.fiotto-kaufman: It just concatenated them together.
j.fiotto-kaufman: and for every sequence that we sample in the batch. The vocabulary is also randomly sampled from from the pool
j.fiotto-kaufman: of of vocabulary. So like, if you have a group like specific group c, 5,
j.fiotto-kaufman: like 1, 2, 3, 4, 5, and 2 af 9, 6 are both equally valid.
j.fiotto-kaufman: So the vocabulary symbols themselves are kind of arbitrary, and the model has to learn how to map like
j.fiotto-kaufman: 2 is actually the 0, the identity elements. And one is, or a is
j.fiotto-kaufman: the 1st element, and and so on.
j.fiotto-kaufman: So here's kind of what some of the data looks like. So we've got these A B equals C pairs. That are
j.fiotto-kaufman: representing these these group operations.
j.fiotto-kaufman: So for like, if you have the vocabulary, 0 1, 2, 3, you get something like this so like 0 plus one. So this is like the cyclic group. So you can think of it as like modular addition, like 0 plus one maths, 1, 3 plus 2 is 5 MoD. 4, mass back to one.
j.fiotto-kaufman: one plus 2 is 3, etc.
j.fiotto-kaufman: and then same thing. So. But if you have a different vocabulary now B acts as like the 0. So like 4 plus B would be 4
j.fiotto-kaufman: etc. And then we we also have been playing around with, like other strategies of
j.fiotto-kaufman: and formatting the data. So there's this one that is like called chaining.
j.fiotto-kaufman: where, instead of having, like these nice separators between every example, the answer to one example is the input part of the input to the next example.
j.fiotto-kaufman: and then we've also got this other setup that maybe we started training like 2 days ago, or yesterday, or something.
j.fiotto-kaufman: Which we're calling multigroup, which
j.fiotto-kaufman: instead of sampling just from a single group. Or you can sequence, you're sampling from multiple groups in the same sequence.
j.fiotto-kaufman: And I don't remember, are they using distinct vocabularies? Yeah,
j.fiotto-kaufman: yeah. So like, the model would be able to tell which group it is based on vocabulary, I guess.
j.fiotto-kaufman: So when you're sampling from the group are using just like, are you literally just randomly sampling valid operations on just the entire group? Or are you trying to sample from a, for example, maybe a subgroup and trying to see if it generalized on the structure, like, for example, if you're on the group of 24 elements.
j.fiotto-kaufman: maybe you're training, maybe you you have it sampled on, you know, increments of 3. So 3, 6, 9, 1215, 1821 0, and then see if it generally, if it's able to do it for 4, or do it for, like, you know, a different or like 6 or something like that?
j.fiotto-kaufman: Yeah. So I I think so. Dave. David's been doing most of the training.
j.fiotto-kaufman: But I think what we're doing is just fixing like the the group size
j.fiotto-kaufman: and or yeah, the order, I guess. And then just sampling like stuff of that order. I don't know if it ever sees stuff of lower order.
j.fiotto-kaufman: Oh, oh, yeah. So during training. So there's 2 questions. So one is, you sample all the groups up to a certain order
j.fiotto-kaufman: right? And then in multigroup you can sample different groups of like a c 4 and dihedral.
j.fiotto-kaufman: you know, 5 at the same time, or something.
j.fiotto-kaufman: But then,
j.fiotto-kaufman: But then, for when you sample facts from the group, it's just uniform random sample from the kley table. You wouldn't want to enforce, like, maybe you want to pick up on subgroups or something like that. No, actually, if you only sampled from a subgroup. Then how would you ever learn what the rest of the group is? Because the subgroup is a group it's just closed under. You're giving no information. It would be like this impossible task, I guess, if there's a pattern, though, where it like loops back. Oh, but it's if it's random letters.
j.fiotto-kaufman: Oh, right? Yeah, so it has. It has no idea. If you've only shown it, the even numbers. It's never seen an odd number. It wouldn't even know what symbols are assigned to the odd number. It is
j.fiotto-kaufman: right.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: Quick question about the data is the hat in the beginning, supposed to be like something that the model would learn to know. It's changing. This is just like a beginning of sentence token style thing. Yeah.
j.fiotto-kaufman: and I guess I guess you could like stick these in the middle of 2, like in the same prompt like. You could have multiple kind of like. This
j.fiotto-kaufman: multigroup thing, if you want it
j.fiotto-kaufman: are, are the models pre-trained? Or is it just oh no. These are trained from scratch, yeah on just on data like this
j.fiotto-kaufman: and they take like, I don't know, like a day or 2 to train or something.
j.fiotto-kaufman: Yeah, they take a day or 2 to train to be okay, and then we try to train them to week to try to get them good
j.fiotto-kaufman: cool. So here. So here's just an example of like the
j.fiotto-kaufman: how they're getting evaluated. So the way you can read this is like.
j.fiotto-kaufman: so here's the example. And then in brackets is the
j.fiotto-kaufman: thing that the model predicted, and then to the right of it is the actual, the actual, correct answer.
j.fiotto-kaufman: So for, like a many shots prompt, you can look at like the end of the sequence like 4, 3, the models predicting B, and the the correct answer was, B
j.fiotto-kaufman: example. So in this, in this batch here, I think it got everything right that I can find.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: you evaluate it on algebraic properties, like, for example, maybe you want to see if it's like learning, or it's falsely learning that the group is Abelian. So the order is the same. So instead, like seeing if it predicts like, the both things are the same. Yeah. So we have this a way to like hold out, to have a holdout. So like.
j.fiotto-kaufman: for example, if you're testing on 3, 6, you hold out any examples of 6, 3 in the in the sequence, so that it can't rely on. Just like if the group is Abelian, just like copying.
j.fiotto-kaufman: And then yeah. And it, some of the models can still do that sort of thing.
j.fiotto-kaufman: So so each of the models is trained to perform a very specific operation on a very specific group. Is that correct? Right? Yeah. So we've trained a bunch of different models. Some of them only see like one kind of group like cyclic groups, or only see like dihedral groups. Some see both kinds of groups. Okay? And the operation is like one operation. Not like, is it all the R's? Or is it just one r
j.fiotto-kaufman: So
j.fiotto-kaufman: I guess it's there's just one R, because, like you're, you're randomly flipping the vocabulary and like for like for cyclic groups. For example.
j.fiotto-kaufman: you only need one like operation to cycle through the whole set, like you'll need one rotation.
j.fiotto-kaufman: But like for dihedral groups, you need like a rotation and a flinch.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: can pre-trained models do this? Have you tested like, llama? Yeah, I don't know. I haven't tested it yet.
j.fiotto-kaufman: It's worth worth trying.
j.fiotto-kaufman: So
j.fiotto-kaufman: one of the cool things that happens when you like train these sort of things is, you see, these nice little like
j.fiotto-kaufman: steps in the in the training loss which we think are like sort of these space changes.
j.fiotto-kaufman: I'll just go through some examples. So here's like a evaluation curve of a similar thing. So like, you see these nice little steps
j.fiotto-kaufman: and this like this, 20% accuracy cliff or like plateau, we think corresponds to like
j.fiotto-kaufman: just randomly guessing vocabulary that's present in the sequence.
j.fiotto-kaufman: So I think this was maybe like a what would be like order 8 or something probably
j.fiotto-kaufman: so like one out of 8 is like
j.fiotto-kaufman: 12%. So it's maybe slightly better than 12. Maybe it's not ordering, I don't know.
j.fiotto-kaufman: But like that's what kind of what we think this little clip is. It's just like it's learned to
j.fiotto-kaufman: that. It needs to predict one of the vocabulary numbers that's actually shown up in the prompts.
j.fiotto-kaufman: and then at some point, then it just like learns what it's supposed to be doing.
j.fiotto-kaufman: You evaluate this. Do you evaluate it on like, let's see, it's a cyclic group. Are you testing it on like raising it to your powers beyond the order of the group, like, yeah. So I'll get to this in some in a minute. I think these one, these ones are not. This is just like the in distribution.
j.fiotto-kaufman: but it held out evaluationers.
j.fiotto-kaufman: Here's some other ones. This one is like
j.fiotto-kaufman: a multi group, one that we were, that they would suggest training most recently.
j.fiotto-kaufman: I think this yellow one is a cyclic ones. I think one is the dihedra one. So there's like these nice school
j.fiotto-kaufman: training curves.
j.fiotto-kaufman: So here, yeah, here's some examples of like what happens when you test like train on like an Abelian group like cyclic groups and then evaluate on like non Abelian groups. So so for the cyclic groups. So like this one was trained on cyclic groups up to order 12.
j.fiotto-kaufman: And so like, it's getting these like C, 8 c. 4, cross, C, 2 and E 8. These are all Abelian groups. And it's like getting basically 100 on those groups. And then when you test it on these groups that are like, not that are not Abelian like dihedral 4
j.fiotto-kaufman: that gets like like 80% or something.
j.fiotto-kaufman: Was it predicting like a billion properties when none existed. I mean, I would guess it right, because it's only ever seen a billion stuff. Yeah.
j.fiotto-kaufman: control for the set, the order center of the group. That is how many a billion pairs there are in the group. Since that has a lot of the rotations are a billion. So it's a lot of
j.fiotto-kaufman: no, this is just like the standard dihedral group.
j.fiotto-kaufman: And then so then on the other side, if you if you like, train on the dihedral group up to order 12 and then test on the same groups
j.fiotto-kaufman: like. Then it can do all of these other ones that the cyclic group struggled with.
j.fiotto-kaufman: I'm actually a little bit confused. Okay.
j.fiotto-kaufman: so so my previous question was, the model has only seen
j.fiotto-kaufman: one type of operation operations, because in your book, in your prompt you have no way to specify the operation other than actually performing the operation rights. Right? So how can you evaluate it on a completely different type of operation.
j.fiotto-kaufman: But
j.fiotto-kaufman: you just give it a prompt that demonstrates that operation, I guess, and just see how it does.
j.fiotto-kaufman: I don't know. I don't. Maybe I don't understand. No, no, that's the question. That's a question. So so think about that. Do you understand? The setting? The in context, setting is cool this way.
j.fiotto-kaufman: So well, how does the operation table size compared to this number of examples?
j.fiotto-kaufman: Oh, yeah. So if you I, yeah, if you like.
j.fiotto-kaufman: increase the size of the group.
j.fiotto-kaufman: let's maybe I don't have a
j.fiotto-kaufman: maybe. Here's a here's a good good instance of that. So like if you increase the order of the group. So these are C, 4, c. 6 and c, 8,
j.fiotto-kaufman: like the you can see these sort of, you need more examples. You need more examples to learn a bigger group. Yeah.
j.fiotto-kaufman: I I have one follow up question. So okay, so while you're training it cute
j.fiotto-kaufman: don't usually expect the in context, learning, ability to develop.
j.fiotto-kaufman: Or let's say, once you have the comma, it's kind of done right it. It's kind of like a different operation with a different
j.fiotto-kaufman: different alphabets in your vocabulary. Right? So you don't expect the in context, learning, ability to develop, to see and learn from example what it's supposed to do. So so within a within, I guess one sequence, at least for the the non multigroup setting. Yeah.
j.fiotto-kaufman: these are all demonstrating the same operation, the same group, same group, same group like, yeah, yeah, I guess my question was, okay. If it can perform other type of operations. Just by looking at the examples, it has developed some sort of in context, learning, ability.
j.fiotto-kaufman: right? It is very surprising to me, because I don't
j.fiotto-kaufman: see obvious how it can develop that, because once you see that comma, it's done like like in the in the 1st example, the 1st purple yeah thing. Purple example. Once it has seen the comma.
j.fiotto-kaufman: the operation that starts after that comma is completely different from 0. 1 equals one. Right? So you can't. You don't really expect it to develop this in context, learning ability to learn from examples.
j.fiotto-kaufman: But it's still doing that. And that's very. I don't understand your intuition. So you see, see the comma? Yeah. And then then 3, 2 equals one. Yep, that's just another operation in the same group. Another operation in the same group. Right? Yep.
j.fiotto-kaufman: So
j.fiotto-kaufman: I mean, I'm just a little bit surprised that how you expect that the in context, learning ability is developing inside the model.
j.fiotto-kaufman: because I mean, you kind of don't expect it to develop in that way, because I mean, after comma, it is doing something completely different. So how is it so like, how is it different from like, if you said.
j.fiotto-kaufman: Hey, you go to llama? Yeah. And you have English word and Italian word.
j.fiotto-kaufman: And you didn't, you know, explicitly tell how to translate into Italian? But you give it 10 examples of like English and the corresponding Italian, the English, and the corresponding time. Eventually
j.fiotto-kaufman: it gets the idea.
j.fiotto-kaufman: You give an English word it says, I know what to do. I'll give you the Italian word correspond to this right. And so this is so.
j.fiotto-kaufman: Now here, it's a little different, right? This. You're giving it a pair.
j.fiotto-kaufman: So you're giving it a pair, and then you're saying on the right hand side of the equals.
j.fiotto-kaufman: oh, this pair means that so? It's sort of instead of English and Italian. You're sort of like I'm going to give you a pair to sum up, and I'll tell you what it means. But this is not completely equivalent, right? Because the pre-trained model has seen a lot of text, it actually has to attend to the context to understand what it's trying to do. But here.
j.fiotto-kaufman: once, after after it has seen a comma. It has no reason to go back and look into its previous context. Does that make sense?
j.fiotto-kaufman: I guess if you train only on one group, right if you train on. Only that's the point I'm trying to make. If you only train on one group, then, once it has, it has seen the comma. It has no reason to go back and look at its previous examples.
j.fiotto-kaufman: So so to be clear, yeah, right? Okay, so there's so it it is when it's trained.
j.fiotto-kaufman: it's trained on distribution of groups.
j.fiotto-kaufman: For example. We take cyclic groups from order 3 to order 10, and we include we sample over all the cyclic groups during training.
j.fiotto-kaufman: And then but then the the killer here is also
j.fiotto-kaufman: every time it instantiates a group, it uses different letters for every one of the group elements. So the group might be represented by the letters b, 7, 4, 1 instead of 0, 1, 2, 3. The next time. There's a training sample. Right? Oh, okay, okay, does that make sense? So it really has to pay attention. It's like, Oh, yeah, I don't know which one's acting like the 0 right now. Maybe B is acting like the 0. I can't tell. You know, it got to understand the structure of what's happening.
j.fiotto-kaufman: And then, after it sees a few examples, it says, I get it.
j.fiotto-kaufman: B is a 0. If anybody adds something to B, it doesn't change anything anymore.
j.fiotto-kaufman: Okay? So the mapping. So the mapping of the pocobilaries and the size of the vocabularys is not constant, for example.
j.fiotto-kaufman: per per yeah. Oh, sorry sorry per model right per per sentence
j.fiotto-kaufman: for sentence. It has to be constant for per sentence. It's constant. But then across across sentences, you keep on, okay, okay, doing different ones.
j.fiotto-kaufman: That's right, the group size changes and the mapping changes. So that we have to do certainly understand?
j.fiotto-kaufman: Yeah, well, yeah, actually, yeah, so even to figure out who is who you need to look at the context. Yeah, to figure who's who? Yeah? It kind of provides all of the constraints.
j.fiotto-kaufman: Yes, the constraints only come from the context.
j.fiotto-kaufman: It's like once you it learn it. The goals, I think, for it to learn the group axiom, and then for a certain number of examples in context, like, it's really easy to see on Kayley table, which is like basically where you have each group element and each group element, and then the each cell of the of the grid is the output of the 2 elements applied with respect to like left to top
j.fiotto-kaufman: And if you have, like a partially incomplete kaley table, sometimes, just because of the axioms of a group, you can fill in the rest of the table just because there's no other like like, there's only a unique way. You can fill it. That satisfies group axioms. So I imagine in this context, like, after you've trained it a while. It picks up on what the rules are of this thing, and then you can give it a sentence which encodes like an incomplete group. But it should have enough information to deduce what the rest of it is
j.fiotto-kaufman: just to clarify and summarize. So yeah, so basically, like, the model itself is learning to train like the I guess it. The type of group itself. And what exactly, is the sequence. And then the in context, learning like
j.fiotto-kaufman: it just gives the context of what the group could be, which is the impression. Okay, yeah, like, if you're doing a group theory test. And you've studied on dihedral groups and symmetric groups of certain sizes, and then your professor gives you some new group of the different order you haven't seen, but gives you enough examples that you like. Oh, okay, this because you have to have an inverse because you have identity. Blah blah, and then you build the rest.
j.fiotto-kaufman: So it's cool, right? So the the so these models have been trained on cyclic and dihedral groups. They've never been trained on direct products of groups, or the Quaternion group, or whatever other crazy groups were in this table.
j.fiotto-kaufman: But then, if you like.
j.fiotto-kaufman: give it a bunch of examples with one of these weird groups that was never demonstrated during training.
j.fiotto-kaufman: Like with some weird multiplication table structure that's never seen. It's like, Oh, I can answer that, even like I've never seen in context Ab before. You never demonstrated. You never demonstrated BA to me. But if you ask me what Ab is, I can get that right.
j.fiotto-kaufman: 99% of the time. So the making should find the group axioms. What? Yeah? So there should like, somehow, it's implementing the group axioms, or some which was only implicit. That's right.
j.fiotto-kaufman: yeah, not cool. But you need it to get the curves up. Yeah.
j.fiotto-kaufman: yeah, unless you have it back
j.fiotto-kaufman: unless there's some bug. Yeah, that's right. I'm still confused when I look at the sequences, because, like.
j.fiotto-kaufman: I'm still confused on what like if is it like the little notes in the graph. Or is it like the rotation? That is the group element? And like.
j.fiotto-kaufman: Yeah, I don't know, like, or if you have a dihedral group, how do you have a rotation and a flip? If there's only
j.fiotto-kaufman: yeah. So I should have. I probably should have put like a big Kaylee table of like how pairs of things map to other things.
j.fiotto-kaufman: But so for like cyclic groups,
j.fiotto-kaufman: it's kind of weird because you can think of like.
j.fiotto-kaufman: since, since you only have like, since the
j.fiotto-kaufman: the way that you transition from like it.
j.fiotto-kaufman: I guess, like transitioning from like states here is like equivalent to
j.fiotto-kaufman: like the operation that happens when you multiply 2 elements of the group.
j.fiotto-kaufman: So I have an idea. Yeah, let's you. We can use Wikipedia. Can you show Kaylee table of the dihedral group.
j.fiotto-kaufman: One good intuition that helps, too, I think, because there's a lot of equivalent representations of group operations and stuff, especially with like dilutial group and permutation groups. It helps to think of them as functions. So like a rotation is a function, and like a flip as a function. And you can compose them. So a rotation and a flip can be thought of as like a composition.
j.fiotto-kaufman: And so, since you know, composing is not usually you can't. It's not. You can't apply a composition in opposite order. It helps with like grounding the intuition of groups. And then it kind of makes sense. Why, you can do these like really complicated geometrics, like rotation and stuff, because each of those are just like an operation. And you can just like.
j.fiotto-kaufman: think of them as functions. Basically.
j.fiotto-kaufman: let's see.
Rohit Gandikota: I'm curious. If Arnab it made sense to you. Why, even after.
j.fiotto-kaufman: Yeah, yeah, yeah.
Rohit Gandikota: Oh, good! Can you hear me?
j.fiotto-kaufman: Yeah, yeah, we can hear you, yeah.
Rohit Gandikota: Even after a comma. Do you understand why the model has an incentive to look back.
j.fiotto-kaufman: Yeah, yeah. Now I understand. Yes.
Rohit Gandikota: Okay.
j.fiotto-kaufman: Okay, so here. So here's the way they, this is proof. Wikipedia, Wikipedia. That's close enough.
j.fiotto-kaufman: And
j.fiotto-kaufman: so here's the way that they represent elements into that group. So they have like an identity element. And then they have A's and B's. So A probably corresponds to rotation.
j.fiotto-kaufman: So like so they have a a squared, a cubed, and then they have B, which probably corresponds to like 1 1 flip, like one of one of they pick one of the axes of symmetry.
j.fiotto-kaufman: Then, like, fix that, and then you can get to all the other elements via composing
j.fiotto-kaufman: the flip, and then rotations so so like. If you take like E, and anything, you'll get
j.fiotto-kaufman: back to that same element, and then, if you like, compose A with something
j.fiotto-kaufman: like you can see like a compose with a 3, gets you back to the identity.
j.fiotto-kaufman: Like a composed with B gets you to like. V. 8, 3, for example.
j.fiotto-kaufman: So these are the things in your ico.
j.fiotto-kaufman: Oh, right? So so you just like pick. So so like one of the Ab equals. X pairs is like
j.fiotto-kaufman: you pick something from the left. You pick something from the right.
j.fiotto-kaufman: you say A, or I guess it's not visible to A and B, but
j.fiotto-kaufman: so pick something from here. Pick something from here like slot, one slot, 2 equals whatever they're
j.fiotto-kaufman: entries in the table.
j.fiotto-kaufman: Yeah. And the vocab size would be just like the number of elements in the grid. Right? So like, here, the vocab size is 8.
j.fiotto-kaufman: Yeah, and just, and you just pick a random that
j.fiotto-kaufman: set of 8 from like the pool of 16. Okay, that makes sense.
j.fiotto-kaufman: Which polygon is this. So this is the square.
j.fiotto-kaufman: So this table encodes all of the symmetric.
j.fiotto-kaufman: all of the transformations that that keep the square unchanged.
j.fiotto-kaufman: like rotation point of view. So that's the
j.fiotto-kaufman: you have a square, and from that you could derive this kind of
j.fiotto-kaufman: thing. You're like, okay, what can I do?
j.fiotto-kaufman: It doesn't change the square, right? Right?
j.fiotto-kaufman: And this is also where this comes from.
j.fiotto-kaufman: I guess. Yeah, we'll see if they have a picture here.
j.fiotto-kaufman: Like.
j.fiotto-kaufman: yeah. So the square here, it's like fixed in spot. Yeah, you still rotate as long as like all the points are like on fixed locations. Right? So like, I think that's something we
j.fiotto-kaufman: yeah. So here, here's like the most intuitive way for me, at least.
j.fiotto-kaufman: so you can like rotate it.
j.fiotto-kaufman: Things of order, 90 degrees like 1, 2, 3, and then, if you do forwards back to the same one.
j.fiotto-kaufman: then they have all these. Yeah.
j.fiotto-kaufman: okay, go back to. The
j.fiotto-kaufman: would be cool if you could see a square in the representation. Like, if you do some. Yeah, exactly like.
j.fiotto-kaufman: let's see, it did. Okay?
j.fiotto-kaufman: So I'll I'll zoom through these other ones. These are just we've trained a bunch of different models and like, been evaluating them on
j.fiotto-kaufman: different kinds of groups. So here's like one that got trained on multi. So rosette is like a combination of dihedral and
j.fiotto-kaufman: cyclic groups that seem both kinds of groups.
j.fiotto-kaufman: And it it generalizes pretty well to all sorts of stuff, except for like this is quasi group, which isn't actually a group. Remember that like.
j.fiotto-kaufman: think we saw at the beginning of like this. So I don't remember what quasi group is, but it doesn't have some of the properties of a group.
j.fiotto-kaufman: It's not associated, it's not associated.
j.fiotto-kaufman: What was that? Okay? No, he would. He tested me on it. I didn't know what it was
j.fiotto-kaufman: that are having, yeah, okay, great and then this is testing it on
j.fiotto-kaufman: groups that are of higher order. So it's only seeing things up to order 10, and then if you give it stuff that's order 12, it could still
j.fiotto-kaufman: kind of do it. But it's not as not nearly as good. Yeah, cool. So
j.fiotto-kaufman: I'll talk a little bit. We've we've sort of been diving in a little bit of like trying to understand what exactly is the mechanism behind how these models do this sort of thing?
j.fiotto-kaufman: So so this the I'll talk about 3 things. So one is, yeah, probably for the identity.
j.fiotto-kaufman: I can then just analyze intention patterns and then do some, some intervention that we've done so.
j.fiotto-kaufman: one thing so like the identity element is like a nice component of a group, and we thought it'd be interesting to see if we could
j.fiotto-kaufman: pick out whether like so, because the vocabulary is random, like the model has to sort of.
j.fiotto-kaufman: We think the model has to figure out like the mapping, but the
j.fiotto-kaufman: elements to which, like the bait, like kind of like the
j.fiotto-kaufman: what would you call it? Like the elementary basis, I guess, of the of the group.
j.fiotto-kaufman: So this is an experiment that rohit
j.fiotto-kaufman: has been working on of basically, you can yeah, train a probe to to determine whether a token isn't is the identity element or not.
j.fiotto-kaufman: And you sam sample these in a way so that it sees like, let's say
j.fiotto-kaufman: you have a vocabulary. So like C, 4, you have like Abcd.
j.fiotto-kaufman: so sometimes a will be the 0, and sometimes A won't be the the identity.
j.fiotto-kaufman: And so, like we we sent, we control for that, and then measure
j.fiotto-kaufman: whether this probe can can detect if something is the identity.
j.fiotto-kaufman: And
j.fiotto-kaufman: it seems like there's sort of this shift at. So these are at different layers, so like layer 0 after layer, 0 after layer, one after layer, 2, and after layer 3
j.fiotto-kaufman: so like after layer 2. It seems like there's a shift of when the model can sort of tell
j.fiotto-kaufman: whether something is the identity or not.
j.fiotto-kaufman: And then oh, and this was for for C, 4, and then
j.fiotto-kaufman: similar thing, for like a dihedral 5, right? It would be very cool to see these accuracies.
j.fiotto-kaufman: Evolve, based on the number of shots
j.fiotto-kaufman: like how accurate is, I mean. I think it's probably the right thing to train a single probe.
j.fiotto-kaufman: but it'd be cool to see what the accuracy of the probe is after.
j.fiotto-kaufman: You know one shot, 10 shots.
j.fiotto-kaufman: shots. I feel like there's like a certain like, there's a point where, when you're given one example, you immediately know the things the identity. For example, if you have, Ab is equal to B, you know by the items immediately, that's an identity. It would be good. Yeah, that'd be a cool thing to do like before and after the 1st demonstration of Ab equals B.
j.fiotto-kaufman: Alright, that'd be another nice one.
Rohit Gandikota: So shots with respect to the Zeros that it has seen, and also the examples that it has seen, so 2 different shots, we can look at.
j.fiotto-kaufman: Sure
j.fiotto-kaufman: like, when you evaluate the group.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So why don't you like look at all of the missing? Or do you look at all of the missing pairs in the table or
j.fiotto-kaufman: randomly sampled. So yeah, so you you can like fix a holdout of like, it's never seen this pair, or it's community of inverse during a sequence.
j.fiotto-kaufman: and then test whether it can get that pair. And then you can just like change that hold out.
j.fiotto-kaufman: So these performance numbers aren't randomly sampled, you know. You know, hundreds of different random holdouts. Okay? But
j.fiotto-kaufman: you would expect the probe accuracy to be higher, right? Like, right? So it's really weird that the probe accuracy is lower.
j.fiotto-kaufman: But maybe it's not it depending on the number of shots, because the accuracy here is like low when you only have 10 shots.
j.fiotto-kaufman: Yeah. But then you probably would include also the shots right for the probe, like
j.fiotto-kaufman: I would take sort of something where they are close to one.
j.fiotto-kaufman: And then look at the
j.fiotto-kaufman: like as you suggested, basically like, after a hundred shots. Yeah, yeah. How does it worse with the shots. Yeah, yeah.
Rohit Gandikota: Okay.
j.fiotto-kaufman: Yeah, I agree, like, it's weird that the the model can like perform the operation of 0. But
j.fiotto-kaufman: like, it doesn't know that the thing is 0. This is a little weird, right? Yeah, I don't know.
j.fiotto-kaufman: Sorry, Roy, go ahead.
Rohit Gandikota: So we do have the same probe accuracies when the number of shot is one like literally, when it just sees the 1st example
Rohit Gandikota: in the entire group, and you can see that all the layers have like close to random accuracies.
j.fiotto-kaufman: Okay.
Rohit Gandikota: Then this is that, I think short, 75 to 1, 50 somewhere, randomly sampled.
Rohit Gandikota: and only at the layer 2 and 3. Do you see a higher accuracy now.
Rohit Gandikota: and still for 0, and one, it still says, stays near the random baseline.
j.fiotto-kaufman: I see. So this is after 75 shots. So it's actually, the accuracy is pretty low, because the accuracy of the model is pretty high after 75 shots. Is that right?
Rohit Gandikota: Exactly. It's 99% accurate
Rohit Gandikota: after 9, 75 shots but still the probe accuracy is at 70 75.
Rohit Gandikota: So there's some gap where the model
Rohit Gandikota: does an operation well, but doesn't know it's a 0 or not, which is kind of wonky.
j.fiotto-kaufman: Is this above like? Does it matter whether it's like above the result.
j.fiotto-kaufman: or like the 2? 0, of like, which which token? Yeah. Which side of the equal side I don't know. Rohit, do you remember? Or I guess, yeah, what are you? What? Where are you picking up the hidden States from to train your probe.
Rohit Gandikota: I'm picking up the hidden States from the
Rohit Gandikota: the 0 operator. So from the left side of the equals where the actual 0.
j.fiotto-kaufman: So like here
j.fiotto-kaufman: or something. Now there's there's 2 spots there, there's the A spot, and there's a b spot.
j.fiotto-kaufman: Are you training the same probe on both spots.
Rohit Gandikota: Yeah, it's the same probe on A and B.
j.fiotto-kaufman: Oh, what if there's 2 different representations? What if there's a left 0 and a right 0? Maybe you need 2 different probes. And then there's also the C spot, the answer spot. Maybe maybe we should try training 3 different probes. We got burned with this in Othello. Right? Do you remember the story there when we trained all these probes on detecting white pieces and detect, you know, predicting black
j.fiotto-kaufman: pieces.
j.fiotto-kaufman: And it turns out that that was totally wrong. We need to detect white pieces only when white was to move and when black is to move, then you should use that same probe to detect black pieces like it's like, there's this parity issue. So so you know, we. So we might want to be careful with that rohit.
Rohit Gandikota: Okay.
j.fiotto-kaufman: Yeah, yeah, did that answer your question. Okay, let's see.
j.fiotto-kaufman: Okay, cool. So the other thing, the other thing I'll chat about before I've done is just some cool attention patterns I found so far.
j.fiotto-kaufman: so
j.fiotto-kaufman: there's there's some interesting heads in Layer 2 so like at Layer 0. All all the attention is just like kind of looking at everything. It's not super interpretable.
j.fiotto-kaufman: And then at layer. One is full, like all the engine heads are just looking at like the very last
j.fiotto-kaufman: query pair which I guess is maybe interesting.
j.fiotto-kaufman: But in layer 2, there's like some nice algorithmic structure to the attention patterns. So
j.fiotto-kaufman: there's 1 head that like attends to like answers. So like, if you have something that's like Abx, it'll attend to the x's, where the
j.fiotto-kaufman: example, looks like something like a and blank X, and then there's another head that attends to stuff where it's like something BX, and then there's there's also heads that
j.fiotto-kaufman: check. Whether this, like the community members, is in the examples, are not
j.fiotto-kaufman: cause. Like, I, I think this makes sense. For like cyclic groups, you can just go ahead and copy the answer. If this thing shows up, you just copy paste
j.fiotto-kaufman: and then there's another head that looks at like all the places where A or B are the answers. So it's sort of like like Michael was mentioning like, maybe it's sort of just like figuring out all the places in the Kaylee table that are helpful.
j.fiotto-kaufman: and then just like deducing what's left and then voting somehow or something. Yeah.
j.fiotto-kaufman: have you tried protrubing these things like these heads like seeing if you protrude the top one and see if the answer somehow relates to. If you had a in one of the slots of the multiplication.
j.fiotto-kaufman: Sorry. Can you say that again? So like it's off one interpret like, if that had fires, then maybe the model thinks, oh, something is getting left multiplied by a. And so maybe, if you protrude that head to make it like for another pair that doesn't have a in it, and you protrude it. Maybe it will then like change its beliefs. I think that A is on the left, or something like that, and you see it in the output based on it.
j.fiotto-kaufman: I see
j.fiotto-kaufman: I haven't. I haven't messed with like the attention patterns. But that's that's I think that's that'd be interesting to do. Yeah, the only thing. So yeah, I'll I'll I guess I'll walk through. This may be hard to see.
j.fiotto-kaufman: Let's see. So
j.fiotto-kaufman: I'll walk through these attention patterns, and then I'll walk through the stuff that I have done with attention. And then, yeah, curious to hear some other stuff, you guys might be interesting. So like this, this one was 2, 7. So this one attends to like a something x, so that here the query is 4, 5,
j.fiotto-kaufman: so like these are attending to like 473-42-6440. It's always attending to the answer. Yeah, it tends to the answer of something that's like a something. X. Oh, that's cool. So you have to squint at this, and like which answers, is it attending to? It's like, Oh, where 4 is the 1st right term. Yeah, that's neat. And so this one, I think this, I think this next one was attending to stuff where the B shows up. So this, like
j.fiotto-kaufman: 2, 5, 7, and then 3, 5 0, were the strongest, and it's like some of these.
j.fiotto-kaufman: It like weekly attends to the equals where the 5 is in the 1st slot.
j.fiotto-kaufman: which I don't know if that's interesting or not. Okay, these ones. So this the l 2 0
j.fiotto-kaufman: looks for the commutative inverse. So this one, if it doesn't show up, it just attends to the last equals.
j.fiotto-kaufman: It's like here I held out 5, 4, so it never saw 5, 4 in the sequence.
j.fiotto-kaufman: But then, if you do stick 5, 4 in there, then it's like strongly attends to the answer.
j.fiotto-kaufman: Let's see. And then 2.
j.fiotto-kaufman: This was similar. So like it attends to stuff.
j.fiotto-kaufman: That's it's kind of a mix between these other ones. So it attends to like the answers where 4 is in like the A is in the A slot, or the B is in the B slot. Oh, like the disjunction of those 2. And
j.fiotto-kaufman: or the the what do you mean? Disjunction? Or, yeah, the the union? Yeah, the union? Yeah. Right? And then it also tends to stuff where it's flipped, so like the A is in the B slot and the and the B is in the A slot.
j.fiotto-kaufman: And then this one, this bottom one, is like attending to places where
j.fiotto-kaufman: the A and the B show up as answers to other queries.
j.fiotto-kaufman: So like it's just a bunch of pores and pipes
j.fiotto-kaufman: so. And this one is a 1. i don't know if I had to send it to those. But
j.fiotto-kaufman: yeah, right?
j.fiotto-kaufman: So yeah, I'll end with just 1 1 passion experiment. So I take 2 different groups. So like a dihedral group and a single group
j.fiotto-kaufman: that have the same vocabulary, but different queries, and then try and check whether I can patch activations from attention heads to get the model to flip the
j.fiotto-kaufman: the group that it thinks is is going on
j.fiotto-kaufman: so like in this case, if I patch from the dihedral into the cyclic, I think 1 6 should flip to
j.fiotto-kaufman: 5 instead of yeah. So instead of from 7, it should flip to 5.
j.fiotto-kaufman: And so like, if you do this a bunch of times for a bunch of different pairs of
j.fiotto-kaufman: like, if you sample different sequences.
j.fiotto-kaufman: You get just kind of like, I I don't quite understand what's going on here yet. I yeah, I know, setting up some more exhaustive experiments, but at the moment it seems a little bit random. There's maybe some structure.
j.fiotto-kaufman: but it seems like there's like you could patch a bunch of different heads
j.fiotto-kaufman: for different pairs of examples, and like, get it to flip.
j.fiotto-kaufman: But it's kind of cool that you can get it to flip at all like you can
j.fiotto-kaufman: make the cyclic group think that. Okay. Now, I'm suddenly a dihedral. I should be doing the dihedral operation instead. And it's like, pretty confident.
j.fiotto-kaufman: yeah. So these are like the clean predictions from the cyclic group. And then when you intervene, that flips back flips to 5. So
j.fiotto-kaufman: anyway, yeah, that's it.
Rohit Gandikota: I have a question. Sorry?
j.fiotto-kaufman: Yeah. Go ahead.
Rohit Gandikota: In the intervention experiment that you've done
Rohit Gandikota: is the the 1 6 operation seen in the context from diethyl 5.
j.fiotto-kaufman: No. So that so it's this one is held out from both both payers
j.fiotto-kaufman: or both sequences. Yeah. So both 1, 6 and 6 1.
j.fiotto-kaufman: So like 6, 1 doesn't show up here, and 1 6 doesn't show up here, and 6. 1 also doesn't show up here.
Rohit Gandikota: Nice. Okay, excellent.
j.fiotto-kaufman: Yeah, but cool. So I'm I'm curious, Michael, the this attention thing. Oh, I have lots of thoughts that could be a group. If you think about firing attention, head, or any of the attention heads as like a sort of like operation. Maybe that forms a group structure.
j.fiotto-kaufman: No, you can apply them together and get another valid output, as if you fired another tension head, and then there's like an inverse, and then
j.fiotto-kaufman: I silly thought, but with the head thing what I was thinking because it actually, you could automate this for the with, because what you can do is
j.fiotto-kaufman: because so say you define a rule, like a question mark equals something. And that's like one head, that into that, specifically, you can think about. So maybe you have that. And then you have question. Mark B equals something.
j.fiotto-kaufman: You can imagine what happens when you fire the 2 heads. Think about all the counterfactual ways they could interpret it.
j.fiotto-kaufman: And then for each counterfactual, for example, it might, it might think, oh, it will only be the operation. A B. Nothing else. It may think it's a question, Mark B. It may think that it's like, just, you know, B gets wiped out. And for each of these counterfactual scenarios you can then, using Kaley table, generate like a whole bunch of valid operations for it, and then test your hypothesis. And you have the tailey table as reference. So you can actually see if it's like learning it or not.
j.fiotto-kaufman: So. And since you have all these heads, you can then extrapolate to like all possible head combinations, which is really cool. It might not be completely well defined, but like, I think, that could be a place to like really just like cool.
j.fiotto-kaufman: Can you show your patching plots again. Sorry? Which ones the heat maps the last ones. Yeah. So the rows are layers here, right? And the columns are heads. Yeah, yeah, layers and head index. Yeah.
j.fiotto-kaufman: and what are, why are there 3 of them? These are just different. If I sample different pairs
j.fiotto-kaufman: so like a different dihedral sequence and a different cyclic sequence, but with the same. And are you patching the head output?
j.fiotto-kaufman: I'm patching. Yeah, the heads outfit.
j.fiotto-kaufman: So the 3 different heat maps are 3 different experiments. Right? But but on exactly the same time. Same experiment, same same
j.fiotto-kaufman: held out. Query, just different demonstrations. Okay, so like, yeah, it's still still trying to solve what? 1 6
j.fiotto-kaufman: is.
j.fiotto-kaufman: Yeah, yeah. So it's kind of a mess. It's kind of a mess. Yeah, well, but like, I mean, the demonstrations are randomly sampled.
j.fiotto-kaufman: Kayleigh Tabor entries.
j.fiotto-kaufman: So it makes sense that different heads would be responsible, maybe for different.
j.fiotto-kaufman: Maybe. Yeah, for maybe the heads. Just look at the position of the examples or something.
j.fiotto-kaufman: you know. And so then you randomly sample your demonstrators, and they end up in different positions
j.fiotto-kaufman: like the relevant ones.
j.fiotto-kaufman: Oh, yeah. Well, so I I don't think the attention, at least of the attention hits I was looking at. They're not. They're not position based. They're definitely contact police, right?
j.fiotto-kaufman: Yeah. One seems to be which one layer, one layer, one. Yeah. Seems to be like, at least.
j.fiotto-kaufman: Oh, this guy. Okay, yeah, yeah. That one shows up sometimes. Yeah, 1, 3. But also like in the 7, it shows up until
j.fiotto-kaufman: 1, 7 and 1 7, yeah, in each of and all of them at least. Layer one has
j.fiotto-kaufman: there this one. Yeah, like, yeah, like, there is some like some overlap. So you should annotate. Oh, no, sorry.
j.fiotto-kaufman: So like all the ones that I was showing, you guys are like in Layer 2, and like
j.fiotto-kaufman: at least in these 3 samples. None of them show up.
j.fiotto-kaufman: but like in a bunch of other ones that I that aren't here like they'll they'll show up as important. There's only 24 heads. Right? You change the query.
j.fiotto-kaufman: So yeah, so I'm really, really curious to know what what sort of stuff is like going through these, how would you like you have any ideas of
j.fiotto-kaufman: algebraic structure that you can test in this? Because, like, I have so many thoughts of like, Oh, you could probe to see if it understands when there's an inverse operation applied, or like. If one elements inverse another, you can think oh, is the is this an even group element? Is this an odd group element? Like there's like a lot of like.
j.fiotto-kaufman: you know, is this one? Is this element in the center of the group. Is this, not in the center of the group, etc, etc, like there's like so many. And with each of these things. There's like a very valid definition of what it means for the thing that means you can generate examples, and also that you can evaluate on. Yeah.
j.fiotto-kaufman: So yeah, I was curious. What? Geometrically or algebraically, you're kind of leaning towards with some haven't done anything with that yet. But
j.fiotto-kaufman: yeah, the probing is powerful here, Buddy, you know, even just
j.fiotto-kaufman: to include probes for all of the possible elements of all of your groups, for something that you consider.
j.fiotto-kaufman: but not just the 0.
j.fiotto-kaufman: Just train it for it. You could have it for once you figured out the 0, you could have it for every
j.fiotto-kaufman: element.
j.fiotto-kaufman: Yeah, it seems like the 0 is kind of a warm up, right? Because 0 is unique and stuff like that.
j.fiotto-kaufman: But yeah, you agree. That's a good idea. Send that
j.fiotto-kaufman: you're going to create like a big like, you know, a giant like to train the model. And then you just like kind of meticulously document like every single like correlation in it, so that eventually, maybe there's some like giant thing that makes sense in respect to how it's understanding like the group algebra.
j.fiotto-kaufman: Oh, this is kind of cool. Yeah, I don't know.
j.fiotto-kaufman: So previously you were also looking at like
j.fiotto-kaufman: Pcas of embeddings and stuff like that. Right? Yeah, you know, maybe you haven't looked at in the current model. I haven't looked at the most in the most recent models yet.
j.fiotto-kaufman: But there, yeah, there's some nice. There's some nice like if you cluster hidden states of different like.
j.fiotto-kaufman: what do you call them?
j.fiotto-kaufman: Like group elements
j.fiotto-kaufman: they like. They cluster nicely and like the like. For example, I think, like the 0 element was like all by itself in that little space when you do Pca on the
j.fiotto-kaufman: and then, like the other ones, are like, I think these are for secret groups. And it's like in a nice like, straight line
j.fiotto-kaufman: and stuff. So there's like. There's, I think there's some cool geometric structure that that the models like
j.fiotto-kaufman: that we can probably account.
j.fiotto-kaufman: but I haven't done it in the most recent ones which are, I think, they're better now. So the new models yeah.
j.fiotto-kaufman: excited to see if, like the classic group stuff like like group product, or like
j.fiotto-kaufman: decomposing groups into subgroups like, whether there's like an algebraic way that it represents that yeah cause it cause it cause we we like tested on some some of the of the direct products and stuff so like like support. C 2, you know, and like.
j.fiotto-kaufman: that's just fine. Even though I never saw that during training.
j.fiotto-kaufman: So like, it's gotta have some some nice little subgroup stuff that it's learned how to do.
j.fiotto-kaufman: Yeah, yeah, it's cool. I think it's a cool setting.
j.fiotto-kaufman: Alright. I think I think I'll do this. Okay, cool.
j.fiotto-kaufman: Thank you guys.
j.fiotto-kaufman: So we'll we'll convene again at noon. I think that we've got our seminar visiting Speaker for this week is this project called the National Deep Inference Fabric.
j.fiotto-kaufman: And they're gonna they're gonna overview you know, the back end design
j.fiotto-kaufman: that right? Very cool. So see, you guys.
Rohit Gandikota: You.