Alex Loftus: Here.
Alex Loftus: But this is good. So you know, we we have been doing so many external talks. And like, I feel like, Oh, we need to, you know, get back to more of the routine of just like talking about research and progress and things that we're thinking about and helping each other think through. You know our our experimental settings, and you know I kind of want to have this meeting. Be a forum where you can come with.
Alex Loftus: like the naughty, you know problem that's just like in your way in research, or the the puzzle that you're trying to think about, just like, just share it, have us just chat openly about it.
Alex Loftus: So
Alex Loftus: so yes, right? So we're gonna try to create more of these kind of things right? Do that every week instead.
Alex Loftus: are we feeling at that?
Alex Loftus: I mean, next week we have. And Sam, Sam, it depends. If you consider him external or internal. And then. June 6, th we do have an open slot for internal. So oh, but you're out of town, which is why we have an oh, I need to make sure we have an external file, ship it across. But oh, yeah, no, I'd be happy.
Alex Loftus: Yeah, yeah. Yeah. So the other thing that we could do is, we could have
Alex Loftus: kind of a talk, you know, like time slot where we have office. Just do you guys, do you guys think that we should do that kind of thing. Yeah.
Alex Loftus: yeah, sometimes I'm done. I mean, there, I mean. So on June 6th you are out of town. Do you think you should do just lining up around then, or do we want to do that? Maybe it's good, you know. I like to
Alex Loftus: be here for that. But let's see. June 6.th We're that's okay. Maybe let me look June 27.th
Alex Loftus: It's so. It's like, because I do feel like that's a little missing right now, like, I don't feel like I'm currently up to date on like what what everyone else is doing, you know. So you could do it both. You could do right. You could do June 6.th
Alex Loftus: So that was because in May it got pushed back so.
Alex Loftus: And it's and with Elliot is more like they're going to bring someone who did something with insight. Oh, and like, give a talk, I see. So it's sort of external.
Alex Loftus: that is true. Maybe we should just give them external slots. Okay? Yeah, it's at 10 am. For not. I've been trying to put that as like a thing. I think the external slots at 10 am. And then internal ones at 12 is is good, because then that lets the internal ones like run way over if we want, and then
Alex Loftus: have that for I guess from the end. Only just because he's flying from like in Israel, okay.
Alex Loftus: is this an internal slide? Should I leave? It's not. Yeah. No. Anyone is allowed to hang out. But
Alex Loftus: yeah.
Alex Loftus: yeah. And it's gonna it's like, much less formal. It's like, this is what I'm doing right now. I don't know what I'm doing. Like.
Alex Loftus: what are everyone's thoughts and like ideas for what to do. Like, you know, on that note? Yeah, on that note like, you know, let's do that for me. Yeah, let's do that. So yeah, Alex, thanks. Okay. So I actually like, haven't really? I think the last time I gave a talk it was like on what's in the textbook?
Alex Loftus: because that's what I was thinking about at the time. So this is actually like, I haven't really talked about my research like at all, really to like everyone else yet. So so anyway, so like 1st semester. David and I kind of like
Alex Loftus: circled around different like settings. Can everyone else see it? And I just want to see maybe at the top. Where's the audio?
Alex Loftus: You you guys, yeah, you guys can hear, okay, good. Good. Good. How do I? How do I go back? Oh, here we go. Okay?
Alex Loftus: yeah. So Dave and I kind of circled around like different general settings. you know, of, like, kind of areas to work in. And we we kind of settled on the code setting, because for like a lot of reasons, that I'm now like pretty pretty
Alex Loftus: black black build. I'm pretty. I'm pretty like, you know, in for the for the code setting, because I think there's a lot of sort of super nice properties of working with code and trying to do interpretability on code that like language doesn't have
Alex Loftus: so you have all this formal structure in a programming language that makes it really concrete to come up with experiments. Because you have all these like formal guarantees with programming languages that, like you don't have in English, and it's just a lot more like tangible a lot of the time.
Alex Loftus: And I also would make the argument that it's like quite understudied because you know, I mean, there's like Francesca in Arjun's lab. There's been there's like some other papers. But you know, in comparison to the number of studies that are, you know, doing stuff with language? It. The language studies kind of dorf it
Alex Loftus: and I and I. The reason my argument for why that means, it's understood, is, if you actually look at what people use language models for a very, very large plurality of like what people are using language models for in practice, particularly like Claude, is writing code, you know. I mean, like cursor has, like a gazillion users. Now, right? And and so code is actually like a really really important topic to to look at because and and like.
Alex Loftus: like some very large like, I think, like 40% of Google's like code now is like, Llm generated. So and you know, if you look at the new like the cloud release yesterday, for instance, if you look at the benchmarks, it was like half coding benchmarks. Right? So all these all these companies are like, really focused on making models better at coding right now. So I think it's sort of like a kind of a hot hot topic to to study?
Alex Loftus: so there's like a lot of different questions. That you can ask in the code setting. So some of the ones that I'm interested in are
Alex Loftus: you know. So the current thing that I've been looking at is Do Llms track variable states. So if you have, if you like to find a variable in python, and then that variable changes over the course of your python program.
Alex Loftus: Does the Llm. Know what all the intermediate states of that variable are, you know, like, does it need to know that in order to produce good outputs? And if so, you know? Can you? Can you localize that? Can you like find a representation for that inside the model? Yeah. And so some other questions are on the training process? Right? So
Alex Loftus: so where in the training process do Llms learn coding abilities? And what coding abilities did they learn? How does that work do they learn like skills, that where you need some set of skills in order to build the next ones and like with human learning, for instance, you know, like psychology, you know, learning and psychology is this huge field that's happened since, like the seventies, right? And
Alex Loftus: and with with humans, you know, you look at like human experts learning chess right, and a novice at chess will be trying to memorize like every individual piece, and where it is on the board and like a chess expert, will you know the the fundamental units for a chess expert is is groups of pieces right? And like
Alex Loftus: patterns of board States right? And that's like chunking right? And so do Llms do something like that as they're learning how to code. Is it like an entirely different mechanism. There's all these sort of questions that you can ask about the learning process here.
Alex Loftus: Is there performance? You know.
Alex Loftus: as measured empirically by like performance on like a coding task. Does that improve continuously over the course of training right like performance on benchmarks? Or does it do like some grocking thing where? You know, it
Alex Loftus: learns very quickly. And if it does, some grocking thing, does that happen on like a subset of problems that all have the same type of property. Right? Or is it like, you know? Like, like, yeah.
Alex Loftus: and then I'm I was also, I'm also really inspired by this.
Alex Loftus: by by this work by Wattenberg and villegas.
Alex Loftus: So I thought this was really really cool, and you know, and I thought, like, you know, if we could build something like this, but for code it would be great. So just to, to, you know, and this will like lead into my motivation for the current project. So just to
Alex Loftus: tell you guys a little bit about what this is. This was this paper that Martin Wattenberg, and, like Fernando Villegas, made like a year or 2 ago, or something that I've seen Wattenberg like present a few times, and it's essentially like
Alex Loftus: on, I think they they it was like on, on, like llama, 2 or something. So it was like a while back, but they they essentially they trained probes. On
Alex Loftus: on what? A chat bot thinks your demographic properties are so so you know, they trained probes at every layer on the age of the person. talking to the model
Alex Loftus: and their socioeconomic status, and their education level and their gender.
Alex Loftus: And then, as you talk to the model you can like, look at these little toolbars like changing in probability, as the model starts to build a better and better picture of you. And so I thought that was really really cool. And I was like
Alex Loftus: man. If we could do something like that and build an interface. But for like, when you're writing code, right? You could, you could, you could have some interface. That's like this is what the model thinks all of the variables are in this python program, and like, this is you know, this is like
Alex Loftus: what the model thinks. All of the types are in the in the program, and like this is, you know, the model knows immediately what like programming language we're working in right? And then, like, you know, you could see it like you know. Say, say, you like, you know you call some some like Triton code, or whatever right like, does the, you know? Do you see some area where it's like, it's like starting to think about C code, or something right? you know. So I think I think there's there's sort of like interesting potential there. And so that was like a little bit of my motivation.
Alex Loftus: So the current project is like,
Alex Loftus: getting slicing off like a very small chunk of this, which is, do language models represent variable states as you change them. And can we track down where? And so this is like super incomplete right now? And you know. So I'll you know. Hopefully have some like more experiments on this. Pretty soon. But
Alex Loftus: so so the problem setup is pretty straightforward. You initially. Yeah, thank you.
Alex Loftus: He has a site here this week. That's true.
Alex Loftus: Yeah, no go for it. Okay.
Alex Loftus: so so here's here's like my, my like, current experimental setup. And like I had this slide earlier that I kind of like glazed over, but like like.
Alex Loftus: I'm super not confident about like any of this setup, and so if anybody has like ideas for how to improve the setup, or like make anything better, or like interesting ideas for experiments, then like, just interrupt me as I'm talking. And yeah. So the setup right now is you initialize some variable. Call it x at some number call it 0 and then
Alex Loftus: in the most basic form of this, you just pick some random number V, and then you either add or subtract V from X, and you do this for, like a few lines of code, and then at the end, X has some internal value, and you ask the model. What's the value of X now?
Alex Loftus: And you know. And and so, in order to know what the value of X is, the model during its forward pass, and and I like take off chain of thought. I'm like, no, just only tell me the value of X. Don't, don't give me anything else because I don't, because then you could just store the intermediate values in the chain of thought, and it'd be cheating. So so in order to get the value of X. You need to like.
Alex Loftus: know, these intermediate states in order to to like, do the calculation during the forward pass. And you know I've tried lots of other versions of this experiment. The 1st version that I tried was like much more complicated and and it it used like it also had like multiplication in it, and then it had, like modulo division, and, like, you know, take the absolute value of X, like you like, kind of mix and match operations a lot.
Alex Loftus: And you know. And then I you know, this is kind of the the very simple version of it.
Alex Loftus: yeah. So this is my prompt. You're given a short python program. The task is to compute the final value of the variable X return. Only the integer without commas, an equal sign, or any additional text. The integer should appear, appear immediately afterward. The word is, and then I put in the
Alex Loftus: whatever the program is into this, and then and then, you know, if all goes according to plan, the model outputs an integer. Right here during the generation and and so, you're not giving the comments.
Alex Loftus: Sorry? What? Yeah, yeah, just the program. Yeah, no, no, none. Yeah. You don't get these comments. Yeah, that would be that would be cheating.
Alex Loftus: Yeah, yeah, these comments are just for illustration. Yeah. So, yeah.
Alex Loftus: whether like any of these are like already memorized by the model or not. And like.
Alex Loftus: yeah, things like that. What is these separate like? For example, the groups add sub multiply division, and you just ask for like a code for that, like the likelihood is that that particular code may have appeared many times in the training. Yeah. And things like that. So does it matter if, like that has been trained before or not? Well, I think that's 1 of the questions is like.
Alex Loftus: you know, did it memorize it in the training process? Or you know? Or is it actually like.
Alex Loftus: does it have an internal state? That it keeps track of X, and then it's like cause like, if it's memory like, say, it's memorized. Say, it has some like
Alex Loftus: some like some table right? Just for like I don't know from 0 to 100 or whatever, and it has, like one of like some inside the model somewhere, some like 100 by 100 table that, like pretty accurately, has all possible summations on like the lower diagonal, for, like every pair of integers from 0 to 100 or whatever right? You know, and like some version of that, exists internally to the model, right? So that would get you like
Alex Loftus: from one line to the next one. But then but then you have, like, you know, you would need like a combinatorially bigger table in. As you increase the number of of lines, so I think it would still be interesting if it if it had that table. But then it was. It was using that table at each line to do to like, add
Alex Loftus: to add each each each integer, like as it as it went, and then it was, keep keeping track of the States right like cause. I I don't think it's possible for it to like, you know, once you get to like
Alex Loftus: 5 or 10 lines, or whatever like you just have like so many. It's like there's so many there's combinatorially more possibilities, right? And and like you, you wouldn't be able to store that or it would be like super inefficient to store that internally. Right?
Alex Loftus: So yeah,
Alex Loftus: yeah. So it turns out, the models are like, not that great at this, like they're okay, but but so I like, I, I kind of like looked at a ton of different models. Just to like, kind of get a sense. One caveat here is I like
Alex Loftus: one. I was like I didn't. I wasn't really using like Nagoya or Hakone that much, because they kept being like full and so so I was a little bit limited to use. Like most of basically, most of these models are like less than maybe 2020 1 billion parameters. So so it's like.
Alex Loftus: roughly between, like, I think, the smallest one here is like this, 1.3 billion deep seat Coder, model. And then the biggest one is, I think, a 14 billion parameter model. And they do get better like as you increase model size, and we'll like, see that in a little bit
Alex Loftus: float them in like the full version, or like, default 16. And things like that. Default. 16.
Alex Loftus: Yeah, so so this is another like,
Alex Loftus: So yeah, the the way the the setup was, I loaded them in B float 16 with left padding and and I had a temperature of point 8 on. I just use the same settings for all of them. So I think I had a temperature of point 8 for all of them and and then each of these, each of these data points is
Alex Loftus: sampled. I think I had it sampled a hundred times for each of these data points, and then each of them is the average of 100 samples of the same sequence length. Yeah, oh, that's number of lines. So so in in this.
Alex Loftus: in this example, sequence, a sequence length of 3 would be x equals 0 x plus equals 3 x plus equals one and that would be a sequence length of 3,
Alex Loftus: maybe counterintuitive is that
Alex Loftus: this is hard, and this feels more of like a math reasoning task than a task. Yeah, yeah, I agree, including.
Alex Loftus: you don't usually see like, who would program that? Right? Yeah, you do it in a for loop or something, or like even you wouldn't even do that. Yeah. So that's a good point. I feel like, if you're testing if you want to test coding abilities. And if you're also using coding models, I would try like a setup where you have like a definition of a function, then call your function, and the function has some like, if else.
Alex Loftus: and then you have to keep track of like, whatever the thing is. And you sum something. And then it's like a yeah conditional. And then something happened. You can simplify it. Just like to start with. But do something that is more code. And yeah, more than just like mathematical reasoning. Yeah, I agree with that.
Alex Loftus: so so like
Alex Loftus: one of the things that didn't make it to these slides that I did during the fall was a bunch of.
Alex Loftus: I looked at a bunch of problems in the human Eval data set, which is you have.
Alex Loftus: it's it's just a bunch of python functions. And then and then it's like a prompt to generate a function that does a particular thing. And then and then the model does it is is, solves it successfully if it passes like a unit test based on that prompt and so yeah, I mean, I have a bunch of stuff. With that, I think that
Alex Loftus: that data set is probably a little bit too complicated. Because, like, I, I want to keep it really simple, right? So that I can, so that I can have like, like, good. See? Like sequences that are,
Alex Loftus: yeah, like, like, kind of concrete. Yeah, so, and I feel like, when you get really simple.
Alex Loftus: the line between math and code gets a little bit blurry right like, because I have. And you'll see this in a bit. I just this week. I did like an experiment where, instead of adding integers, it's doing different Boolean operations on like true false variables. And like, that's kind of more coding. But you could also argue that that's like Boolean logic, which is like a math. Right? So yeah. So I think I have to think about like.
Alex Loftus: yeah, like, what is a coding setup where you can still do like basically exactly this experiment. But it's
Alex Loftus: it's something that like you would see in in the wild, you know, during like a software engineering
Alex Loftus: thing. I don't know if you know what about something
Alex Loftus: where you kind of have a
Alex Loftus: for a loop, and then you're printing stuff.
Alex Loftus: When a condition is met, given some input variable.
Alex Loftus: And then you ask the model, what's the output of these? Or something? Yeah, yeah.
Alex Loftus: yeah, that that could be interesting.
Alex Loftus: let's see. Yeah. Yeah. So yeah, you loop through some list or something, and then when you hit like.
Alex Loftus: I don't know some, it's like a list of like strings. And then, yeah, yeah, you could do something like that. Yeah.
Alex Loftus: check like every model.
Alex Loftus: 3. Do this, if whatever else? Yeah. And you have like 3 variables, and no one has to like keep track of those 3.
Alex Loftus: All right, if you can start with one. Yeah. Yeah.
Alex Loftus: Yeah. I think that would be interesting.
Alex Loftus: yeah, I have. I have like a bunch of of
Alex Loftus: you can. Also, I also was thinking about doing something with like types, right? Like, what's the type of this variable? And how does it change over time? The reason I didn't do that initially is because Francesca, in Arjun's lab already did like she wrote a paper that was all all about like type representations. But yeah, I wanna I should go reread that paper, and there's plenty of stuff to do there that doesn't like isn't isn't overlapping. So
Alex Loftus: yeah, yeah. I mean, there's a lot of different forms that this experiment could take. So yeah, I think
Alex Loftus: that's a good point.
Alex Loftus: Yeah.
Alex Loftus: yeah, I was gonna ask what sequence life meant? Yeah. And we're just adding and subtracting in that that chart in this case. Yeah, yeah. So I mean, you can see these operations down here. That I just like commented out. So I was doing like a lot of other stuff besides adding and subtracting. But I've like heavily simplified it because models just like we're not able to solve this task at any reasonable accuracy.
Alex Loftus: and and like, even even when you simplify this much. They still can't. Really. I mean, it's okay. Right? Especially sequence lengths of 2. There's like a big cluster of models that do like quite well. You know, like this pink one is Quinn, 2.5 like the Quent models typically did like pretty well here.
Alex Loftus: and at like low sequence lengths which kind of like it makes sense, because, you know, if you have x equals 5 x plus equals. 3. What's X? Then you can just refer to some lookup table internally,
Alex Loftus: and then and then they kind of even out around like a sequence length of 6 ish, right? Like they kind of drop pretty quickly from like 2 to 6 down to like a 50% accuracy, which is still above random chance. You know. I mean, random chance would be close to 0. Right? So so it's still above random chance, and then it kind of evens out?
Alex Loftus: Or are you just trying to add base addition and subtraction? Oh, without any like sequence, length, number, just as you pose. If you pose semantically the exact same math problem, but but write it, present it like a math problem as opposed to like a code problem.
Alex Loftus: What do you mean by present it so like just a mathematical expression.
Alex Loftus: Oh, I see! Oh, like not on different line, you mean, on the same line rather than different lines. I haven't checked that. I've I've checked. I'll show you guys in a second. I I like checked doing this in a for loop instead of doing it in different lines of code, and it gets a lot better when it when it's doing it in a for loop, which is sort of interesting.
Alex Loftus: I also. So I checked in a for loop rather than in different in like its own line. And then I also checked you. Just add the same number every time within that for loop, and, like models, get to like close to perfect accuracy. I'll show you guys in a sec. And then I also checked. You have a for loop, but then you still sample, and they do like much better if it's like.
Alex Loftus: even if even if it's like sampling. And it's a for loop, which is like, kind of weird. So
Alex Loftus: the the these these values, each. V yeah. That's like a parameter in the function that I have set up to do it. But I've been using either 0 to 5 or 0 to 10, depending on the experiment. Yeah, because if the values are large, then it's
Alex Loftus: yeah, it won't have a lookup table. Yeah. But if they're small, yeah, it would have a lookup lookup table, probably. Yeah.
Alex Loftus: Alex, I came in a little late but
Alex Loftus: the your goal is to understand, if there's a
Alex Loftus: if these models keep track of their variables. Yeah, my goal eventually will be
Alex Loftus: say, say, you give this entire sequence to a model and then you you say, what's the final value of X
Alex Loftus: during that forward pass. Can you go back to this spot? And and does the model know what X is at this line of code?
Alex Loftus: right like, does it? Does it keep track of all of these, or or or like? Does it just somehow just
Alex Loftus: I don't know know the final value from, I don't know. Magic. What? Exactly. Yeah. Yeah. So, totally. Yeah. So so yeah, you're totally getting it. So so that's why this original experiment was much more complicated.
Alex Loftus: Was this setup? Had this like super long function that I spent a bunch of time on to make sure that it like couldn't be associative. So I set up these operations so that you picked an operation, and then at every line it deleted all operations that it could pick that would make the program associative. And so every program that it made was like, fully non associative.
Alex Loftus: And you know, and they and they like didn't do well. And so now I went back to like, Okay, maybe it's fine. If it's associative. You know. And that's that's this addition of subtraction thing, and
Alex Loftus: you know, and and this it's it's a little bit better when it's just addition and subtraction. And I've like gotten rid of the whole like associativity, constraint, thing. But but like they're not. It's not that different. You know, the associativity Thingy was like basically this, but slightly worse, but not that much worse. Yeah, they're they're pretty close to equivalent. Yeah.
Alex Loftus: And 1 1 other thing you might be interested in is like.
Alex Loftus: what other kinds of properties. So instead of just like
Alex Loftus: taking one variable, that's an integer and adding a bunch of stuff like.
Alex Loftus: maybe you have some list, and you could check like, whether the model knows the length of the list or yeah knows the sum of a list. Yeah, yeah, I was thinking about that, too, know as well what the elements in the stream are. Yeah, like that. Yeah, like.
Alex Loftus: I think I should just have like 2 days where I just like, sit down with a chat bot, and like, try a bunch of these different like forms of this. Because, yeah, I think there's so many different questions. Yeah.
Alex Loftus: yeah, yeah, right? So that I think, the 5 month alpaca data set might be useful, because I just noticed that it has, like some instances of like the setting where you have like a list that you just either want to print or modify the string. Okay, and it remove punctuation and things like that. So yeah.
Alex Loftus: just run that data set and see how well it doesn't.
Alex Loftus: Yeah. Yeah. Another thing I was thinking about was like giving it like numpy arrays, or like pythorch pytorch tensors, and then doing some reshaping operations. And then
Alex Loftus: and then like, does it know the shape? Yeah, that, I mean, and that seems like a much more like standard coding thing that it that like you, would expect it to be able to do if it like, needs to generate pytorch code, and that, like knowing the shape of a tensor seems like
Alex Loftus: really important if you want to generate good pytorch code, and like in practice, it feels like they do know the shape of tensors when they when I like, have a model, write pytorch code for me. So, yeah, maybe that can be like a thing to try. Next, I think I would start with, like.
Alex Loftus: just type some code. First, st just basic, yeah, because it's more. It's like, not yeah, like, specific. Yeah, obviously, having like.
Alex Loftus: really use all these models. But I have at some point tried with like church, Pt. Or DC. As in Hightorch questions.
Alex Loftus: and they're not as good as in all the other stuff like sometimes she's like that's wrong, and no matter what I tried to say, that doesn't work. It's like, I don't. I'm not sure in those cases. Maybe it's so complex, and they don't have this like
Alex Loftus: latent representations of what they need. Instead, I would start with like simpler python setups and then move on to that. Once you have like a better sense of.
Alex Loftus: yeah. Yeah, that that makes sense. Yeah.
Alex Loftus: especially because, like, during the training process, it saw basic python stuff, you know, orders of magnitude more than it saw, like shape, changing operations and pytorch. So
Alex Loftus: are the tasks. These, like the code, only models? Are they generating code? Or are they answering questions about code. What are you trying to do?
Alex Loftus: But what are they trained on? They're trained on?
Alex Loftus: I think it include, yeah, yeah.
Alex Loftus: lots of code generate. Yeah, there's a lot of like.
Alex Loftus: it's like a mixture of like
Alex Loftus: cleaned up like code data sets pulled from Github like. And and like a lot of synthetic code that is like like you. There's a lot of like rejection sampling stuff. Where you like, you know you you have a model. You know, you ask a model to like, generate, like a bunch of different classes of
Alex Loftus: like ideas for functions that you would see in the wild or like classes that you would see in the wild or whatever, and then unit tests for those. And then you like. Do resession sampling with this generation process where you like, get rid of anything that doesn't like, pass the unit tests and then you like, create a data set out of that. That's like pretty common.
Alex Loftus: probably not a huge deal, but like on the margin. If you set up the prompts to be less like, here's a here's a question I have about the code, but more like complete this code, and you can only complete it in certain ways that are enforced by
Alex Loftus: the program that you gave.
Alex Loftus: So like the type, the type annotation would be the same. Yeah, yeah, yeah.
Alex Loftus: yeah. I agree. Yeah.
Alex Loftus: A setup that seems very similar to me
Alex Loftus: is, I don't know. One of
Alex Loftus: our lab members had done the setup. There, there's a box A, there's a carrot in the box. Person. B. Came and removed the carrot, put something else, and somebody tracked it over the time the model knows what's in boxes. Yeah, that was Nikhil. That was like Nikhil's stuff. Right? There's entity tracking. Yeah, yeah, it seems very. Yeah. It's pretty similar. Yeah, I was actually thinking about that. Yeah, maybe I should.
Alex Loftus: It's a shame. Where is Nikhil? It's like not. Oh, he said, oh, yeah, he's doing an apple entry. Yeah. Well, that's a shame. Because, yeah, I I have actually had that thought. I haven't really talked to Nikhil about this, but some of this stuff feels kind of similar, but like in code, instead of like in rabbits in boxes and stuff.
Alex Loftus: Did did you try like regular language model? Not a cold model
Alex Loftus: cool? I tried all these, and some of them are code models, and some of them are just normal language models.
Alex Loftus: Like, I think, like most modern models like are kind of our code, like they like people have been putting a lot of effort into making like models. Good at code now. And so like, when you see a new model, it'll typically have been like
Alex Loftus: it'll be good at code, even if it's not called like a code model, you know.
Alex Loftus: yeah. So
Alex Loftus: it was a thought. It was like there was like a parallel to theory of mine. But at the same time I think this goes like
Alex Loftus: much more beyond it, and it's like a more control setup, because, like in theory of mine, you have, like
Alex Loftus: all the compounds with reality. Right? Like is the folks transparent or not, we're like in code. It's like, no things are the way they are. And like there's no misinterpretation. This this color combines, and the mode is like.
Alex Loftus: has seen so much code that it like really knows all this insects and so on. I like the logic right? So I think you get like a lot more than like sure. This is like the 1st setup, right? Or like what? What the the model knows, what the variable encodes. But
Alex Loftus: you can go like way beyond with like
Alex Loftus: for loops. Yeah, yeah, totally. Yeah. Along along those lines, and also along the lines of
Alex Loftus: having the question that you're asking being enforced by the program. I wonder about like, just give it a simple, just give it a simple program and a compiled language, and just ask it, does it? Does this code compile? And if no, what is the error? Yeah, yeah.
Alex Loftus: yeah, that's I mean, that's an interesting thing to like train train a probe on, too. Right? Like, does this compile? And then the probe is just binary like cause. That's like a yes or no. Will this run into an error, or will it not? Yeah, that's I like that task. Actually, that's like a simple task. And like, very easily checkable. And it's like a binary, very solid question. Yeah. So
Alex Loftus: yeah.
Alex Loftus: I want to see the rest of your yeah. So yeah, so let's let's keep going. We do. We do this a lot. I know, I know. I know. Yeah, yeah. Cause there's gonna be like a whole other project that I you know I've been doing with Steven as well. You know that
Alex Loftus: hopefully, we'll have time to get into. Okay. So so so you know. So I like test all these models. And then I kind of zoomed into the quent series, because they were typically just like
Alex Loftus: pretty good and it was either Quin or Gemma, and they both were pretty good. So
Alex Loftus: So this is just you know, looking only at Quen, we can see that as you increase model size. Directionally they get a little bit better at this task. So maybe if I use the full big boy version of Quen you know it would be I haven't. I haven't checked that but you know, because it was annoying to load onto memory. But but you know, maybe maybe they'll get better.
Alex Loftus: Yeah. So this is like the 14 billion perimeter Quinn here. The red is the average of all the Quinn models.
Alex Loftus: and then so this is what I was talking about before. If you do this all in a for loop, and so you generate some random Z between one and 10, and then you don't change z at all. So it's you're literally adding the same value every time. And then your program is for in range, whatever your sequence length is x plus equals. Z. So if Z is 2, then it's just adding to, you know, sequence length number of times.
Alex Loftus: So that just collapses into a multiplication problem right? And so
Alex Loftus: seems like the models know that. Right? So the smaller model, the smaller Quin, doesn't the the 500 million parameter Quin like doesn't realize that that's a multiplication problem. But as soon as you get into like.
Alex Loftus: you know, more than like 1.5 billion parameters or so they basically get perfect accuracy. If it's just a for loop with the same number.
Alex Loftus: and then literally the same problem. But instead of a for loop. You're doing the new lines thing. It gets much worse. So it stops being able to. I like my guess is like this one. It goes into a multiplication table. and then this one
Alex Loftus: it I don't know doesn't, or something. so so it gets it gets a lot worse. Here when when it's the previous thing except
Alex Loftus: with with different lines of code.
Alex Loftus: And then now, if we sample ZZ is a different number every time. Now we're kind of back to the old problem, and it gets even worse. And yeah, so here's here's Z is the same at every part of the for loop, and then here is Z is randomly sampled between one and 10 for each line, and you know, but still lines of code.
Alex Loftus: So in in these cases.
Alex Loftus: does your prompt change. And how like, how does the model? How would the model know what? The random?
Alex Loftus: Well, so so here you're you're giving
Alex Loftus: The model is given a sequence like this
Alex Loftus: right? Whereas whereas here? 2, it's 2, 2, 2, 2, 2, that kind of thing.
Alex Loftus: And then and then here it's not 2, 2, 2, 2, 2, 2. It's like, you know, a for loop.
Alex Loftus: yeah. And then I also have this Boolean setup I also like was a little bit irresponsible scientifically. And I added, a
Alex Loftus: in context, learning little bit thing here to, you know. So I should probably do it again without this. But but this is like a Boolean task. You know. We do Boolean operations on true false. Now we have some other variable Y that we don't want to predict. But we just use Y as like a method to, you know. Do Boolean operations on X,
Alex Loftus: And yeah, I
Alex Loftus: I should probably give an example of like one of these sequences, but it's the same as before. It's like a bunch of new lines, but each new line is either not and or with x paired with so x equals x and YX equals x or y, and x equals not x. So you sample from one of those, and then that's the new task.
Alex Loftus: And so I should probably have the old pass and the new one next to each other. But it's a bit better here. Right? If you look at the Y-axis values. You know, we're we're a little higher, but also, you know, predicting the final value of X is you're predicting a Boolean. So now, random chances of 0 point 5. So it's it's a little bit harder to like. Compare apples to apples here? Right?
Alex Loftus: yeah, I think so. Yeah, maybe it should just be like X equals X, and true, or something X equals X or false. XX equals true. So so instead of saying, like y, it should be like x equals
Alex Loftus: x, and true x equals x, and false x equals x, or true X equals X or false. Right? So you sample one of those? Yeah. Yeah. Makes sense. I'm really curious how you did the evaluation with, I'm guessing would by accuracy. Is that the logic of the token that corresponds to the correct 0? Prep. One, or you actually like sampling repeatedly. Oh, I'm sampling repeatedly, oh, okay, yeah, yeah. I didn't. I didn't look at like logics at all. This is each one of these is like 100 samples with like a temperature of 0 point 8.
Alex Loftus: Okay, I guess I was thinking you could probably simplify the pre-promp considerably if you just if you're sampling and you're using like Vlm or something just highly logic bias, true and false, or what? Or whatever. Yeah, or just actually measure the logics you want to sample. I don't know if that'd be easier or not. Actually, I don't know. Yeah, yeah, I think it would also be interesting to look at the logics. And then how does that compare to the accurate accuracy?
Alex Loftus: Yeah. Cause like theoretically, they should be close to each other. But maybe in practice they're not. And then that would be kind of weird, you know. But yeah, that that's sort of yeah, yeah, that's I haven't. I haven't thought about that. Actually, that's that'd be interesting to look at.
Alex Loftus: So I have a bunch of experiments plan that are like actual interpretability instead of like evaluation stuff. So one of them is, train probes on the last token to predict the final value of X at each layer, and then at what point in the forward pass for successful runs. Does it know the value of X. Another one is you know, a causal mediation thing where?
Alex Loftus: you know you do. Forward 2 forward passes and then you diverge. The forward passes at like a particular point, and then the rest of them is the same and then?
Alex Loftus: so, and then you're asking, does patching activations from the second one into the 1st one cause the model's final prediction to shift towards what it would be if the model had processed the second sequence right? And then that kind of gets at
Alex Loftus: this, like internal state representation question. David and I were talking about Atticus Geiger's stuff, which I've been slightly irresponsible and haven't. You know, dove given into doven into but yeah, das and hyperdas. And that stuff, I think, is is like an interesting
Alex Loftus: set of methods to try out on this problem.
Alex Loftus: yeah, it's I guess. Like, if if the model can't do it, it doesn't seem worth it. Yeah, I agree. Yeah, yeah. So the 1st thing is still like, PIN down something where it can do. Yeah, yeah. So yeah, and you have, you have the for loop. I have the for loop. It can do the for loop. Yeah. Yeah. So I got one. I got one thing that it can do.
Alex Loftus: So you know, that's cool. I was really curious, as like, what are the failure cases like, do you know, like what the distribution of the wrong answers are like? Is it? Is it going like? Sure? Here's the like. Is it starting a longer response that's not conforming to your free prompt ever? Oh, saying the wrong answer. No, it's not that I have. I have like post-processing code that uses Regex to like pick out to like, pick out the thing. So even if it does that.
Alex Loftus: I mean, unless, like my code is bad, which that's like definitely possible. it should be picking out whatever the model thought it was, even if it has some weird prompts
Alex Loftus: so, or some weird output.
Alex Loftus: As long as it like gives you a number. Yeah,
Alex Loftus: But but yeah, I mean, I I haven't. I haven't looked at like, I haven't like gone into that as much. So yeah, I should. Probably I should like, look at a few manually and check that out.
Alex Loftus: Yeah. So I guess, like general feedback on, this is probably switch out the task so that it feels it's like more in line with what you would find in the wild, but still
Alex Loftus: but still has the property where you can do these kind of causal mediation, analysis, experiments on it, and these probing experiments and stuff.
Alex Loftus: Yeah, this is actually natural. But like.
Alex Loftus: well, I don't know. Maybe maybe it's because I'm not used to prompting models, but like.
Alex Loftus: for some reason, just like the autocomplete, seems
Alex Loftus: better. It seemed more natural to me like predicting
Alex Loftus: you know, like the next line, or whatever, and so like
Alex Loftus: like, do models actually see like like a python console, or like
Alex Loftus: you have, like the prompt. And then you can. You can just say X, and then the next line would be like the value X.
Alex Loftus: Or is that like, I don't know a more natural way of like showing what
Alex Loftus: is in the variable X, if that makes sense. Oh, I see what you mean. Print print X, print XX, yeah. Yeah. And then the model would output, yeah, yeah.
Alex Loftus: that instead of saying, like, tell me what the error is like, you just got to complete the sequence that's about to output an error. That's yeah, yeah, I should switch it out to be like that, because then I could have the whole prompt. Just be python code. And it would be like a lot more pure instead of like, Hey, model, here's this set of python code. Could you like? Tell me this thing about it? Yeah, I see what you mean
Alex Loftus: the models that you were trying? They're mostly like chat and instructions. A lot of the ones that I've been trying are instructuned. There's a way you can still prompt it to an instruction. One solution, just use a base model. Yeah. The other thing is very cool. Base models.
Alex Loftus: Yeah, not necessarily new ones, I mean.
Alex Loftus: most of well, I haven't checked. If Quinn has, I mean, almost definitely has their base model, because I've played with that a lot
Alex Loftus: mo. I mean, most model providers that I've seen that are like open source, like Quinn and Gemma and stuff.
Alex Loftus: I think most of the time have their base model on hugging face. One has base models. Yeah. Yeah. But even if you only have the instruct model, you could like have a user. Prompters like, Provide me some arbitrary python code. Maybe not that, but something with that effect.
Alex Loftus: and then simulate as though the Lm. Has already produced like the beginning of the python. Id like you said, I think is a great idea. Yeah. And then just go all the way up to not the end of the assistant response. So it's as though the assistant started to give a response. But it just needs to finish it and get that final output of like an example of like, you know, you have the X equals one X plus equals. And then it's just that final little thing that has to output right here. Yeah, right? Right? Yeah.
Alex Loftus: No. I also like your current comments that you showed for our sake. I'm sure you're not showing that to the model. Wait! Which? What? Comments in the comments? Oh, oh, yeah, of course I'm not. No, no, maybe you could.
Alex Loftus: and then let it finish the comment, like, what's the next in the sequence? Oh, yeah, yeah, that's another way to point it out. Yeah. Yeah.
Alex Loftus: But I wouldn't want to give it the intermediate values in the comments, because well, because the the point is to figure out if it's representing them internally. So if it's if they're stored in a token. Then? Then then you're cheating. You could give it. You could give it one or 2 of them just to show. Oh, yeah, yeah, right to get it going? Yeah, that'd be kind of interesting. Yeah, yeah.
Alex Loftus: Can you? Can you train a probe that lets it, that where it knows where you lied? That could be that could be kind of an interesting experiment. And that's how stack overflow shows all the answers right in the comments, yeah, yeah, yeah, that's that's sort of yeah, that could be an interesting direction is like.
Alex Loftus: yeah, how well, how robust is it to being like misled? Yeah.
Alex Loftus: on the task choice. I don't think this should be dispositive, but I think it'd be really cool to have a task that has an analog
Alex Loftus: natural. Let's move.
Alex Loftus: Yeah, yeah, multilingual. But instead of English, French, it's like English code. Yeah, yeah, that's a cool question. Yeah, the box idea. Nikhil's box thing. Yeah, yeah, you could get a code of cool under that. Maybe it's the same circuit. Yeah, that would that would be very cool. Actually, yeah, yeah, that'd be interesting state bracket. Yeah.
Alex Loftus: And you have all these like pointers. Yeah.
Alex Loftus: yep, yep, that would be very cool.
Alex Loftus: okay, I I have like an entire other section of this talk that's like the stuff that Steven and I have been doing. So I'm gonna run through that quickly. So this is like, almost. So to give you guys like a sense for the setting. This is like looking at training dynamics stuff.
Alex Loftus: so for anyone not familiar with the Oma models. AI. 2 put out these models, and the nice thing about them is, it's sort of like the successor to Pythia, where, you know they have their full training run with every 1,000 gradient updates, they have a checkpoint that you can find online. And then they also have their
Alex Loftus: entire data set, also online. And then I check this. It's nice that you also know exactly what data point was given to the model, at what gradient update.
Alex Loftus: And and it's like, fully random seated so you know. So so you can. You can go to, you know, gradient update number, you know
Alex Loftus: 1,500,000, you know.
Alex Loftus: 400 and and and you can see what what data was added at that. At that update. So their recipe was pre-training for 4 trillion tokens, where you just throw tokens at it. And then, what they called mid training, which is the last 90% of the the remaining 10% last 10% of the pre-training run. They switched to higher quality tokens
Alex Loftus: and then, and also at that run, they have. These they they did 3 different randomly seeded, post-training, mid training runs, starting at 90% of the way into into pre-training that they called ingredients, and then they they did like this model merging thing to get to the end of pre-training to get their base model and then and then they did for post training. They did sft.
Alex Loftus: And then they did dpo, and then they have this new thing that they called Rlvr, which is reinforcement learning with verifiable rewards. And that's like, generate a bunch of synthetic data that you can verify and then give the model that. And so the instruct model is after Rlvr. And the Dpo model is after dpo. So the difference between them is this Rlvr thing they're like.
Alex Loftus: I I ran these a while ago on this is all on human Eval,
Alex Loftus: the human eval data set. And the and I looked at the correlation between pass rates with these different post trained models, and the Dpo and the instruct are like basically the same, as far as I can tell, in terms of like which problems they in terms of which problems they like get correct.
Alex Loftus: And then the sft kind of diverged from the Dpo and the instruct a bit. And then this is the base model. So of course, the base model was like way different than any of the post-trained models. But so you know, just looking at this, it looks like Dpo made a huge difference. Rlvr.
Alex Loftus: as far as I could tell, didn't really change that much.
Alex Loftus: yeah, sft plus Dpo, so it's it's sequential. So they did sft first, st and then they took the sft model, and then they did dpo on it, and then they took that. And then they did rldr on it. That's the instruct. Yeah. So that was their training run,
Alex Loftus: and you know. And so so this this set of stuff. So like I said, so, Steven, over there has been working on this a lot as well. So so the the problem setting here is, I've been doing a bunch of evaluations on on different parts of the pre-training run, and post training run on this crux Eval data set. So this is I don't know if you guys can read any of the text here, but
Alex Loftus: but each crux eval sample has a function, a python function that's typically pretty simple. And then it has some input to that function. And then it has the ground truth output. And so you know, and they kind of take all shapes and forms. You know, the inputs could be strings sometimes they're lists, you know, it's all kinds of different stuff. And so I'm testing on whether the model correctly gets the the right ground truth output.
Alex Loftus: What's the generated using some other model? What generated?
Alex Loftus: Oh, oh, this is this is what the model! This was like, an actual example. Run where I was actually like evaluating model. So this is this is like me. Me running it and then generated, is like what that model
Alex Loftus: produced as the output. And then the true output is the ground truth, so like this one would be false, for instance, because the model didn't get the correct, true output.
Alex Loftus: and then, like, I guess all these are false. But anyway.
Alex Loftus: so. So so you know, this was just kind of like a big picture observation of like what's going on with this? you know, abilities in different sizes and fine tuning setups. So I think Steven actually
Alex Loftus: generated this. This plot. So so yeah. So this is like the 32 1 billion, you know, does better than the 13 billion does better than the 7 billion. And then the instructs typically do better than the than the base models. I have been kind of looking at the 13. So oh, weird thing! Here the 13 billion did like better than the 32 billion. I don't know why. I think it's just random chance. Yeah. Oh, sorry.
Alex Loftus: What is it? Stag? Oh, yeah, right? So, yeah, so right? Right? Right? So
Alex Loftus: so the orange is so the the orange and the the blue are.
Alex Loftus: I was looking at the overlap in problems that were solved? By different models. So the
Alex Loftus: how is this set up? This was a while ago. Do you? Do you remember
Alex Loftus: the the orange is basically all the problems that the model that only that version of the model solves. Right? Yeah. And the blue is like something that at least one other model solves. Yeah, yeah.
Alex Loftus: because I I wanted to. Basically, I, I'm interested in like robustness of like, like, if I want to ask questions about like a model's ability to solve a particular type of problem. I want to know, how how robust?
Alex Loftus: how robust that model like like, if a problem is solvable by a model in one setting. I want to know, like, what's the chance that that problem is going to be solvable in another setting, because then you can get a questions of like, do models like, have some kind of like semantic ability to solve this class of problem, kind of thing. Yeah, yeah, basically, that's it. Yeah, consistency and generalizability.
Alex Loftus: so this is sort of what? Yeah?
Alex Loftus: So it would be like.
Alex Loftus: like, you just determine whether it gets the correct answer. What if it like? Has that tracking, semantic tracking of like X or whatever you're considering? But it just ends up getting like something wrong.
Alex Loftus: Yeah, we're just measuring whether it gets the correct answer right now. Like, maybe it does. But but we're just looking at like, can it get the ground? Truth?
Alex Loftus: So
Alex Loftus: 1 1 kind of interesting thing that we were looking at is, if you look at the pre-training you have this sort of a very interesting phenomenon. So this is all base model. But this is these, these data points. Here are all you know that 1st 4 trillion tokens of pre-training. And then right when you hit mid training right here, you immediately jump up to like from 0% of these problems being correctly solved to like.
Alex Loftus: not 0%, right? 6%. So not high. But but there's 800 problems total and 6% of 800 is like 60. Right? So you within, like a thousand gradient updates, you go from being able to solve no problems to being able to solve like 60 problems. So you know, like, we thought that was sort of interesting. So we've been, we're like zooming into that a little bit. Here.
Alex Loftus: are you showing examples, or are you just giving the question to the baseball?
Alex Loftus: I am giving?
Alex Loftus: I have been messing with that, and sometimes it's few shot, and sometimes it's 1 shot.
Alex Loftus: I think typically I've I've switched to one shot because
Alex Loftus: I don't know. I just. I don't.
Alex Loftus: I feel like a few shots for this graph for this graph. I think it's 1 shot. You mean 0 shot or 0 shot. Sorry? Yeah.
Alex Loftus: Yeah. I guess it might be good to disambiguate between, like, can it follow instructions?
Alex Loftus: Yeah. Yeah. So as in like doing, few shot would do that. Yeah, yeah. Yeah. I mean, I've done few shot like I have a function that does best of K sampling and I think I just had the hyper parameter for that set to one here but
Alex Loftus: but I was playing around with like
Alex Loftus: 1040, and like a hundred you know, in terms of like K shot success rate. And but do you know what it'd be like by by fee shot like
Alex Loftus: like, where the input has? Here's here's like, you know, 3 solved problems. Now, here's a 4.th Well, that's what you mean. Oh, oh, I see, I see. Yeah, yeah. Oh, yeah, I haven't. I haven't. I haven't done any like in contest learning stuff. But that yeah, be good to try out baseball. Sometimes you get just like it'll give you another question, yeah.
Alex Loftus: yeah, yeah.
Alex Loftus: And I think I'm also curious. If
Alex Loftus: if nothing really happens before the 5,000 steps, or maybe you can. Yeah, maybe if you kind of like include some easier problems or easier settings to test out the earlier steps.
Alex Loftus: Maybe there will be different. Yeah, yeah. So so we actually,
Alex Loftus: so I'll just well, I won't skip ahead but but you know just a little bit more like information here. These orange dots are those 3 ingredients that I talked about before, and then at the end. Here, that's you know, the base model is this final orange dot, and that's the average weights of the training runs for these 3 ingredients,
Alex Loftus: and then
Alex Loftus: they're just different random seeds. So they're just that you all start with with the model as it exists at 5,100 tokens, 5,100 billion tokens trained. But then you you give different random seeds for the gradient updates, and then you just train them all separately, and then you combine them. So that's what they did. They average them. And I like didn't know that that was a thing, and it just works. And I was like what you can do that.
Alex Loftus: And that's like just what they did and like. I was as shocked as you are. I like. Didn't know that that was a thing that
Alex Loftus: companies did in pre-training, and apparently you can just average weights after doing this, and it like
Alex Loftus: makes a better model, which was like shocking to me. But apparently that's true. So yeah, pretty weird. That's also something that I kind of want to like
Alex Loftus: explore a little bit, because that's weird. Oh, it's so. This is this is like, I want to understand how?
Alex Loftus: How a model learns to code over the course of training and and like, more specifically, does it? Does it learn like? It's like, sort of like some like grocking questions. Right? So like, does it? Does it learn particular things within like small periods of time? You know, what is it. What does it learn? Does it learn? does it learn? Kind of how to solve code in this
Alex Loftus: sort of you know chunk chunking way where you where you learn like basic stuff first, st and then more complex stuff as the training run goes on or like, how does it work? The questions are like pretty broad right now. So like one of the things that I want to do is, you know, get a little bit more concrete. It's a little bit of like a look at the training run, and then and then you know, and then kind of like.
Alex Loftus: get a little bit more concrete with. I can tell you what we wrote on the grant. Yeah, I don't know if we'll be able to get there. But it's
Alex Loftus: you know, the the basic idea is, you know, if you if you you have a code. Lm, that you know, suggested an answer to you or helps you out in some code task.
Alex Loftus: There's always this question like, How did it know that?
Alex Loftus: You know it? Did it know that because of some example it saw in the training data.
Alex Loftus: So there's this, you know, training data attribution question and and so sort of it seems like one elementary part of training data attribution. We should. We could actually look at the training process to see like.
Alex Loftus: what did it see in the training data that caused it to be able to solve one specific problem.
Alex Loftus: You know, can we even like get a single one of those
Alex Loftus: cases? Yeah, I understand that. Yeah, yeah. 4th generation it involves. I'm assuming that it involves a lot of subtasks. And you see this rocking phenomena when the model kind of like learns to solve all of these subtasks, so that now it can combine all of these capabilities to something something like 4th generation. Yeah, yeah, right?
Alex Loftus: Do you have an idea of what these tasks might look like, and maybe
Alex Loftus: like, what? What are the subtasks that I that you are looking for like when you say that.
Alex Loftus: when the model has gained this capability for course generation, right?
Alex Loftus: I feel like, I don't know what that. Yeah, yeah, I know. I know what you mean. Yeah. So there's a cohere paper that that looked at this that like you know, laid out like, here are some subtasks for coding that you can like break things into. And you know. And so there's and there's like 2 other papers that I found that are like.
Alex Loftus: here's here's like some ways to categorize, you know, different code problems. So so I think, you know, once we get a little bit further, we're going to go through some of those papers and probably use one of their setups for for how to
Alex Loftus: you know how to actually like operationalize like, what does it mean to, you know? Break.
Alex Loftus: break, you know, prob problems into different sub subtasks. Kind of thing. But I think it's like it's a bit of an open question, too, because it feels like a little bit arbitrary, like
Alex Loftus: what you know. Cause you could also, you could also
Alex Loftus: do some clustering thing where it's like, instead of like
Alex Loftus: putting a label on different types of problems. You could just like you could probably get it sort of dynamically, where you you know where you where you kind of figure it out.
Alex Loftus: you know, with some clustering setup. But I yeah, I don't know this. It's this is like a little bit like further down the line to like, nail that one down.
Alex Loftus: yeah. Yeah, this is zoomed in looking at every 1,000 gradient updates. So basically, it goes from can't solve it. And then you get to
Alex Loftus: you get to mid training and then it solves 8%. And then it kind of like stays roughly there like it doesn't really get that much better. And so
Alex Loftus: So when you zoom in. And so Steven just did this work like this week. So we just added this to the slides at like the last minute. So David hasn't actually seen this yet. But but when you, when you look at some of the specific generations, at the during this pre training run. It turns out that we have this situation where it actually still has the correct output.
Alex Loftus: But but there's this like crazy amount of like weird noise that's that is added in in its output generations that doesn't exist. Once you start mid training so like. Here, for instance, you know you have this this generated output right?
Alex Loftus: which still isn't correct, but it's, you know, kind of in the right ballpark. But then you have, like all this random crap after after it. That like you don't have any of that noise when you move into mid training. And that's true, for, like most of these things, and so and then, if you look at the stage, 2 starting. So this is in stage one.
Alex Loftus: I think it works.
Alex Loftus: Sorry. I don't know what happened.
Alex Loftus: Okay.
Alex Loftus: so this is which checkpoint is this in stage one like, where? How far along? Okay, so this this is like the last checkpoint of stage one. And this is the 1st tech checkpoint. So these are only a thousand grading updates apart, and you go from like a bunch of noise in these outputs to like immediately having like clean outputs, which is like kind of to me. At least, that's like pretty weird that, like it only takes a thousand gradient updates of like higher quality data to to like.
Alex Loftus: Go from this crap to like this much nicer crap. During beta.
Alex Loftus: yeah. So the stage 2 is like a curated selection of higher quality data set that I have a slide for that, like at the end of this, that like goes into exactly what's inside of it. But but it's basically like
Alex Loftus: all the pre-training is like they just took the Internet and threw it at the model. And then the last one was like, they have these nice curated data sources that they're like all shutting in at the very end. And yes, yeah, exactly. Yeah. And but it's like, but then, like in a thousand update steps, it like
Alex Loftus: is able to get to this from that. So who is it just
Alex Loftus: now? Knowing how to output end of sentence?
Alex Loftus: That could be it? Yeah, yeah, it just like, keeps going. Yeah, yeah, that that might be it. Yeah. Yeah. Maybe it's just maybe it's just like figured out how to do end of sentence or something here, that's a good question. So you know.
Alex Loftus: Yeah, I don't know how they prepare the their data. But
Alex Loftus: could it be that one during stage one, just like concatenated all the documents that they had. And so there's like, No, yeah, I feel, I think, yeah. Yeah. So I think so. So like I said so. Steven's been doing a lot of this work. So you know. So I think, like, maybe one thing to do is like, look at a bunch of examples of the pre-training data sets and the mid-training data sets and see how they're different. Because, I haven't really like
Alex Loftus: manually gone in and like looked at a bunch of the pre-training data. And I I think, like definitely, it's time to go do that. And cause you might be right. Maybe they like
Alex Loftus: it's all like concatenated crap. And you know, so yeah. And then you can also see this with this generation length. The the stage. 2. Beginning like this is a thousand gradient updates. Later, it has, like a much smaller number of tokens that are generated than the stage one. So the stage one, I guess, just outputted a bunch of noise in addition to the in addition to the like generated tokens.
Alex Loftus: you know. And you can look that over here, too, again, like Steven did this like yesterday. So
Alex Loftus: yep. And then when you filter out yeah, you'll never see him again.
Alex Loftus: So then then, when you filter by the 1st instance of a new line, and then you reevaluate accuracy. You know it goes. It goes way up. And yeah, I mean, Stephen, if you want to, you can. Also you can explain what's going on here, if you if you'd like, in this graph
Alex Loftus: for each generated output, I just like cut it off at the new line and then reevaluate the accuracy.
Alex Loftus: And as you can see, it's like much closer to the stage. 2 starting checkpoint. Now, although, yeah, so.
Alex Loftus: and I think you just learned when Michelle.
Alex Loftus: yeah. But it learned very quickly. I think that's the interesting thing. It's like it didn't learn over, like, you know, a million tokens it learned in like a thousand updates like a thousand update steps. Yeah, sorry. Sorry each step has.
Alex Loftus: How many tokens per step. It's like 8 million per step. So so sorry. So I misspoke. It's still a lot of tokens. It's 8 million times a thousand tokens but but in the grand scheme of things that's not that many tokens. So I missed the results. So even so, you're saying that before
Alex Loftus: this checkpoint it actually had the right answer. But it was. It was putting extra garbage. Yeah. Yeah. And then after it, it's just cleaned up as garbage. Yeah, so it is a yeah, interesting. Yeah. Yeah. So it's it's like, yeah, a garbage cleanup
Alex Loftus: situation is, is, you know. So so if you just clean up the garbage for it, then, even without the gradient, it has actually pretty good accuracy. Yeah, yeah, you can reconstruct close to the accuracy. Yeah, I see. So you might have to look back further in history. Exactly. Yeah. And and clean up a little bit. Yeah.
Alex Loftus: so I mean, so 1 1 thing that Steven and I were like brainstorming earlier is like.
Alex Loftus: you know the like actual learning is sort of blocked here by this noise situation so like you can't really. Do you know this like
Alex Loftus: black box evaluation, in order to figure out how much a model knows about about code, because it's blocked by all this noise, even though it like secretly actually does know, you know. And so, if you could make an interpretability based evaluation here, where you actually get the true accuracy
Alex Loftus: right, even if the output is noisy, I feel like that could be kind of an interesting paper. Because now you're like, Hey, guys, when you're early in pre-training like evaluations, actually aren't telling the whole picture because it's noise. But the model actually does know how to solve this problem.
Alex Loftus: you know, if you take out the noise. But then, if we use, we can use interpretability techniques instead for evaluation early in in pre training. And you actually do get it. The the like. True you know, accuracy of the model. So I was, we were like, you know, just thinking like that could be kind of an interesting direction. To go here is like you using interpretability techniques as like an early stage evaluation method.
Alex Loftus: but yeah, I want to know 2 things, though one is
Alex Loftus: sounds like adding a bit to the earlier comment that I feel like
Alex Loftus: given that you're looking at pre-training.
Alex Loftus: It wouldn't give the prompt as you're giving it. And I'm actually surprised that the model is this good training? Yeah. Bearing in mind that the Y-axis is not accuracy. The Y-axis is number of problems solved. And there's 800 problems. Total okay. Okay? So so this is this is this is like 6% accuracy. Yeah, it's a little bit misleading. I would try to make the example on the prompt
Alex Loftus: In such a way. That is something that you would find in the Internet kind of thing, like no instructions, because then you could get like even earlier, probably.
Alex Loftus: when the model does not need to understand. Oh, yeah, use of these kind of construction? Yeah, yeah. And then, what was the other thing.
Alex Loftus: Yeah, yeah, I was gonna say that there was a paper, remember exactly, but it was censored at some point in everyday group where people have like made the point that
Alex Loftus: so like emergent abilities are not so emergent. If you
Alex Loftus: look at a metric that is not so binary. Basically, they're emerging. And you're seeing like this plot because you're looking at accuracy. 0 1. Yeah, yeah. Emergent abilities are a mirage. The model does learn this pretty linearly, and it gets better and better. It's just that for accuracy, for getting like the one right token, it takes
Alex Loftus: a while. Yeah. And then you only see it from 0 to yeah. I think that that one like Icml 2023, best paper or something right? That was like a whole debate like 2 or 3 years ago. Yeah.
Alex Loftus: yeah, yeah, that's a good point. Just so I always have it in mind. But yeah.
Alex Loftus: it's like, maybe it's still growing inside. It's just that you're not saying it because you're looking at accuracy. And maybe that's something that you want to do. And that's fine. Yeah, yeah. Makes sense.
Alex Loftus: Yeah, we also did some robustness stuff robustness to and yeah, to like, you know. So like, if a model actually is good at at solving these problems earlier in the training run. But but you can't tell that it's good, because the outputs are, you know, noisy.
Alex Loftus: You know, but you can predict that it's good with like, I don't know, like probing or something. Then, you know, maybe you can create an evaluation technique where, where, like, actually, it's better to use probing early in the training run than it is to to do these black box evaluations because because that gets it like the true knowledge level of the model instead of you know this like, whatever the output is, you know.
Alex Loftus: I hope basically before fine tuning, probably the capabilities already present. You just need a way to
Alex Loftus: bring it out. Exactly. Yeah. Yeah.
Alex Loftus: I guess the thing I worry about there is that the baseline you have to compare to is pretty strong, which is just fine tuning on the tasks at that checkpoint.
Alex Loftus: And so, yeah, whether you can improve on that with probing or something like that.
Alex Loftus: But it's good stuff. Yeah, yeah, yeah. Getting close is still at pretty interesting from the base.
Alex Loftus: from the base model getting close to your fine tune model without really fine tuning it.
Alex Loftus: Yeah, that would be cool. Yeah, yeah, if it if it happens. Yeah, yeah, it's a big if but but you know. But that would be cool. Yeah. But yeah, that maybe that's the upper one. Yeah.
Alex Loftus: I guess it's like you gotta cause you're still spending some compute.
Alex Loftus: So it's like hard to know
Alex Loftus: how you compare those measure flops in training the float probe versus flops to fine tune. Yeah, yeah. If you could find you could do that, that would be interesting. Yeah.
Alex Loftus: yeah. Cool.
Alex Loftus: this isn't that interesting. This is looking at the different ingredients in the stage. 2. This is
Alex Loftus: This is the intersection over union. You take the 1st stage, 2 checkpoint. So there's 3 total. And then you look at the intersection over union of of that with other, the other 2 checkpoints. So this is just
Alex Loftus: you know, another kind of like problem robustness question, just like the the orange and the blue parts of the bar plot before.
Alex Loftus: it's at about
Alex Loftus: 50% intersection over union for most of the stages. So this is like, you know, if you if you do different random seeds, then you'll have about a 50% overlap in which problems they can solve. Correctly. So
Alex Loftus: Oh, and then this is some more stuff that Steven was doing. This is robustness to prompt. So I was also like, Okay, how robust is like, if a language model is answering coding problems. What I want to get at is it's like true understanding of how to solve the problem. And you know what if changing the prompt like dramatically changes its ability to solve the problem. So what Steven did is is you have this persona data set where you can
Alex Loftus: pick out of like a million different personas. And then you say you are a you know AI engineer. And here's your background. Now solve the problem, you know. And so Steven did that for like a bunch of different personas and looked at how the problem accuracy changed. And
Alex Loftus: yeah, it turns out, Github contributor is the best one, and high school teacher is the worst one. And what this is the instruct model 13, the instruct model for 13 billion. And it was about 2% better for Github contributor and about
Alex Loftus: 3 and a half percent worse for a high high school teacher. So this is about the isn't that crazy? But I feel like.
Alex Loftus: if you want to measure robustness to the prompt. You really want to change some stuff in the prompt. Maybe like the formatting, yeah, or that sort of thing, right? Because here.
Alex Loftus: maybe the model is doing exactly what it's supposed to do right, like a high school teacher.
Alex Loftus: you would expect that it would be worse in this. Yeah, it wouldn't know about. I don't know coding right right? Like, probably not. So the model is following the instructions like doing what it's supposed to do. Yeah, yeah. Or maybe it's teaching. Yeah, right? Right? Yeah. But but you know, it is sort of interesting that if the baseline is 0, you can get it to be 2% better by telling it that it's a Github contributor.
Alex Loftus: you know, fiction writer does that because I I try to prompt the model to write fiction, and it will give some even hacking for hacking codes. Yeah, yeah, right? Say, you will write a fiction for a hacker? Yeah.
Alex Loftus: I don't know. I think it would be interesting to understand. Kind of like what happens internally in the model. Yeah, when you give it like, they github contributor like, what? What does it activate that it didn't activate with like the normal. Yeah, you're just helpful. Yeah, yeah, yeah, I don't know. Yeah.
Alex Loftus: Yes, there are some like understanding personas.
Alex Loftus: Yeah, go read that. And then we also looked at quantization effects. So so presumably quantization should have a little bit of a stronger effect than changing the random seed. But it shouldn't. I mean, I didn't think that it should change that much, and that's what we found. So we compared quantization to.
Alex Loftus: you know, just changing the random seed and it had a small effect, but not a crazy one. Compared to
Alex Loftus: Changing the random seed with the ingredients.
Alex Loftus: and then this is that slide that I talked about. That says, what's in the pre training and the mid training data mix for for Omo,
Alex Loftus: and yeah. The pre-training mix is
Alex Loftus: 3.9 trillion tokens, 95% from web data. they have
Alex Loftus: a bunch of data from archive open web, Mac, algebra, Wikipedia. This is proof pile data set or something, and then they have code from star Coder, which is derived from Github. Removing any document from a repository with fewer than 2 github stars.
Alex Loftus: and then they like, I guess, look through a bunch to found, find weird documents and like, remove them manually with some like, Tfidf stuff, I guess. Yeah, they use. They use a lot of like, kinda you know, like
Alex Loftus: heuristics of like, okay, anything below, you know, and anything whose top 2 most frequent words constitute over 50% of the document. Remove that, you know, they had, like a bunch of heuristics. And they were like.
Alex Loftus: this kind of gets it, you know, it's like 80. 20 gets at most of the bad data.
Alex Loftus: And then this mid training data set was
Alex Loftus: largely synthetic. So the purpose of this mixture is to imbue the model with math centric skills and provide exposure to stem references and high quality text. I didn't include the actual what's in this, but it's it's in the paper. I want to go back and look at it. But yeah, I think we should spend a little bit of time like
Alex Loftus: exploring this data and seeing what the heck is in it.
Alex Loftus: So
Alex Loftus: yeah, yeah. And that's that's pretty much kind of the 2. The 2 things going on right now. Yeah.
Alex Loftus: overall question. So like, I guess
Alex Loftus: a lot of this data you've been looking at was more about understanding code or like.
Alex Loftus: and rather than writing code, do we feel like those
Alex Loftus: happen at the same time or at different times, or that they're intertwined at all like I feel like you could write code without understanding it.
Alex Loftus: you know.
Alex Loftus: Could you?
Alex Loftus: Do you think? Do you think you could write code without piles? With that understanding? I'm sure
Alex Loftus: I yeah, I guess. So. I guess. Like, yeah, curious.
Alex Loftus: Yeah, I haven't. It's not really a question that I thought about that much. I mean, I guess the main thing that I've looked at is
Alex Loftus: and again, this didn't make it onto the slides. But like a a while ago, like a few months ago, I was looking at the human Eval data set instead of the crux Eval. And so human. Eval is like, you have the model come up. It's for writing code. Yeah. So the model comes up with its own function for writing code. And then you're testing whether it passes this unit test.
Alex Loftus: and
Alex Loftus: But yeah, I mean, I, yeah, it's I haven't really
Alex Loftus: got. I kind of just switched out of that because because
Alex Loftus: why did I switch out of that? There was like a good reason for switching to proxy. Well, I mean evaluating written code seems harder than
Alex Loftus: evaluating like and output of some code. Yeah, that you give it, that's true.
Alex Loftus: But like, if you if you want to have like general claims about
Alex Loftus: like capabilities emerging about codes like, maybe that's a distinction to think about of, like yeah, writing versus understanding. Yeah, yeah, that's a good point.
Alex Loftus: And it.
Alex Loftus: are you still interested in the the training data application?
Alex Loftus: Yeah, I've I've moved away from it a little bit. But I I am like interested. But Luther is doing this
Alex Loftus: thing. They're they're building like a whole repo right now. on their interpretability channel. Yeah, Nora Bell, rose is is kind of making. And and Stella asked me if, like, you know, I wanted to help build that. And so far I haven't really so. So, you know, like, with this new result from Steven. So you know, I
Alex Loftus: I
Alex Loftus: yeah. So I was sort of like, yeah, I'm not sure. But actually, now, now that we see that oh, it's actually like the setting that you're looking at. You have a pretty crisp
Alex Loftus: view that. Oh, it's just it's just cleaning up the result. Yep.
Alex Loftus: it's probably not a cliff you probably have like a smoother
Alex Loftus: you know. Probably when you get into the details you'll find that it's it's more smoothly learning things like, I'm I'm actually more
Alex Loftus: interested in the in the attribution stuff again. Yeah, maybe as a tool, right? Right? Yeah, it could be interesting for us to like, dive in and help build that. Yeah. Yeah. Great yeah, when you when you're saying like the Grant proposal, I thought it
Alex Loftus: sounded like an interesting setup to study like the whole
Alex Loftus: all debate of generalization and memorization. Because, like with code, you can
Alex Loftus: kind of like, maybe look at it more easily, more measurably. Yeah, like, it's I feel like it will be easier to say whether it's really copy pasting. Because, exactly, yeah, yeah, and which parts could be facing. And at what point starts generalizing. Yeah, yeah, more. And I think there's like a
Alex Loftus: decent amount of papers on that. So maybe I should go through them. And then, like, think about how to just literally translate the experiment from that paper into like code land, it could be, yeah.
Alex Loftus: yeah, right? Yeah, yeah.
Alex Loftus: but yeah, I mean that it. Yeah, I agree, I think I mean, that was like one of my motivating questions originally, for, like being interested in interpretability was like, I remember I had this debate with my boss when I was working in this in the biotech startup about like he was like, Oh, L. Ones are just stochastic parrots, because he had like. Read that, Emily Bender paper, and I was like, I don't think they are. And and then I got like
Alex Loftus: so engaged in that that I like started reading interpretability papers. So like, you know, I am sort of interested in that. Yeah. So
Alex Loftus: wait. What is it that convinced you they are not
Alex Loftus: using them a bunch. And
Alex Loftus: I mean, okay, I I think my opinion now is that like.
Alex Loftus: like, say, like, the word stochastic parrot is like
Alex Loftus: just doesn't make that much like like, I feel like it's just so ill defined like, what are you talking about when you say that they're like
Alex Loftus: just statistical, like output, like they are. But that doesn't mean that like that's not interesting, I guess, is is like my current opinion. They still might be.
Alex Loftus: Yeah. But well, I I just think that the question the like measuring
Alex Loftus: what you're talking about when you say they're
Alex Loftus: a stochastic parrot is just too ill-defined of a question like, it's just not a very concrete, you know, and I think that's why it got popular and like somehow remains popular is because it's not disprovable, because it's not because you're not really saying anything like what you're saying is like Llms. Use statistics to generate outputs, which is like trivially true.
Alex Loftus: They go through this 3rd experiment of like.
Alex Loftus: if you're never seen that world, would you have models of that world, can you? From just statistics, from just statistics. You can't, and that's their point. And then, if you don't think that there's fantastic because you think that they have these models of the world, and they can connect things to the real world and understand them. Yeah, yeah.
Alex Loftus: yeah, I don't know but
Alex Loftus: but like, just from an information compression standpoint like there, you know, I mean, like.
Alex Loftus: at what point do you say that something is a world model.
Alex Loftus: you know. Like, if you're compressing information
Alex Loftus: like, there's a certain level of compression where I feel like you say it's a world model, because because, like, that's what we do in our heads. It's all about like compression. Right? Like, like, we have a compression algorithm in our heads that, like takes all of the like very high dimensional input that we receive throughout our lives and turns it into like, you know, you connect to like some mental model, you know, and like to me, it feels kind of like. There's no way that Llms don't do some kind of compression because they have to, because that's like.
Alex Loftus: did you know.
Alex Loftus: there's a limited amount of space in the weights, and they clearly are successful at like repeating stuff, you know. So
Alex Loftus: yeah, like.
Alex Loftus: I don't know anything for coding problem. The interesting thing is that everything is in test. So like, if you want to one model to interact with the world, maybe one simple which just to look at, or if it does not compile. Okay, what's the error message? And that's the feedback from the world. But it's still in the form of text.
Alex Loftus: So yeah, yeah, I think it would be interesting if, if, if like.
Alex Loftus: take a look at how the
Alex Loftus: if there's some error message, holder, model
Alex Loftus: like, locate the error in the call like, how the error message and the code are related.
Alex Loftus: Yeah, the pattern of the like
Alex Loftus: attention. Yeah, yeah, yeah. Yeah, I do think that that like
Alex Loftus: binary probing experiment. For like, is it gonna compile like that's kind of an interesting like
Alex Loftus: thing like, does the model know that in advance that a program is gonna fail, not just whether the compiler not but the exact error message with the code. Yeah.
Alex Loftus: Yeah. But well, I, the 1st thing would be, is there an error at all? I feel like that's like the easier version of it. And then you can get into what's the exact
Alex Loftus: type of error, I guess.
Alex Loftus: Yeah, if we can let the model predicted the error message.
Alex Loftus: Here's the code pre-generated error message, yeah, yeah, right? Is that correct? Yeah, yeah.
Alex Loftus: that's that's all I got guys.
Alex Loftus: So you have a little sense for what? What Alex is working on? Yeah. So if you got any cool code, Lm, ideas or attribution ideas.
Alex Loftus: then Alex is the guy you share them with.
Alex Loftus: Thank you, Alex.