David Bau: So today we have.
David Bau: So we have David Atkinson presenting today.
David Bau: And so, David, you. So, David, are you going to talk about 2 things or one thing, 2 things. Oh, okay, great. So I'll let you introduce your own your own topic. But I think that David did a bunch of interesting work at epic. Is that right? Before he started as a Phd student that I've always been interested in learning more about. And so he's offered to talk a little bit about that. Yeah. Yeah. In addition to his, in addition to his current research.
David Bau: Yeah. So so most of this is gonna be that ebox presentation which I'll get to in a bit. But I guess I wanted to take advantage of all of the patching experts here to just present kind of a very quick
David Bau: setup and sort of puzzling set of results that I'm getting, and and maybe get some ideas for what I should do next.
David Bau: So here, what I'm interested in is
David Bau: the question of if you have a reasoning model.
David Bau: One sort of model of what it's doing is it's generating a bunch of answers it then somehow chooses between these answers, if it conflicts and and you can get into like, when does it choose to generate more answers? When does it choose to to terminate? But ultimately it has this, this choice at the end, where it wants to identify the good answer and distinguish that from all the bad answers.
David Bau: And so here's like, here's just a straightforward example of of the setting that I'm working with this is Gsm, 8 k,
David Bau: it's another basic interesting.
David Bau: Oh, it? Oh, maybe it just slept.
David Bau: And so
David Bau: yeah, so it was this problem. It's kind of like, you know, maybe an arc problem. And the good answer just solves it
David Bau: perfectly. The bad answer, this is an organic rollout
David Bau: which model this is on
David Bau: 1.5 feet. So the r 1,
David Bau: when 1.5G distill correct, go.
David Bau: maybe I just need to click through everything.
David Bau: okay, so. And and you know, I've highlighted these in red. Generally the errors tend to be you know, either arithmetic errors, although that's less common, more common? Is it just? Does the wrong
David Bau: the wrong operation. And then we have this this like even worse answer, which is egregious. And so we have this set up where it hallucinates 900 red squares as a result of multiplying by 4 by 15.
David Bau: And so then, we have this this question of like, how does it choose between them?
David Bau: and oh, yes, you slow down a little bit. Yeah. Go back to the previous slide. What's the what's the difference between a bad and a worse answer?
David Bau: Yeah, that's a good question. I will get to that in a second. So I guess the 1st thing that I wanted to do is some questions. Yeah. So there are like 3 types of answers for the same question you got. Is it through sampling, or how did you? Yes, it's through sampling.
David Bau: I see. And what do you mean by choose between these answers? Yeah. So if you look at that setup right there. Sorry. So what we have is, I've taken the blue or 2 different organic rollouts and I then concatenated them together. So like I've added the black text. And then sort of added this prompt to to choose between them. And then.
David Bau: importantly, here I've I've forced it to choose an answer so normally it would roll out and give the full answer. Here, I've said, stop thinking, and then immediately give me the 1st token of the answer.
David Bau: I see. So you like, force the thing tokens with these 2 answers, and somehow force it to like, decide between these yeah
David Bau: and the black text that you put in is that like typical of text that it might have might have generated on its own. Yeah. So I
David Bau: I played a little fast and loose. With this I found like 2 examples of, and then I I inserted it. I see so it does exist in in the wild. I'm not sure how important it is. I see that's great.
David Bau: So I guess here the 1st thing I wanted to check is
David Bau: whether so you know what happens when you do this. So do we actually have this ordering and basically, what you can see is like, so when you run this model?
David Bau: When you just give it the good answer, give it a normal end of thing token, and then let it run normally. It almost always chooses the right answer, which is 66.
David Bau: When you give it the bad answer or the the bad rollout. The bad rollout has an associated answer of 70 or an implied answer of 70.
David Bau: It
David Bau: usually does not choose that answer. So it usually kind of like, knows that something is wrong. Self corrects, but like doesn't fully self correct, and ends up somewhere where it should.
David Bau: Sometimes it does fully self correct consumer.
David Bau: And then the worst answer is,
David Bau: mostly gets throwing it mostly gets does the same thing. Sometimes a little bit more likely it fully self corrects it completely disregards what like what it has generated in the thinking process and goes back to the right answer, yeah. So in the like non-thinking process, it will say, you know, here's the answer, and then go through the steps.
David Bau: Oh, so it'll generate a chain of thought outside the thought token, yeah. Yeah. So it it will present that as kind of like explaining the answer. But okay.
David Bau: And when you say good, bad, and worse. In the last slide you saw 2 explanations.
David Bau: But when you say good, bad. What does that mean? Yeah. So here, what I have is just this is the good explanation. This is the correct one. And then this is the bad. This is the bad trace. Okay? Alighted the the worse
David Bau: trace. So in theory, there's like an even more egregious
David Bau: set of blue text that you could put in there and you get different results. So here, I'm just trying to explain, like, what happens when you combine these 2. Okay? So in the table when you say good, both of them. Both of these are good.
David Bau: So in the table in this row I'm not. I'm not doing this, so I'm not combining, and then you end the thing
David Bau: the good and bad like. Yes, yes. So you'll see here. The next row is okay. I concatenate the good with the bad
David Bau: and then they're after that I concatenate the bad with the good and
David Bau: I can't tell so either here or the next slide. There's this preference for
David Bau: the second rollout, but that might only happen when you when you port it to answer.
David Bau: Also, I might have missed something. But what are these multiple values for run results like multiple couples? Yeah, so these are these are
David Bau: this is the answer. And then this is the count of the times, you see. So I rolled out.
David Bau: yeah, okay, 30 times. Okay, okay. And 30, is the answer. And this is the number of roll hours. 6. Okay, yeah. So so 30 is the answer. And then we saw that 28 times, and then 66, we saw 3 times, okay, what's the baseline?
David Bau: What do you mean? The baseline? So these are your think force right like, what? What do you consider the baseline for this is
David Bau: I'm not sure I fall if you just let it generate its own thoughts freely. Yeah, then, how how often does it get this one right? Oh, I see. Yeah, so about 80% of the time. Or the other question is, do you think letting it run? The thought is your baseline, or just forcing it. Not to think at all is the baseline.
David Bau: I think letting it run normally is should be the basis. But if it depends what you're trying to do.
David Bau: And so yeah, I guess here, what I take from from this ordering is
David Bau: we don't quite have a full kind of like total ordering
David Bau: But I wanted to get a total ordering. So so what I did is I so so step through. Why, it's not a total ordering first.st
David Bau: so it might be a total. Let me let me double check. So
David Bau: here, if we have good and bad, we have 66. First, st we have bad and good.
David Bau: I see. So here it does not choose 66, which is the right answer, despite the fact that 66 is in the chain of thought.
David Bau: so you know.
David Bau: Possibly it's it's having. It's it's being confused by the bad. Possibly it's self correcting but
David Bau: if you have the setup where you force it to answer.
David Bau: then you do have this kind of total ordering that you would like so oh.
David Bau: so force it to answer without further explanation,
David Bau: yes, yes. So sorry. Yeah. So here, if you just do the box, it doesn't. It does not have that. So here you have this. This problem of like, it still chooses
David Bau: bad.
David Bau: But if you prompt in self reflection, you do get this this total order.
David Bau: So
David Bau: yeah, I'm not. I don't. This is just for this particular example. So I'm not sure it's like super crucial to understand the specific results. But this is kind of how I got to this this prompt setup where you know. Here, if you have good and bad, it'll choose good if you choose, bad and good, it'll choose good. If you have bad and good and worse, it'll choose good and then bad and worse. It'll always choose bad
David Bau: and for all of these for all the examples that I'll I'll show you.
David Bau: That's I kind of like verify that for
David Bau: so before this, what were you doing? There's no I need to select the right answer. So here I have. You know this like, let me check, and then I force the answer.
David Bau: Here I have stopped thinking, and then force the answer. Previously I just had stopped thinking, and then did no forcing of the answer.
David Bau: So you just present 3 examples, and then you don't say anything after the last one? So I present 2 concatenated examples. And then, yeah, right?
David Bau: Yeah, I mean, I think the thing to take away from from here is.
David Bau: you know, in the setup that we have.
David Bau: which is, we have a good rollout. We have a let me double check. We have a bad rollout, and then we have this like reflection prompt.
David Bau: It will always choose the good role, or it will, it will
David Bau: empirically 90% of the time. Choose the good rollout.
David Bau: and then, if you concatenate the worst rollout with the bad rollout, for example.
David Bau: it will almost always use the bad answer.
David Bau: So I have this patching stuff where what I would like to do is bypatching from this concatenated
David Bau: good rollout. Let me double check bad rollout. Think
David Bau: to the bottom row of the worst, plus the bad. I wanted to change the answer from the bad to the worst answer to the associated
David Bau: and the logic here is that you know.
David Bau: Here these are. These are token token aligned the good rollout, and the worst rollout have been chosen to have the same number of tokens.
David Bau: and so I'm just patching from the exact same bad token to the Bad token.
David Bau: and the theory is that if you do that and you get worse. Answer. What you've patched in is the sense of
David Bau: the rollout that I'm in right now is not the good one. It's kind of worse than the previous one, and so I'm not patching in anything about the content. I'm just patching in like some sense of relative validity.
David Bau: Does that make sense? Do people have questions about the patching setup?
David Bau: I'm I'm not 100% here. So okay, so
David Bau: So
David Bau: both of these things that they are exactly the same. They only differ between the good rollout and the worst rollout. That's right. Yeah. Okay.
David Bau: And
David Bau: and the 4 arrows that you have drawn you will perform this kind of this all of this patching simultaneously. So
David Bau: the I guess the 4 errors are a little bit misleading. Basically, what I'm doing is like the 1st token of the good rollout. I patch to the 1st token, the worst rollout. Okay. The second and I vary like which tokens which positions depending on which setup I have. So each tokens get patched to their corresponding tokens, one at a time, one at a time. Okay.
David Bau: okay, but the issue here is that oops. Sorry- sorry. The
David Bau: you can only do that when the when
David Bau: when they, when the set, when the information in both of these rollouts are kind of sort of like template.
David Bau: So you have an idea that you are.
David Bau: You are only changing one thing or this token here. It has a corresponding token here, which kind of is similar of similar type, but they're only their information is different
David Bau: here. You don't have control over that.
David Bau: Yeah. So I agree. That's a problem in the good rollout to worse route setting at least the way I have it set it up right now. I don't think that's a problem in the bad rollout to bad rollout setting.
David Bau: Yes, because everything is literally the same. Yes, yeah. But couldn't you draw the same conclusion by just patching at the last token
David Bau: hatching? At the last token, because you you force all the way? A table? Yeah. Answer, yeah, or something, or boxed
David Bau: right? And then, if you were able to pitch something here, some position and pointer, yeah.
David Bau: I think that's that's right. Then you would draw the similar conclusion. Right? Yeah. And I do do some of that. The advantage of the setup is you can do that, but you can also account for cases where, like one thing you might imagine is like during the bad rollout you're keeping track of like
David Bau: Well, not so much whether it's correct, because then you just, but whether it's better. And and so, if you don't know where that information is stored. Wouldn't it be nice if our one could annotate its own thoughts?
David Bau: Yeah, yeah, that'd be really nice. I would cite that. Yeah. So I have. I have more feedback. Why don't you just remove the bad rollout from the equation?
David Bau: Just just change the order battery, load, force, result and battery load boot. Result.
David Bau: You don't have to do patching over the batch rollout and let me double check.
David Bau: I'm not sure I understand, so what would I change? You? Just swap the order of good rollout and bad rollout and swap the order of post rollout and bad rollout.
David Bau: yeah. So I would be patching from good plus bad to bad, plus worse. Yeah.
David Bau: So yeah, I guess the
David Bau: the thing that I was worried about there is that you would be patching both the
David Bau: good question. So what was the recommended setup? Good, bad, and then, what's bottom? I was saying that just just for the 1st row, swap, bad rollout and good rollout, so bad, good, and for the second row swap bad, and both so bad words so for the so now, if you look at column wise. The 1st column and the second column is literally the same. It's an autoregressive setup. Patching does not change anything.
David Bau: You just change good rollout and force rollout. Oh, so you just have less to patch because you have this tautology where there's nothing. Yeah. And now and then you you just. But eventually, I guess you also want to understand the setup as well.
David Bau: But as a 1st experiment that makes things a little bit easier, maybe.
David Bau: Yeah, yeah, yeah, it's fewer dependencies
David Bau: that's interesting. So I'm gonna have to talk with you about that. Because I I
David Bau: I have a memory of
David Bau: drawing out all the possible Patrick combinations. Okay, I decided against that. But I can't remember why. So maybe, okay, okay.
David Bau: So I guess. Yeah. So so what happens when you actually do this patching?
David Bau: the main thing that happens is you get something that's that's
David Bau: impossible to visualize in a in a small screen.
David Bau: And so I guess I've I've extracted kind of good.
David Bau: maybe interesting parts of this. So so what I do is I have the tokens going vertically and on the left column. So if we're just looking at the the
David Bau: left plot here, the left column is the the good rollout. The right column is the worst rollout, and so I'm patching from good to worse.
David Bau: And then the colors tell you
David Bau: what the changes in the likelihood of the worst token.
David Bau: So you know, this is very red. And basically, what that means is like your
David Bau: this, this 0 token, which is supposed to be the answer for worse.
David Bau: You're clobbering it with the
David Bau: with the 6, which is the answer for bad. So it's like it's never gonna output 0
David Bau: And then
David Bau: the the x-axis is the number of layers. So that's why it's very dark here, and then it.
David Bau: and that gets lighter as you go to the right
David Bau: and then on the right. Here I have
David Bau: just kind of further down and so, as you descend, as you know, as you go through the the rollouts and number of tokens. You mostly see nothing. And then eventually, you see this little patch of blue, and then at the end, predictably, you see this this red. So you know, when you patch in basically, whatever was going to be output by the
David Bau: by the
David Bau: good worse rollout, you're basically you're patching in the good answer. And so the worst token gets much less likely.
David Bau: So the answers actually appear at the same token position. Yeah. So the answers appear at the same, because the good and the worse have the same number of tokens.
David Bau: So do do we need to somehow make sure that that happens? Yeah, enforce the data center.
David Bau: So I just I do a ton of sampling basically to get these.
David Bau: And what about the suggestion that Chris mentioned patching at the last token.
David Bau: Yeah. So shit.
David Bau: I think my takeaway from at least this single example.
David Bau: Is.
David Bau: You don't really get what you want from patching up the last. No, I mean, this is the last of the I mean, last token of your good adverse
David Bau: last token of your good and worse. Yeah.
David Bau: this is that? Yeah. So on the right here. This is the last token of the good plus worse. And I'm patching into the last token of the
David Bau: that or the other. Sorry.
David Bau: Yeah, the
David Bau: Oh, the worst was bad. Sorry. Okay.
David Bau: This on the left here is the bad is the good plus bad.
David Bau: and then on the right is the worst, most bad.
David Bau: Oh,
David Bau: cool plus bad and voice plus bad. Yeah. My question is, is that token block token.
David Bau: if you go to your previous slide, is it that backslash blocked token? Or is it
David Bau: the last token in your voice, rollout?
David Bau: It's the boxed. Yeah, yeah, no. Yeah. So the last token. And the worst raw is
David Bau: right around here. That would be bad, right?
David Bau: That's in bad. Yeah. Yeah, worse one.
David Bau: This would be bad rollout.
David Bau: So yes, these are both the bad rollouts. Yes.
David Bau: worse, and the good last token of good and worse. Oh, yeah, where that? Where is that? So? The last token of good and worse is
David Bau: probably doesn't have anything.
David Bau: So it's new line, new line, new one new one. Yeah, no, not much. Okay.
David Bau: Before confusing me that you are patching from good to worse.
David Bau: Right? But you are. You're tracking
David Bau: the probability of worse result going up and down.
David Bau: That does not make sense to me.
David Bau: If you're passing from good to worse, you should track the probability of good results going up or down. At least that is how I understand, but he only wants the pointer, but he's looking for a pointer. The correct answer there is bad without any intervention.
David Bau: And you just want to flip to the other one, the one to make the answer, the good one.
David Bau: It's similar to what? Yeah, okay, okay, yeah, yeah, okay,
David Bau: yeah. So I guess I guess what I take from this is is mostly that you cannot make
David Bau: worse. More likely. So like, you know, this is maybe a little bit promising. But
David Bau: you know, you get to maybe a a 5% difference
David Bau: actually less than that you may say it's a lot because the rollouts are long.
David Bau: You're only patching a single token.
David Bau: maybe. I mean, I guess, would it would. It
David Bau: communicates is like, if there is a signal signal, it's probably distributed like, not a single place. Yeah.
David Bau: Might be.
David Bau: Wait one second before you go, for I think I have been in a similar situation. What I would
David Bau: recommend to do is so. If you patch in all the tokens of your good to us, it is bound to change at 0 clear,
David Bau: I think that's your starting point do that
David Bau: or start with a 0 clear, it's gonna work
David Bau: let's say, all the token, just check check till which layer it is working. That's gonna give you some intuition and then start to filter out the tokens.
David Bau: I mean, or the very 1st thing you could try is
David Bau: patching in. Only the last token of your voice, and good does not work. So patch the last 2 or last 3 or last 4.
David Bau: Oh, all at once, yeah, all at once I would assume the the information of the sentence may not just be in the last token it might be a little bit distributed across the last dish tokens.
David Bau: Yeah, that might work. Okay, cool. Oh, that's that's helpful. Thank you.
David Bau: I guess. Let me just
David Bau: yeah. I won't have time for the full thing. So let me just speed through this.
David Bau: I the other thing I did is, I took kind of like
David Bau: each of each of every possible head. What their patching score was. If you just patch that head sorted them by defect. And then kind of
David Bau: you know, added the 1st patch, the 1st 20 together patch the 1st 40 together, and here again I'm plotting the the change in the worst output.
David Bau: And yeah, you really you you don't get too much. These are all the heads in the model.
David Bau: So these are all as the model are candidates for this. Yeah, at all token conditions.
David Bau: and you kind of start leveling off after you get there.
David Bau: I also tried doing evolutionary strategies for a com. All the combination of heads like, maybe there's something nonlinear going on cool.
David Bau: you get this Pareto frontier, which is get to about
David Bau: 11% is the best you can do.
David Bau: And that's with 32 patch components. So I think that's somewhere around here
David Bau: and the best results that you get
David Bau: with that is this guy right here where?
David Bau: you know, 0 is the Mlp, so the Mlp. Turns out not to be
David Bau: important when I do that. And then, you know.
David Bau: maybe some indication layer layer 2223, 24 are important. But
David Bau: not sure what to make of that.
David Bau: And yeah, so then, this is, this is kind of like averaged over the data sets.
David Bau: you know, you get similar kind of like.
David Bau: Nothing particular, particularly dramatic to the folks
David Bau: so I think that's that's all kind of the empirical results I had.
David Bau: I yes, and Nikhil suggested, starting with the the 1st token or the 1st layer, and
David Bau: going from there, do people have other suggestions? I see I have another one your setup actually reminded me of when I was looking into the Mccu setup. Okay, it was like one like option option. And then finally answer, which is kind of similar to the
David Bau: little bit like you have 2 options, and you need to choose one of them.
David Bau: Yeah, we may. If you want, we can discuss it later. But yeah, that setup and the setup same similar. So you may. That might give you some
David Bau: good intuitions to start with or looking for the dilemma camps. Okay, yeah.
David Bau: I have another suggestion. I think
David Bau: catching at those tokens only makes sense.
David Bau: If these tokens are of same type, and you assume them to the
David Bau: assume them to have similar types of information otherwise matching across tokens
David Bau: to me, it does not make sense.
David Bau: Yeah. So my basically what I was. My next steps were I was going to change the
David Bau: the difference between the good and the the worst rollout to
David Bau: have that same property, but be just fully aligned and just have different digits
David Bau: with that satisfied? Yeah, yeah, yeah, I think so. And also, like, you can probably have
David Bau: similar templates like this. Does. This Rollout has to be something like that is model generated. You can. You can generate them yourself.
David Bau: Yeah, I can. It's harder to build a data set. And I'm also just a little weirded out by.
David Bau: yeah, I feel like there might be interesting things going on with like steganography or the model, like choosing particular tokens. Maybe you don't even need templates. You need. Let's say you decide on 6 lines. 1st line says something like that
David Bau: second line says something like that by the final line you actually have to answer, and then you will patch across lines
David Bau: all the tokens in a line. All the tokens in the line is kind of thing. Yeah, definitely. Yeah.
David Bau: I wanted to.
David Bau: Actually, before seeing your head fetching experiments.
David Bau: I wanted to suggest. Like to do something like patching a single
David Bau: principal component at the last token, only right to simplify all of this yeah. Token position, mess.
David Bau: But I think it's not gonna be more expressive than what you did with the heads.
David Bau: I think what you did with the heads is pretty nice.
David Bau: Yeah, yeah, I think so. So.
David Bau: actually. So, I think this is
David Bau: so. I did the the evolutionary strategies, both at the last position and over the full model.
David Bau: Yeah, they end up quite similar, which implies that the last token is is kind of where it's happening. Yeah. And so, yeah, I guess that was my conclusion.
David Bau: Yeah, so you can still try it. I mean, people are also trained now saes right on on
David Bau: one. Yeah. And then there is like this recent. I sell our workshop papers by by Josh. 1 1 by Josh, and one by Ben Wu.
David Bau: where they look at se features that are just always on or more on more than 10%. And there there they find a lot of those are actually these positional
David Bau: like these pointers.
David Bau: Yeah, that's a good idea, among other things. Yeah. But yeah, I
David Bau: yeah, I started doing the essays. But I haven't. I haven't done any analysis.
David Bau: Also I trained a tune Glen Setup for this, but I also have not started to use it.
David Bau: I don't know, so I think I think really my next steps are, you know.
David Bau: I I feel I feel a little bit limited in kind of coming up with
David Bau: theories for what underlying algorithm is is being chosen here.
David Bau: And so yeah, I'm kind of, I'm gonna start with these like unsupervised methods 1st and then have that inform the patching.
David Bau: David. What's the algorithm you're trying to analyze here. What's the main question?
David Bau: Well, yeah. So you can do that. Like, I guess, different levels of abstraction.
David Bau: here the main question is like.
David Bau: is there some elegant mechanism by which it chooses between 2 different answers? Can you manipulate that? How does that change? If you get to 4 or 5 different answers, I see, does it naturally happen in these rollouts that it comes up with 2 different answers?
David Bau: So yeah. So if it, if it kind of like gives an answer, and then says, Wait, let me check sometimes the answer will be the same, but a lot of times the answer will be different. And so, you know, after the after the end of think tag, it kind of has to choose
David Bau: right, but usually on those like self correcting like I made a mistake here. So maybe I should correct myself. Yeah, yeah, right? So so
David Bau: right? So I guess you're saying like, maybe it just always chooses the last answer.
David Bau: Yeah, I don't
David Bau: think that's the case. Maybe that is the case. And maybe that is kind of the mechanism, right? Because right now it looks like starting the rollout from scratch again. Yeah, yeah. And it's like, yeah, I guess, like, one indication is.
David Bau: yeah. So if you do this.
David Bau: you know if you have the good and the bad, and you do the standard rollout.
David Bau: It doesn't always choose just the second answer.
David Bau: So there's some like discernment going on.
David Bau: and when you have the deliberation it it is very clean, right? It it doesn't always choose the last
David Bau: right, right? So like, if I replace that with, Oh, wait, now, I have the answer. Then would it be always the right last answer?
David Bau: I haven't read that like. Usually. That's how the rollout finishes. Right? Yeah, like, now, I'm satisfied. And then it's like, Yeah, I don't know. My guess is like you could. You? Could.
David Bau: That would be some signal that you could like. Put truly terrible rollouts in there, and it would self correct right? So then, like Pointer, should be somewhere here in this, I have the answer, or like, let me choose the right answer, yeah. Yeah. Oh, you think the pointer is actually in here somewhere? Maybe like, if you change it, and it changes to always the last answer.
David Bau: So don't you? So don't you guys think it's surprising that
David Bau: that David hasn't found a pointer at the last token.
David Bau: you know, at the box token the the right right before the answer
David Bau: is copied there wouldn't, wouldn't you expect? And maybe there is just no pointer
David Bau: I don't think so. What did you find at the last token position? Did you bring up the heat map or
David Bau: the previous slide?
David Bau: Yeah, but all you're doing is you're you're eliminating the probability of choosing the worst answer, because you're increasing the probability of choosing the good answer.
David Bau: So it's not really what I want. Yeah. So you're expected. You expected a blue spike here
David Bau: I was hoping for. Say, like, like, obviously, there's going to be a red spike at the end. Yes, but I was hoping for something like a blue spike, you know, maybe at the middle layers, or like, or you know, at earlier tokens, or something like that.
David Bau: What do you get for? Like the token like, do you like? What what results did you get for that like if you go back to the template that you have like at the token for let, and I guess the token
David Bau: so the token for let is here.
David Bau: So yeah, not much. And then.
David Bau: hmm, is you're also not that much.
David Bau: Yeah. So just to confirm red, here is the logic of the good answer. Blue, here is the logic of worse answer. They're both. They're kind of the change in the logic. They're the change in the probability of the worse answer
David Bau: change in the probability of worst? Answer. Yeah. So red is the worst answer is less likely. Blue is the worst answer is more likely more likely
David Bau: it. It might be interesting to just look at that last token, and then plot the 3,
David Bau: you know, all 3 probabilities of of a good, bad and worse. Yeah, yeah.
David Bau: Oh, yeah. Another thing that this is probability. Right? You probably should. Wanna look at logics as well, because probability, because small changes in logics will not get reflected here.
David Bau: Small changes in logic will not get reflected here, because probably it will. Just it's an exponential scale.
David Bau: does is that that seems desired by me right, because ultimately what I care about is probability. But if there is some small changes in logic, something is systematically getting promoted in the logic space, you will not capture it here.
David Bau: Yeah, yeah. I mean, I guess, like, yeah, you'll get stuff like, like, with this. There's some small change of probability. So you're saying like, I'll pick up on more Logos. I guess the question is like, Do I want to pick up on that in this setting.
David Bau: Yeah, yeah. For now you're like, I don't see anything. And they're like, well, maybe maybe if you take off your sunglasses, you can see something. Yeah, okay. But yeah, how useful that will be. I have, I have no idea. Yeah.
David Bau: yeah, I can try it. I would be very open minded to other solutions.
David Bau: Good other solutions that the model is implementing. Yeah.
David Bau: like what? I don't know. I thought you were
David Bau: the the model in the default processing. It just keeps spamming these answers right? Like, a lot. Yeah.
David Bau: modern tool. Yeah, yeah, lots. And they repeat.
David Bau: sure. So I could imagine that like while doing that it just
David Bau: maintain some belief about what is the what is like?
David Bau: The best answer, or like how the answers relate to each other
David Bau: like, if I generated 5 times. The answer is 3,
David Bau: and with with 5 different algorithms, which is not the case, like often it does like, Oh, let me use a calculator, or let me use the whiteboard
David Bau: right.
David Bau: but at least in the chain of thought. You you have like 5 times and 3 occurring, I think, that should do something.
David Bau: With regards to whether free is a good answer or not.
David Bau: Right?
David Bau: yeah, along the way a lot.
David Bau: But in. But in this. But in this setting, where you've got just 2, which happens sometimes right?
David Bau: Well, maybe there is a confidence associated with each algorithm. Right? That 1st answer I derived by
David Bau: I don't know.
David Bau: Applying induction proof, and second answer, I did
David Bau: prove my contradiction, but I know I am worse at that, so
David Bau: maybe I trust the 1st answer more because it's
David Bau: okay. But but still so even even though there's some logic there.
David Bau: If if you say I'm going to trust the 1st answer more.
David Bau: Then at the very end it's got to copy the answer from the 1st answer, right like, which is moving along all the time. Right. So there's so there's 2 pipe. I guess it's like, I can only think of 2 ways that information get goes. It's either
David Bau: by reference. And it's gonna dereference, this pointer at the very end, or it's by value, and it's literally copying the number 0 along.
David Bau: you know.
David Bau: you know, so that by the time it gets to the end. It like has the number 0 here, or it has a number 66, or whatever
David Bau: right?
David Bau: And and so like, I I was really surprised, not
David Bau: like it seemed weird and impossible for it to be bringing the number 66 along, for for it to be by value because of all of Nikhil's stuff that, like everything, is relentlessly by pointer. But did Nikhil do? Quinn?
David Bau: No, didn't look at Quent? Maybe Quinn is by value.
David Bau: Okay, so like, these are really small models.
David Bau: Yes, the small models have this. How? How big is it? Only 1.5. 0, yeah. And then there is the other thing right? I mean, it's just a Gsm. 8 K example. So the model might just look at its Mlp, it's like, Oh.
David Bau: I know, it's like this. Gsm, 8, k, example, right? It's 66.
David Bau: I just memorized this.
David Bau: Yeah, I guess. Intuitively like.
David Bau: And then there is no pointer.
David Bau: Right? Yeah. So oh, I have one alternate hypothesis which might explain
David Bau: why we are not getting the blue one. So maybe the correct answer
David Bau: for the good one. The correct answer, like the number token in the good one and the worst one
David Bau: have different pointers. So I think here right now, we are thinking pointers of like a 1st and second option, first, st option on the second option, but maybe the numbers have their own
David Bau: pointers.
David Bau: And
David Bau: The answer of in your good rollout.
David Bau: The answer token, let's say, is the 5th number.
David Bau: and the num like in your worst rollout. The answer token is, let's say the 7th rollout.
David Bau: Sorry token. Sorry. 7th token, yeah. And hence, when you're patching it at the last token, the model might just be looking at the 5th or yeah, the 5th token of the worst rollout.
David Bau: and hence not getting the the correct token, which is the 7th one.
David Bau: This is just alternative hypothesis.
David Bau: Yeah, no, that's that's that's a good point. I think the the fully templatic. Good.
David Bau: Yeah. So is it like fully templated right now, or you're planning to do it? No? So right now, yeah. So right now, the good and the the bad are standard rollouts aligned, whereas if they only differ by the concrete token. Yeah, yeah, like.
David Bau: okay. So if the pointer is pointing at the wrong spot.
David Bau: you would, would you expect to boost like just a a random logic
David Bau: corresponding to that spot. And we are not measuring this right now, because we look only at the 2 candidates.
David Bau: Yeah.
David Bau: yeah. Maybe starting with the counterfactual, where your good and the voice just differ by one token, one number.
David Bau: and then checking out if you see a blue one there or not. At the last token, I think, that might give us a subsequent
David Bau: oh, God! How would you force your thing tokens to be templated?
David Bau: Wouldn't that be unnatural?
David Bau: Of course, my think test is concerned. Yeah, it might be a little unnatural, but because our one cannot do that
David Bau: right. I mean, I think the the natural. What I was gonna do is is just because the tokenization is per digit, which is nice. You can just change 2 to 7 here or 2 to 9.
David Bau: And then yeah of
David Bau: oh, sorry. I meant like template in terms of like, I don't know what I'm not saying. They all need to differ in like just one.
David Bau: Oh, well, has a nice temperature, I mean, probably you can start with that 1st and see how it goes, and then you can make it more natural. I mean as natural as possible. Then it's it's harder to
David Bau: yeah, I can try it. You might not get this, the scenario that I want where you have order, invariance, actually, probably that's not what you want. You want template by, but line was line, wise, template, not like token, wise or word wise template.
David Bau: Then you will have information which line is communicating this pointer information.
David Bau: Yeah. And then, once you understand that line, then you can.
David Bau: I don't know. Try to impose some template on that specific line, which seems much easier.
David Bau: and I hope that it will be at the answer line, and then it will yield to exactly what you just suggested.
David Bau: Yeah, you could do that.
David Bau: how would you feel about Tall's automatic span labeling solution?
David Bau: Oh, man.
David Bau: so I I like. So I like the advice that you're generally getting from the group, which is, instead of making it harder. Yeah, make it easier.
David Bau: right? And you. I feel like, you know, you're you're sort of like. Oh, but it makes it less realistic. And then they're they're sort of saying, Well, you can make it more realistic later, right? Like, if if you don't, if you don't have signs of life.
David Bau: make it super easy until you see signs of life, or or maybe you still don't see signs of life, even if you make it super easy, then the mystery deepens, and you can make it easier and easier.
David Bau: Yeah, right? Yeah. You know, that makes sense.
David Bau: I think, in speaking of like different ideas.
David Bau: I suppose if we were to think about how like, if your correct answer is 60, and your incorrect answer is, 33, or whatever it's helping either 60 or 33 like, I would imagine it's 1 of these attention heads that are promoting either one of these. And so I think I would focus
David Bau: there and look for specific heads that are
David Bau: like, I suspect you'll find the attention head that not only attends heavily to 66, the correct answer, or 60, or whatever the number like 66, it would not only attend heavily to 66, but by doing so it also promote the likelihood of 66.
David Bau: So I would like, if you, instead of like trying to localize by
David Bau: activations. Maybe you can try to localize by detection heads.
David Bau: Now you have some attention. Head analysis. You did. Does any of it come down to that.
David Bau: Let me think so.
David Bau: Click on the next slide.
David Bau: Yeah, these kind of things. Did any of that come down to that, or is that it didn't seem like there was
David Bau: a single attention head which was right. But I imagine, if you look at some of these that you highlighted, I would imagine if you look at their attention patterns you'll see focusing on, say, 66. Yeah, I imagine. I suppose there are either, maybe 2 scenarios. One is like.
David Bau: I don't think this is likely, but you have one attention head that attends at 66, as well as 30, the incorrect case, and like
David Bau: that attention had. This is like solely choosing which one to output. I don't think that's likely, I think what instead, you'll find is some heads attending to 60, some heads attending to 30. And then it's kind of this competition
David Bau: of like which one wins out. And so
David Bau: do you think this is like kind of just additive process of each attention had. I think so. And if that's the case, you could try hacking these heads to see if you can switch like reliably
David Bau: make it output 60 or 33, and so on.
David Bau: Yeah. And and how would you intervene on those sets
David Bau: intervene on? How would you intervene on the head to make it outfit 66. So either like you can.
David Bau: If
David Bau: if you have, if you see attention heads heavily attending to 66. You can like dampen those attention patterns, maybe.
David Bau: or other way around for the ones attending to 30. You can heighten their attention patterns, or vice versa. You have like their W out vectors, maybe you scale those larger. I think there are probably different ways to go about it. But
David Bau: but instead of trying to localize with activations.
David Bau: If we think attentions are what's outputting these tokens, I would. I think I would try to localize the attention. That's first, st
David Bau: yeah, okay, yeah, I can. I can. I meant to talk more.
David Bau: No, I think that's great. Thanks, cool. Thanks. You guys, thanks for writing back. Yeah.
David Bau: I guess. How much time should I? Should I allot for this? Oh, I'm not sure. So you can. Maybe half an hour or something like that. Okay, all right, I will do my best.
David Bau: okay, so let's see.
David Bau: Alright. So here's this like obligatory slide that I have here. And
David Bau: you know the the alternative opening picture I was going to use is this tweet I saw
David Bau: by Phd. Student, and the Tweet was essentially Phd. Students on the West Coast are very different from Phd students on the East coast.
David Bau: and my my hope is to try and
David Bau: give some quantitative explanation of what the worldview is here, and maybe like how much support there is
David Bau: for it. And so so west Coast Phd, students are different from East Coast Phd students, that's that's the take? Yeah. So when, when, like, Stanford steals.
David Bau: you know shooting from us. Yeah. Then he transforms as he crosses the Mississippi. Yeah. Right? Yeah. Yeah. We can ask him, we can ask him, yeah, so I guess just you know, here is this this kind of like attempted history of the past.
David Bau: 3 years. And so, you know, he's talking about
David Bau: the hype over language models in 2023, you know. Are they? Are they?
David Bau: Are they actually thinking, are they not? All this Vc money? 2024. You know, the apps aren't really working, but you know the chat bots seem to work great. 2025, you know, we get to this like chain of thought process. We get our one. We get rollouts. And
David Bau: you know the punchline is that this was written in August 2,021. And this was, you know, this is roughly when the stochastic parrots paper came out.
David Bau: yeah, it's a little bit tricky, but I think the the kind of like overall picture is quite good. And I think this picture is like impressive evidence of being able to say something useful about AI over the next, you know, 6 years. And so the guy who did this
David Bau: recently released this this new what's the name of this guy? This is Daniel? So he used to work at Openai, and he he left as a as a whistleblower.
David Bau: and all right, yeah, yeah, and so he has this forecast of, you know. March 2027. Yeah, you can do the job of the best human coder on on taps involved in AI research, you know, in
David Bau: what 8 short months we get to a super intelligent AI researcher. And then, you know, less than a year later we get this artificial superintelligence so better than the best human in every cognitive task.
David Bau: And, like, you know, it's very, very scientific as log normal distributions, confidence intervals. And so you know.
David Bau: what do people think of this? Some people hate this. I assume this is what most of you are thinking. I'm I'm very sympathetic to this response.
David Bau: but I'm hoping for this presentation to be kind of an intro to what the scientific evidence is for how things are evolving in AI,
David Bau: okay. And
David Bau: maybe the last thing before we get into it. Why should you actually care about this, you know. I think it's like intrinsically interesting. It's, I think the heuristic of like Know what's happening in your field tends to perform pretty well over time
David Bau: in particular. I think it's hard to understand how like a lot of the field is approaching things. If you don't have a model of what they care about. Why so like? Why is anthropic pushing so hard on saes? You know some of that comes down to timelines that they have.
David Bau: This, you know. Maybe fanciful. Fanciful. Take this. This guy is a Ph student. Phd. Student at eth, you know he's like he's a real dude. This is Daniel Palaka. He was a co-author on the refusals mediated by single direction paper. He co-authored a lot with Nicholas Carlini.
David Bau: But, like the actual answer to why you should care is because if you disagree with the forecast that I showed you, you can bat with them, and you can get $100 if you're right, and so I encourage you to pay attention. And if you see something that you think means that, you know, like this approach to forecasting is is wrong or or
David Bau: fundamentally flawed. Then you can get a hundred dollars. Maybe you can give me some, but as a my understanding.
David Bau: So here's this big picture that I have when I think about.
David Bau: you know how AI progress proceeds.
David Bau: So
David Bau: you know, we start with like essentially Moore's law for Gpus. That's compute we have. How much do you spend on Gpus? That's investments?
David Bau: That combines to get to physical compute
David Bau: when you combine that with, you know the actual, the flops that you're generating, the algorithms that you have determine how effective that can be at transforming flops into loss. Numbers.
David Bau: loss. Numbers go down as you add more data, as you have more effective compute, and then the loss
David Bau: is transformed, I think, relatively, predictably into benchmark performance, and then benchmark performance, I think, less predictably transforms into real world performance.
David Bau: And then we have these 2 kind of like like, why do we care about real world performance? And so I'm going to talk about this less. But
David Bau: you know, one answer is obviously investment. So like as AI gets better. People spend more money, and you have this flywheel. The other, I think, like fundamental assumption of this worldview, is, as you get, better models, you will get better algorithms. So you will have these AI researchers. And the question is like, as you increase the effectiveness here, how does that actually like feed into this model as I've set it up. You have a feedback loop. Does this work. Do you have diminishing returns? That's that's kind of the question.
David Bau: And you know
David Bau: this is pretty pretty broad presentation. But I'm happy to dive into stuff if people have particular interests. And then also like definitely push back on anything you think is incorrect, or or
David Bau: having a or maybe like fundamentally flawed.
David Bau: Were you defining self-improvement previous slide
David Bau: this one, the feedback loop. Yeah, am I defining it?
David Bau: Is that what you're saying is that? I think that is what is implied by this diagram? Yeah.
David Bau: because of the arrow up. Yeah, yeah.
David Bau: I mean, also because of the arrow down. Right?
David Bau: You know, if you go from yeah colleagues, is it possible that this
David Bau: cycle breaks because we have limited data.
David Bau: Yes, yes, and I will talk about that. So there are a bunch of things that it could. It could
David Bau: make this stuff. So the 1st thing is compute. This is mostly. This is just like pretty empirical. You look at Gpus, and you look at like how fast their flop per dollar is improving, and you get a slope of about 2 and a half years. So here they they say it's about 3 years on the right. They look at updated numbers. It seems to be closer to 2 and a half.
David Bau: those those are pretty close in log space. For context. Gpt-four costs about 50 million dollars to train
David Bau: if you I guess if you if you in 10 years, just relying on this. Nothing else. You could train Gpd 4 for about 3 million dollars. So that's kind of like
David Bau: rough sense of the effect here.
David Bau: And I plotted these numbers here so like 1.3 x
David Bau: every year. That's about a 10th of an order of magnitude, and then that doubles every 2.5 years.
David Bau: So then we have investment. So investment doubles every 9 and a half months.
David Bau: we plotted all these these numbers here. Yeah. So one question is like, How do we get these numbers? So sometimes people will just report them publicly. They'll they'll say, like we spent, you know, X number of flops on llama. Sometimes
David Bau: you can get it privately through the grapevine, sometimes you can infer it from public statements. So like, I think, Mark Zuckerberg said that they train one of the llamas on like X number of gpus for X amount of time, and you can kind of like infer. And then sometimes you just have to guess, based based on loss.
David Bau: and you know, like inference about the number of tokens that are used.
David Bau: the other thing. This is a little bit sensitive to amortization choices. So if Gpus are improving really fast, then you use them for less time, and so they're more expensive because you want to replace them quickly.
David Bau: In this case it's it's less important because we know exactly how fast stuff is improving. But you know, for forecast, that's something you want to take into account.
David Bau: Okay. And so when you combine investments
David Bau: and compute, you get this this combination. So we're at about every year.
David Bau: The amount of like actual flops that we spend on frontier models goes up by about 3 times that doubles every 7 months.
David Bau: okay? And so then we have this question of like, that's our calculation. Does this match what we see empirically?
David Bau: it actually undercounts a little bit. What we see empirically.
David Bau: So one reason is, is quantization.
David Bau: Another reason. So so you can get kind of like more flops out of out of the same Gpu. Another is is just data center efficiency. So in general, like, you know, memory gets better. You move things back and forth. And
David Bau: you know, I guess, for now I'm gonna stick with the the slightly lower numbers and and the
David Bau: interest of of conservatism. So I'm just going to add these together. But these are roughly what what you expect to see in real life.
David Bau: So let's take a look at algorithms now.
David Bau: So this is the first, st the 1st one that I was a co-author on. So I'm going to go into a little bit more depth. So you know, we have this one little question like Gpt is, Gpt. 4. Is a lot better than Gpt. 2.
David Bau: How much of that is compute just like raw compute. And how much of that is stuff like, you know.
David Bau: group query, attention, glue rms, norm, you know, whatever fancy algorithms people are using now.
David Bau: the answer seems to be that it's mostly compute. So you can see at the bottom. Here we have this. This is physical compute ceiling, and then algorithmic progress in particular. It's the amount of compute needed to reach a fixed level of performance on halved every 8 months.
David Bau: And
David Bau: My big takeaway from this paper honestly is that the transformer is is alone 20% of the algorithmic gains. So it does a really good job of converting flops to loss reductions.
David Bau: But how do? How do we actually know this? So
David Bau: we took all these language models. We evaluated them, or, like took their public evaluations on entry Bank and Wikitext and for all of them we collect parameters, data and loss.
David Bau: and so we start with this this Hoffman scaling law. This is the chinchilla scaling law. And so you know what that says is that you start with the irreducible loss. This is roughly the entropy of the data set. Then you have these constants, and then you have these exponents for parameters and data.
David Bau: And so our model assumes that you know
David Bau: the kind of like multiplier for parameters and for data grows exponentially
David Bau: over time. And so we have this model. You know, y. 0
David Bau: doesn't really matter. But we chose, I think, 2012 here.
David Bau: and we fit this data. So we have. We're we're estimating alpha year.
David Bau: alpha parameters, beta data and beta here
David Bau: and we have. We have variants of this, but most of them performed roughly the same
David Bau: and so you know, that gives us this, this top level number of algorithm progress house every 8 months.
David Bau: We also want to disentangle like you know how much of it is parameter scaling, how much of it is data scaling how much of it is parameter efficiency, data efficiency. And so what we do there is we use Shapley values. So we have this like nonlinear nonlinear model. So we can't just take the coefficients of all these things.
David Bau: And we
David Bau: maybe I I won't get into details here. But we basically take the average marginal contribution of every every
David Bau: player that we have here over all the combinations that you can imagine. For
David Bau: you know, first, st we would do parameter scaling, then data scaling, or 1st we would do data scaling and then parameter, scaling, etc. And so what you see is like it varies pretty widely across models, but generally parameter scaling is more important. Data. Scaling is, you know, the second most important and data efficiency parameter efficiency does not really matter that much.
David Bau: and you know some of this is a little bit confounded, because
David Bau: earlier in, you know, the the 2015 to say 2022 we were much more doing parameter scaling. Now we're much more doing data, scaling after Pinchilla.
David Bau: But overall that those are results results that you get, how do you define efficiency?
David Bau: So this is kind of the
David Bau: You know how how you convert data into loss reduction.
David Bau: So if if it's
David Bau: you have higher data efficiency. As you increase data, you get more loss reduction for every doubling of data. So the 2 columns on the right are the kinds of things that Phd students might hope to work on.
David Bau: Is that right? Yeah, that's right. That's right. And you know, I think
David Bau: I guess, like 1 1 takeaway I have from this worldview is mostly you don't want to be working on capabilities because it's it's pretty boring and pretty predictable. But
David Bau: I think people could disagree with it.
David Bau: I guess the
David Bau: the transformer to to get that in particular, you know, you can add a a indicator variable and refit the model. And then look at, you know, if you click on, flip it off, we get what I said, which is about, you know, 7 times less compute to get the loss reduction that you would you would want. And then,
David Bau: that's 20% of algorithm progress. Yeah, interestingly, so, yeah, so one limitation is, we use the chinchilla scaling law. It turns out chinchilla scaling law, which was a correction of the capital scaling law is a little bit wrong. They they didn't let their their optimizer run a convergence, apparently, and so it maybe underestimated a little bit. The importance of data, and so maybe, like you should,
David Bau: should shade these up a little bit.
David Bau: And then we also, you know, we're just looking at like
David Bau: the number of tokens to loss reduction. And so that doesn't distinguish between data quality. And, you know, like an incredible sweet glue
David Bau: innovation.
David Bau: So I guess you know
David Bau: I 1st heard this from Matthew Barnett, who worked at ebook, and his thing was that
David Bau: you know in general you have a lot of these domains where algorithmic progress tends to grow at roughly the same rate as, or, you know, close to the same rate as compute progress.
David Bau: And so you know, if you look at chess engines like as you get more compute, you get better algorithms.
David Bau: And you know, there are a couple of different things you could imagine for that for like, why, that is but you know, one is just like you have this problem of like you need to experiment. You need to like, try lots of algorithms. And for that you need to compute. And then you have this twin problem of like, maybe algorithms only apply to certain scale. So they don't become relevant.
David Bau: I think this is particularly important, because, you know, if you think AI researchers can be automated, you and you think this is right? You're going to have to account for this problem where you're capped by experimentation. You're going to need to figure out some way that you can get kind of like better sample efficiency for your AI researchers without just spending on physical compute.
David Bau: So just we. So this, the previous slide that goes against the beta truth, right?
David Bau: This one, this one, the next one, this one. Yeah. If the algorithmic improvements is us.
David Bau: I mean is is on par with the compute one.
David Bau: Doesn't that go against the beta truth?
David Bau: Yeah, I mean, I think it. It depends on
David Bau: a what your baseline was for what you think the better lesson was telling you. But
David Bau: you know, I think if you were trying to recover that you would say something like, you know, all this algorithmic progress like, it really only works because they're just trying lots of stuff. And there's kind of like, you, you know.
David Bau: a lot of these, these experiments are automated. So they're trying like 12 different kinds of swiggly variations.
David Bau: But you know, certainly, like insights are important. Yeah.
David Bau: 1 point I was reading this book, the human computation, a human compatible. Yeah, it's 2 or 12, I think. One example that
David Bau: is coming on to my mind from that book is what he says is, I think he was trying to explain how research generally works, that we try out a lot of thing in research. Lab. Only one or 2 things actually get get ended up in a real actual product. And he was actually giving example of a chess engine.
David Bau: I don't know the detail, but I think what he has mentioned there is the actual algorithm
David Bau: was more or less known by 1960 S.
David Bau: And it was only when the compute become more and more accessible, the look ahead algorithm become more and more like actually useful that it can
David Bau: be on par with the human, and then later be able to beat up the human in a chess game. Yeah.
David Bau: And that seems to again align with the better truth. One?
David Bau: yeah, yeah. So I guess. My, you know, I don't remember. This, this cut your Grace paper.
David Bau: super. Well, but my memory of that paper is what what she found is that
David Bau: you know that was the bulk of how you improve chess engines. But, like, you know, if you took algorithms from today and applied them 20 years ago, like you would get serious improvements.
David Bau: So you know, there are like little things you can do on the edges. You know
David Bau: ways. You can be smarter about like which nodes in the Mcts tree you're choosing.
David Bau: Oh, okay, so
David Bau: that's algorithms. When we combine algorithms with with physical compute, we get effective compute. And
David Bau: you know, this is interesting, because, you know, for one thing like Sb, 1047. If people remember that bill would peg the trigger at which these models became subject to state scrutiny at, you know, I think it was 10 to the 26 flops.
David Bau: but that was effective compute that was not physical compute. And so, as you know, other way around
David Bau: that was, they regulated physical compute right? I think they regulated effective compute. Oh, really, yeah. And so,
David Bau: oh, sorry. Yeah, you're right. You're right.
David Bau: And so you know. But the implication is like, as time goes on the the amount of compute that you would actually need to reach the same performance would fall by about 3 times every year. And so you have to like, make this choice of like, how do you actually want to? You know which one of these is relevant? And how fast. Do you think algorithms are going to improve?
David Bau: Can you remind? Or can you explain briefly how to read this?
David Bau: Yeah. So now, you have like 8.9 x 0 point 9 5 oms and 3.8 months or something. Yeah, what does this mean now? So you know, the 1st line is the multiplier per year.
David Bau: So every year you get 9 times more effective, compute the second one. These are all just different ways of expressing the same number. So you know, roughly, you get 10 x per year. Yeah, it's point 9 5 orders of magnitude.
David Bau: And and then every 4 months the amount doubles.
David Bau: So that's very fast, you know, this is like,
David Bau: yeah, the you have to get like very rapid diminishing returns on loss to keep that up?
David Bau: So I'm not gonna spend a ton of time on data. But just, you know, as a as a takeaway, like people use a lot of it. It's going up exponentially like all these things.
David Bau: and there's this question of like, are we going to run out? So if you you know, if you estimate, based on the total stock of public data.
David Bau: what you get is about a median date of 2027, when we run out.
David Bau: Some of that depends on you know how much you expect models to be overtrained. So
David Bau: this is assuming that they're going to be trained with 5 times more tokens than chinchilla would would tell them to.
David Bau: But you know, it turns out not to be super sensitive to that number.
David Bau: and yeah, the fundamental dynamic you get is like this, this thing is going up
David Bau: inexorably. And you're hitting this like, you know, we are creating more text. But we're not creating it that fast.
David Bau: And so what do you do?
David Bau: so you can under train, so you can return to parameter scaling, so you can use the same number of tokens, but just more parameters.
David Bau: You can pay for more data. So like, pay scale AI pay New York Times. All these private data stocks.
David Bau: You can use synthetic data, which is what we're increasingly seeing. And you can train for more epoch. So it seems to be like I don't know. Somewhere between
David Bau: 3 and and 12 epochs does not seem to be. You don't seem to get diminishing returns
David Bau: and then you can also use other modalities. So there's a ton of genomics data. There's a ton of like video data out there. And and you know, I I think, unclear how much that'll be useful. But
David Bau: epoch, when you account for all that. You know this is this is not something I worked on, but they estimated that you would get you can scale up to about 2 to the 2 times 10 to the 30 flop
David Bau: without hitting diminishing returns. And so you know, that's like.
David Bau: maybe you know, 4 more years of scaling something like that.
David Bau: so yeah, we have effective compute going very fast. What did they estimate diminished
David Bau: just because it hits like human generated text?
David Bau: Take the comment
David Bau: so like, why you get diminishing returns when you train on multiple ebooks, or just in general. I mean, why is human generated public text like the cap like, why is there no synthetic sealing? Oh, so
David Bau: this this image right here does not talk about these other things. So this is purely on.
David Bau: If you only want to use public text.
David Bau: yeah, so I won't go into details here. But when you plug these numbers here into the chinchilla adjusted chinchilla scaling laws, what you get is that the reducible loss goes down by about 33% a year.
David Bau: And so.
David Bau: you know, the question is like, is this really something we should care about is loss like relevant
David Bau: in some theoretical sense. Yes, so you know, in principle, like
David Bau: 0 loss implies indistinguishability. Right? And so, if you have a loss. You also have a number which tells you how many tokens it will take to distinguish between these 2 distributions at a certain, a certain level of confidence. And so you know
David Bau: what this approach is is doing is it's saying, like, well, if you assume that you know you have some number of tokens that you care about. You know, for example, like, if you have length of an average scientific paper which is about 5,000 tokens, and you assume that humans are some multiple worse than an optimal predictor at distinguishing between distributions. So like you get a token you're like, is that a scientific paper? Is that not how many tokens will it take for you to get like, you know, 10% confident?
David Bau: and what this model tells you is like, it's it's pretty uncertain. So a lot of that is, you know, uncertainty what this model does. It has uncertainty, and, like all the scaling law parameters, uncertainty, and the the slowdown parameter. How much worse humans are. And
David Bau: You know, you'll notice that it has like substantial loss on numbers that we've already reached.
David Bau: It also has substantial
David Bau: probability mass on numbers that you know we will never reach, or just like certain physical ones.
David Bau: I'm totally not following what this is what this is about. You talked about. How many tokens I need to distinguish.
David Bau: distinguish a generated paper from a human paper.
David Bau: and then you switch to this plot on the right side. Yeah. So so,
David Bau: this part on the right is saying.
David Bau: if you want, it's trying to quantify. Given a like non ideal predictor. So given a predictor that needs, you know, 2 times as many tokens or 10 times as many tokens to predict
David Bau: And you use the scaling laws that we have. How many flops will you need to train a model such that it can fool this predictor basically over 5,000 tokens so like your and then your your predictors, like
David Bau: you're trying to estimate how many flops would it take to pull a human?
David Bau: Exactly right. So so we'll just
David Bau: we'll pause it, that you have a model of a human here, right? And so you're saying that there's some uncertainty in that, because here's your bell curve of like you would need.
David Bau: you know, under some range of assumptions, you would need something like this number of flops to pull a given
David Bau: where the human couldn't tell the difference between a scientific paper made by an AI and a scientific paper.
David Bau: Yeah, that's right. That's right. Now. There's there's a little bit of a gap here, right? Because we we all know that
David Bau: you know the point of the review process
David Bau: is not to find scientific papers that are in error
David Bau: right. The point of the group process is to find scientific papers that are
David Bau: written well, and you know, use the idioms correctly and and are self consistent. And you know, do the all this type of stuff. But like the the sort of findings of truth
David Bau: like come outside of reading the papers they come out of like a much slower process of like reproducing the results.
David Bau: And that type of thing, right? So I I'm like, I'm not sure like is this is like is like getting through, like the equivalent of peer review, of like making a readable paper. Really, the thing that we care about, or actually, is it like producing true research, which is actually.
David Bau: you know, it's it's like a harder thing.
David Bau: so I guess my take is like you should not. You should not trust this for a bunch of different reasons.
David Bau: that's 1 of the reasons. I wouldn't trust it, but it's not the highest reason I wouldn't trust it, I think. Like if you could get.
David Bau: you know, according to you know, this kind of predictor, a indistinguishable scientific paper, you
David Bau: would see quite a bit of progress like that would be significant. You know, you wouldn't necessarily have to like, just grab the papers, but it could do a lot in terms of like you know, writing up results proposing new results. That kind of thing.
David Bau: I guess. I guess you know.
David Bau: for for those reasons, and others like this is more kind of a theoretical argument
David Bau: in practice. There's this question of like, does loss predict benchmark performance, you know, empirically.
David Bau: and the answer to that is is mostly yes. So the right plot here
David Bau: is plotting compression. So bits per character which they get out of cross entropy loss, and against the average benchmark performance.
David Bau: And so they look at a few different domains. So knowledge and common sense.
David Bau: coding, mathematical reasoning. You know the correlations are all above point 9
David Bau: and then here is here is a project that where he was looking at scaling a sigmoid to Big Bench and Mmlu.
David Bau: And what he found is that you know you have this like reasonably good fit. And then, if you predict on held out data, these yellow dots. You also get kind of roughly what you would expect.
David Bau: so you know, not perfect, but like
David Bau: seems better than a lot of things that we use as proxies.
David Bau: And so, okay, I guess I'm
David Bau: running up on time here. But I will skip through.
David Bau: Yeah. So I guess like, that's why I have a solid line between loss and benchmark. Performance. But there are a lot of different things that go into benchmark performance. And so you know. One thing we've seen with oh, one is like, how long you run the model
David Bau: in this this
David Bau: project that we did. At epoch. We looked at. You know in general, how do you trade off between inference and training compute? And so that has a bunch of implications? Which I'll get into in a second, I guess, to to tutor on Horn a little bit. We started this, I think. A few weeks after Chat Gpt came out, I think turned out to be
David Bau: relatively prescient in terms of like how you know
David Bau: o 1, and and r, 1
David Bau: start operating. So yeah, I'll I'll kind of like skim through these, but with overtraining you train for longer and your inference is therefore cheaper at a constant loss rate. So like we started to see with llama. Now we see this lots of places because you want to deploy lots of places and only train once.
David Bau: The second is Mcts.
David Bau: And so, you know, you can search through the search tree at more nodes.
David Bau: so skip pruning period sampling. And so yeah, we have the summary here of like what we found in general. It's it's, you know, something like
David Bau: one to 2 orders of magnitude before you start
David Bau: leveling off with the with the
David Bau: exception being repeated sampling, but for various reasons I would trust that was
David Bau: So I guess my speculations as a result of that are like, you know, there are limits this approach. So we're not gonna get. Oh, one super. o 1 by just like continuing to run it.
David Bau: And you know, you also can't get like a cheap inference frontier level model.
David Bau: here we have this like this seems to be clearly true. Labs are going to keep their best models private. So we're not going to have access to best models. They're going to run them 2 times and then use that models as training data.
David Bau: And then the last thing which I think is kind of cute is
David Bau: you know in general, the total amount of training on inference and compute. If you assume this substitutability should be roughly equal. For a particular lab. And so, you know, as you get more users, you probably want to like, overtrain more and get cheaper inference. And so it's like, maybe a little bit of a quality inducer.
David Bau: Okay, so I guess, like the top level the top level takeaway here.
David Bau: you know, unclear, how we get from benchmarks to real world performance. But I feel like academia kind of has that real world performance to algorithmic progress, I think, is like, kind of the
David Bau: taxi topic in a certain community right now. And you know, I don't know if people saw the the meter study which
David Bau: is maybe a little bit less externally valid than people think, but I think still pretty interesting.
David Bau: And then the final thing is this, this you know, link between real world performance and investment, which I don't have time to get into. But I would encourage people to check out this model. That epoch released called gate, which kind of supersedes an earlier model, I did, which looks at
David Bau: as you get more
David Bau: economically valuable. AI. How does that translate into investment and compute in progress? And I think uses pretty respectable kind of economic models that real economists like some of them at least.
David Bau: And it's it's fun to play with. It's interactive for them.
David Bau: Great. Yeah. So that's that's my presentation.
David Bau: So so if you're thinking, is so like, do you? Do you feel like
David Bau: So this so is
David Bau: so you start off with the story of the 2027 essay. Yeah, so is the author of that essay basing their work on on this kind of yeah. So they say, a lot of epoch work. And it's pretty like constant with this kind of general approach. I see. Yeah.
David Bau: I you know, to be clear. I don't wanna like fully endorse this. But I do think if you're
David Bau: more than 85% confident in either direction, like, I would love to argue with you. I think I think you should be open to many possibilities here. What's the other direction?
David Bau: Well, you can either think you're gonna get
David Bau: self-improving recursive superintelligence, or you can think like.
David Bau: it's just gonna fizzle. And we're gonna get like, basically what we have. Now, can we bet on this?
David Bau: Yeah, I will send you the link. I can put it in the discord. Yeah.
David Bau: they just want to get the money. They do say you're more likely to vet you if you're prominent. So you might need to get like, you know, David, to launder his betting market
David Bau: places right? Yeah, I don't know if you can do real money. I guess you can do on like
David Bau: call sheet. Or I, yeah. But they don't have real money, do they? Yeah.
David Bau: But yeah, you can. You can win lots of fake money nice. So did you argue that? It cannot self-improve
David Bau: or it can.
David Bau: So yeah, I kind of punched on that. I think it can.
David Bau: But
David Bau: I think you have to be really careful about accounting for diminishing returns in a lot of different ways.
David Bau: and you have to account for a lot of uncertainty on those. And and if you do that rigorously, what you get is just this like incredibly wide distribution of effort.
David Bau: like, I have a quick question. Yeah.
David Bau: So like, given this context. And maybe you know, AI doesn't happen the way we're thinking, you know, like
David Bau: 85% likelihood compass whatever like, who knows what's going on? But if there is a, you know, like a 2 to 4 timeline of Agi to the people in this room. What is the point of interpretability like like? In what? In what ways does the research? Everyone here or the peers at other universities are
David Bau: the research that people are doing. Now, how does that like directly
David Bau: track to like the impact? For, like.
David Bau: you know, once we hit Agi like a safe trajectory.
David Bau: So yeah, I don't want to answer for everyone. I guess my answer is like a most people here do not buy into that. And and I also do not buy into like 2 to 4 year timelines. In general, I think. Like if you did buy into 2 to 4 year timelines, you could be saying something like you want to do research that
David Bau: will accelerate interpretability for Ais when they're doing cool.
David Bau: That's I also think this argument kind of hinges on the fact that like interpretability is going to be a silver bullet to solving like alignment. But I think interpretability is more useful, like. I think Neil had a really good recent post, where it's like interpretability is useful for just like being generally useful for safety techniques and
David Bau: at like understanding and debugging weird behaviors.
David Bau: And this is like kind of thing to work on. I think it's also necessary to get past the laws and regulations right?
David Bau: Alright, you're never gonna be allowed to use your model
David Bau: if you don't know what it does
David Bau: in order to predict my credit score.
David Bau: at least in recent number of countries. Yeah, yeah, I mean.
David Bau: some of some of the. I think the worldview here is that you only need one country.
David Bau: and then that country outcompetes the other countries.
David Bau: But I agree, that's silly.
David Bau: Yeah, I mean, I I don't know.
David Bau: Yeah, I never know which which like side to push her. But
David Bau: I think it's it's unlikely. But it's possible.
David Bau: David, what was that capability argument you were talking about? We should not work and get.
David Bau: Oh,
David Bau: I think if you, if you look at like, you know, this plot of the relative contribution of
David Bau: algorithmic progress to total progress.
David Bau: I think the regularity implies that
David Bau: there was kind of just like
David Bau: the, you know, there are some dynamics that are self reinforcing, and you don't really need to attribute
David Bau: in that way. You should contribute in ways that kind of would not get rewarded so reinforced.
David Bau: It's a little bit of a hot take like I think you can.
David Bau: You can
David Bau: have problems with that. But I think I think it should imply that, like, you know, you have this firewheel going of economic inputs and and kind of like
David Bau: experiments are limited by experiments. There are all these ways in which, like it's, it's priced in. Maybe
David Bau: you're you're arguing for the agency. So so you sort of then you're saying that. Well, you don't need to do it, because if you didn't do it, somebody else will do it.
David Bau: Yeah, it'll just happen. Yeah. So they economic forces, economic forces. And then also, like, if you didn't do it, they'd get like slightly worse at it. But they'd have more compute. And so like it wouldn't go down. That much like compute is kind of the limiting factor.
David Bau: and you know, there are lots of fields where you can have like
David Bau: counterfactual impact that you would not have if you're trying to find a better optimizer, I see.
David Bau: Thank you, David. Yeah, thanks.
David Bau: And if people want to find me and and argue. I'm happy to take either either side. You take whatever others, whatever the other side is.