David Bau: Are you? Are you recording.
177 Huntington: Yes, I just started it.
David Bau: Excellence.
177 Huntington: Start. Sure.
177 Huntington: What's the best way to join the northeastern dot zoom.us slash my slash, David Val.
177 Huntington: recording in progress.
177 Huntington: Alright!
177 Huntington: The people on the zoom! Can you hear me? Fine? Is am I audible?
David Bau: Sounds, great sounds, perfect.
177 Huntington: Okay.
177 Huntington: Alright
177 Huntington: so Hi, everyone for people who don't know me or didn't get a chance to chat before. My name is Adam and I'm an engineer on the and this team. I primarily focus on the Insight library. But we've been doing everything. Everybody's been involved in everything these days.
177 Huntington: So for the people who don't know maybe didn't attend Jaden's talk 2 weeks ago. So we have a new version of insight coming up open 5. It's pretty different from the current version of the library, and a lot of exciting new features and different like design implementation. And thoughts have gone into this version. So
177 Huntington: Jaden already presented most of it last 2 weeks ago. But what I'm gonna try to do is as I've been working on really finalizing the library and stabilizing it. There's like some things that come up. And I want to get like user feedback on like, what's the best syntax. What do you think of this? And also just like
177 Huntington: give, maybe. Oh, why is it not full screen?
177 Huntington: Joe says. Full screen. Maybe like unshare and try again.
177 Huntington: or maybe I should share my entire screen.
177 Huntington: Yeah, alright, great, cool, and sort of also, maybe just give a few example of things that like
177 Huntington: in case you think of doing it, and 7.5, it will not work
177 Huntington: alright. So the 1st I'm gonna start with is skipping. So just say, quick refresher. So this functionality allows you to completely like skip the the 4th call of of a module so that way you just like, save on time and and and compute but you still need the module to return something so that it can be passed on downstream to the rest of the model. So
177 Huntington: this is a very simple syntax. You can pass in an output, and then the module is good.
177 Huntington: Now let's say you are. Oh, David, you muted.
David Bau: Yeah. So I have a question. So wait you you don't. You didn't pass in the module name
David Bau: you passed in this output thing. So what is so that what layer? 5 output
David Bau: like? Normally, if I were at to access the output, it would be layer 5 dot output right.
177 Huntington: Yes, so here I'm calling skip on the module itself.
177 Huntington: and I'm just passing in the output that the module picture.
David Bau: Also is there, so is there a oh, I see. So, layer 5 is the thing you're skipping.
177 Huntington: Yes.
David Bau: And then the thing that you're passing as an argument is a tensor, which is the output. Okay? So I was just confused by that you're not saying skip the output you're saying, skip! And then
David Bau: yes, got it. Got it?
177 Huntington: And then skip the module.
David Bau: Got it.
177 Huntington: So so like what's different from like just replacing the output of the module by layer? 5, the module will still be will still be executed before the entire forward method of the model will be called, and everything else inside of it. But if I do skip, it will not be called. It will just like
177 Huntington: the the next module research will receive this output that I'm specifying. So think if I'm executing like a layer versus skipping a layer. So you save some flops, basically. Yes. So let's say, if I wanted to only start executing the model from like layer 5
177 Huntington: so I could like skip all the the modules before it. Here I'm just skipping layer 5, but I could be skipping all the layers before, and that was like, save I don't know, for Gpt to maybe like 40%
177 Huntington: of the of the runtime.
David Bau: It's, you know. It's funny. Just when you just showed it to me. I thought, Oh, I if this just just from the syntax of it, I thought, oh, the thing you're skipping is in the in the parentheses.
David Bau: If if you had a longer method, name like Skip to output.
David Bau: or something like that. Then then it wouldn't have that optical illusion.
David Bau: Right? Layer 5. Skip to output parentheses, some, some tensor.
177 Huntington: Okay.
David Bau: I don't know.
177 Huntington: Then it may be something to do with the example as well. Yeah, that could be fine
177 Huntington: somewhere. Comment on it or something. Yeah. Okay.
David Bau: Yeah. Layer 5 output layer 5 output equals, you know.
David Bau: tensor zeros, you know, something like that.
177 Huntington: You got it. Okay?
177 Huntington: Alright. So okay. So this was already presented by Jaden. Now, the thing I want to bring up to you is. So let's say you're doing interventions on different interventions, on different prompts within the same forward pass, so that would involve using invokers. So maybe for people less familiar with insight. Basically, you can pass multiple prompts to the same forward pass. But still like, do different interventions on them, like basically just narrow, like
177 Huntington: the content of the batch. And you isolate these interventions is pretty neat, because then you can just like parallelly, like, run your interventions on multiple prompts.
177 Huntington: Now, if you skip a module, the module is going to be skipped for all
177 Huntington: invokers, you can't like. Skip the module for one prompt, and then like not, skip it for the other prompt, if it's truly like parallel
177 Huntington: so we need. So what? What kind of syntax are we going to adopt to basically say, for all the invokers skip this module.
177 Huntington: So one way to do it is that you just have to skip in one of the invokers.
177 Huntington: and we'll just catch that, and then skip the module for all the invokers, which is what I have here. Now, you will still need to pass in
177 Huntington: an output, so that that contains that reflects the entire batch.
177 Huntington: So if you have 3 problems, then you still need to pass in an output that has a batch size of 3.
177 Huntington: So this is one way to do it the other way to do it would be
177 Huntington: you still specify it in only one of the invokers. But then you can just pass
177 Huntington: the same number of inputs as invokers that you have.
177 Huntington: and then we'll handle the concatenation under the hood. So here, if I specify the skip in the second invoker, then I can pass in the output from the layer the previous layer
177 Huntington: from the 1st invoker, and then also from the second.
177 Huntington: So here I am skipping, and I'm saying, take the same output that layer 0
177 Huntington: had, and return it as the output of layer one and I can.
177 Huntington: and and under the hood the concatenation is handled so that the whole batch spot the whole batch output is passed to the next layer.
177 Huntington: the 3rd way to do it.
177 Huntington: Sorry can you repeat the previous one again? So like, what are the 2 arguments with the skip like. So so the arguments to the skip needs to be the same number of invokers that you have.
177 Huntington: Okay.
177 Huntington: So here out, one is like the output that I want to pass
177 Huntington: as the output from the 1st prompt or 1st set of prompts in the 1st invoker and then out to is
177 Huntington: the set of pro like outputs from the second invoker.
177 Huntington: Okay, so is it like batching? Essentially.
177 Huntington: Yes, we'll ha, yes, yes, you. The batching will happen under the hood in the gate, but then I can just pass them as isolated so I could.
177 Huntington: You know it's it because these outputs are not just like one tensor. They're usually like a tuple of a tuple with a tensor and all of that. And so in the 1st case, you would have to look into.
177 Huntington: how, how, how the models output or input are handled, and then
177 Huntington: and then you do the concatenation and do everything yourself. But then, in this case it's sort of neat that, like each invoker.
177 Huntington: you pass this output independently.
177 Huntington: Sorry. I'm still a little bit confused. So what's the situation like.
David Bau: And what's what's what's the 3rd option? Maybe the confusion is a good note.
177 Huntington: Okay? So in the 3rd option you would, you would be required to specify the skip in each invoker and just pass it. The output for that invoker
177 Huntington: that makes sense. I have a an alternate, a 4th suggestion.
177 Huntington: What if you don't want to specify a skip value? Can you just have it default to the
177 Huntington: module that came before this one of like the this module's input just gets forward it as its output.
177 Huntington: Yes, but sometimes it may not be the case, like, if you have like, inside of an Mlp layer like
177 Huntington: or inside of like an attention, the output of the C tension is not the input to the C project.
177 Huntington: And so you'd like, like, it depends also on the models. But there is like intermediate things. Yes.
177 Huntington: I guess I just feel like you wouldn't skip only an outreach if you're
177 Huntington: not keeping it in touch. I don't know.
177 Huntington: Well, we could have it by default. Do that. And then, if you pass in a value, then it takes in the value.
177 Huntington: But essentially here, what I'm trying to focus on is that we'll basically, you'll be required to specify skip in each invoker. So it's it will add a little bit more code. But I think it's also like. It makes it clear that you're like skipping this module
177 Huntington: for all the invokers.
177 Huntington: And you can also give, just like the output required for that invoker.
177 Huntington: And and that will handle basically concatenation. So I feel like it's
177 Huntington: it's a better version of version 2 once, if, instead of having, because the thing confused me with the Pre, the one where you did the skip in the second invokers indent. Yeah. So at the bottom, there it's in context of the second invoke. How come it has to be there as opposed to outside of the con, like in the main, in the main context.
177 Huntington: That way, it's kind of clear you do this invoke to get these intermediate things, and then you're kind of applying it to all the previous invokes. Yeah, that makes more intuitive, since I was kind of getting confused about like how? Why, we need that layer skip out one out to within the invoke
177 Huntington: of the second one.
177 Huntington: So the reason for that is because if I put it somewhere else, it will be out of order. So so one of the biggest updates in open 5 is now like exactly the
177 Huntington: execution order of the interventions strictly follows
177 Huntington: the execution order of the model. And so, if I put it somewhere else like if I put skip layer one on top of, like
177 Huntington: the invokers where I'm specifying layer 0. So it happens afterwards, that would still be the case, if I have some intervention on layer 2, 3, and 4 inside of the invokers.
177 Huntington: But couldn't you throw an error for that like this would? Yeah, exactly. It will throw an error. It will not work, basically sure. But
177 Huntington: see? So this, oh, I see. Do you have it in? Could you have? Well, in this case it makes sense for it to be outside. So maybe having that as an option that works but would throw an error for out of order. And then, if you did need to do something where there was further stuff happening dependent on it, then you could put it inside the in hope, using your way of having it in both. So that's more readable. I'm not sure it seems complicated either way.
David Bau: Which one which one do you think? Things which which one do people think is the best out of these 3.
177 Huntington: Think maybe that's the 3rd one.
177 Huntington: the 3rd one, the 3rd one seems the most clear. Yeah.
David Bau: Yeah, I think clarity like overrides everything else when it comes to like code. Apis, you know.
David Bau: I think clarity.
David Bau: we don't want like you want to avoid a situation where you have write only code where it makes sense to the person who wrote it. But it doesn't make any sense to the people who are reading it.
177 Huntington: I agree
177 Huntington: the biggest user error would be they, for they put a skip in one. They don't put a skip in the second, and it doesn't skip because you can't both skip and not skip in this batch context. And that will just be a tiny performance hit that they won't notice
David Bau: It doesn't notice, or or you can. You can throw a warning or something so that they'll notice
David Bau: like, if if it's not skipping, you can say you should just set the output because you're not skipping.
177 Huntington: Okay, okay, so that's for skipping with multiple invokers.
177 Huntington: Okay, next, I'm not sure if Jaden spoke about this feature in the presentation which is gonna give like a quick
177 Huntington: overview.
177 Huntington: So like a lot of users sometimes are just interested in like caching activations of like all the modules. So Jitter added this like new feature, where you can just call cache on the tracer, and it's going to grab the input and output of every module in in the model. So whenever a module gets executed, we grab the input output and then and then cache it in this dictionary, and you can just access it this way. So cache model dot layer 0 dot output.
177 Huntington: You can also specify to cache specific modules. So here I am, looping over all the layers. So like I'm not caching the Lm head, the embedding positional embedding all that. I'm just grabbing the layer and and not going deep into like the layers like the cage and tension. I'm just grabbing
177 Huntington: each layer's hidden state, and and Cashina can access it
177 Huntington: after after the context. And I need to call dot save on yeah. And I need to call Dot save on this like cache object
177 Huntington: any questions about the overview? I really like the cache all activations thing, if like, you want to get everything I think for, like the cache activations of specific modules, I'm kind of just interested to know more under the hood like, is it
177 Huntington: any different than just like saying cache equal to layer for layer, and gpt, dot Gpt. 2 dot layers dot save
177 Huntington: like
177 Huntington: like, doesn't dot save? Isn't it meant for caching or like? What? What's the difference in this? So like dot, save usually is usually for like saving the activations for the you know, for whatever layers we care about broadly, is to save anything that you want to reuse outside of the context. So even if I have a list with like
177 Huntington: a string I would want, if I want to use it outside of the context. I would not need to call dot, save on it. True, true. So like, basically, if we were to cache it like before, this whole dot cache function was there like we would have to just say, Dot, save like in some random activation. Right? Yes. Yeah. Yeah. So like, let's say, for like these layers.
177 Huntington: oh, oh, okay, sorry. Yeah. I was gonna mention that like we probably when previously needed to do like for each of these layers, just do dot save. So it's so Dot Cache is just trying to. It's it's essentially the it's essentially the. But then it's like already format for you in in
177 Huntington: in a dictionary where you have key is the path to that activation. The output is this like entry object which you can call dot output and then inputs on, okay, does
177 Huntington: dot. So in this case, if, like cash is kind of also meant to be saving so like, what's the difference between dot, cache and dot save.
177 Huntington: or is dot save necessary here? Yeah, yes, because it's created inside of the context. So to be able to use cache, the cache object outside of the context, you need dot save on it when you need the cache inside the context, but not outside of it. Like, when would you want to not save it? I guess.
177 Huntington: Yeah, that's a good point.
177 Huntington: That's a good point. But I think it will have to be that way, because the way the dotsave right now is implemented. So there's no more intervention graph, right? So we're just using like plain python objects. And so the way dot save is implemented is that
177 Huntington: we go into the object class of python, and we add dot, save to it.
177 Huntington: So every class now has a dot save to it. So
177 Huntington: maybe we could. Yeah, we could exclude cache or something like that. But it's a good point that, like you would always need that. So maybe in the dot cache, like in the under the hood will always have dot, save for all that's true. You just have tracer Dot, save.
177 Huntington: I'm pretty sure they'll say 5
177 Huntington: can you specify if you don't care about saving the input? Of the yeah. So right now, no. But I was gonna get to that later. But yes, that's a good point.
177 Huntington: because it will be very redundant, especially when you're grabbing just the layers. Right? The output is the input of the next layer. Right? Can you specify if you want, like it to be on saved on Cp, like, if you want to move everything to the Cp. Yes, you can. You can pass in the device to the cache function. You can pass in the D type and the device, and it will move everything you can also like. There's a keyword for detach.
177 Huntington: I detach the oh, yeah, what about to disk? Can you save it and then load it in like serialized? No, no, it's not doing that at the moment.
177 Huntington: Any questions about anything, insight in general.
David Bau: Yeah. Cash cash question, cash question. Hold on, how do I? So
David Bau: So the left and the right are different from each other. So, in other words, the cache is not recursive by default. If you want the recursive cache, you have to do the thing on the right.
177 Huntington: No, if you want the recursive cache, you do the thing on the left. If you want to specify which modules
177 Huntington: to cash, then you do the thing on the right.
David Bau: Oh, so the thing on the left caches, all modules.
177 Huntington: Yes, all activations of all modules and all activations means all inputs and all outputs.
177 Huntington: Yes.
David Bau: That's a lot.
David Bau: Go ahead, is it? Is is it typical for people to want just the inputs or just the outputs.
177 Huntington: Possibly yes.
David Bau: I wonder if one of those should be the default.
David Bau: and then and then there's an option to say you want all like I'm not really sure, like I kind of feel like this is a simple little choice
David Bau: to do both of them that will like it'll like double the memory requirements of the default.
David Bau: Right? So like, like, what is like when you use transformer lens. And you say, you know, get everything, are they? Are they caching the inputs or the outputs.
177 Huntington: I think it's outputs because they make hook points
177 Huntington: so that they don't have to worry about inputs.
177 Huntington: I think, yeah, I think the default hook is the output trust one of yours.
David Bau: Yeah. So I wonder if I wonder if we should?
David Bau: I wonder if we should make that
David Bau: the default, or something like that?
David Bau: And then
David Bau: and then there's some flag or something that you can say that cash your inputs, or whatever.
177 Huntington: Yup. No, I agree totally agree. So let's maybe talk about the syntax, and then maybe 2 slides from now.
David Bau: Okay. Okay. Cool.
177 Huntington: Okay.
177 Huntington: Okay. Now, again, like cash in with multiple invokers. So this is not like a very difficult technical
177 Huntington: implementation. But what what makes more sense in this context as in terms of syntax, to provide to the user.
177 Huntington: So if you have multiple invokers just like one, you want to
177 Huntington: do specific interventions on specific set of inputs. Maybe you'd only want to cache one invoker, or maybe both.
177 Huntington: So in terms of syntax, do we want the cache to be inside the invoker which gives you flexibility? Or do we want the cache to be outside and just to be in the tracer in the tracer.
177 Huntington: Okay, I'm confused. Why is it not with tracer dot invoke as invoker and then invoker dot cache so that you can specify specifically what
177 Huntington: context that you want to cache in. So if you want to save everything, then you would do tracer dot cache dot, save if you wanted to only do one of the invoker context, then you just do. You would have it. Have the context manager have a name, and then you apply it to that.
177 Huntington: Which context manager, so there's nothing there is the invoker. So when you do with Tracer Dot invoke, you'll do that with Tracer Dot invoke as invoker, and then you can do. Invoker dot cache dot save or dot cache.
177 Huntington: Yeah, th-that's possible.
177 Huntington: So then, and the pattern is, whatever con the the context manager that you cache on
177 Huntington: is going to. It's going to apply to everything for that specific context. And you can. The same thing that we were just talking about of how the pattern of of saving can apply specifically to each thing. I guess.
David Bau: That's cool.
177 Huntington: And so essentially, what you're saying is, we should have both one and 2 implemented and
177 Huntington: in the case where you want to save or cache a specific
177 Huntington: this activations of one invoke. Then you should call cache on the invoke invoker.
177 Huntington: But how would you handle? For, like I I agree with that, like having the named modules as like the named parts, but like in the case for like, let's say, the second one like, would you, then error out if they don't have the names for those invoke functions like, how would you access that cache to actually know that you're getting the activations from one invoke versus the other.
177 Huntington: So like, let's say, we did do cache tracer dot cachesafe in the second option. Yeah, how do I know what activations is what like or which invoke you mean? Yes? Well, you will. Just you will get the entire like batch input or output, and then you just need to index for
177 Huntington: right.
David Bau: It's like the zeroth one, and the 1st one.
177 Huntington: Exactly it will. It will be like, just like what the model sees as an input and output which is concatenated. And then you will just need to index.
177 Huntington: okay, with with the with the batch index that I mentioned.
177 Huntington: Okay, any other like observations, do we have like a consensus that maybe something.
David Bau: Sounds like that.
177 Huntington: Yes. Yeah.
David Bau: Yeah.
177 Huntington: Okay, so more more thoughts on like, what features we can add to this
177 Huntington: caching functionality. So the one we we discussed earlier is like, what if I just want the input or only the the output. So initially, I was thinking of like a keyword, either input, only or output only, but I like the idea of just having output as the default.
177 Huntington: And then if you want all or the input, then we'll have keyword arguments.
David Bau: Yeah, yeah, don't. Don't, don't, don't have any keywords that are, say, only
David Bau: right? You you can say, you know input or include input.
David Bau: is true or false, include input it or include output. True or false, right only is always like this weird thing like, what does that mean? Like you? You'll never add another keyword if you guys say, only
David Bau: yeah. Anyway. Yeah. Only only is just generally not not a good thing to have in people arguments.
177 Huntington: Okay.
David Bau: That makes sense. Yeah.
177 Huntington: So does it make sense to have, like 2 keyword arguments, one that, like includes
177 Huntington: the input instead of the output, and one that has both.
David Bau: No, no, no.
David Bau: One. Keyword argument that it says include input.
David Bau: And then one keyword argument that says include output, and then both of them can be set to true or false. So I I think
David Bau: right. So then you can say, include input equals. True include output equals false or something like that. Don't! Don't you think.
177 Huntington: And by default the include output is, and by default the include output is true. Yeah.
David Bau: Yes, and include input is false.
177 Huntington: Okay.
177 Huntington: I thought I had, which is similar, is having something of the form of a cache pattern, and it would be something like an enum that specifies how the caching will take place. So there would be like the input one. Only cache inputs output. Only cache outputs input output. I guess the the thought there is in case there's more patterns that potentially would be wanted like, maybe there's 1 where only, like the the like.
177 Huntington: like, you don't want to cache the intermediate stuff only input output of like the attention or the the like. Just each layer instead of like the inner of attention, and Mlp, etc.
177 Huntington: yeah. So you could.
David Bau: You could have include you, you could you could have an include equals, and it'd be a string.
177 Huntington: Yeah.
David Bau: Or something like that. So then it's like, Oh, include equals input or include in equals input comma output, or something like that.
177 Huntington: Good or like, yeah, yeah, basically.
177 Huntington: okay.
David Bau: I don't know, and maybe that's ugly. I don't know what people think. I'm not coding. What am I saying?
David Bau: You guys should decide.
177 Huntington: I think I think there's a non ugly way of doing that.
177 Huntington: like the idea of it is essentially just. You specify in the in the method.
177 Huntington: What? How you want it stored, I mean, if you're doing that fancy trickery previously shown, then it gets ugly quickly, so I guess I see it as like in most cases. If you have the good. If you have a good default, then you don't need to see it, and if you do want to be a bit more specific. Then you kind of have to crack up like the the like strings, or like the the specified patterns.
177 Huntington: yeah.
177 Huntington: Okay, any.
177 Huntington: What about the thoughts?
177 Huntington: How do you load them in the caches later on?
177 Huntington: Is that in a future slide? How do you like load a cached tracer like later
177 Huntington: is the point of it, so that you could like use it down the line like maybe.
David Bau: What is it?
David Bau: Oh, yeah, what I that's all I'm saying. I'm asking the same question, what's this? Impacts of dereferencing the cache.
177 Huntington: You just you just you just reference it like cash. And then
177 Huntington: like, it's it's a dictionary. So.
David Bau: You have to use diction. You have to use dictionary syntax.
David Bau: You can't just say cache dot model dot layer 0.
177 Huntington: That that would be possible. That would be possible.
David Bau: That might be cleaner.
177 Huntington: Yeah, my thoughts, really, this caching is very fundamental to like a sort of future technology with insight which is the ability to
177 Huntington: save and experiment in the format that is like standardized. And if you have caching, then sort of the next step is like, Okay, you can cache it to CPU, you cache it to whatever device you want, you can cache it to disk, and then there's a sort of serialized format which is like the dictionary that you have which could be written to a Json object somewhere,
177 Huntington: or or and and you have the deserialize, which is just the tracer itself. So you could, you could load it into a trace context and basically start from something you already have.
177 Huntington: And I feel like that's like the general. That's the generalization that allows for a lot of really cool
177 Huntington: kind of
177 Huntington: kind of corollaries to to the the tool. Like being able to share your experiment, and have someone loaded in directly and continue from there. Sort of thing or I mean on the end of side, like
177 Huntington: you could have experiments that you like. You have parts of your experiments cached and then use later on, sort of thing. So I guess my thought is, it's a very cool thing, and I imagine having a serialized, deserialized format of it would be very valuable to go alongside it.
177 Huntington: Yeah, I think it's
177 Huntington: maybe a little bit. It's it's an anticipating, maybe, like very future features with within this right now, like, I think the cache is basically just like convenience method, just to be able to like instead of having to define, say, like, input output dot, save for multiple modules. Basically, it's just like tracer dot cache. And then the user can save it to disk, or however they want, in whatever format they want.
David Bau: So research research students, would you guys use cash.
177 Huntington: Oh, yeah, definitely, right. Now, I'm doing like a hacky. Other alternative like, instead of accessing it as a dictionary, basically like, call dot model dot layer 0.
177 Huntington: I'm not sure if it would make a difference, but I didn't think like
177 Huntington: it. It might make more sense to do, and not the dictionary, but also in some cases it would be nice to have, like the dictionary format if I was like saving it somewhere.
David Bau: It might be. It might be nice if it was accessible both ways. At the same time.
177 Huntington: Okay, I think this that's definitely possible and easy to do.
177 Huntington: Yep, no, that's great feedback.
177 Huntington: Okay?
177 Huntington: And see you here.
177 Huntington: Okay, this sort of other
177 Huntington: thought about caching is so we have this feature called rename, where you can basically rename the path of of like a module path in the model. So I've been using layer 0 just for convenience to make like
177 Huntington: to to make to make these code snippets short. But technically, let's say, using Gpt. 2. Then I have to call dot transformer dot age and then index 0. But then I could rename all of that just to layer underscore 0.
177 Huntington: So.
David Bau: Is there? Is there?
David Bau: So this this rename has existed before right.
177 Huntington: Yes, in the last version.
David Bau: Is, is there any way? Because the common thing is for like H. Is an array right?
David Bau: Is there? Is there? Is there a way for me.
David Bau: So let's say H is an array, and then it has Mlps and HH. Right?
David Bau: So I guess I have 2 questions. One is, is there a way for me
David Bau: to rename layers to h without the thing? So that, like like layers, is an array.
177 Huntington: So what do you mean? It means like standardizing. You take an arbitrary model, and then you can like automatically detect where the layers are.
David Bau: No, no, no, no, no, no, that's not what I'm asking. I'm asking a simpler question
David Bau: you're anticipating. I'm I'm asking. I'm asking
David Bau: on the left hand side you have a string
David Bau: right, which is not an array. Right? That's just that's gonna be a Singleton.
177 Huntington: Yep.
David Bau: Right. Is there a way of putting a thing on the left, which is, gonna be an array.
177 Huntington: Yeah, it's just the name of the module.
177 Huntington: So yes, so is you can straight away cheers and then transformer dot. H, yeah.
David Bau: Does it?
177 Huntington: And then index into layers.
177 Huntington: Yes.
David Bau: Cause like transform, like transformer dot H is not a module. It's actually a module list or something right.
177 Huntington: But but I think it implements from the module right? It has an output. Of.
177 Huntington: Yes, but yes, it is. It has, you know. I I see what you mean, but it has a path. So you can definitely do that.
177 Huntington: And then and then, basically, the name layers will be for the module list, and then you can just index it just like you do.
David Bau: Okay, then I have another question. So if if it's more like dot transformer, dot h bracket, 0 dot, Mlp.
David Bau: Right could I say rename equals Mlps. Colon.
David Bau: something that would make give me an array of all the Mlps.
177 Huntington: No, not at the moment. No.
David Bau: I see.
177 Huntington: You could, you could. You could with like
177 Huntington: like with an inline for loop you can like with the list comprehension you could. You could do that.
David Bau: Oh, but like Rename doesn't allow us to do this.
177 Huntington: Yes.
David Bau: Do you think that we have a need for that.
177 Huntington: I I think so. I think people people I mean it would be. I'd be surprised if somebody wants to rename specifically only the Mlp. Of like layer, 3 in. In the model.
177 Huntington: but
177 Huntington: same thing here with caching where I'm specifying the layers manually, or if I wanted just to grab all the Mlp activations. I would need to do something this way. I feel like the user can just do a list comprehension.
David Bau: Oh, no, but that! But then they would have to have this underscore 0 underscore one underscore 2, and they wouldn't be able to do convenient list indexing later.
David Bau: You know what I mean.
177 Huntington: Like in here in the list. Comprehension you would have like
177 Huntington: you, you would have this like layer underscore 0. You would just have like a.
David Bau: I know I know, I know. But but do you do you understand what I mean? Like later on inside your code? If you want to have a for loop over, you know, a range of 5 layers.
David Bau: Then you can't say, like layer bracket.
David Bau: 3 layer bracket, 4 layer bracket, 5 anymore. You have to. Let's say, layer underscore
David Bau: 3, which means that now you have to say, layer underscore percent, d percent 3. Whatever you know, it's just like this ugly
David Bau: think like like you lose the array
David Bau: if you use a list comprehension, anyway, just.
177 Huntington: Well, are you saying like here, when you're doing the least comprehension over the Mlp. Specifically like you? I just want to grab all the Mlps. Is that what you're saying or or like?
177 Huntington: If I were to rename age, if I were to rename age, or which which one are you.
David Bau: If if so, if you rename H, then you're fine, that you still have the array.
David Bau: But if if you, if you have to make an array out of things that are inside. H.
David Bau: Then there's no way of making them into an array.
177 Huntington: No, it still it still remains the array like, if I have to access other things, I will still need to call dot H, and then bracket 0. It only renames the path to that specific module.
David Bau: I know. So so when I say so I can't. I can't have a name called Mlp.
David Bau: Bracket. 5.
David Bau: Expand out to transformer dot H. Bracket, 5 dot, Mlp.
177 Huntington: Yes, you can. Yeah.
David Bau: Right, which but but so if I so I I could only say, Mlp. Underscore 5
David Bau: expands out to transformer dot 8 bracket, 5 dot mlp, but then but then, you see, I've lost my functionality, because now now I if I want to do that, then the Rename is no longer just a pure win. I now now I have to like, do string, interpolate interpolation every time I want to, you know. Go after the 5th Mlp.
David Bau: It just seems like there's this loss.
David Bau: And
David Bau: and and so it which doesn't matter for the rename that we put in manually. But you know, so, anticipating our standardized Renames, where somebody is gonna go through and say, Hey, you know what? It doesn't matter how ugly you've made the thing. But I'm gonna clean up the Renames. If we don't have the ability to do that, then it's very easy to make an ugly
David Bau: You know hierarchy which you can't. You can't unroll because they have an extra dot somewhere.
David Bau: Does that make sense.
177 Huntington: yeah.
David Bau: Right, anyway. So I'm just yeah.
David Bau: It seems like something that like.
David Bau: you know, I I don't know. Maybe it's not. Maybe it's not not a real issue. Like, I think that this this, like this feature.
David Bau: Okay, so 1st of all, are any students using this feature? Right now, did we.
177 Huntington: Can you read it?
David Bau: Yeah.
177 Huntington: I I didn't know it existed.
177 Huntington: Yeah, no.
David Bau: Everybody is using it right?
177 Huntington: We have other users there. But yeah, I guess nobody here.
David Bau: Yeah, I wonder if anybody's using it. But but but it's like, it's this feature is really powerful. Once you have Standard Renames. But we have no idea what the customer for Standard Renames is because nobody has attempted to do the standard rename. But my guess is that.
David Bau: yeah, you you may have a need for more powerful renaming than what you've got, because
David Bau: when you want to really do the standard renaming well, then, you'll you'll be forced into situations like what I'm describing.
177 Huntington: Hmm.
David Bau: Does that make sense.
177 Huntington: Yep.
David Bau: Right, anyway. So that's that's my main feedback. Is that maybe the right way to getting feedback on this is not
David Bau: to ask people who aren't using it, but.
177 Huntington: Hmm.
David Bau: To think through this, and to renaming.
177 Huntington: Make, sense.
David Bau: So.
177 Huntington: Yeah, so- so okay, it's good. So we're getting like meta feedback on things. But here, basically.
177 Huntington: I wanted to ask users if it would be like. If you rename a module then and you, if you cache activations, including that of the module rename, would you want the name of the activation inside the cache to be that of the like the new alias, or or to be the original path
177 Huntington: to the module, like
177 Huntington: probably the renamed version. Just so that like, so the way I imagine this could be useful of is, let's say, I want this one code to be run in like multiple models. But multiple models have different ways of like referring to layers. Then I could rename that at the top and then at the code of intervention. Then I could just keep that the same that we need
177 Huntington: yeah, I think that will be
David Bau: That's cool.
177 Huntington: Hello.
177 Huntington: okay. So now move on to a few things that like related to new features of insight.
177 Huntington: dot.
177 Huntington: You shouldn't like basically attempt or that are like will not yield the results that you're expecting.
177 Huntington: So quick overview. I have a side for this. But the dot source functionalities basically allows you to intervene of like inner operations of a module.
177 Huntington: So I don't know. Here, for example, the attention it's actually the C attention. This is not accurate. But no, actually, it is nature. Okay, the attention module in layer 0 has a split function. It's torstensor split function. And I wanted to like rewrite the output of that, so I could like
177 Huntington: call the dot source of detention, and then call that split and do an output.
177 Huntington: But let's say I wanted to rewrite an operation inside of that split function. I don't know some.
177 Huntington: either a transpose. I don't know what split is calling contiguous or something, and I wanted to rewrite the output or input of that inner inner function.
177 Huntington: But I don't know what operations are inside of the split.
177 Huntington: So for for let's say, like the source of modules. I can just call print
177 Huntington: layer 0 dot attention dot source outside of the trace and get an idea of the operations inside of it, and then call them in my trace.
177 Huntington: But for inner inner operations I cannot do that. I can only get access to them within a trace.
177 Huntington: And that's because it needs to be executed. 1st for me to be able to gather.
177 Huntington: The the definition of the function
177 Huntington: is, is that clear.
177 Huntington: You can't. Basically, you can't call dot source twice outside of a trace. Yes, it's because he needs.
David Bau: If you put, if you, if you put both of them inside the trace, would it work.
177 Huntington: Yes.
177 Huntington: is it just simply because the forward call on attention is needed to do the recursive split
177 Huntington: need to do okay.
David Bau: Would it make sense just to tell people that everything has to be inside a trace.
177 Huntington: Clarity, basically.
David Bau: Would you say you think it would be less clear.
177 Huntington: No, I said, I said, like for for clarity, right like, for the sake.
David Bau: For clarity, so that there's not 2 different ways that we're teaching. People do it this way, except in that case do it that way. Just tell everybody just always do it this way.
177 Huntington: Yeah, I was. I was thinking about the answer of like.
177 Huntington: here's my let's say, if you were to go that way, he's my counter arguments number one is, it's like really rare to do like a recursive, you know, like source, because then you will get into like other like operations. So it's it's fairly rare. Number 2 is. I would want, for example, to
177 Huntington: access the source of a module without executing the model.
177 Huntington: and that's why it would still be helpful to have that functionality outside of the trace.
David Bau: So
David Bau: when when Jaden presented it right, we had this whole issue of like, when you actually look at the real source, then for a complicated source.
David Bau: like often, like 75% of the code is dead code.
David Bau: But but like you won't know what's dead unless you run it.
David Bau: It seemed like the one of the most valuable reasons to do this source thing inside
David Bau: the Api, instead of just like looking up the code yourself. Is that
David Bau: like we could tell you what's good? What's dead?
177 Huntington: I see. Okay.
David Bau: But then, but then, if you, if you don't have it inside of trace, we can't do that right.
177 Huntington: Yes, that's correct.
David Bau: It. It seems cleaner to me just to say you just always do it inside. And one of the reasons you do it inside is because we need to tell you what's what's live and what's dead. And
David Bau: that's just how it works.
177 Huntington: so you just run it on a dummy, prompt or something.
David Bau: Yes.
177 Huntington: Yeah, I feel like I would believe that
177 Huntington: I know I would. I would buy that as an explanation like if you just told me. I can only do it.
177 Huntington: You'll keep trying to do it outside of it like I wouldn't try to. I don't know. I was like, Oh, I just
177 Huntington: do it, you know, I don't know. Is it?
177 Huntington: Okay?
177 Huntington: Moving on?
177 Huntington: Okay, okay. This one, I guess. Pretty self explanatory. So just calling things out of order strictly forbidden and prohibited in inside. Open 5. Technically, even inside open 4. But there was like.
177 Huntington: it's just your code breaks. There was like no warning, no proper like error if it happens. And we had some like
177 Huntington: like.
David Bau: Yeah.
177 Huntington: Like heuristics going on that the users are not fully aware of. And so.
177 Huntington: even if you some some code, even if you write it out of order, it would still execute properly. But here it's like pretty strictly
177 Huntington: private.
177 Huntington: So that also applies to like inner modules of modules, if you like. Call the output on layer 0. But then you call the output of the attention, and this will throw an error
177 Huntington: with a clear message.
177 Huntington: So so how does this interact with like caching? If I want to save attention outputs and Mlp outputs, for example, like, maybe I have like, do I have what I have to pass
177 Huntington: all of the modules into a single cache, or could I call like dot cache for Mlps and dot cache for attention.
177 Huntington: Yeah, you would need to pass it to the single to the same cache. So there's not a way to have 2 caches. No.
177 Huntington: but so that the cache handles the ordering for you.
177 Huntington: We'll put the cash handle the order in for you
177 Huntington: like, if I just give you
177 Huntington: a hundred modules, but they're not in order. Pass as a list to the cache. Oh, yes, absolutely. Yes. Okay, yeah. The the order does not matter, so you can. I don't know, for the sake of it. Just give me, give them to me in reverse order. And it will they will not throw an out of order.
177 Huntington: Derek is David.
David Bau: Oh, no, that that's all I I was. Gonna I was just gonna ask, you know.
David Bau: Eric was asking. You can't have 2 separate caches. Eric, do you? Do you want to? Would you want 2 separate caches.
177 Huntington: Not necessarily like I can always split it later, I guess.
177 Huntington: Yeah.
177 Huntington: like I get me so I could see the case where like, maybe you want to save some things to
177 Huntington: like the CPU. But then keep some like cached.
177 Huntington: And so like for that, maybe you would need 2 caches or something I don't know.
David Bau: Cool, cool, cool. That's a request. That's a request.
177 Huntington: Okay.
177 Huntington: And so would you like, for example.
177 Huntington: is it like difficult? For example, if you like.
177 Huntington: have just 1, 1, 1 cash call? But then you're able to like, specify these like inner dictionaries like
177 Huntington: for Mlp. I want these modules, and then for attention, I want these modules, and it will be in one line
177 Huntington: instead of instead of having multiple cache objects you would just have like this cache. You can index Mlp, and it will have this like dictionary of like all the Mlp. Paths
177 Huntington: with with their activations. I guess how I understood the cache was. You pass some list of modules to it and then it creates a dictionary with the module name as the key.
177 Huntington: That you can either choose to like.
177 Huntington: You can like, choose the device that it gets sent to. Can you? Can you send it to a file?
177 Huntington: The cash? No, you you can. You need to handle that alright outside of the trace. Gotcha.
177 Huntington: That's what I was suggesting, because, because, like these things have are like, not just sensors, there'll be like couples of multiple things. And then maybe even like
177 Huntington: dictionary or something.
177 Huntington: So so I mean, maybe it doesn't make sense to want 2 caches and like, just
177 Huntington: just deal with it process it outside of the trees. Yeah.
177 Huntington: I don't know. I I could see people
177 Huntington: like when they 1st see this cash thing be like, oh, I'll have a cash for my mops, and I'll have a cache for my attentions.
David Bau: Yeah, it's cool. It's it looks like a constructor for something right?
177 Huntington: But but then, like when they try it and get in there like, Oh, I have to stick these into the same one.
177 Huntington: Maybe that's fine.
177 Huntington: I don't know. It's something to consider, I guess, whether it's worth a union of caches. It seems it seems like to guarantee like
177 Huntington: not things not being out of order, it might create a lot of headaches to allow unlimited caching or something.
177 Huntington: But
177 Huntington: hmm! I don't know if that causes an issue. So like right now, for example, let's say I am accessing
177 Huntington: later to be output. And then I call Cache.
177 Huntington: Okay, only only everything below that point will be cached
177 Huntington: so it doesn't. You don't have to call it at the beginning. Let's say you only wanted to cache things after day or 4.
David Bau: Oh, so if you oh, so if you call cash too late.
David Bau: then then you don't get the early steps.
177 Huntington: Yes. Oh, that seems like a problem.
David Bau: Yeah. Do you wanna have that? You wanna have that trigger warning or something?
177 Huntington: I guess. Oh, that's very interesting. So like you do some intervention, and like you, you have some cash, but your cash has
177 Huntington: many modules in it that maybe some are before your intervention, so it wouldn't get any of those. So if if the module is like, I'm like in its inner module. This one I called cache.
177 Huntington: like, for example, its input will only have none
177 Huntington: as its value. But then the output will be populated, because then I will be like
177 Huntington: indenting out of the models. Computation. But then let's say for layers. If I'm like starting to cache after layer, 3, then layer 1, 2, 3, and 0 will not have
177 Huntington: anything. It will not even be present in the cache. But then there are 4, 5, 6, and everything else will be. So you basically have to stick cache at the top of your trace. Here's a solution to that. Instead of doing the dot cache, you just have it as a keyword argument to anything. Trace, invoker, and then the thing you pass to cache can be a variety of different things. If you just put like true, it will go to the default pattern if you pass in like a
177 Huntington: I mean some sort of like dictionary of like or whatever of things you want to cache, then it will accept that if you want to put in a string that dictates the caching pattern that would work too. So that way. It's clear and obvious. It's right at the beginning of it, and if there's only one way of doing it, then you'll run into the error at all.
177 Huntington: Hmm!
177 Huntington: So the.
David Bau: This Api. This Api is okay. This Api is okay. It's just it. Also, I mean, I think that would be nice, too.
David Bau: But it. But this is, it is pretty clear to read. So this is like.
David Bau: this is readable. Code, is. It? Might be hard to write because you didn't know. It has to happen at the start.
177 Huntington: So.
David Bau: How readable it is.
177 Huntington: So so here's the reason behind this sort of implementation is that let's say, I only wanted to catch things after layer 4,
177 Huntington: okay, and we, instead of like saying, specifying all the modules after layer 4, then I can just like stick it there after doing some intervention on layer 3, and then I will only get things below it, and I think it makes sense.
David Bau: Yeah, is that is that what is that? Is that like a pattern that you really want to do, though.
177 Huntington: What? What do you mean?
David Bau: I don't know. Like you're you're you're you're filling with layer 3,
David Bau: and then you only want to do something with layer 4 beyond or something like that. It seems like it seems a little contrived. It doesn't. I'm not sure it's actually something that a researcher like wants to do all the time.
177 Huntington: Like. Let's say I patch into Layer 3,
177 Huntington: and then I only want to get like the activations of everything below it.
177 Huntington: And I think that may be like a neat way to like cache things below it
177 Huntington: below that point of the competition.
177 Huntington: So I'll do whatever I want in the earlier layers, and only after it I call cache.
177 Huntington: I'll save, and then it gives me everything below it.
177 Huntington: Here's another.
David Bau: Maybe.
177 Huntington: So if if in your cache you pass all of the layers.
177 Huntington: can you do anything after cache? Because just delete order, yeah.
177 Huntington: Like, if I try and do any interventions, stuff will be out of order, because I would have passed like layer 30. No, no, it's fine. It will not be an issue.
177 Huntington: Yeah, you can. So you can. You can define your cache with specific modules and then do the interventions later on on those modules. And it will be fine. So so I guess I'm struggling to understand why I can't do interventions first, st and then cache later.
177 Huntington: and hope that my cache just grab stuff from the beginning, because it will be too too late, like those modules, will have already been executed for the cache to like cache, to like catch it, because everything inside the tracer is.
David Bau: Right, right, so.
177 Huntington: Executed before you reach the cache.
177 Huntington: compiling my with context, it's still executing by line.
177 Huntington: You see, it is exactly executing by line. So that's why like oh, so sorry.
177 Huntington: Oh, I guess. Okay, let me let me clarify this. Then basically.
177 Huntington: your lines in your tracer are executed at the same time as the modules are executed.
177 Huntington: So if I have a line, would like to an intervention layer one that line only gets executed at the same time that the module is like has reached layer one.
177 Huntington: So it doesn't. So it doesn't like
177 Huntington: Go through your whole with block and like check. Whether I've got late modules and early modules ex exactly. If that happens, it will happen later. So basically, if I define intervention, layer 2 and then and then layer one. It will execute layer, 2 at the same time as the layer, 2 on the model, but then the next line is a layer, one intervention, and it will be like, wait. I already visited Layer 2
177 Huntington: throne error.
177 Huntington: Gotcha. Okay? So so maybe if this is the thing, then just say like.
177 Huntington: you need to stick your cash first, st if you
177 Huntington: like, I guess, like you should always stick your cash first, st even if you're doing interventions.
177 Huntington: Yeah, if you didn't refresh it. Yes. But then, if you like, want to skip some
177 Huntington: some activations to not cache them. But you just happened earlier. Don't pass it to your cache.
177 Huntington: Yeah, you could do that if that's a little bit more manual, or you can just like stick it below whatever layer you want. Basically. But yes.
177 Huntington: like, I don't necessarily need to do some intervention. I guess this. The the thing that seems silly is like you can specify modules to the cache. That won't get populated if you call cache late.
177 Huntington: Yes, but if you call cash early, then it will always get populated.
177 Huntington: Yes, which is like, yeah, I don't know.
177 Huntington: I I guess in that case it should throw a warning. If, like, there's a module specified, and then it's not visited. It should definitely throw a warning. Yeah, something like that. Okay, cool. I think. I understand. Now.
177 Huntington: Would you think of again? You were a little bit reluctant about like sticking the cash like below something like, would you like not use it yourself?
177 Huntington: I guess if I was only trying to cache like it? One individual thing. Then I would probably just call it like dot output dot save the dot cache for me. Seems useful to like
177 Huntington: just like grab everything that I want and not have to worry about whether it's gonna get
177 Huntington: stored or not.
177 Huntington: So that's why I would stick it at the top, probably.
177 Huntington: But I maybe people that like
177 Huntington: wants to grab activations after some intervention.
177 Huntington: but like want everything afterwards, I could see that being a use case good.
177 Huntington: Any other alterations. Thoughts on this
177 Huntington: I thought the value error thing was a bit like maybe a custom error name would be cool. So it's a bit kind of is a call here, but not. It's a bit of a like catch. All makes sense.
177 Huntington: Yeah, like, like module specified out of order or something.
177 Huntington: Yeah, it. It does have like a message error message.
177 Huntington: which is the error type. Basically.
177 Huntington: Okay, okay, lastly, so, on the left hand side is basically the thing not to do, because
177 Huntington: it'll be considered technically out of order. So imagine you want, if you're familiar with the iter syntax, you can basically specify which iterations in the token generation to like intervene on, so it might be tempted to say, want to iter on 0, and then 3, and then do everything from 2 upwards, and then do some specific interventions for them, just like if you had like.
177 Huntington: 2 for loops, and I think this used to work in open 4 but then.
177 Huntington: instead, what you what you can do is this, the dot iter call returns to you this index
177 Huntington: just like a for loop, and so you can do if statements conditionals to specify the interventions for specific
177 Huntington: iterations.
177 Huntington: so the left hand side will will not work cool. When would you use dot iter instead of just like a for loop.
177 Huntington: For I and range yeah, yeah, the for the for loop will only work in the 1st iteration of the token generation.
177 Huntington: So so here the error is not like a regular for loop.
177 Huntington: It basically. Sorry I missed. It's within a dot generate. Yes, it's with that generator. Thanks.
David Bau: Do any of you use this.
177 Huntington: I I will use it now 3 times steps on.
177 Huntington: so you don't have to name it as idx. It can be anything we like, find the name for you, and you can reuse it inside of your function or inside of your cool context. Yeah, I didn't realize this was within dot generate before. So I think this is nice. Yeah, this this is this is new basically
David Bau: So so the the thing on the left doesn't work anymore.
177 Huntington: Yes.
David Bau: What's what's the Colon needed in everything on the right there.
177 Huntington: Online, 3 on the right hand side.
David Bau: Yeah, yeah, yeah.
177 Huntington: It's equivalent to tracer the all basically just saying, for all
177 Huntington: iterations of the generation do this. And then I'm doing things conditionally.
David Bau: We.
177 Huntington: Clicks.
David Bau: I I guess what I'm saying is we
David Bau: can you put anything other than the Colon in there?
177 Huntington: Yeah, I would. It's.
David Bau: Oh, oh, so! Oh, so you still can't say 0. So you still can't say 0 Colon 3.
177 Huntington: Yeah, yeah, I can. Yes, yes, just can't have conflicting indices exactly.
David Bau: Oh, that that's all so like it. So the the syntax on the left still works.
David Bau: It's just the fact that it's conflicting and doesn't work.
177 Huntington: Yeah, so technically, you could rewrite that as this if you want. But then oh, oh, oh! Oh! Oh!
David Bau: The other one is better.
David Bau: Okay, yeah, that's fine. Seems fine. And then so if you do the thing on the left, you just create some sort of error or something.
177 Huntington: Yeah, it. It breaks.
David Bau: With with an error.
David Bau: Cool.
177 Huntington: Yep. So technically you could, you could do this. Just don't have conflicting indices like alright.
David Bau: Cool.
177 Huntington: I'll put.
David Bau: Yeah, does it need to be? Does it need to be in order if you put them in the wrong order? And you said with editor to 1st by mistake. Is that also an error.
177 Huntington: Yep.
177 Huntington: because things are executed sequentially. So basically here it will do everything for like iterations. 0, 1, 2. And then you're trying to do something again for like iteration
177 Huntington: before, and then.
David Bau: Yeah, that seems pretty cool, both both these in. Yeah. Both the right hand side here seems cool, and
David Bau: the the one on the next slide seems cool. They both seem cool, nice.
177 Huntington: So yeah, that's it. So aiming for a pre-release on next Wednesday, I think.
David Bau: Cool.
177 Huntington: Please book me with any more feedback, any errors, any help with your code.
David Bau: Did I? I know I was I I was trying to create a thread with you and the good fire guys to review open 5. Did that ever happen.
177 Huntington: I don't think so. I did not see any emails or messaging.
David Bau: We will you ping Jayden, about this.
177 Huntington: Yeah, sure. Perfect.
David Bau: Okay. Cool.
177 Huntington: Sounds good.
177 Huntington: We are meeting with entropic with the Michael Anna's team, and he's already like, show them open 5 things, and they're gonna be using.
177 Huntington: Yeah, that's great.
David Bau: Yeah, I love it. That's wonderful. Yeah. But good fire is pretty major user. So.
David Bau: That's great. And then and I'm sure Emma's trying to get you guys in front of.
David Bau: you know, some of the some of our research users, too. Right?
David Bau: Cool.
177 Huntington: Okay, that's it. For open 5 oops.
David Bau: Pretty good.
177 Huntington: Oh, I didn't realize we went like way over time.
David Bau: Yeah, yeah, really, slow. Yeah, it's okay.
177 Huntington: Yeah. So we have. Yeah, Michael's gonna have some content as well. I thought I didn't have that much. I guess conversations. You know, we can have 10 min worth of content and still talk for an hour. So it's okay. That's what happens.
177 Huntington: Okay, that's
177 Huntington: alright. So as context, I made these slides very last minute, and I tend to do more impromptu than slides, so I may not follow it to complete team. But I was going to just give a few updates about some sort of things that that's kind of a mix of things that previously got finished or more recently got finished some stuff that is in the process of getting finished, including like right now.
177 Huntington: and some stuff that will be in the near future coming out to be excited about, and I will try and go quick. If I speak too much, let me know, and if I speak too little, ask questions.
177 Huntington: so sort of the agenda. So I'll kind of focus on like the past. And now so this is like end of in the back there. Nice little
177 Huntington: fabric and some nice changes that have happened to it, including stuff aws, stuff with Delta automatic redeployments and end of queue.
177 Huntington: And then you got this futuristic looking. And if you see it's going 3D it's popping out and I mean, there's a bit of a you know. Some of this stuff is closer and more sooner than others. But there's more these 2 big areas which is on demand jobs which I think is really exciting. And the oh, point 5 relay changes which, to be frank, it will be a pretty brief covering, because it's more what James, working on right now. But I'll try and just like
177 Huntington: illustrate, I guess what he's thinking about a lot and what I'll be thinking about as it goes and gets created.
177 Huntington: So just for reference this is from a previous presentation in January. But there is this kind of old looking architecture of it. It's mostly still relevant or still accurate where you have your your local workstation, which is really just your your code, that you're writing in your on your laptop.
177 Huntington: and
177 Huntington: the experiment gets created to an intervention graph, I mean 0 point 5. This will change. But at present you have this intervention graph which gets sent to our front end server which gets routed back to the back end and based on the model you request. It gets routed to a particular worker which computes the request.
177 Huntington: serialize it and sends it back to you. And there's a whole lot of fancy looking stuff in here. But the main sort of key things that I want to point out are the separation between front, end and back end. So the back end is where all the gpus lie. It's the very expensive set of servers. It is managed in Illinois. It is
177 Huntington: Delta delta AI for those familiar with the names, and then their front end is where essentially all the user facing stuff and telemetry and user metrics and whatnot are stored, captured, etc.
177 Huntington: So just keep that in mind in the back of your head as we go through this presentation. But you don't need to know it in much detail for what we're gonna talk about.
177 Huntington: So our 1st major update is that our front end
177 Huntington: went from previously it was on a single virtual machine Openstack instance.
177 Huntington: and everything was on there. So you had your Api, you have your message, broker. You have the all the observability stuff like stuff like metrics, and like logging captured logs, etc.
177 Huntington: And our objects were all stored there as well.
177 Huntington: And we went ahead, and we migrated to aws.
177 Huntington: And so now every single service. So the Api for fauna, which is like a dashboard, visualize visualizer for metrics, etc, etc. They all have their own. Basically, you can think of it as a virtual machine. So they're all separate. They have. They're all managed separately.
177 Huntington: And, moreover, we previously had a all of our user data stored in a service called Google firebase. You can think of that like a a thing that stores Json objects. It's so. It is basically a a
177 Huntington: what do you call that object database. But
177 Huntington: the problem with it was it was hard to visualize actual user. Like, if a user made a request, it was hard to tie it back to the user because there was no easy join operation from
177 Huntington: any of the metrics that we have captured with an Api key and an object in firebase. So now we have a proper schema. It's a it's it's it's a postgres database. So it implements the SQL. Query language, and it's a lot more. It's a lot has a lot more interesting things to it. But from the point of view of
177 Huntington: well, really, the the really big update is that it's a lot easier to tie stuff and get more insights into the usage based on individuals.
177 Huntington: And for aws and a whole. I mean, it's a bit of a big, you know, it doesn't immediately necessarily indicate any like real value if you just think of it as like, oh, we just switch one thing to another but I think that it is pretty. It's a it's a pretty big shift. It allows for a lot of things like by default, which
177 Huntington: you know, make life easier, like being able to automatically back up.
177 Huntington: Like volumes. There's more monitoring that can take place. You can have better alerting for specific services. It's a lot easier to configure things have a lot more resources now. So, for example, a problem that we faced was, we had these big queries that we would try. And we do big time series metrics, and we'd want to do these complicated queries on them, and we'd hit ball next after like going back like 3, 4 weeks. So
177 Huntington: just resource wise it wasn't there. And now Grafana, for example, has its own Ec. 2. Instance, and it has which it vm, basically, it has a lot more RAM. So you can do a lot more quick queries and and operations. And as time goes by, this will only become more useful. So essentially, we have the tooling now in place to be able to really smartly scale up if we need to and allocate resources where needed. So it's
177 Huntington: we're in the very early stages of it. But it's an exciting time for us, as we
177 Huntington: are creating a lot more of a standard sort of stack for for and if and I, the reliability side of this is is definitely a big selling point, too.
177 Huntington: So
177 Huntington: that was one big update. And this is something that's already in place. Right now. The second one is actually in the profit process. In fact, something that is pending on my mind right now is migrating
177 Huntington: so
177 Huntington: migrating to back to Delta. So a little back story so previously we so in like you may, you guys might remember around November ish, we went from having Nagoya, and like some of the Batlab machines as our like production end of infrastructure. And we went and we removed
177 Huntington: our our production code from there. And instead, we move to this compute cluster called Delta AI. The AI parts, the emphasis, it's it's the same. It's the same people, but it's just a different part of the whole, like
177 Huntington: whatever data set or whatever you call their entire unit. And specifically it was with nodes, compute nodes that had a very specific type of gpu on them, called Grace Hopper Nvidia Grace hopper whatever gpus
177 Huntington: These things sucked. They were a huge problem. So I actually will skip the slide because I go over a little bit. But there was a huge problem. So a lot of engineering time went into basically some very silly problems that were basically caused by Nvidia. So
177 Huntington: to give some context. So these machines had this sort of feature, where, if you copied some files, you did some like I/O operations
177 Huntington: that to speed up the copy and speed up some, some related stuff. It would cache the like intermediate data, both on RAM, which that's normal, but also on Vram. So you'd have, like you do a file copy, and the V. The Gpu would fill up.
177 Huntington: And because it was a system process, you would have it that you wouldn't be able to flush it afterwards. So in practice, what that would look like is, I try and load my model in, and very quickly it would just get out of memory, and I would have like 5 nodes, which have, like
177 Huntington: an insane, insane amount of vram, and you would still have the issue that, like 4 or 5 B would not fit in and after, like maybe half of the weights load in, it would just get out of memory. And so this problem led to us, having to do some really clever hacks, one of which was that we would block the Gpu. So we just fill them with art like, just use pytorch to fill up all the gpus completely full. Then we would copy the model weights over completely
177 Huntington: once the deletes are finally copied. We then release the the tensors. So then the gpus would open up, and then we'd very quickly have it load in all the stuff.
177 Huntington: and and there is still the problem with the distributed 4 5 b, where
177 Huntington: it for some reason was doing some I/O stuff. So when you were loading in the shards one by one it would start to cause the spike again. So what we had to do was not only have the weights copy over to each node, but as we load them in one by one, we had to then delete the the weight so it would like free up the the device again. And so what that meant. So so what this really means is that once you deploy once, it's really hard to redeploy again, it's really
177 Huntington: icky, especially with well, with 4 or 5 specifically, it's really slow to deploy because you have to copy the entire model weights over like that's really slow operation.
177 Huntington: and there's just all these more issues with them. They they happen to be slower, like performance, wise for some reason, even though they're like state of the art, they happen to be way slower than just older, like Nvidia, like Chips.
177 Huntington: and one other small but also important thing is because there were. There was only the nodes had only 4 gpus per per on them, so we had to distribute 4 or 5 b 4 or 5 B would not fit on a single node it had to fit on 4
177 Huntington: this in itself caused a lot of I mean this. This was a lot of extra innovation that need to take place, and as I will now discuss, our move to Delta is so relieving because they happen to have a node that fits for 5 B on it completely. So what that means is that you don't have to distribute it over multiple nodes. You can. There's no network communication. It's all local to the one
177 Huntington: single node, and it is amazing. It is so good. Not only is it that not only do you have the. You know you have the speed up of inference and everything operation wise with it, being on a single node. But you also don't need to copy over the model weights, too. You can just load them directly from like off the node on like a a network file system. And it's really fast. It takes 20 min to load in 4 5 b like start to finish.
177 Huntington: and the rest of the stuff. The the whole deployment spins up insanely fast. Now, like it's within a minute. It's it's ready like, not the whole thing. But it's spun up. Yes. Well, that's pretty one. That's pretty good. 2. I maybe you're gonna talk about this. But like I remember the one issue with Delta before that there was like some
177 Huntington: time limit, like, how long access you have does that? Is that still an issue, but I mean we were a little bit we we knew we needed to redeploy, anyways. So it was sort of this like, well, we could fight a lot with them, but we kind of have to redeploy in the 1st place, like we could have it so that a job has we could have them change it and make it arbitrarily long.
177 Huntington: But in practice we would have to redeploy anyways. So it was. And the second you what the second the the caching happened like it's over. You need to redeploy. There's just no way to flush it so
177 Huntington: that will come. That that point is is a good one to keep in mind, it will come up in a future one of the next updates.
177 Huntington: so. But yeah, so this migration is great. So we also have. We happen to have a bit more resource wise. So 4 or 5 B is on this H. 200 node.
177 Huntington: and we now have also these a 40 nodes. A 40 is another Nvid Gpu, H. 200 as well. We have
177 Huntington: basically a lot more compute. I mean, right now it's roughly the same.
177 Huntington: But going forward.
177 Huntington: these nodes are very easy to get access to the A 40 ones. They're like a lot smaller. They're very small lightweight. They're not as fast but they're they do the job for things like A, B 70 B, and they're very easy to acquire, so it'll make for us, being able to have a lot more models, a lot more easy, like the even the Gracehoppers. Like they they were not. They were reluctant to give us like 6. They were very reluctant. They kept.
177 Huntington: you know, there it took some time to badge them badgering to get them to like, you know. Give one more, node. Give one more node. Here. It's a lot easier for us.
177 Huntington: So this is a really exciting migration. I think it will. And and, moreover, just the fact that you now can load like there, there is no
177 Huntington: thought at all about memory leaks and stuff which means that you can update the state of things and not worry at all about. Oh, oh, no! I have now all this memory that is in the view, like all those Gpus are filled up. So that will. I'm hinting to a thing that I'll talk about later on. But switching a model is, there's no, there's a much less overhead. Which yeah, it's it's to. We'll talk about that in a second.
177 Huntington: To your point. So another thing. So a big, so a sort of pattern here is that we are using someone else's resources, this big cluster. And so we have to play by their rules in general. We have, we? We don't have full permissions to do what we want. We have to go through a formal process of getting, you know, a reservation for some nodes.
177 Huntington: You know there's there's a lot going on that we don't have visibility just from like a you know. I don't have root access. And there's, you know this for a good reason. One issue that had came up was the sort of need for us to auto manually redeploy when something went down. So
177 Huntington: the reason so 1 1 thing that this whole time that was a bit of a burden was, you know, when I wanna if I want to go onto this cluster. I have to use multi factor authentication for good reasons. But I can't just write a script that goes connects on checks how things are going goes back offline. And that's good. That's for security reasons. But an issue was that I wanted to be able to automatically spin up a job again, and
177 Huntington: you know, if something went down I would want. I don't want to get up at like one am to go like replay. So
177 Huntington: finally, after a lot of bureaucratic discussion, we finally got the means to
177 Huntington: run basically a bash script in infinite loop on on their login node. And so now it's currently running on Delta AI, and we'll be getting set up on Delta once the migration happens. But essentially, we have a very basic thing that checks to see if the model, if if the 2 day limit is reached, basically if
177 Huntington: for some reason the job submission is gone, usually it's the 2 day limit could be other reasons. It'll just automatically do it and re redeploy and another one that's pretty common, especially with the distributed stuff is 4 or 5 B will break and crash, and usually we just end up having to redeploy anyways.
177 Huntington: So now it just checks the Api. It makes a call to the Api checks to see if 4, if 4 or 5 B or a 4 5 b model is listed within the list, if it is then or then it continues, if it doesn't it redeploys? Now again, very simple stuff. This isn't very complicated.
177 Huntington: And if you have, if we were self hosting. This would be something that would be done on day one but it's just a good update and a good reminder of the process on Delta, and how you know some things you can't take for granted, and that even something as small as like a quick bash script that checks things is worth celebrating, because it does add, it really adds a lot both for developers and for the users.
177 Huntington: Yeah.
177 Huntington: And the 4th update for now the end of Q. So this one is.
177 Huntington: I'll try and keep it concise. From the point of view of a user when you make a request to end. If remotely.
177 Huntington: there's different, there's different stages. You get received. That's basically the second. The server gets the request. It's a confirmation essentially that yes, you indeed hit the right endpoint.
177 Huntington: and following that, there is an approved stage which essentially your Api key is good, and it's put on the request. The existing request deployment queue thing.
177 Huntington: And this approved stage is actually very ambiguous. So a lot of times what happens? A user will come and complain to one of us about how they make a request, and it gets stuck on approved. This is such a common pattern. It's so common. And the problem for us and for them is we don't know why they're stuck on, approved.
177 Huntington: It could just be because there's just a really long job going on in the pro in the background, and they're just waiting. It could be because the model that they requested is is currently broken, and they're like unable to reach. The thing is just silently failing, there's no indication. And, moreover, there's no prediction or information about how like, when they can expect their job to run.
177 Huntington: they also can't cancel it. So, you know, if you know, they're sitting there waiting. And they they're like, Oh, well, I'll just run this later.
177 Huntington: It's it's there forever, like they. It'll run. They can cancel their thing. But the the actual, the job itself, will still be on end. It's still waiting.
177 Huntington: And I mean the way that what I'm alluding to is the existing off the shelf, queuing that we have which is using, basically deferring to ray which is our compute
177 Huntington: the thing that manages all the compute stuff. Deferring to it also means we can't prioritize requests. So if a user makes like 20 requests and has been using a model for like an hour, and a new user comes in with a job.
177 Huntington: There's no way to just like skip that user to the front of the queue and then have them do their thing.
177 Huntington: So this is a problem. And so this turned into a project of creating a standalone queue micro service, which is what I've been working on recently. This actually, essentially the way this works is, you kind of think of it just like a simple web server that has you can. You can make requests to it. Well, you can send requests to it, and you can also get query it for for analytics.
177 Huntington: And this thing is quite simple in terms of what it does really. All it does is it keeps track of so has a queue for each requested model.
177 Huntington: And what it'll do is, it'll see whether it has dispatched a request to
177 Huntington: to ray or to to be run or not. So if it had, it makes sure that the request is, is.
177 Huntington: it checks to see if the request has finished or not. If it's if it's not finished, it will not dispatch any further request until the job is complete.
177 Huntington: If it is complete, then it dispatches the next one, and it keeps track of it keeps a reference of it, and the same pattern repeats
177 Huntington: so essentially, it's sort of stealing the rate, queue and kind of taking over it ourselves.
177 Huntington: And this is a really powerful thing for us. It allows for a lot more things based on how we implement it. It allows for full visibility on. You know where user is in the queue. We'll have it so that the users are notified as their request bumps up. You know I'm in position number 3. 0, you're now position number 2.
177 Huntington: We'll be able to know the last time a model was used. So it'll tell like that. That's a that's a detail. We don't currently have. We could get estimated. Wait times. That is definitely a function of this queue. It's not an immediate focus, but it's something that is it. It comes from this like having data on on the queue.
177 Huntington: And there's there's just a lot of other potential cool things that can come down the line from this as well.
177 Huntington: One small thing but potentially important is the fact that if all the requests are on ray, and then the 2 day limit reaches. You know, if Ray goes down, all the requests go down to this service is separate, so it's sort of persistent, and I use the term persistent a bit kinda tongue in cheek, because it's not storing anything at this point. But it's it is keeping track. It's keeping the state of what users request, irregardless of what's going on on the back end.
177 Huntington: And yeah, I think this will help a lot, too, with
177 Huntington: it'll help with the users as they view the request in real time, and it will also help with the status page. As you'll get a lot more, you get a better sense of like, okay, currently, 4 or 5 B has, like a bunch of people using it. There's like 8 requests on it. Right now, there's been run one that's been running for like 20 min. Maybe I won't use 4 or 5 B right now. It's it's it's it's painting a future where there's a lot more transparency for the users, and for us, too. It really will help with the observability, I think.
177 Huntington: And there will be a lot of really cool things that can kind of derive from just having this. So getting this out and ready is going to help for us now. And in the long term, as it will enable a lot of cool things built on top of it.
177 Huntington: So yeah, now, I missed the transition here. But there, these updates. It was a bit hard to make these, because some there, there's all things change all the time, like Jaden just works nonstop. And you know there are. There are things which kind of become seemingly more short term than long term. And I kind of put this on demand thing as this long term. It's it's a big project.
177 Huntington: Jaden is chugging along on this one. But it is a very like the thing I think I'm most excited about in all this is what we're soon to be offering, which is an on demand model hosting service.
177 Huntington: So basically, the goal is that any? Well, Jaden's ambitious. I am a bit more reluctant to to state it this way, but I will state his way of allowing a user to make a remote request to an arbitrary, hugging face model.
177 Huntington: And when I say arbitrary, I mean literally any anyone. Security concerns pet like there's definitely some there. But this is this is the goal. This is a big change. It will essentially get rid of the need for users to
177 Huntington: in any way request for a type of model. I mean, there are exceptions, of course, like, you know, gated models and things that are too large to fit. There will be some restraints there, but it really democratizes like, what end? If is because right now it's arbitrary decision to posted models that seem kind of correlated to what users want
177 Huntington: it doesn't actually tell us really what the users want. We have to manually go and ask them. But having a service like this actually literally is pulling the users what they want because they're constantly getting, you know, new every time a mall is deployed. That's data. Every time it's used as data, you're you're literally getting a live
177 Huntington: sense of what is wanted. And it's it's very, very powerful.
177 Huntington: there's this. So so so that's a super nice thing. It's very adaptive as well. That's another aspect. So it's it's it's it's perfect. It's closer to perfect information, you know, like it's
177 Huntington: a new release happens. Given, it's not a game model, someone. You can just start using it today. In theory. You know. Obviously, things will break and we will have to debug a lot of stuff. And that's actually a very valuable part of it is, it's testing, really how strong and robust our infrastructure is to just arbitrary models, but it does allow for the users to kind of evolve their experiments over time, not us evolving for them.
177 Huntington: And there's this illusion of more. So something I'll talk about a second is we'll be able to start doing this caching.
177 Huntington: and it will, and especially with these Delta speedups, it will. You'll have an illusion of having an on demand model, because some of them will just load up really quickly, especially smaller ones. It will be something where you might not even you may barely notice the wait time for it to get deployed.
177 Huntington: So here's the sort of something that we've been talking about, which I think is currently going to be the sort of thing. There was some uncertainty about the part of the caching, but models will fall into 5 categories while there'll be a 6th one. But this is like the on demand sort of side of it. So
177 Huntington: a model will be hot if it's already loaded. So basically, any loaded model is hot.
177 Huntington: It will be warm if it's
177 Huntington: on some node in CPU, so the model is not an envy. It's not on the gpus, but it's pretty close. It's on RAM. It's quick to load in.
177 Huntington: It'll be cold if it's already downloaded, but it's not loaded yet.
177 Huntington: It will be glacial if it needs to be downloaded from hugging face. But it can be. And then there's some that are not supported so like any really big one that isn't a dedicated like 4 5. B, for example, probably won't bother downloading it. Especially early on. There might be some weird ones with unsupported code. So it's not complete democracy, but it's certainly a lot closer to it.
177 Huntington: And based on usage. It will. This whole, like on demand aspect, involves a lot of scheduling, so demand will move things up and down this queue, excluding the the 5th
177 Huntington: alongside this we're going to host a set of dedicated models, probably mostly just like things that are used for tutorials constantly, or like programming, like 4 or 5 B. But these models will not be affected by user usage. And we'll have a calendar which there's already prototype for where someone can just like an end. If
177 Huntington: end of team member can go in and update a Google calendar. And that will just automatically tell our deployment what to do.
177 Huntington: That's kind of cool. It's I'm like worried about like oops. I accidentally dragged this thing over here and now something is happening, but I think that with a little bit of oversight and precaution, it's a pretty convenient way, and it just makes it possible, for you know it's no longer me going on and having to do something. It's like. Emma can just put something in a calendar.
177 Huntington: and I don't even need to know, and it just will. It will be the ground. Truth and other things can can sync to it.
177 Huntington: It can be a a calendar which has, like a public facing sort of artifact, so that users can literally just see. Oh, I see the 4 5 b instructors up for this date. And yeah, it's it's it's an exciting thing that seems to be coming much sooner than I would expect.
177 Huntington: Now, this is the part I'm the least familiar with. This is a lot of what Jane's working on right now. And this is stuff in context of insight, 0 point 5. So a big challenge. And this is a really ongoing thing is because we're shifting from intervention graph to basically like a pile of code.
177 Huntington: It's such a such a sketch. Sketchy thing. You know, a a user can just very easily, without any proper setup to use use the OS library to like, you know, or they could. They could use. They could start a shell, for example. They could like look and delete all the files and the whole, you know, it's a very interesting security problem. Actually, how do you like the the crux of this is like, how do you sandbox arbitrary code execution?
177 Huntington: And so a simple starting point, I mean, like the 1st sort of basically what previous and and if did was, it has a white list of supported functions and types in python. So things like, you know, pytorch different pytorch
177 Huntington: functions classes, etc. And you know, basic python types, etc.
177 Huntington: That's cool, but there's an issue where you know there might be some sneaky trick where you can get the OS. You know. Maybe you can get the OS module
177 Huntington: by using some reference in pytorch where it uses it, and then you can just like
177 Huntington: have a call to it. There's a lot of very sneaky stuff that Jaden. Soon I will be thinking about deeply.
177 Huntington: And I think this is a this is pretty. There's a nontrivial problem. And if anyone has any thoughts or insights here, or just general, you know curiosity
177 Huntington: one. We can talk about it, and 2 as this thing gets kind of created. I think one thing will be cool is having people try to break and find vulnerabilities. So it may be a fun kind of, you know little hackathon thing we do at some point where I mean, I'm just saying this right now. I don't know what the process will be like, but you know, we may want to really
177 Huntington: have a lot of people mess with it and try not to, or try and
177 Huntington: find these things for us. So
177 Huntington: yeah, that's that's sort of the
177 Huntington: this is more of like the looming future thing. Not the next few weeks. Kind of thing. But
177 Huntington: yeah, that's all I got thanks for listening, and I'm happy to answer any questions before my laptop dies. Well, I mean I'll still be here, and everyone will still hear me. So it doesn't ignore that point
177 Huntington: for the link.
177 Huntington: Hmm.
177 Huntington: So, for example, for like building in models that are like too large with the on demand
177 Huntington: service. Are we like going to like cap it to a certain like
177 Huntington: file size of like ways to download does like the hugging face. Api, give us that information, or do you need to like all download and see how much you're expected to like
177 Huntington: downloading? And then we just like interrupt it.
177 Huntington: Yeah, I think the I think hugging face there's like a there's like a config dot Json, or there, there's some metadata that you query for and I think if you just grab that you could see the size of the bottle somewhere, there's
177 Huntington: there's probably a few ways of going about it.
177 Huntington: but I presume yeah, you'd have a cap on. You know what your nodes are, so you'd have a you'd have a node understanding of what like the the resource, like the Max amount of vram, or whatever that you could fit, and then you would just check the you would just check
177 Huntington: from the hugging face thing there. Yeah.
177 Huntington: So for example, let's say, we have 2 nodes dedicated to
177 Huntington: on demand models. Would we allow? Like, for example, one sort of like
177 Huntington: asked model to just fill up the
177 Huntington: both nodes. That's a good question. So that depends on the schedule, or sorry. It depends on the usage. So there's a concept of there'll be a concept of not supported models, or I forget the term that I use for it. But models that are we just can't allow. And and one reason for that is, they're too too big. But
177 Huntington: you might have a case of like 70 B, where, you know, there's just only one.
177 Huntington: There's only really, you can only really get one device
177 Huntington: at a given time, because there's just so much demand. And so it, despite being able to fit on the node and being a model that we do support, it may just not be you may have. The user might be told that either wait or it will say, currently we can't support and the job will get cancelled. So it's it'll be a combination of demand. It's like they use. There's a combination of yeah, the demand. And also what actually, we physically can host or commit to hosting.
177 Huntington: Probably.
177 Huntington: Yeah.
177 Huntington: So do you think maybe also, like keeping tab of some of like
177 Huntington: these requests that were like responded to as like not supported. And if we see, like one is like getting a lot of like traffic. Then maybe you should like consider adding it to the white list of like
177 Huntington: models to be supported. Absolutely. Yeah, I think, having a a feedback loop of well, a having a when you get that error, it redirects to our discord or our our for our discourse. That's like one way the user can immediately. If they're like, really, it's really important. They can start asking about it. They can get immediate feedback, and then 2, as you said, storing and keeping that sort of stored and accessible, and some dashboard would be useful too. So you get like a
177 Huntington: transparent sense of like, oh, yeah, these are what these are, the users who are telling us.
177 Huntington: or who are complaining loudly, what they are wanting and not being able to get. And it would give, yeah, I think that's a good yeah. A good thing to keep in. Mind. Yeah.
177 Huntington: Okay, so
177 Huntington: at 12. Yes, we we have Karen. Yeah, we'll probably just maybe start 5, 10 min later. We have probably.
David Bau: I I just want. I just want to say, congratulations. You guys on the end of guys. You know, Michael, this is it's awesome, you guys. I'm I'm really happy about all the aws and delta move you guys are doing.
David Bau: I it it sounds it sounds awesome.
David Bau: It sounds like you're
David Bau: pretty happy with the Delta move. I if I at least from the tone of your voice, is that right? Am I? Am I reading.
177 Huntington: Yeah. Once I once I once I finally figure it out, they'll be happy. Currently, it's like a challenge I'm working on. But it's in theory. It's very.
David Bau: Good. So it sounds like we we've we've we've cleared the path for you guys to do good good things. And it sounds like, it's going well so far. So it and it's wonderful. I think that that yeah, I I think that my prediction
David Bau: is that the students are going to be very excited to have
David Bau: any model that they want to look at available and end of
David Bau: and so so that it's so. And and it's the whole thing.
David Bau: wouldn't. It? Sounds like it wasn't possible on Delta AI to do this kind of thing. So it sounds like the whole move to Delta, and everything is, is enabling this great
177 Huntington: Yeah. Well, let's come together.
David Bau: Yeah, congratulations on all this.
David Bau: It sounds really great.
177 Huntington: Exciting times for us. A lot of stuff to fix, but it is. It's going to be fun to chat through this in the next many months.
David Bau: So so on the security stuff that you're worried about while we're just sitting here chatting you. You know that Northeastern is one of the top cyber security universities in the country. And we've got a lot of you know, grad students and professors
David Bau: who are pretty well versed in a lot of different security issues. So if you haven't already, you know I so so
David Bau: you know you you can.
David Bau: You know we could try to connect you with
David Bau: people who think about the specific kind of security issue that you're worried about. I don't know if Arjun or John
David Bau: has already talked to you about this or or if not, then we can.
David Bau: We can connect you to more resources.
177 Huntington: I used to do research at the Cyber Security department. So I know a lot of professors there and Phd. Students.
David Bau: Oh, that's great. Yeah. So yeah, we should definitely be doing that.
177 Huntington: Yeah, it's a great idea. Yeah, yeah, we'll mention that again
177 Huntington: before we get some like professionals just poking at it and giving an insight
177 Huntington: may may have been something that's been tried before.
177 Huntington: or imagine remote code code execution as a thing
177 Huntington: has been thought about a lot. So I'm sure there's
177 Huntington: some good insights from those people.
177 Huntington: Yep, yep.
177 Huntington: yeah. Also. I know the president of the undergraduate Cyber Security or Captain Flag Club, so
177 Huntington: maybe you could set up like a little hackathon with them. I know they do one every week. So that's a good idea. Actually, yeah, yeah, I know somebody from like the clubs. Oh, me, too.
177 Huntington: Yeah, that's a great idea. Actually having one where we have a version of end of on a local machine or a somewhat sandbox, ish, machine, and kind of also go off with that good good faith, assuming they are going to just like use the use that to play and exploit, but, more moreover, like a temporary thing, that the the back door, the the way in, will be removed. What's acting on the back? And use that to gain insights of like, what is?
177 Huntington: Yeah, that's a great. That's a great idea. I don't know if the barge entry would be too high with the insight syntax. But well, technically, like all they need to do is like, do with model dot trace. And then right, the goals are going to be. They're going to write whatever python code they want. That's true.
David Bau: Yeah, it's great. Yeah, yeah, yeah, is quite, quite hackable.
177 Huntington: I I think I think knowing hackers, they're probably going to pull the source code up and start messing there, which also is good. But it kind of will get a bit. Besides the or it'll kind of I don't know, actually, that I just take that back. I think that's also very valuable, and something that, like most
177 Huntington: perfect.
177 Huntington: Do you catch yourself off guard?
177 Huntington: Because there's some there's some
177 Huntington: with insight right now, with end of that like is sort of just like, you know, like there's like different settings that aren't actually in practice like used at all, and kind of just like our legacy that Jane had in so like.
David Bau: Oh, I see!
177 Huntington: There are different.
David Bau: Yeah, like.
177 Huntington: Interesting. Okay.
David Bau: Cool. Yeah, yeah, our secret backdoor setting.
David Bau: They'll find that that that's great.
David Bau: Yeah, that that sounds like it'd be fun, a very unique opportunity.
177 Huntington: Slash.
177 Huntington: I have to.
177 Huntington: Okay.
177 Huntington: Slide, good. How's the movie doing?
177 Huntington: Still growing?
177 Huntington: I mean, I I they just don't do anything they just like, sit there, you know, just just sleep. Yeah, and cry, yeah.
177 Huntington: So
177 Huntington: also, is your wife doing okay?
177 Huntington: Nice.
177 Huntington: So how is catching your daughter's name? How's she reacting to this whole situation? She's she's obsessed, she's obsessed. Every morning she wakes up. She like holds her arms like because she wants to hold the baby. Oh, that's like that's really cute! That's very nice.
177 Huntington: like the best toy we could have gotten. It's like more interested in the baby than I am.
177 Huntington: Oh, hi! Aaron!
Aaron Mueller: Hello!
177 Huntington: Maybe. Maybe. Yeah, maybe like, 3 more minutes cool.
177 Huntington: I've seen it.
177 Huntington: please.
177 Huntington: He is online. So he's kind of basically there.
177 Huntington: yeah, actually, are you still recording? Yeah. Oh, yeah, I am still recording. I can't like, stop that one and then record a separate one.
177 Huntington: Yes, we'll start recording.
177 Huntington: Alright, I'm gonna share my screen.
177 Huntington: Wait. So
177 Huntington: Hi, hey, everyone I'm keram for people who don't know so I will be today talking about the research that I've been doing for the past couple of months at the lab
177 Huntington: awesome.
177 Huntington: Yeah. So the project is about the emergence of shared multilingual concept representations.
177 Huntington: So I've been working with Chris Aaron Sherdon
177 Huntington: air. Adam Yannick and Constant also recently joined. So yeah, we've been working on this for the last 3 months. So I'll be sharing my updates and hopefully get some feedback for people.
177 Huntington: So to start with basics when I say concept representation, I mean not only semantic level, but also grammatical concepts that are that we can extract from the model. So a concept, a representation of a concept could be a word, level concept, like meaning of word. When we talk about semantics, it could be a sentence level concept, the meaning of a sentence. And
177 Huntington: also we could be a concept that are related to the grammatical concepts of a sentence like the tense of the sentence, whether the sentences in passive or active stuff like that.
177 Huntington: and by shared multilingual, casual representations of
177 Huntington: it could mean a couple of different things. But at least the
177 Huntington: oh basic thing that we can say is, oh.
177 Huntington: the same. The similar structures across different languages are being reused for throughout different languages. And you can like. There, you can observe this from different perspectives. So if you can, if you can check, you can extract some embeddings from
177 Huntington: similar meaning words for different languages, and you would observe that they are more similar to things that are that do not have the same meaning, and this generalizes to sentence level as well. You can also
177 Huntington: extract some embeddings. So this is from Yannicksburg can extract some embeddings, and, for example, you can, by giving English sentences and training a linear probe from the activations of the mid mid layer.
177 Huntington: You can train a linear probe to predict the tense of the sentence.
177 Huntington: and then, if you use this probe to predict the tense of other languages. If that language has a
177 Huntington: past, that structure, it
177 Huntington: it that linear probe does also reliably predict different languages, tenses, tenses as well. So it seems like there's this shared concept representation across languages.
177 Huntington: and therefore in our project we wanted to see what are the major contributors, contributors of the mergers of this shared multilingual concept representation during training.
177 Huntington: So to start off in order to accomplish this, we have to check some multilingual metrics across different checkpoints of a of a model to see to as a 1st step, to see how they evolve.
177 Huntington: So mainly. I've been looking at the Sads and the Lrds scores that are from a converging to a lingua Franco paper.
177 Huntington: So so I wanted to like 1st explain this metric to see if it makes sense. And
177 Huntington: yeah, if there's like any issues with the metrics that you're thinking of.
177 Huntington: So the 1st thing 1st metric would be using is the sads metric
177 Huntington: the Icds metric. First, st let me try to explain it in a high level thing. So when you, if you see this formula, the 1st part
177 Huntington: is basically telling this, we will get positive tears from with the same semantic meaning, but from different languages.
177 Huntington: So the 1st language. The 1st sentence is, the cat is sleeping, and we get its French version.
177 Huntington: So we collect activations for the sentences to have representations for these sentences.
177 Huntington: For the sads. This is a positive pair, and the second part, which is the negative pair.
177 Huntington: is a pair that has both different language with different semantics, and maybe take the difference
177 Huntington: so we can think of it as a way to see how
177 Huntington: our words with sentences with similar meaning, are more alike compared to sentences with different meaning in different languages and the collective activations by.
177 Huntington: So we collect the activations in this format. So if if you give a sentence like this
177 Huntington: for every layer, use sum the embeddings
177 Huntington: to get like a holistic representation
177 Huntington: for each layer, then they concatenate. These layers. Sorry in the previous slide. This one. Yeah, for the Lrds. Does it consider the same language? For let's say, like, I guess
177 Huntington: like American, English, and like Shakespearean, English. No, no, okay. The data set comes from a, so the data set that we are using is a Bible data set. So it's just basically the English translation of the Bible versus
177 Huntington: the French translation of the title. And like, they're parallel sentences. That's like what we're calculating on. Yeah. So we don't consider that much, that granularity. Yeah.
177 Huntington: Also, you send these activations and then concatenate them to get a
177 Huntington: gigantic vector for the sentence, and
177 Huntington: they are take. So we're taking a cosine similarity to get the similarity score.
177 Huntington: Oh.
177 Huntington: so for the positive pair, we're taking the cosine similarity. And we're subtracting, subtracting it from the negative pair.
177 Huntington: And we can think of this as a as this increases the similarity of the same sentence,
177 Huntington: but a same as meaning difference. Language sentences are getting closer
177 Huntington: for the second metric. The Lrds.
177 Huntington: They are looking. If you're looking at a different thing. Now this, this metric, like we can in in a high level, we can think of it as.
177 Huntington: oh, how similar the subset bases are.
177 Huntington: because the way we're calculating it is as follows, we get 2 different sentences from the same language.
177 Huntington: and this is a positive pair, and
177 Huntington: for the negative pair. We are getting both different language and different meaning things. So we can think of the high level meaning as if we.
177 Huntington: if they have no similar meaning. The only thing that connects them is their language. And how similar is, how different are the language spaces between these things. So
177 Huntington: the an increase of Lrds score would mean that the
177 Huntington: the difference different languages have much different subspaces, but if it decreases, it would mean that there's no meaningful difference between
177 Huntington: between different languages, if
177 Huntington: if these, if this difference goes to 0. So that's those are the metrics that we were looking at.
177 Huntington: So if I just go there. So the left one is like controlling for so well. So these things are 0 to one. So it's you're subtracting sub 2 numbers from 0. And one.
177 Huntington: there is something that is basically assumed to be much less likely, which is where you already have some condition. And you're making a more unlikely condition. So in the bottom one, the left one, it's the semantics are different, and then the right one. Well, actually, both of them, both language and semantics are different. And the idea is
177 Huntington: you're in the top one. You're introducing either semantics or language as a prior being the same. So if the semantics are the same, then that means that you are trying to see if the languages are the same or not, or sorry to see if there's similarities between the languages, whether there's a meaningful distinction relative to the baseline.
177 Huntington: and then for the right one, it would be if it's like a syntactic thing, almost if there is different senses, whether there's a similarity amongst the sentences. Yeah, I think if I understood correctly, yes, basically, that's what it's doing. So the second, the difference is basically, yes, it acts as a baseline, a lot like a normalization sort of like, we just don't get
177 Huntington: random values at different checkpoints because these values do change a lot. But when you're like taking the difference, at least like, now, you're seeing a
177 Huntington: more relative difference between these things. So yeah.
177 Huntington: for this one. I'm not sure if it like only captures syntactic things, but at least like the language information
177 Huntington: separate from me, so like the Englishness of the sentence or whatever. That's how I think of the picture.
177 Huntington: So okay, wait.
177 Huntington: Okay. So I can talk about this one, too. So we've also like checked several metrics across these checkpoints the one that was interesting with
177 Huntington: the calculation of the Cdslr Ds metrics for the token loss difference.
177 Huntington: Because when we calculate this token, token, loss, difference
177 Huntington: we see that the phase change like starts at around the 500 step and like the
177 Huntington: yes, I can.
177 Huntington: Okay, I've sent it. Does it work?
177 Huntington: Okay? It works. I think I have, like, allowed everyone to use the link.
177 Huntington: No problem, of course.
177 Huntington: So
177 Huntington: the phase shift that for token loss, difference starts around step 500, and it starts to stabilize at around 7,000
177 Huntington: and one thing that we have noticed when we check the multilingual metrics across shake points is that especially for the or this for the moment? This phase shift starts
177 Huntington: the or these scores now start to decrease, which is the
177 Huntington: language meaning the language difference. So what's the loss on the previous slide. Oh, the token loss! So this is next to a prediction loss.
177 Huntington: Oh, this is like, if you have like a
177 Huntington: not a prediction. Like, while you're training the model, you just take the loss value of the 500 token and the 50th token, you know, like
177 Huntington: subtract them to get a difference. So this was like used in entropics induction. In context, learning paper.
177 Huntington: And it was like they're using it as a way to check. How induction has it for me? Because as the model as this difference, so I'm like, reversing the way they're subtracting. But if the difference between them increases. Now, the model is like having
177 Huntington: is better at predicting when it has, like a hot, more context. Sorry. How do you calculate the loss. Ictx is equal to 50. Is it like, the few short example index or something? No like while you're like training the model you got. Give the, for example, sequence length, 1,024 thing. And like you just like, take the loss at that
177 Huntington: token like some random document. Yeah, some random documents, the loss at the 50 token. Yeah. Subtracting the loss at the 500 token. What that entails like a practical example, like, if you have 2 pairs like, what would. Where would you like prototype? I guess examples of when the loss would be high, the difference would be high, and when it would be one.
177 Huntington: Oh, could you repeat that like either, for like specific token pairs, like what would be an example of a token pair, where the loss that the loss
177 Huntington: would be similar, and when their loss would be different
177 Huntington: or like the specific token period doesn't matter. The thing that this metric is showing is when you have like, when you're at the 500 token, if you have a less, if you have like a smaller loss. It means that the model is able to predict better. We need to seize more context. That's that's like what it's. Yeah.
177 Huntington: Also, I think for the loss of the 50th token in the paper. They do like the average. The value is like the average over the entire. Whatever data set that you're trying to get. We are doing the same thing as well.
177 Huntington: and let me do this.
177 Huntington: So what they observe is that at the 1st 500 steps. The Lrds score is increasing. So meaning that the so what? How I'm reading this graph is that
177 Huntington: until 500 step the differences between the language subspaces are increasing for for French and English, and so this is averaged over several languages. But the moment
177 Huntington: that
177 Huntington: this starts to go up the phase shift. Now our Lrds score now starts to come down so we can read that as the difference between subspaces are getting lower when this happens.
177 Huntington: and for the sadds, for after 5 step 500,
177 Huntington: so from the beginning to here the increase is around 0 point 0 0 5.
177 Huntington: But after this happens at at again like 500 steps in the 1st 500 steps it
177 Huntington: the amount that it increases triples after this happens.
177 Huntington: And so also we have checked one other metric that was just from Yannixford.
177 Huntington: So
177 Huntington: If you remember this previous setup where we like train a linear probe to
177 Huntington: predicted tense of a sentence, and then
177 Huntington: I'll now use that on another language to see if it's like able to predict tense in the other language.
177 Huntington: The
177 Huntington: the accuracies also start to go up after this accuracy, start to go up after 1,000 which is, when.
177 Huntington: Now the phase shift stabilized. So one thing I want to add, this is the X-axis is log scale. So this is not as dramatic as this seems, but still, in the 1,000 step, the accuracy of the of this grammatic sentence, prediction doesn't change at all. But after the 1,000 step, now, it's able to increase.
177 Huntington: So when we looked at these things like we saw that there's at least like
177 Huntington: they're somehow happening at these similar times. And
177 Huntington: we wanted to check if there is a relation between these, the emergence of the induction nets, or increasing these in context, learning score, and our shared multilingual concept representations.
177 Huntington: And so.
David Bau: Oh, is, is was it induction score on the same graph? Or am I misreading
David Bau: these? All look like linguistic things? Oh, you said this happened at the same time as the induction score. But I I just don't see the induction score here.
177 Huntington: This one like the token last difference, I think, like.
David Bau: Oh, oh, this is oh, oh, it's on this graph. Okay, it's just on a different graph. I was looking around and I didn't see it. Okay. Great thanks.
177 Huntington: Oh, that's like.
David Bau: I see.
177 Huntington: Bye.
177 Huntington: Okay, I think I'm missing it.
177 Huntington: Yeah, whatever. So the the question that we are asking is if, since, like, they're happening at relatively similar times. If there's a relationship between them.
177 Huntington: And the hypothesis that we think that this could make sense is that a combination of parallel multilingual data and induction could cause the model to create a big drive for the shaded multilingual concept representation.
177 Huntington: So an example sentence where the existence of induction would be helpful, is as follows, if the model observes a instance like this, which is
177 Huntington: one, I think this is
177 Huntington: but the second sentence is English, so
177 Huntington: this has anchor token, so it's actually an easier setup. But the moment where induction happens, the model to induction, for example, this word, the attention pattern will copy the next token to here, and this could happen arbitrarily at arbitrary lengths, and it could, like induction, could be actually copying these synthesis from one language to another, and it could, like
177 Huntington: I drive the model to create these shared representation spaces, and it could happen at arbitrary lengths.
177 Huntington: and so it could even generalize to sentence level translations, paragraph of translations, and therefore we thought it could be helpful. And that's what we wanted to test.
177 Huntington: So did did you have I thought you had another induction score as well. Not just the token loss difference. But I don't remember the the thing that I'm calculating for the heads. Yeah, right? That also goes up around one. K, right?
177 Huntington: Yeah, I think, yeah, it goes around. I think I have some things. Okay, so this is like a different thing. But yeah, they they all differ. But like, they go go up around again. 500 and 1,000. Yeah.
177 Huntington: But yeah, yeah, yeah, I forgot to put that. Okay, yeah, also.
177 Huntington: we wanted to test if connection is.
177 Huntington: So if there's like a causal relationship between the emergence of these shared multilingual representations and the existence of these production nets.
177 Huntington: So in order to test this, I'll be.
177 Huntington: we thought, of 2 possible pathways, for now
177 Huntington: the 1st would be to reinitialize induction nets that are responsible. Yeah, the index nets. And
177 Huntington: to see how the metrics for the multilingual.
177 Huntington: the multilingual metrics would happen. So it will mean re initializing the query key value and the upper projection for a specific head.
177 Huntington: And the second thing, second thing we wanted to look at would be adding a loss function that discourages induction, to see how it would affect.
177 Huntington: So I will talk about what didn't work and what's
177 Huntington: kind of work. But still, like, there are certain issues that I want to talk about.
177 Huntington: So the 1st thing that we have tried is reinitializing the induction nets.
177 Huntington: So the way we're doing it is. So we're calculating the induction score for every attention
177 Huntington: the way we're calculating induction score for these heads.
177 Huntington: I think I have.
177 Huntington: Okay. So
177 Huntington: this is a repeated random sequence, the one that you're seeing. So the tokens, the 1st half is repeated in the second half.
177 Huntington: And the way we're calculating these induction scores, is you?
177 Huntington: Some of these attention rates here. So don't mind the process right now. This for in another slide. But you're looking at these attention rates to see if this head attends to these parts.
177 Huntington: That's how we're calculating the next scores for each end.
177 Huntington: And the way we're now doing. The reinitialization is
177 Huntington: the moment where an attention, its introduction score goes above 0 point 3
177 Huntington: we would re initialize the our weights, the attention it to see
177 Huntington: how it would affect the multilingual metrics.
177 Huntington: So when we did this, oh.
177 Huntington: we observe that the induction scored doesn't go down immediately, but it just stabilizes.
177 Huntington: Also, we think that this could happen because the you're not re initializing the atom.
177 Huntington: and it could immediately call it pull the weights back up.
177 Huntington: and therefore the score just stabilized at 0 point 3
177 Huntington: for heads that are turning into induction heads.
177 Huntington: And also when we do this, the number of induction has just jumped. So normally, the number of heads that go above 0 point 3 would be like 5 induction nets. Now, when we do this reinitialization. The number of induction nets start to increase, it goes up to 25 induction nets. So the model just tries to create new inductions. When we're doing this reinitialization.
177 Huntington: and therefore we couldn't get a oh, yeah. So
177 Huntington: we wanted to see if this actually
177 Huntington: if this stabilization makes sense. So
177 Huntington: if you've done. Sorry. Could I just ask the question you?
177 Huntington: I think I missed the explanation. So the idea is, you reinitialize, and then you don't see a drop because of Adam.
177 Huntington: But why wouldn't there be initial? There'd be like an initial drop. And then, Adam, take some time to reinitial this.
177 Huntington: I had the exact same question. So like, this is not what I've expected. I thought like it would drop. But I'm not really a hundred percent sure of the training dynamics. But and the second part, like, I've tried one more thing to see like if this makes sense.
177 Huntington: Oh.
177 Huntington: so for the second part, backup induction heads, and when you are initializing the induction heads, you have identified the backup heads. Don't fire in presence of this main induction heads. Oh, but like this is for the same induction example, attention. For like layer, one head, 9.
177 Huntington: Correct. Yeah.
177 Huntington: It's weird, like, immediately goes back to the next step. Yeah, it like, yeah, immediately goes back in the upcoming training step. Because, I'm not calculating the induction score right after I reinitialized it. So I like.
177 Huntington: there's a training step. I reinitialize the heads. Then there's another training step, and then I calculate it again. So in the next training step, it somehow like keeps the weights back up.
177 Huntington: Yeah.
177 Huntington: And like to see if that makes sense. I've also.
177 Huntington: we initialize the optimizer states that correspond to that induction net. Now, like they drop.
177 Huntington: And like, that's why I think it's not a problem with the code. But like, it actually happens. So maybe, like we initialize the optimizer states that correspond to those attention rates
177 Huntington: they drop and like. Now it like takes more time to go back up.
177 Huntington: And this like doesn't affect oh.
177 Huntington: heads that don't have like induction score. So if it's like just
177 Huntington: around this score, it doesn't
177 Huntington: cause them to go back down. But now, when we do the optimizer state it again like
177 Huntington: kind of failed, because now the number of we initialize the number of induction has jumped to 143 3, which is the number of attention it's. And so the model just like, seems like it's just looping. Maybe we initialize the optimizer state. So this
177 Huntington: set up the reinitialization didn't prove that much helpful of.
177 Huntington: And yeah. So for the second part.
177 Huntington: what we have done is we have
177 Huntington: added a loss function that discourages token copying.
177 Huntington: So the way we did this is so. We have calculated the induction score for a given induction net
177 Huntington: the way we do it is we give it random sequence like this one, and then
177 Huntington: the for each attention at and for for the, for each batch.
177 Huntington: What we're doing is we are looking at the attention rates that this model is putting. So we're basically calculating the same conduction score.
177 Huntington: And if we are computing for these sets.
177 Huntington: and we are summing that up for for each attention, that!
177 Huntington: And then we are putting that score with a hyper hyperparameter lambda to the loss function and
177 Huntington: yes. So the way we can visualize this is that so we're telling the model to not attend to
177 Huntington: the preceding token of the previous sequence. So if you have, like ABC, ABC
177 Huntington: for the token for the second, C, you're telling the model to not predict A or yeah, that's so, we're basically telling the model to not pretend to these parts with it. Induction. So one
177 Huntington: thing with this scheme is that it does not discourage the old circuit.
177 Huntington: It only discourages the second part, which is
177 Huntington: attending to the next token, and then copy trying to copy that. So it's it would only discourage
177 Huntington: those kinds of attention, so we would expect this to at least damage the
177 Huntington: emergence of these induction metrics. But you would also expect, like the model
177 Huntington: could figure out some other way to do this.
177 Huntington: And so maybe you did that.
177 Huntington: So there's like a sanity sanity check that we need to do when we do this, we, the induction scores for a given attention. They all go down to 0 because we basically are optimizing for that thing, optimizing this to go 0. But the training loss also gets a hit as well, which is expected.
177 Huntington: But the thing we wanted to see is if this difference for
177 Huntington: The loss is only caused by
177 Huntington: metrics that are related to induction tasks? Or is it? Did we just cause the model to
177 Huntington: become.
David Bau: How, how do? How do? How do we read the change in training loss on the right hand side.
177 Huntington: All the changes training loss So
177 Huntington: I think I have a clean run setup as well. The way I'm telling it is because of that. So
177 Huntington: this no, this is not the claimer.
David Bau: I see it would it would it'd be it'd be cool to grab them on the same plot, so that we can actually see, or or at least put the plots next to each other.
177 Huntington: sure. So this would be like the
177 Huntington: this would be the loss graph that is similar to doug, actual clean room.
177 Huntington: but so like at around step 5,000. The loss usually like goes to 3,
177 Huntington: but with this one, now we have
177 Huntington: the model is at like step 5,300. It's 3 3.20. So not not a big hit. But
177 Huntington: so there's a difference in this loss. And
177 Huntington: to like check causality. We have to like, make sure if this only affects
177 Huntington: induction scores related things, or if it affects the model in general.
177 Huntington: how how do you define induction related things.
177 Huntington: Yeah, that's a great question. So maybe
177 Huntington: not induction. But like copying tasks. Basically.
177 Huntington: So so this was my impression. And I might be wrong. I mean, induction seems like such a basic thing that it is kind of like a core sub circuit of them, of a lot of things. That was the impression I had. So if you, if you end up damaging the induction, has everything as a whole.
177 Huntington: probably should go down.
177 Huntington: And then yes.
David Bau: But the loss. But but the loss is going down to pretty not too far.
David Bau: 3.1. It's like pretty good.
177 Huntington: So it is.
David Bau: If you're more gentle on the lambda, can you get the Us. To go down even faster.
177 Huntington: I think so. Yes, so I haven't done a lot of hyper parameter
177 Huntington: tuning. That's true. Yeah. But that's like, I think we've been talking about that as well. So induction does cause a lot of could like cause a lot of circus to
177 Huntington: emerge. But the thing is, we are only
177 Huntington: doing this for the random sequence thing. So we're only not. You're only giving the model a signal to
177 Huntington: oh.
177 Huntington: not copy the next preceding thing. But like, if if it's like a circuit, we're not, we're messing with only the second part of the circuit, so
177 Huntington: we would hope that the damage is not that much, but yes, that's what people deserve as well. So like, that's what we are actually like, kind of struggling right now.
177 Huntington: So yeah, that's true.
Aaron Mueller: Yeah. So I think this gets one of the core things that I would really like to get feedback on from people. It's like, I think Arthur was right. That induction is kind of like a core ability that's going to damage a lot of stuff. So it's kind of trivial to conclude that you know, removing induction will also remove crosslingual alignments. And so I think the more interesting question to ask is something like, in what way is induction leveraged to Bootstrap these crosslingual alignments, and I'm not yet sure how we could start like addressing that question. But maybe something to think about.
177 Huntington: That's yeah. That's exactly what you were thinking. Actually, yeah, should be like, the
177 Huntington: main thing. I think we would want to show is that
177 Huntington: maybe, instead of like damaging induction nets.
177 Huntington: We can like show the actual mechanism we can like show actual evidence that it does off.
177 Huntington: behave in this way like it actually does copy one thing from the from the parallel sentence, and like it actually like starts to build up these engrams. Maybe that could be like another way to calculate it. But yeah.
177 Huntington: that's that's an issue. That's definitely an issue that we want to think about.
177 Huntington: Okay of.
177 Huntington: yeah. But there's 1 more thing. So my induction loss doesn't completely damage the induction scores as well.
177 Huntington: So when we look at the multilingual metrics.
177 Huntington: It like it definitely decreases. So the Lrds constantly stays at around 0 point 2 higher than the
177 Huntington: regular clean run, and the sad score this one seems to going up. But like for the
177 Huntington: Pythia with induction loss it. There's a spike at the beginning, but then it just seems to stay at a relatively at the relatively same score.
177 Huntington: But the thing is
177 Huntington: so. I'm not sure about this one. But so the way token loss difference was used in the anthropic paper was like, I need to see if induction is for emerging.
177 Huntington: So the token loss difference, although, like it, gets damaged, it does go up at some point.
177 Huntington: but I'm not sure if this is directly related to
177 Huntington: the induction heads emerging or not.
177 Huntington: So, therefore, like I have checked some other metrics as well as a sanity check. Oh.
177 Huntington: and so, for example, these are common benchmarks, the lambda.
177 Huntington: the arc, and this is, I think, I got it from Eric's the truth, first, st of 3 accuracy. So I have some things I want to discuss with this as well. So these 2 are seemingly
177 Huntington: tasks that do not require that kind of copying at least the way you're like we're damaging it. But they get damaged as well.
177 Huntington: But we see there they start to increase that after some point. So it seems like we
177 Huntington: my, the way I'm adding the induction loss
177 Huntington: did cause the model to
177 Huntington: It caused the model to
177 Huntington: stop, or or at least like delay, some of the fundamental things that it's trying to learn. But then it it seems like it's somehow figured out some other way.
177 Huntington: because.
David Bau: So these are the. There are 4 tasks here. Is that right? What are the 4 tasks, or the 3 tasks.
177 Huntington: Yeah, these are treated. So this is the lambda task. Let me try to remember the tasks.
177 Huntington: yeah. So the only thing I remember is that one of them is
177 Huntington: basically a question answering thing. I think it was the arc Aaron, I think, remembers it also. Please correct me. If I'm wrong.
177 Huntington: it's arc. You're giving a scientific question, let's say, and there are 4 answers to that question.
177 Huntington: or you calculate the you calculate,
177 Huntington: the accuracy on those 4 choices to see how how about the model performs on this scientific question thing?
177 Huntington: So this doesn't seem like it's a copying task, and it also gets damaged
177 Huntington: the accuracies do go down. But
177 Huntington: so for the lambda tasks.
177 Huntington: What was the lambda task?
177 Huntington: Can I quickly check with the lamb to test these sometime.
177 Huntington: I don't remember.
177 Huntington: It looks like it's like sentence close. So you just give it some sentence, and then you ask it to break some token.
177 Huntington: Yeah, yeah. Okay, so okay, the arc definitely doesn't seem like it is related to token copying lambda
177 Huntington: could be because, with the lambda you're giving the model a big paragraph, and the paragraph has, like some
177 Huntington: name of people. And
177 Huntington: so and then at the end, you only leave the last word. And what is argued is that humans are able to predict that last word. But so we they are also calculating how models are doing
177 Huntington: so this might have this might have some relevance to the token copying, or at least being able to attend to
177 Huntington: those previous things. So they get damaged.
177 Huntington: The thing I wanted to. Kind of discusses that the
177 Huntington: have this choose 1st of 3 accuracy thing that you've gotten from Eric. This is a in context. It's an Icl task. Right?
177 Huntington: So I'm I'm
177 Huntington: I only realized this while I was preparing the slides. I'm not sure if this is like.
177 Huntington: if, like looking at in context, learning is a
177 Huntington: good way to check. If the model is able to do that token, copying or not, it's it's more
177 Huntington: it's more complex.
177 Huntington: But still, maybe oh, we should just look at more plain things other than Icl.
177 Huntington: But at least the way that the
177 Huntington: understood is that the model is
177 Huntington: the model does suffer a lot in the 1st couple 1,000 steps, but after some point it starts to go up.
177 Huntington: So it seems like
177 Huntington: My addition of the induction loss did not cause the model to
177 Huntington: did not cause it to completely.
177 Huntington: not completely, not being able to do induction, but it kind of I think it kind of figured after some point to trigger induction loss, because
177 Huntington: to be. We're adding, the induction loss is
177 Huntington: probably not capturing everything. Can I ask something? Sure. So you are. You are applying this induction loss
177 Huntington: only at the second head. Right? Oh, yeah. Okay.
177 Huntington: Oh.
177 Huntington: okay. So this this is not really question, like, it might be interesting, but probably not very useful. Like to find out how the model finds ways to trick that like. Probably it distributes that load across multiple heads, says that one injection score of one head does not go up, but it can still somehow implement this.
177 Huntington: using maybe 5 heads instead of one hand. Yeah, that's what David kind of suggested as well. I think at some point like the way that the model, being able to figure out some other way could be interesting. So actually, I have some results of attention patterns on both these things. Maybe I can show that so they could be like more helpful.
177 Huntington: But
David Bau: So, Karim. So there's so this is I. I think these are quite interesting. And I I know that
David Bau: you know the the project is motivated by the multilingual a question.
David Bau: But but there's this very broad claim
David Bau: that's just out there. You know that. You know the the anthropic people have hypothesized this claim.
David Bau: but also the Harvard people have hypothesized it, and many countless other papers.
David Bau: if hypothesized it. That says, You know this induction copying.
David Bau: you know, process is instrumental to a broad range of capabilities.
David Bau: And I and I think that that in itself is a pretty darn interesting paper
David Bau: or a darn interesting question,
David Bau: and and this. So if you back up to the slide, so you're about to show it one more thing. So maybe before we have a discussion. Maybe I'll let you finish here
David Bau: your your, the thing that you want to present. But I but I just wanna comment that you're sort of looking at this. All these other damage tasks as a negative
David Bau: right now. But it they seem interesting, and the only negative seems like
David Bau: somehow we need some better controls
David Bau: over, you know, some other different ways of
David Bau: stopping induction, or something like that, like like, if but but you know, this this broad damage seems to be, you know, largely consistent with
David Bau: this hypothesis, this unproven hypothesis that people have been making out there.
David Bau: It it seems like it would be interesting to go ahead, and, you know, confirm the hypothesis is just. The only only question is like.
David Bau: How do how the heck do we control this experiment.
David Bau: Let me let me let you finish your slides, though, before.
177 Huntington: Thanks also. I think, yeah, the that's that's like.
177 Huntington: definitely it seems like just a minor part of the paper. But there's a
177 Huntington: interesting thing that there are like a lot of interesting direction that we can go with this one. So yes, we have to like.
177 Huntington: said, and take a direction. But yeah, we can definitely talk about it afterwards.
177 Huntington: So the thing that I can like show the attention patterns.
177 Huntington: Oh, so this is on a random sequence that we give to the model the repeated random sequence.
177 Huntington: This is just the attention patterns on a random sequence of a clean run model. So it everything like behave. This is the regular behavior. Some of the tokens are just attending themselves.
177 Huntington: Some heads such as these ones are just induction heads.
177 Huntington: There are other random things
177 Huntington: but when we added the production loss function on the random sequence.
177 Huntington: now the there's like a weird behavior that on this random thing
177 Huntington: nearly a lot a lot of the tokens and a lot of the heads are just are either attended to themselves.
177 Huntington: or they are attending, attending to their previous themselves. So like the induction that would attend to these parts.
177 Huntington: But when we add the loss function.
177 Huntington: they start to attend to themselves.
177 Huntington: And a lot of has just become, has that behave that way that couldn't figure out why? Because it seems weird that the model is
177 Huntington: just attending to. Each head just attends to themselves, maybe just figured out a way to do that, copying by
177 Huntington: up
177 Huntington: writing the value to its own previous self. I don't know. But yeah, that's like that was the potential pattern. I don't know if it like.
177 Huntington: oh, make sense or clear some things up. But yeah, this it causes the model to do some
177 Huntington: other kind of attention patterns. And oh, yeah, it's able to.
David Bau: Whoa!
David Bau: It's very weird. Can you flip back to the so the healthy attention head looks like what.
177 Huntington: The healthy, healthy induction. It looks like this one.
177 Huntington: And yeah, the healthy ones. They're they're like, very.
David Bau: And and so and so and so that diagonal shifted over to the right by one as as it should be.
177 Huntington: No.
David Bau: And here it's not shifted over to the right.
177 Huntington: Yeah.
177 Huntington: Hmm, it's attending to itself. It's attending to itself. Yeah, that's it's attending to previous self.
177 Huntington: But the but the ones that look like diagonals. Are they really an off diagonal?
177 Huntington: Oh, these were like head 5, 7, for example, head 5. So leave
177 Huntington: Layer 2 had 5. This one no like Layer 5 had 7 like that one
177 Huntington: like, is it an off diagonal, or is it actually diagonal like, I've checked them one by one and
177 Huntington: oh, you're asking if it's like this, or like this, like, if it's attending to, it's that.
David Bau: Token pretty gets token.
177 Huntington: Instead of itself.
177 Huntington: for example. Oh, yeah, no. Like, I've checked them one by one all these. So this one
177 Huntington: it tends to itself.
177 Huntington: Also, I've checked them manually one by one. There's like, maybe it's just like my bad eyes.
177 Huntington: I can like open the helpful thing as well. Maybe let me just pull up your slides. This is all the heads in the model, right
177 Huntington: between the ones that previously had diagonal and the ones staying the same as single diagonal. Oh, we can check. Sure. Yeah. So this one's like a off diagonal. But yeah, this one's
177 Huntington: a diagonal.
177 Huntington: So the shifted diagonal. It did not change position, or it did. Can you go back to the healthy pattern, please? This is
177 Huntington: the tokens are not meaningful like 3, 3, 7, 0 2. Oh, yeah, the tokens are not meaningful. These are not meaningful. Yeah, these, these are just the token ids. And they're they're not meaningful. Okay, okay?
177 Huntington: And this pattern it did not shift but introduce it introduced another pattern
177 Huntington: another kind of like ray of pattern in the diagonal. Is that what you're saying? What I'm saying is, so this is the pattern of an induction. Yeah.
177 Huntington: For the damaged one. This diagonal, this one.
177 Huntington: Okay, not not like actual diagonal of the matrix, but like this one switch to here.
177 Huntington: But the oh, yeah, that's Daniel to H.
David Bau: Need about 15 min more, not not too much longer. But if is it possible to give me half an hour.
177 Huntington: Oh! What!
David Bau: Okay. Great. Thanks.
David Bau: Thanks.
177 Huntington: Yeah.
David Bau: Great. Great. My name is just right over great thanks.
177 Huntington: I'm just wondering if the model has found a way to like, just look at itself as a register or something.
177 Huntington: Yeah, just just to find workarounds around your induction score loss. Yeah, I guess like that. That's what
177 Huntington: could happen. Because the model can do these tasks now, like, although it's like Lower School. I guess I get.
177 Huntington: instead of like looking at the actual token, maybe just writes this value to here, and which is it? Tends to itself like figures out. But yeah.
177 Huntington: so that was like the thing that we've realized.
177 Huntington: And other than that, we.
177 Huntington: So what we can do is we could like still go on with the adjusting, the loss function. Thing.
177 Huntington: There have been some things grading based approaches there, Aaron suggested to
177 Huntington: attribute like if it actually, if they're like.
177 Huntington: correlated the the multilingual metrics and the next units. And if it's like the
177 Huntington: if connection nets are the highest contributors, or one thing that I want to try with this is to get a model
177 Huntington: get 2 different data sets from with different alphabets, such as one of them is in Latin, and one of them is, Ask Greek alphabet, and then have true training models.
177 Huntington: In one of them the model
177 Huntington: sees some of the languages in the same example, but in the second training the model never sees them in the same instance. Now we're doing this, because if induction has an effect.
177 Huntington: it would have an effect in context, and if the model is able to figure out how to get these shared representations, and by not ever seeing them. In the same instance, then like that, would be that will definitely falsify our hypothesis. But if the model is only able to form these shared representations when it sees them in context that would be a the but still
177 Huntington: a positive for our hypothesis. So just like what we wanted to try potentially.
177 Huntington: But of yeah, that's that's all I have to say. I think,
177 Huntington: yeah, thank you. Take a listen.
David Bau: Excellent, excellent.
177 Huntington: Can I ask a silly question? So
177 Huntington: can you remind me what? The what? The question you were investigating like? So there are these shared multilingual concept representations that are happening in models. And
177 Huntington: we wanted to see what components in the training dynamics are.
177 Huntington: what drive, what drive? Basically, the big question is, what drives this straight concept? Representations like forming the enabling the model to learn this shared representation? Yeah, is it like just gradient descent? Is it like is there like a specific component that helps it? Like.
177 Huntington: we believe that there's like there could be like these.
David Bau: And and when you drop out induction heads these they they do kind of emerge right? What is the graph when induction heads of these 2. Do you have these 2 metrics graphed.
177 Huntington: Oh, yeah, so
177 Huntington: yes. 1st of all, so yeah, it does. It does emerge. But yeah, they suffer. But yeah, that's like, there's, it's not like they stay at 0.
177 Huntington: So yeah, there, there's like an emergence. But yeah, of course. So
177 Huntington: there are a lot of things that could allow this to happen. So
177 Huntington: yeah, we didn't expect this to go 0. But yeah, maybe it would be
177 Huntington: like, maybe we would expect it to be a bit lower
177 Huntington: like. So it didn't suffer as much as we thought. But still there's a suffering of the multilingual metrics.
177 Huntington: Yeah. So a lot of things could be contributing because of
177 Huntington: what Chris was talking in the beginning of the project is that like even just grading descent because of the way optimization works, it's just like it's the
177 Huntington: to encode them in. The same space is just like a huge leverage for the model. So you just like plain, gradient descent. Without these induction mechanisms
177 Huntington: could could be enough.
177 Huntington: And the induction on all this stuff might not again be that important. But
177 Huntington: so like we see that it's suffering. So I think, like, I believe it's
177 Huntington: it's like suggesting at least that there's like some interaction for sure. But yeah.
177 Huntington: I wonder if like, you just hard coded the induction head
177 Huntington: into like you had some set of hard coded induction heads that you didn't actually have to like.
177 Huntington: Learn that functionality.
177 Huntington: But then you like froze the weights in some way that would help you like pinpoint.
177 Huntington: What part of this head needs to be learned for these like multilingual things to converge. Yeah, actually, like, freeze the keys or the values. Yeah, that's or something like that. Yeah, I'm happy that you like talked about this at some point like one of them
177 Huntington: things I I did like wanted to freeze the models, but
177 Huntington: I wasn't sure about the exact setup like because so the thing that I've thought about was I could like freeze the attention. It's before inductions immersed and freeze the attention. It's after inductions immersed like just right around step
177 Huntington: 1,000 versus step 800 like, just freeze the attention. But let the model, for example, put to your training to see like if
177 Huntington: they would still create these multilingual metrics. But, like
177 Huntington: I'm I'm not sure if
177 Huntington: seems like we're just freezing them at a previous point.
177 Huntington: Is it like the same thing? You'd be just like not letting the model to get better in terms of the attention? It's so like. That's 1 thing that I couldn't figure out. But
177 Huntington: Do you think like, what? What could we do to follow
177 Huntington: to like overcome with that issue? Or is it an issue? I'm not sure about that one.
177 Huntington: So I guess I'm trying to understand, like what the what the hypothesis is around like. Why, you think induction heads are linked to like the convergence of multilingual representations.
177 Huntington: like, yeah. Is there? Is there a specific way you could phrase that
177 Huntington: of like, maybe maybe the way to see it is like
177 Huntington: once the model like learns how to do literal copying, then it can like align
177 Huntington: stuff in such a way that the representations converge across languages.
177 Huntington: But like, if you just
177 Huntington: forced it to do literal copying without having to learn it.
177 Huntington: Maybe from step 0, like, okay, yeah, maybe
177 Huntington: that could like, definitely try that 1. 0, I don't know.
David Bau: So so so is. So I I've got. I like that hypothesis. It's it's cool, so is is here.
David Bau: If if you you stop your graphs. All at 5,000 iterations is is that?
David Bau: Do you feel like? Oh, that's that's a good reason to stop there, or do you like, you know, when, like the multilingual score is kind of low here on the right. This add score, but.
177 Huntington: Yeah, this.
David Bau: Do you? Do you wonder if you like? Go to 50,000 iterations if it'll catch up or.
177 Huntington: Oh, yeah, yeah. The so for this one, I think I definitely need to train it more so for the regular run when we were when we were not training it on our own model. But we were just like getting checkpoints from hugging face. We did look at all the checkpoints like the 500,000 checkpoint the 50,000 checkpoint. The reason
177 Huntington: I was training until 5,000
177 Huntington: is like both training takes time, and also until step 5,000. We could like see that phase shift
177 Huntington: or like. We can see the major difference in the 1st 5,000 steps. So yeah. But for the
177 Huntington: for the induction. Last thing, I think that maybe if I did training for more steps it could be helpful to check. If if it's ever gonna catch up. Yeah, yeah, I can.
David Bau: I think, because it might just be it might just be delayed. Right? Let's say 1. 1 hypothesis is that
David Bau: it? It might be delayed the
David Bau: you know. I I feel like you've got different hypotheses here, right? It's like one is.
David Bau: are the injection heads part of what you need for the multilingual thing? Or is it part of learning?
David Bau: And and and if if it's part of learning, there's this question, is it like a temporary part of learning
David Bau: like. So, for example, you could try different things like you could say, oh, you know what? We'll let it do the face shift, and we'll let it learn this multilingual thing, and then after that, we'll hit hit it with his loss to make to make it forget how to do induction.
David Bau: So it doesn't know how to do that anymore.
David Bau: But maybe that's too late to make it forget the multilingual thing. Maybe you don't need induction
David Bau: to do the multilingual representation. You just need it to learn it right?
177 Huntington: Sure.
David Bau: You know, so you could do different things like this.
David Bau: do do you think that?
David Bau: So? I I think that
David Bau: like Sheridan, had a slightly
David Bau: unresolved investigation of training dynamics in her dual route paper.
David Bau: right? Which is clearly it's like related to multilingual representations right? Where she says, well, it it sort of looks like some of these dual route induction heads, you know, this concept induction has sort of maybe come from
David Bau: regular induction heads, and some of them don't. It's sort of unclear.
David Bau: right? But you you might be able to settle that question if if you.
David Bau: if you like, you know, prevent the induction heads from showing up ever
David Bau: and then show over a long, long period of time. You never get these concept induction heads, but maybe if you prevent.
David Bau: if you allow induction heads for a while and then you kill them off later. Maybe it doesn't matter, because now a lot of them have turned into concept induction heads, and once it has concept induction heads you don't need.
David Bau: you know, literal token induction. That's those are the multilingual ones. Those. You know, your loss doesn't kill those. And
David Bau: you know they're gonna survive. And that's all you need.
David Bau: Right?
David Bau: yeah, I I think that would be pretty interesting, right like, if you could come up with some conditions to
David Bau: to show that I that that might be that might be pretty clean effect.
David Bau: You. You have this, you have these, you have these patterns that you showed that you're concerned about about that. Maybe your induction loss is doing something weird for the random text patterns, you know, in the next slide, or something like that or previous slide. Yeah, I'm not. I'm not too worried about the weird patterns, because you're only testing on random text, right? Like, if you put right. If you put like natural texting in here, you'll have, like lots of natural
David Bau: natural patterns. Right? It's it's only for this, like weird, random text. Random repeated text. You have these weird patterns. And
David Bau: I think it's okay.
David Bau: It's like, this is out of distribution, right?
177 Huntington: Yeah, like. The only thing that's weird is like this is also a random sequence. But like behaves
177 Huntington: more natural with the clean run. It just doesn't behave. It's just like, bh, but yeah, we're directly optimizing the model to not do this. So I guess it's not surprising some weird things happen with the random sequence.
David Bau: Yeah, yeah, this is this is you've you've brought this into into the training data. Now.
David Bau: right? The normal training data doesn't have this.
177 Huntington: Yeah.
177 Huntington: Oh, yeah. True.
David Bau: And so right? Right? So it's it's not too surprising that you've created some effect on this type of data.
David Bau: you know, you might. You might test it on instead of on random sequences. You could test it on, you know. Natural, repeated text, or something like that. Not not really sure what. But I'm I'm not too worried. I I think this is, it's okay. I mean, it's, you know, as long as if. But if but if you
David Bau: if you said, Oh, here, I'll just take attention on our representative pieces of of natural text.
David Bau: and it looks very different. That would be more concerning yeah.
David Bau: I I love the setup. I love this. i i i think that I think it's cool. I I don't.
David Bau: I don't see a huge concern of it, like, you know.
David Bau: I feel like do you have one way of knocking out induction? It seems
David Bau: to work really nicely, but I feel like so have. I'm sorry I had a couple of phone calls the hotels like kicking me out of my room. The
David Bau: But so I missed. I missed some of the conversation. Have have you guys like suggested to program different ways of
David Bau: knocking out induction? Besides, besides this loss that he's using.
177 Huntington: Yeah, so Eric had. Did you like catch what? Eric suggested.
David Bau: No, no, but I'll I feel like that's the main thing that is needed. You you maybe you need that 2 or 3 ways of
David Bau: trying to knock out induction, and and then just compare them. And so that, you know.
David Bau: like a reviewer would look at this and say, Well, I don't know if I were knocking out induction I would do some other thing.
David Bau: But but if you if you have like 2 or 3 ways, and maybe you'll capture all of the different things that a reviewer would imagine would be the thing that they do. You've already done a couple. You've you've done this random initialization. You can tell people well, that doesn't work, but but you know. But you can have a few other ways, and
David Bau: and maybe more than one of them work right. Maybe some of them work better than others. But you can kind of just you can map that out a little bit.
David Bau: I think it's cool.
David Bau: I it's very neat.
177 Huntington: David, do you think you're more interested in the angle of what happens to a model without induction rather than the multilingual focus.
David Bau: I'm I'm a little bit. But the multilingual focus is a really nice lens. So I I don't want to dissuade you from it either. Right? I I think that I think that, like what happens to the model during induction
David Bau: is of like this really fundamental
David Bau: import to all sorts of like, you know what happened to what happens in context, learning what happens to other types of reasoning, what happens to it's sort of almost anything
David Bau: in the model. But but I think that, like, I really like
David Bau: this stitching together with your story of the dual track where
David Bau: it looks like. There's this intimate relationship between induction and multilingual.
David Bau: You know, representations which is just. It's just this beautiful thing. And and you could pick up that narrative.
David Bau: Yeah, yeah.
177 Huntington: Oh, yeah, I think the other interesting thing is like, I. I like
177 Huntington: the the example that Karen had of, like the parallel texts of like this is actually why induction could help a model learn something? Because it if it sees these sequences, then it's going to make these mistakes. Because, like I feel like when people are like, Oh, induction helps the model do things. It's not exactly clear, but like this is the 1st example I I've seen which we talk we were talking about of like, why, induction would actually like help. A model like learn a specific capability.
177 Huntington: So maybe, like the multilingual thing is a good like case study or something to focus on, but more the broader
177 Huntington: you know.
David Bau: That's that's exactly how the Rosetta Stone worked right. It's like the Rosetta Stone. It took them a while to decode it, but it was anchored in all the
David Bau: proper nouns, because the proper nouns were pronounced the same in Greek and Egyptian, and
David Bau: then it allowed them to see that these were the same words, and then they could use that
David Bau: to to show where their induction head was failing. You know it's it's all you know. We needed multilingual induction had to like, do the rest.
David Bau: Yeah, yeah, it's exactly it. Yeah.
David Bau: yeah, you could. We could. You could tell that story.
177 Huntington: Yeah.
David Bau: And but so you could tell that story. But then you have this interesting
David Bau: like, you might be able to tell the story, but then also
David Bau: quantitatively like demonstrated by like knocking out the induction heads. Later, after after they've been used to
David Bau: trying the second route. Oh.
177 Huntington: More.
David Bau: Oh, Consenza has a question, Consenza. Sorry I'm I'm I'm paralleling on too much, Consenza. What's your question?
constanza fierro: No, it's fine. I was just thinking that maybe we it could also make sense to do the opposite of like
constanza fierro: adding to the loss like, so that the language
constanza fierro: stay far away from each other, so that we really like are optimizing against Ldr.
constanza fierro: the Lda score and then see what happens with the induction heads like, does this create more induction heads? Because now it needs a per language, induction heads.
constanza fierro: And maybe we could draw some conclusion there, but maybe it doesn't and then we don't.
constanza fierro: We can't see much, but maybe it's worth it.
177 Huntington: Oh, I guess another thing. I thought Eric was saying this, but then I'm not sure if he said this, but like what if you just make the model start out with induction heads
177 Huntington: right away, and then when you see everything goes up faster.
Aaron Mueller: Into like subtracting the induction loss term from the loss function, like basically encouraging it.
177 Huntington: Oh, yeah.
David Bau: Yeah, unfortunately, that's.
Aaron Mueller: You can.
David Bau: Do anything other than induction heads.
David Bau: Yeah, exactly.
David Bau: How? How long does it take to do this? 5,000 training run.
177 Huntington: Oh, like. So right now, I'm running them on a smaller model. So it's 160 million parameters. Also, it's like a couple hours, 2, 2, 3 HI think.
David Bau: Oh, oh, that's that's great! So you you could. You could really fiddle with, like, you know, in a in a few days you could fiddle with a bunch of different loss. Ideas. You know the kinds of things that Constanza is talking about, the kind of things that Aaron's talking about, or whatever right you could.
David Bau: you could fiddle this like this. This, this loss here you have is very fast
David Bau: now, like like in in
David Bau: in Sheridan's paper when she defines her induct her new wave induction score. I don't know if you have a name for that.
David Bau: Share it in your your intervention. Based induction, score.
177 Huntington: Oh, yeah, I think we were just calling it the
177 Huntington: causal score. Actually, I don't remember what we called it.
David Bau: Should be able to. Yeah.
177 Huntington: Concept, copy. Score, yeah.
David Bau: Okay, yeah. So like, I like, I wonder if.
David Bau: like, you know, to to this different possible losses, like, you know, a causal loss
David Bau: like like what Sharon is set up there and then the other. Funny one is.
David Bau: since the causal loss allows you to target the second dual route.
David Bau: You could you could prevent the dual route?
David Bau: You know, induction heads from showing up potentially.
David Bau: Yeah, I'm not sure but it seems like you could. You could fashion, a loss for that kind of thing, too.
177 Huntington: Right. I think we were talking about that cause cause the the hard part is like to calculate the causal losses you have to like, run the model
177 Huntington: like
177 Huntington: a million times to do, like all the patching, to calculate losses, or to calculate the score for each head.
177 Huntington: and then I don't know if it makes sense to do that while you're training the model or not, I think, is, why can I've never, never ended up doing it.
David Bau: Oh, I see, I see! So it's a per head. It's a per head score.
177 Huntington: Yeah, yeah, just kind of annoying to if you want to use it as like a training objective.
David Bau: Alright, right, that makes sense. You could. You could do it stochastically.
David Bau: But yeah, it would be a week
David Bau: like, pick a random head, and then do it in.
177 Huntington: Oh, yeah.
177 Huntington: or like, maybe look at attention scores, because the attention scores when you looked at them like they were sort of sort of correlated with the causal scores so like, maybe do it.
177 Huntington: I don't know.
177 Huntington: Based on that.
177 Huntington: You can.
177 Huntington: Christmas.
David Bau: I see. Yeah, yeah, you could do it stochastically. You could draw, based on the attention score or something like that.
David Bau: Yeah, yeah, it's good, reusable.
177 Huntington: Yeah, something.
David Bau: Yeah, some weird thing I have to think about it.
David Bau: Interesting.
David Bau: But I I yeah, I'm not. So that you know, it's it's gonna cause damage. It's gonna cause damage all over the place.
David Bau: But that's sort of the interesting thing. What damage does it cause?
David Bau: And I guess so. So it does. It seems like the challenge is
David Bau: when you cause too much damage.
David Bau: Then you get into this domain where you're just kicking the model in the head.
David Bau: and it's just not interesting.
David Bau: And so you need to figure out how to
David Bau: like gently cause damage to see what damage is like.
David Bau: fundamental and what's not fundamental. What role it is May. So maybe this temporary thing is a thing. So maybe maybe maybe this idea of like scheduling.
David Bau: scheduling, the removal, the
David Bau: the induction heads for certain periods of time. You could. You could withhold them at the beginning, and then allow them later, or
David Bau: with all the later, and then measure, or something like that.
177 Huntington: Cause cause right now you just do it for the entire run. Right? You know. Anti induction head box. Oh, yeah, yeah, it's gonna be there for the entire training. Yeah.
177 Huntington: yeah, maybe. Yeah, I think so.
177 Huntington: like, we could either like, just copy some deduction heads to step 0. Or we can like.
177 Huntington: just re do the minus loss. So they just like encourages induction and like, see how it happens. But like.
177 Huntington: Oh, yeah, just doing it at at different steps could be good as well.
177 Huntington: yeah, we we can try all these ones. I think it would be helpful to answer the question.
177 Huntington: Wonderful.
David Bau: Now you you showed some tasks where you showed a bunch of damage. Did you happen to measure any tasks where there's not any particular damage.
177 Huntington: Oh, it's
177 Huntington: these 3 were the only tasks that I have checked sadly. Yeah, I can increase the number of tasks I I was trying to find certain tasks that wouldn't like
177 Huntington: benefit from like, at least doesn't seem like it benefits from induction.
177 Huntington: Oh, yeah, like, right now I didn't see anything that like just kept being the same thing.
Aaron Mueller: Yeah. So I think we talked about including tasks that don't really require icl abilities. Possibly blimp
Aaron Mueller: because we know that like pretty small scale models that don't have induction can do subject to agreement or things like it pretty well.
Aaron Mueller: So we could check the performance. Tests like that is preserved.
David Bau: Cool cool. That's a good idea.
David Bau: Yes, the fundamental multi monolingual grammar tasks.
David Bau: Yeah, that's nice.
Aaron Mueller: Yeah, or even something that's sort of actually, things like Lambada are pretty good for this setting, because they are just testing like generic language, modeling abilities. But I guess Lambada is actually kind of
Aaron Mueller: like, it has a lot of very context, sensitive things, and I guess we could consider like a much easier version of that task where we just curate some very, very simple inputs where, even like a super small scale model should get things mostly right, and then just compute something like a token, accuracy, or probability, weighted accuracy, or something like it.
David Bau: Yeah, yeah.
Aaron Mueller: But so do people think that it's like worth investigating? Both of these research questions, mainly, like, you know, given a single checkpoint like, how is induction used in multilingual contexts versus like, how is induction leveraged during the training process? Or should we focus maybe more on like the latter question.
David Bau: I I think I think there's an interesting contrast between those like, in other words,
David Bau: like it. It seems like a reasonable hypothesis, and I don't know you guys can disagree with me. I I think that the reasonable hypothesis sounds like what you guys are advancing is that
David Bau: induction is needed during the training process, even though it's not literally used in these multilingual translation tasks. It obviously can't just do token copying. It has to do this language, independent thing.
David Bau: but that that your hypothesis is, it's important, for in the process of learning
David Bau: how to do it multilingual, you'd have to learn induction first.st
David Bau: But then but then, after you've learned it, maybe you don't need it.
David Bau: And so there's this, there's this potentially interesting contrast of like.
David Bau: if, after you've learned the multilingual capability.
David Bau: And then you knock out attention heads later.
David Bau: Maybe the model is fine
David Bau: and it can still do a bunch of these multilingual tasks. Okay? And the representations are good and all that stuff. But if you never. But if you took the same.
David Bau: you know 20,000 training iterations, and you just never let it have induction heads, or.
David Bau: or, if you did, the opposite.
David Bau: and and and a lot of having induction heads in the last 5,000 iterations, instead of the 1st 5,000 or something. You know that that you know at that point it's too late, you know, it doesn't help anymore. Yeah, it, you know it's not. It doesn't need it, but
David Bau: but but it it really needed it for
David Bau: for for learning, for for as part of the curriculum, I think that'd be.
177 Huntington: Dude. Your voice is getting.
David Bau: The cool story.
177 Huntington: Sure.
David Bau: Oh, sorry. Sorry.
David Bau: Yeah, sorry about that. But, Aaron, you heard me right. It's it's it's just like, Oh, if you have 2 different schedules.
David Bau: one.
177 Huntington: Okay.
177 Huntington: One where?
177 Huntington: Oh, it.
David Bau: Okay. Yeah.
177 Huntington: Okay.
David Bau: Think it's on their end.
David Bau: Are you able to hear me?
David Bau: Yeah, something wrong.
David Bau: It's zoom.
177 Huntington: Maybe it's hear completely what David was saying.
177 Huntington: If you did, then it's definitely our end.
177 Huntington: Okay? Sorry.
David Bau: Okay.
177 Huntington: We have them.
David Bau: Okay.
177 Huntington: No, no.
177 Huntington: Can you hear us?
177 Huntington: Okay, David?
Aaron Mueller: We can hear you, but maybe.
177 Huntington: For that.
Aaron Mueller: Speakers there, off.
177 Huntington: Okay, David, saying, I'm just saying that if you have 2 different schedules for the induction head knockout, and they show different behaviors.
177 Huntington: dot. Dot. I don't need seconds.
Aaron Mueller: Yeah, this has, like a very, very similar flavor to that Angelica Chen paper and Naomi Zapper paper from a couple years back, because, like, you can basically define a loss term that encourages or discourages the phenomenon you're studying. And then you can try like injecting that new loss term at various points throughout the training process. Like you can start adding in that extra loss at like step, 3,000 or something, or you can like subtract it. Starting from different points to encourage or discourage the rise of like these components.
Aaron Mueller: and then you can sort of see how that.
David Bau: We should.
Aaron Mueller: Multilingual stuff. It's basically the same idea. I think, actually.
David Bau: Oh, that's cool! And and did they already? Did they already do induction heads in that? In that in that paper.
Aaron Mueller: So in that paper they were concerned with the correlation between attention, weights, and dependency tree structures. So they were studying.
David Bau: That's fantastic.
Aaron Mueller: For an attention heads, but they didn't do any.
David Bau: I see.
Aaron Mueller: No.
David Bau: Yeah, it actually is simple. Yeah, maybe it's worth another look. But yeah, definitely worth reading. Graham, I look. I don't remember that paper. Graham, have you? Have you read that.
177 Huntington: I haven't read it, but I think Aaron sent it. Yeah, I I'll read it. Oh, is it the syntax acquisition? What was that? The name Aaron.
Aaron Mueller: Yeah, 7 drops and loss.
177 Huntington: Okay.
David Bau: Yeah, 7 drops in the last. I remember that paper
David Bau: cool. This is great.
David Bau: I don't know. I I think there's wonderful. I think it's on a good path. You guys.
177 Huntington: Thank you, David.
177 Huntington: Okay, okay. So yes, I think I
177 Huntington: don't have any, did I? Well, oh, okay.
177 Huntington: yes, that's about it. Thanks. Everyone for listening. Thanks for all the feedback. Thank you.
David Bau: Okay.
Aaron Mueller: Thanks. Everyone.
constanza fierro: Thank you.
constanza fierro: Nice.