David Bau: Okay?
David Bau: And then so so we have. Hey, Archie, come on, there's a there's still a chair here.
David Bau: You can introduce yourself before we get started here, I'm going to stop the recording here, and we'll start again.
David Bau: Okay, so so Koena is a 3rd year. Phd, student in my lab.
David Bau: She is very interested in problems where you treat models sort of like data
David Bau: instead of model instead of just the data as data, right? So what happens if you have a lot of different models?
David Bau: She wrote a paper called Model Lakes, where she thinks about different problems that you might want to solve. If you have this problem where you have tons and tons of different models. And so one of the issues that she's been chasing is what happens if you have
David Bau: pairs and models
David Bau: like a fine tune model and a base model. You know, what can you? But somebody doesn't do a good job at documenting what they did.
David Bau: Then what can you learn? So this is this is a theme of for talk, or I don't know. I I imagine this will be the theme of talk, or at least it'll be the theme of
David Bau: the entire general spectrum of what I guess I do. But, like again, I feel like most research projects depending on like what you get here and there. And so this is, this is a talk. This is very much work in progress talk where she's so you know where we? So we do. A lot of like finished research talks in this seminar normally. But then for our own students.
David Bau: We're just encouraging people to just say, Okay, here's my half done thing. Give me some advice. So this is the talk that's sort of in that in that spirit. So everybody thank you to.
David Bau: Okay, well, yeah. So the title of my talk is recovering sensitive, fine-tuning data via gradients. And so to kind of just start off with, of course, like an AI generated image. But like the idea is like what we have this like sensitive data in like the wild and like, what do I kind of mean by that? Like we have on one side, like the input problem, where, like, you have the system, Prom or like
David Bau: the user input that input it prompt. That may be used for like training for, like other future iterations of the model. Or you just have the training data itself. And like part one of that is like the pre-trained data, and like another is like, if you're continuing to find tune, then you have a specific like, fine-tuned data. And you know, people have been concerned about like these leakage of like sensitive data, both in terms of like input. From
David Bau: and then as well as like with respect to like training data. So there have been tons of works. But like, what if we don't really know what we're looking for like this. Most of the previous work has been like, okay, we care about like this category of data. We're gonna try to like, find ways to like, prompt and like, break it, or like, try to like, say something that we care about sort of thing. But like, what if we don't even know what that sensitive data would look like.
David Bau: And so that's kind of generally the theme that I'm trying to like get into like based on that. And so like, given a fine base and a fine-tuned model, can we recover sensitive data that the latter was trained on so sensitive data could basically either be like personal identifying information or like, it could also kind of be
David Bau: per. Okay, this is not like sensitive, but it's like persuasive data. Where, like, you know, let's say, there was a model that was trained like, Okay, at the end, I want you to convince this person to buy this product. Let's say then, like, it's written like, you know, it's trained to like, kind of say that to you when it's not really like, basically like biasing you to like, do get into that. But for now, I think, like, I'm focusing on sensitive data cause, that's like more
David Bau: like viable to broad categories of sensitive data. Sure. Yeah, yeah, so like, I mean, like, email, id security numbers and things like that. But essentially anything personal, identifying. But again, like right now. So it's trying to go back to this thing of like, how? What do we like? What should we look for when we don't know sort of thing. So like sensitive data can also be
David Bau: just as broad in terms of whatever you consider sensitive. But at this point, like, I'm just considering anything that's like personal identifying as like sensitive but this is still a very broad question, because, like.
David Bau: how can we really know where to find that sensitive data. And so, right now, what I'm trying to do is given a base and like a gradient to like fine tune, the model. Can we actually recover sensitive data that was trained on that ladder model. So I'm trying to focus on gradients and like, see, what can we get out of that, because, like, that's like the one thing like while you're training, that's like usually tend to, you know, have access to and like change
David Bau: and to be more specific. Let's see, did I have anything else to say in this particular one? Not yet. But yes. Yeah. Question, yeah. Just reading the devil's idea. I mean, if you're assuming access to the gradient, why wouldn't you have access to the data?
David Bau: You see, this is the ultimate goal. This is the current goal. The idea was the idea was that is, that like, if you can do through gradients, and if at some point you can do an approximation of gradients or like, let's say, some sort of difference between the 2 models. Can you like kind of get like an approximate of what sensitive data might have been trained on. It's like, it's a very step by step, iteration of like a.
David Bau: I have a different question. Yeah. So the underlying assumption here is basically, the gradients for the sensitive data is different from the rest.
David Bau: Is is that correct? Yes and no. Like, the idea is, I get okay, we'll see that in the later in a bit. But like the idea is, if I have the gradient, can I have a method that can somehow like
David Bau: sift out what could be like the sensitive aspect, or like the non-sensitive cases? Because, like, let's say, like we are, we are like, Hmm, this building is northeastern. And currently we, the people residing here are row hit.
David Bau: Shoshanta, Shoshanta, and like and like Carson, and like. And yeah, and so the personal identifying information in these days are like your 3 3 names over here. But like, let's say, that entire sentence was there. Can I actually get those instead? Just those 3 names
David Bau: without, because the other ones are like kind of like fluffy information. Yes. So I'm gonna ask the same the same question, but more concrete. Yes. Are you specifically interested in data that is memorized by the model or
David Bau: any sensitive data that was that was in the training data. Perhaps maybe the model is not
David Bau: memorize that. But you still want to somehow extract.
David Bau: So this is a little bit in the fuzzy ground like at this point we are seeing what the model has like learned. Now, at this point we don't know whether it's like learned or memorized, but like it's just an information that we can get right. By that I mean, the model can utter the names of me. The gradient would have significant information of saying that we do not know if the model will actually ultimately be able to say that.
David Bau: yeah, yeah, and so, yeah, that's that's the quote unquote current goal. And so the main like I felt I feel like the infrastructure of my like methods and project is with this paper, which is the language modeling version paper. And you know, just to kind of give like a understanding, brief understanding of what this is is like. The authors of this paper they try to find a way where, like
David Bau: if you have the next token, probabilities of like, whatever prompt like a user may have given, you don't have access to what this is, whether you can actually reconstruct what the prompt could have been. And so the way they did that is, they basically trained like an inversion model. So it's essentially like a Meta learning approach where, like they took in the next token probabilities and had trained like with a bunch of like
David Bau: ground truth prompts. And then ultimately, this was like, like, I guess, like robust enough for that particular setting that they had to essentially reconstruct what the prompt was so. The difference now. And what I'm trying to do is this like, now? This.
David Bau: I should say fine tuning data, just to be more specific, but like assuming that is hidden. And you have your gradients. Can we like also train a similar inversion model to essentially get back what the
David Bau: input like the fine tuned data was.
David Bau: And so and so, yeah, and I see, this is our 1st setup. So like, so what I'm trying to like see over here is hints like, without even knowing whether it's sensitive information or not. Can I actually just reconstruct within, like what the fine tune data could have been and so just to like, make it more concrete about the gradient itself.
David Bau: point like these are like, if you want to take a closer look you can. But the point of this is, I'm just going one data point at a time. And I'm taking essentially the rank one
David Bau: one step gradient. So like, it's like the like, I guess. How do I say? Like the smallest.
David Bau: like, gradient sort of input you can get and just trying to see like what you can get, get gather out of that. And so just to hold on this any questions.
David Bau: Yes, okay, yes.
David Bau: So what do you envision that you will find different or why gradients? Why not just output?
David Bau: Why not just the like? The inversion paper? Yeah, as it from the output logic distribution right? But you want to do it with gradients. Do you see any advantage that you would get from gradients right? Because if you're like, if you're kind of doing step by step, and like using gradients, you can maybe see at what point it actually like, learned that sensitive data
David Bau: and see that because, like with the next token probabilities. You don't really know that in the sense of like, whether so like as this kind of goes back to the question of whether the model has learned or memorized or maybe not learned it at all. But, like maybe the gradient could have that information, regardless of what like the model
David Bau: kind of so can I say, like, you're trying to make sense of what the gradient is doing during a training step. So that's the interpretability perspective. You can think of this as 3 perspectives. The interpretability perspective is like one like you have this gradient. What can you understand about this gradient, with the difference of the 2 models. And with respect to like the data, it was like trained on, you can think of it from an attack perspective, because, technically speaking, if you do have access to this as an attacker, you may want to see if we can access some set of data.
David Bau: but you can also think of it as like from a government point of view, where, like, Hey, you know, I have these 2 models. I want to see whether it was actually trained on a sensitive data. And so there's perspective on that as well. But right now, since we interpretability folks starting off with the interpretability perspective, I guess. But yeah, any. Oh, you had also a question. Yeah, yeah, I mean, if I understand your work correctly. So what you're trying to do is, basically.
David Bau: you have access to a gradient per input and you now want to decompose the gradient into sensitive and non-sensitive information is that essentially essentially yeah.
David Bau: so the decomposition part is like doing this. Svd, not not really. The decomposition part is just for me to like work with fewer dimensions, like just fewer data. The Svd is not the trick yet. Right? What is this Svd for?
David Bau: This we need for is to just like reduce, like the like, and just have more sparsity, and like what we want to see for the gradient and like, look into it. So what? So you're you're collecting this.
David Bau: you I? Is it the you that you're collecting? I guess you know. The funny thing is, I've never seen this code before. So this is really informative. So you're collecting this, you and does is this
David Bau: you? Gonna your plan is gonna be to use this, you
David Bau: as input to a model that you train later. Is that right? Yeah, okay, like, this is like the red use Brad thing. That's what we actually wanted to ask this question as well.
David Bau: we usually don't care about the you.
David Bau: We care about the direction which is usually B.
David Bau: Oh, did your stuff work better if you train on V instead of U,
David Bau: I can. This is what happens when you show.
David Bau: you know. Yes, what does you signify? Yeah, you signifies the mixture between these directions. And we are the actual directions. Yeah, yeah, you're right. So like.
David Bau: okay, there's this, like, the other thing is, we did this, some in pytorch. In some models things are transposed from the way that a linear algebra person might read it. So I don't know which one. This is. Okay. Yeah. Sometimes the ways they're actually transposed. Yeah, so like
David Bau: looking back at the pass code because it has been months, I did have a there was a comment saying, Hacky way to get the smaller vector of the gradient. And and yes, that's what that's what happened. And then like ultimately, yes, this is this particular low ranking is to get the reduced gradient.
David Bau: Sorry if you're interpreting the transformation as column spaces, then you care about the view, but if it is transposed, then you are actually doing it. Row spaces. Then you care about the U. So the view of the U. So there's 3. There's 3 results from this Svd. US. And V. Yes, and you might want to be
David Bau: training on the other one. We just don't. Yeah, we care about whatever the dimension of gradient is, if it's you or we whatever that is. That's what you care about, right? We care about the direction of the change right?
David Bau: And I thought that V is the one that gives you the directions
David Bau: like it depends on what the G, but you care about what the hidden dimension of the gradient actually is.
David Bau: The dimension is always one here.
David Bau: What do you mean? Always one. The rank is always one here. No, no, the G, not the UG.
David Bau: G is the input, the gradient, the matrix, the matrix. Yeah. Yeah. So G is, probably it has like the hidden dimension of the model itself. Yeah, like, 4,096, yeah, yeah.
David Bau: Whatever U or v, 1 of them would be 4,096. It's actually not the gradient of the model. Dimension G is the gradient of the whole weight. Right? So it's a projection. Yeah, it's a matrix. It's a matrix of 4,096, 4,096, or rectangular matrix. Yeah, it depends on the matrix. Yeah. So if it's like a fan in or fan out Mlp, or something like that.
David Bau: it could be like 4,000 by 1,000. Is that right? Yeah, yeah. Yeah.
David Bau: Hmm, that is right? Right? Yes, yes, it's just. I have not printed the values in so long. So like, yeah, that. But yes.
David Bau: granularity of the gradients. Is it like per input always, or per batch per input
David Bau: oh, wait. Oh, for now you're you're not batching inputs. Yet, yeah, the batch is one. Okay, okay, maybe maybe later on in your, you will do this. Yeah, yeah. So the whole idea. I guess the intention behind. This is like, if you are able to gather anything from this rank one like, you know, representation. And like, you're still able to grab like
David Bau: some like data. But I want to clarify so. But you're you're doing this on an autoregressive language model right? And so, even though you only have one, it could be a long piece of text with like lots of cross entropies added up. I mean, even like a single sentence is basically a batch of tokens right? Yes, that's why hint, and we can recover things. But with the gradient the gradient wouldn't have all those batch dimensions right? Gradient is just aggregated once, and then it gives
David Bau: it. It adds, it adds everything. Once, yeah.
David Bau: that's right. Actually, do you want to go? You've been waiting? Yeah. I was just wondering, for you know, if if we're thinking, let's say the column we're thinking in terms of like column vectors, and we're looking at like some one rank one like modification, then we have, like a 1 vector that will modify all of the column vectors with like different weights. But
David Bau: if, if, like, if it's not like neuron basis aligned like, if it's just like, if it's all in superposition, then why do we care about
David Bau: the modification along like each like, why do we care about? Only one of the
David Bau: the right singular. Vector, why not? Yeah. Yeah. Why not both of them? Yeah. Why not? Both of them? Oh, okay. Oh, yeah, okay, I got confused. Oh, but it's just quite as big. So maybe maybe you'd want to take both of them as inputs. Maybe. Yeah, maybe. Maybe. Yes. Yeah. So kind of related to how big it is. I'm curious, like, what's kind of the order of magnitude of like the base gradient.
David Bau: the like reduced gradient that you're trying to use. And then, like the probability vector that that paper that inspired this uses.
David Bau: Well, how big are all these different pieces of data.
David Bau: Good question. It's like power of 10 is fine.
David Bau: So
David Bau: it's so for context, the model that we're using in this one is like 50 or 14 million. So even amongst like models itself is like one of the smaller ones. But I think
David Bau: the gradient
David Bau: I can't recall the number, but it was. It was still pretty big. But maybe I can get back to you. The size of the gradient would be equal to the size of the model. Because, like, basically, when you train a model
David Bau: you get an update value, a gradient value for every single parameter in the model. Right? So so like you said, it's 50, 14 million, that's 14 million numbers.
David Bau: So 14 mega, you know, 14 times 4 MB.
David Bau: Right? You know. So 50 MB of data.
David Bau: Would be the full gradient.
David Bau: And then, if and then you're doing this, svd, like, what's what's what's the dimension of 50 or 1414 B. Like, is it a 700 dimensional? 7 7, 68, or something like that? So it's 700. So so if you're doing a rank, one reduction
David Bau: that would basically divide by about 700
David Bau: about this thing. So it's like, so one, maybe 1, 28 smaller. So it's still a hundred. So right? So you're going from?
David Bau: whatever it is. 50 MB down to 100 times less than that. So I guess.
David Bau: What is that it's good to do this back of the envelope.
David Bau: What's 50 MB divided by 100?
David Bau: 500 k, 500 k, yeah. So she still has a 500 K,
David Bau: input. And then when you guys are like, Hey, maybe we should do both you and V, well, instead of 500 K, maybe it's a megabyte, and she's like, that's a lot. But maybe it's okay.
David Bau: Yes, yeah. And are you doing every layer, or are you picking? Are you picking out layers? I'm picking layer. 4. 0, so you're just doing one layer. So that's another reduction in like 20 fold or something like that. Yes, right? Okay, yeah.
David Bau: Yes. You'll sweep right ultimately, ultimately. Yes, yes, yes, there's a lot of ultimately. Why did you pick layer 4 because usually like the semantic content like from the previous layers. Yes. Oh, there's only 6 layers in this model. Oh, so it's not a factor of 20 factor of 6.
David Bau: So ideally by that time it should have like got it. Yeah. So in terms of designing experiments. Do you suggest taking the 1st layer, last layer, or something in the middle?
David Bau: Absolutely.
David Bau: Oh, for this type of thing. Yeah, I think that Coen is on the right track. Something in the middle.
David Bau: Right? Yeah. Yeah. But again, it has to be a full sweep eventually, for like more robustness. But yeah, okay, that is, that is, with this just any other.
David Bau: Yes, I guess not. To blur the point too much. But another dimensionality reduction could use. Have you considered like using a Laura or Loriq adaption? Fine tune instead of a full fine tune, and then later, try to adapt some. Oh, I haven't considered that. But yes, we could do that as well. Yeah, it's just yeah. I think. So, yeah, okay.
David Bau: I, you're answering all my questions that I had towards the end as well. So that's great. Cool.
David Bau: okay. So in terms of the initial experiment setup.
David Bau: kind of went again. Since it was trying to recreate like the papers set up, I used what they had, so that was like 1 million instructions data set. The Inverter model is the T 5, a small model. And then the language model that we used was the 50 or 14 million. And yeah, just to give a little bit more context, it was like about 25 epochs and like 1 million training data, and like about
David Bau: after every like 2 week, pox, and so to kind of just give like a result of like the initial thing. This is not sensitive data, this, at this point the initial setup is just to see whether it can recover
David Bau: con, like information like whether it can recover data. And so yeah, these are like some instances where, like, I have another question before you show you the results. Yes, so because
David Bau: I wish I understood this better. So you have a
David Bau: you're. This is output from a fine-tuned T. 5. Small. Is that right? That you're fine tuning the T. 5. Small? Yes, yes, and
David Bau: and so normally, the input to T 5 small
David Bau: is like a bunch of text or a bunch of you know, a bunch of tokens right? So. But you. You've been saying that you have this input which is your you.
David Bau: So how do you feed your you thing
David Bau: into T 5 small. How is that in. We have a embedded sort of thing to like. Ultimately change it to the dimension of 768, I see, and then we make into a series of tokens. Yes, and then, like, just run it into a series of tokens.
David Bau: I thought that you were just doing model stitching.
David Bau: Sorry. Sorry, like the gradients that we have, we just essentially like shape it into the t, 5 into into like one token, input or a bunch of tokens
David Bau: One token, I think, really just one, because that's only 7, 68 numbers the last token of it. So I guess it's the entire thing. Sorry.
David Bau: but you have but you. You said that you have gradient for the whole sentence, and then you, the gradients are accumulated now. Yes, she's taking the next token
David Bau: thing.
David Bau: No, no, no, so your gradients are definitely over the weights. Right? So she's she's got like, you know, this loss taken over the entire sentence, and it turns into this giant, you know, 50 MB gradient over the weights, and then she reduces it down to yeah.
David Bau: 7, 68 numbers. Is it only 7, 68 numbers, because it's because it's 1 layer one I see. So maybe that is all the way down to a single token for T, 5, is it? Could it be that small?
David Bau: Okay, so I don't even know, okay, okay, that's okay. Yeah, yeah, it's interesting. Yeah, no, I guess you're right. You know, I don't know.
David Bau: like you could like T. 5 is used to like pretty discretized results. And so I imagine
David Bau: that if you just feed the vector. In the T. 5. It might not be accustomed to, you know, thinking so hard about a single token like, I can kind of imagine
David Bau: they're breaking it up into. I don't know.
David Bau: 64 dimensional pieces or something like that. And then, like, say, Okay, this 1st 64 dimensions will give you this token, the second 64 dimensions will give you the next token, and just like spreading it out. So the T 5 has more space to, you know, specialize and think about
David Bau: where all the different parts of the vector are, what did the previous work do? It's just the last token. Yeah. I mean? Yeah, they just yeah, they just take.
David Bau: Yeah, they just take a lot like they just take how did they beat this entity? 5.
David Bau: don't remember, had to go back and look. Yes, it's okay. These are all good questions to really these are good questions. Yes. Another thing is that T. 5 is a model that is encoder, decoder model, trained with masked language, modeling objective? Right?
David Bau: So so when you are actually fine tuning the model
David Bau: first, st how? Yes, how are you feeding it? And second, how are you? Fine tuning the model? Is it still the mass language modeling, objective? Or is it now an extra prediction objective. It's still the mass prediction. Objective. Okay? Yeah. Yeah. But it is supposed to generate autoregressively.
David Bau: Yes, you you're you're training the autogressive part of T. 5. Right? Yes.
David Bau: just the decoder part. Yes, yes. Why can the inversion model be the same as the original model? Why can the inversion model be the same as the original model. Why cannot you mean, yeah, why not just use the same model? Yeah.
David Bau: 12.
David Bau: That that I mean intuitively. That makes more sense to be.
David Bau: I think. Okay, yes, I think the
David Bau: okay. I think the idea initially was to continue with the setup just to like, kind of see the transferability of like what the findings could be.
David Bau: And then yeah, because, ideally speaking, it would be nice to have this generic Inverter model that could take ingredients. If you use that setup, you're actually proving the model using its own.
David Bau: Yeah, there's an elegance there. But your but your model is very, very underpowered. It's this 14 million parameter model. Right? You go big, like 70 M, or like 4 10 M. Yeah. But then you make the problem harder, too. At the same time, right? It allows you to. It allows you to scale these things. If I were to start with it, I would start with electrical 1 billion.
David Bau: I think it's a question. But yeah, but your strategy allows you to make a more powerful use, a more powerful model to understand an easier one. Yeah.
David Bau: yes.
David Bau: And this is like model probability probabilities for the next token.
David Bau: Yeah.
David Bau: So they just get it for the next token.
David Bau: at least from that information over there.
David Bau: Gives the probability of each of the possible next tokens.
David Bau: Yeah, let's see.
David Bau: So is it loading?
David Bau: Yeah, it just seems like all it's like mixed up in probabilities.
David Bau: Right? And how do they feed that into T 5. They use a single logic. Vector oh, they just they just feed it directly in as one token, one token.
David Bau: So they transform that 50,000 vector to some stuff, 7 with an embedder which they find train themselves.
David Bau: Yeah, they're training T 5. So they're also training. So
David Bau: yeah, so you're doing the same thing. Yeah, I mean, like, they're one good thing about their code base is just generally kind of like like you, you can. You just need to add a new class, find it for the gradients itself. And then.
David Bau: yeah, did you scroll up to that equation? You just went past it looks like they're oh, wait! Do you know what they're doing with the Argmax X there
David Bau: is. Isn't that just the the predicted next inversion?
David Bau: Yeah? Or because that's like the output, like the because the V is the the like predicted probabilities that we've been talking about. I think right. And then C is just like a transformation of that.
David Bau: And then it's decoding. C, and then this X value. I'm just curious what that X value is.
David Bau: Next token prediction. I believe it's the next. Yeah, it's it's just the next token probability, at least from this X over here.
David Bau: So TT. 5 is a mask model, right? So it predicts a mask.
David Bau: whatever the Marshall credit. So they are.
David Bau: They're using like 1st 10 tokens and masking the 11th token, letting T. 5 predict the probabilities and just greedily sampling the top probability as the next token
David Bau: they appended. It's like using Tifa as an auto regressive.
David Bau: So is X, the 10 tokens that you're conditioning on or x is the
David Bau: I mean X is the text right? And you're just taking the one that maximizes essentially. Whatever your decoder probability is, you're just taking. You're taking the text of all possible text that
David Bau: is most probable according to your decoder.
David Bau: Yeah, that makes sense.
David Bau: You're just generating the text auto regressively and prep like appending each token at a time.
David Bau: Yeah. Basically, greedy decoding.
David Bau: Wait a second. So wait X is the input or the output to a default.
David Bau: X Star X star is the complete. I think it's a complete text prediction
David Bau: or approve, or like the entire sentence.
David Bau: I think it's the network prediction because it generates one by one.
David Bau: the text that is the predicted inversion?
David Bau: Yeah. So I would assume, like, the 1st token which is protected is like the.
David Bau: and that the is also being fed into the decoder model, and then the model generates some other open.
David Bau: Yeah.
David Bau: star, is the predicted output. Right? So
David Bau: all good. No, no, no! Hold on. Yes, sorry. We have to unroll the vector into a sequence of pseudo embedding Ci. Oh, oh.
David Bau: oh, what's that? Okay? It was just constantly bouncing. Oh, yeah, sorry. Sorry. No, no, it's okay.
David Bau: Speaking condition as well. But it's up
David Bau: full part of the vector B,
David Bau: let's see. Oh, do you understand? Yeah.
David Bau: what is your, what is your reading. So the the output logic distribution is like a very huge number of vectors. V vocabulary size. Yeah. Instead of using an embedding vector to reduce it.
David Bau: They are breaking it into multiple chunks and using them as like tokens like just what you so kind of what I was describing. Right? Yeah.
David Bau: And then that is fed as an encoder in encoding input to the T 5, right?
David Bau: That's what they call unrolling.
David Bau: I see. So X star is unrolled. X star.
David Bau: Is that right?
David Bau: No. So I don't understand the X Star part. I understand the Ci part. So that is just them chunking the big vocabulary output into multiple tokens
David Bau: and then pushing it into the encoder of T. 5.
David Bau: I I don't know how decode Rock T 5 works if anybody else. Yeah, I think X is like an empty thing. Let's assume that as an empty set in the start. So in the 1st time step
David Bau: you give that encode C, as the input to the encoder and X is null and basically empty set. And the model gives you the 1st token.
David Bau: Oh, so this is the auto regressive. So you have to iterate the second thing just to get the generation out of it.
David Bau: I see, I see, I see.
David Bau: So there's the encoder of T. 5 and the decoder of T. 5.
David Bau: And I see, I see.
David Bau: Alright. Okay, all right.
David Bau: So what's up with that sectioning or categorizing it
David Bau: of the probability splitting up, splitting up
David Bau: like. So that 1st line of the equation they have this
David Bau: big V, which is for them. It's the logic vector, or something, the probability vector, V, yeah. And and then they're saying, Hey, that vector, is so huge. And it has so much information that we don't want to feed it in as a single token.
David Bau: They say that we're gonna split it up into
David Bau: lots of tokens. The number of tokens is I going from one to
David Bau: line d divided by D at the right. So see it right? So this thing at that. So that's the number of tokens that they're going to use like
David Bau: size of vocabulary divided by D,
David Bau: which is 42. 0, 0, yep, cool right? And then
David Bau: so they're going to use 42 tokens stop
David Bau: just just used to Eric being the mathematical genius. Actually, he's just reading so then. But then, you see, on the to the left of the all it says, log v. Sub d. All these subscripts right? So d parenthesis I minus one colon di right, that that business right? So so they they're just inventing this notation
David Bau: of V sub. I being the probability of the I word. And they're saying, Hey, you know, you know python style d
David Bau: parentheses. I minus one Colon di, will indicate something like the 0 to, you know.
David Bau: 100th Dimension, or the 101 to, you know 200 200th dimension, or whatever
David Bau: And so so they have eye of these little chunks of
David Bau: subsets of the probability. Vector so they take the whole long probability vector, and they just literally just chop it
David Bau: into subsegments.
David Bau: And then they log them for whatever reason. And then they stick them through some Mlp, which maybe is like a trained thing that they're gonna turn that into
David Bau: some other vector right? Do some learned embedding.
David Bau: And then and then they get this, you know, pseudo embedding ci.
David Bau: which is d dimensional, which you can. Then you can feed into tis
David Bau: the I token. But you know, for out of 42 tokens. So they have these 42 tokens, and then
David Bau: and then so you have these 42 tokens, and then there. There'll be like the fixed
David Bau: conditioning text that T. 5 is expecting to see.
David Bau: And then T. 5 can then also just do this generative process. We're given a fixed conditioning text. It can like
David Bau: write out a sentence autoressively about it. And the second equation is that process is just saying, All right. Starting from an empty string x which they don't write down
David Bau: you know. How do you know what next
David Bau: thing to say in your generation? Well, you just keep this conditioning constant, this encoding of the 42 word C,
David Bau: and and then you just keep on
David Bau: getting X to be longer by predicting the next X token.
David Bau: I have 2 questions. The 1st is, why do that? Something is still not clear to me. Why not just use the entire thing? Is it for the computational reason or something else.
David Bau: They say that their reductions rank. There is what large reduction in the rank and loss of information
David Bau: they say, if you try to put it all in one token, then you lose too much information. Oh, I see.
David Bau: Right? So there's another variable that you could play with, because I don't know how many tokens you're using it. But but like you could potentially spread out to more tokens if right? And so there's like there's another hyperparameter. I don't understand that choice of DI mean, like, why would you choose? D equals to the model size? It seems like
David Bau: there.
David Bau: The 1st model is like Pts. 70 M. I'm assuming in that case, these like, I think in this case it's
David Bau: T. 5 T. 5 C. Oh, I mean like, but like that particular number seems to be the model size of whatever the 1st models. No, it's the second second model.
David Bau: Is that the reason? Yeah. But I also feel like, you're right. It's arbitrary, right? They to whatever right? It's just like, just like how much they want to spread it out to multiple tokens. So so it's another thing for you to play with with your with your training process. Right? How do you fit it in? Otherwise
David Bau: it needs to be the model size, I mean, but you need to fit it into the encoder. Latent, ultimately, output has to be technically the output of the Mlp. Could be different from d right, but it could be nothing. It could be a fan out. It could be fan in right then. You need to train another thing. But that Mlp. Sub, I is the thing. That's the train. That's the thing.
David Bau: Yeah. In fact, it looks like they don't just train one. They train I of them. So they train 42 model. If the downstream model is expecting a 768 dimensional input, then you need to give it one. Even if behind the scenes. You're taking multiple steps to get it to that size. You said. d. 2, 1. You're like
David Bau: you're feeding all the all the things in your vocabulary model output. So essentially, they're they're just training square Mlps. Yes, from Sasha Rush's group. What hackers they're just hacking, right? So you know, it does like they take the wrong data type.
David Bau: And then you're just like turning into a bunch of tokens. And they're like, How do we do this? All is just split up into 42 tokens. And then, yeah, I don't. It's not going to be the wrong thing. We'll just learn some mlps, and you'll do it right? And then does it? So does. Now
David Bau: you're you're using this setup for analyzing a gradient instead of the probabilities. Yeah. And so so now let's go back to your results. Yes, but before that I think, Carson, you raised. Oh, I was. I was just wondering if they had like in the Appendix, like tests of different numbers of like instead of 42, like half of that double that. Yeah. But maybe they just chose it.
David Bau: So they they didn't test. Yeah, you have a very, very high expectations for these Phd students. I think that. no, I'm sure that they didn't. We tried it for other data. So that's what the appendix is on like. How it's more. Yes.
David Bau: alright. And just to clarify your setup is, instead of that, 42 tokens is one token.
David Bau: it's I guess it's I guess. Yeah, I have to check your your code to see what it turns out. It's probably probably I think, still, gonna like, because I didn't change anything off the other code. And so like, literally, it might just be one token, because your vector is a lot smaller, like, right? Right? And so you might. It's something that you might want to adjust to adjust. Okay.
David Bau: yeah.
David Bau: Good questions, good questions. And it's also bringing me clarity. And so yes, in terms of performance. This is kind of like, oh, let's look at the example. Yes, so this is so. This is your your data on
David Bau: yes, and so like initial, it gets like the initial stuff, like, in terms of like, Oh, describe the current shipping processes when Amazon, if that's the original one, the decoded one is described, the process for applying for a job sort of thing, and like, like, I think, the general, like instruction asked.
David Bau: seems to be like like instruction. Template seems to match up. Is this restaurant review positive title, Great Experience Review. We had a yeah. So great review I had a great experience with, I see. So there's yeah. It's kind of feeling like they're like, completely inverting the semantics. Yeah, there's something in there. Yeah, like, absolutely opposite.
David Bau: Yes, and like. So I also like continue to like, run it on like again.
David Bau: since it was like a reproduction replication I tried on, like the other data sets that they had, and in terms of performance
David Bau: like it does. But it's not surprised. It does relatively well, for, like the 1 million instructions ones, but then, like the ones that you see are super low, are like basically almost everything else, except for, like the platform called alpaca, and essentially the reason why it is like that is that it was trained on 1 million instructions. So it's more like likely to
David Bau: give back similar like instruction, style, problems and everything else like it was just really bad at like actually recovering. So yes, the red thing is what it was trained on, or which one which data the red thing is what it was trained on. The V thing is the the purple thing is the Val, and then higher is better. Yes, higher is better.
David Bau: Is this more similar to the reconstructed one
David Bau: that that makes the original one that makes sense. So it's trained on red, and which is similar to purple. Yeah, that's the hold out. And then what is so the other? There's the yellow thing that's pretty good. So that's like python called alpaca, which is also very much instruction. Oh, alpacas are training like everything else over here like ag news. Summation like summarization. Doc, like that's like kind of in the lower bits. I see it's not instruction. Ish? Kind of yes, exactly.
David Bau: Yes, is cosine similarity. How the paper that inspired this evaluated so actually, that's how I think, okay, this one, I think, like, they did it by f 1 score. And like, I guess, like root score and all, like, yeah, basically by root score.
David Bau: I did it by cosine similarity, just because I thought that would be like better to just see like contextually, this cosine similarity of some some representation. Vector yeah, yeah, it was basically of the of just the decoded outputs that that was given. And like I ran it on like an like like a sentence transformer to like, get the embeddings and then do like a, yeah.
David Bau: yeah, okay, so that was this one. So yeah, this is a base model. You're training on top of your fine tuning data set. Right. This is a base model, like with your 14 is a base. Yeah. Yeah. So
David Bau: for an instruction tuning, the 1st few tokens are all obviously going to be like the most lossy as they go. Probably the loss would go down is that, do you think that could be the reason why you're seeing this.
David Bau: Yeah, the 1st few sentences are like, well represented, but then, as it goes, it just drops down. So there was actually no like in general, like in terms of token size, I would say that could be the case like that is the case, and, like instruction like style, wise, it was more likely to get it. But, like, you know, there are cases where it still got the entire thing, or like, even when it was instruction asked. It's still butchered the entire thing. I have like qualitative
David Bau: instances for that. So speaking of that in terms of like, I just like, got like, best instances. And so, yeah, like it was able to like, you know.
David Bau: get this like ditto, sort of thing. And like, just like token, original and decoded wise. It's like, you know, same and in terms of like
David Bau: worst examples. Yes, it's like, yeah. So the longer sentences are the worst examples. Yes, they're generally worse. What is the distribution of similarity like? Is it mostly tail heavy or like
David Bau: so I do, so that I do have a graph for that.
David Bau: So this is like, wait, do you ask for token? Lens? No, I'm like the cosine similarity distribution.
David Bau: Yes, I mean, is it mostly on the better side or the worst side. That's what I'm saying
David Bau: on, like, how many instances on my test set? Yeah,
David Bau: I guess. Like, if I have to consider all the other data sets, probably on the worst side.
David Bau: Yeah, yeah, but on the on the instruction ones is on the much better side, like, like, if you look at the graph over here like it's probably like, it's generally like, Yeah, yeah.
David Bau: but you're mostly concerned with. If you look at the graph. It looks like mostly on the politics side, though. Right? This is kind of it's not his grandpa. If you look at the line chart you were showing. It looks okay. I had 2 different line charts, because I kind of skipped and was going to show extra results. But let me well, you have this for cosine. So you have blue score here. You have this kind of standard plot for cosine. Yes, I did.
David Bau: Yes, one second.
David Bau: Oh, hold out
David Bau: well? Oh, yes, well, but
David Bau: yes, it's a blog, is it like? Is the X range from minus one to one? I created 0 to one? And and was this was this, do you know, if this was for instruction text, or for other types of text
David Bau: instructions? Because, like, what is this? So this is 2 like this. What's that other one that looks a little, that one's 7,001 was like literally everything like like 1,000 of each of the data set, including the not instruction text. And so this one down here is just instruction text. Yes, this one is just instruction. I see. I see.
David Bau: Okay, interesting.
David Bau: Yes, alright.
David Bau: I have a bit of a novice question. Yes, so like
David Bau: cosine similarity versus like I don't know. Like like you put in your like l. 2 distance between like representations. This kind of thing like do you ever use like? L. 2. Is it always cosine similarity, or
David Bau: well, I think so.
David Bau: Hmm.
David Bau: I probably no, yeah. I probably just was focusing on like semantics. So like juice, for I have a math professor answer for you just did the same.
David Bau: So you know, you know, in in high dimensions. If you write down the formula
David Bau: for cosine or dot product, you're, you're, it basically discards one dimension of information from, like the the L 2 distance, the Euclidean distance. And they're basically, you know, the same like sort of 2, you know, one minus the cosine, or something same thing as the distance. And so or or maybe the square root of it, or something. Right? So
David Bau: and so.
David Bau: So, yeah, not that much different? Okay, thank you. Yeah.
David Bau: And what's the cosine similarity? What's what are the 2 things you're measuring? So the original output and the decoded output like I said, no, literally, the output literally text to text. One thing I wanted to say is that it mostly looks on the positive side, because cosine similarity is from minus one to one, and you are all of you are chopping from 0 to minus. One
David Bau: seems like shifted to positive side.
David Bau: So at least it's not reversing.
David Bau: Well, okay, well, it's all relative. When it comes to this for me. I like, look at this and think more. It's like still. But yeah, I mean, of course, like on the instruction side, it's definitely positive. But like, just generally like out of domain, it's like, not that great.
David Bau: Yeah, that's how about that? That is like the positive and negative answer for that. And so this is still the initial setup, as you could also probably see from, like my other, like, you know, a skip slide sort of thing.
David Bau: So yeah, basically from this is like the general stuff is like, okay, it can sort of be used to recover training data. But can we specifically get sensitive data? Now, how does that really work for us? So case study is, I call it like the person finder sort of like, yeah, case study, because you know, rather than collecting email ids, social security numbers and everything. One thing that is
David Bau: like, I guess more like, I guess easy to get would be like person names. And so, yeah, that for that I needed to this, I spent a good chunk of time trying to create a good person finder data set. And so you know, when it comes to person names like you could technically use these existing like, nor like name, entity, extraction methods. But if, upon like analysis and things like that, they don't actually perform that. Well.
David Bau: And so, yeah, I essentially had to do this whole like Llm looping to like, essentially, have a more positive thing, positive, like person finder automated process. So like for creating a clean training set. Yeah, for yeah, for creating a clean training set. And so like, the 1st part of the thing is like, well, is there partial name present? And so the reason I look for is there partial name present is because I want to reduce
David Bau: use the possibility like of like the model, thinking that, hey, this could be a name. This could not be a name. I want it to be like, specifically like, Hey, this is really a name like, oh, so you're so to set context here. So you're you're describing your prompt engineering process for creating
David Bau: like, what is it? Is it Gpt-four prompts, or is it? No, no, I I used llama 3.1. 0, the open model. So you're you're using the open allay open to help you help help you filter your data set. Yes, yes.
David Bau: of person to create a person finder data set. Because right? Now, we're using some text.
David Bau: Yes, yes. So I'm using the Dolma data set. So like, yeah, I was initially working with pile. But then, apparently because of some copyright potential issues. Just switch to Doma, because it's like, what are you using? Lama 3.1 8 b instruct? It's not good.
David Bau: You should. You should at least go to 70 B to get like a better thing. But I actually suggest using Gpt for I mean, I in my experience, 8 B is trash.
David Bau: Well, it worked out with, like these multiple chain ones. After suffering through several weeks of getting this to clean your data, found a way. Yes, it's all right, it's all right, it's all right.
David Bau: and so yes, and then I have. So I guess that's why I needed like 3 prompts for it. Right? So 1st is like seeing if there was partial name present, second, extracting all the names. Now he was right in that sense, like, sometimes it's like doesn't like really like it repeats the names or things like that. So or like. Sometimes it doesn't really give a name.
David Bau: Oh, so I have like another thing, another time. Verify the name I go through each name and be like, Hey, is this really a name? And so that's why I have like this. 3 change change process of like both verifications before and after, and then in between, to just get the names. Yes.
David Bau: sorry. So why didn't you just use like a data set that was made for named entity recognition. Really like that. That was like, good in the sense of like, yeah.
David Bau: yeah, I think there are replacing pi data sets.
David Bau: But okay, let's see.
David Bau: Well.
David Bau: maybe granularity, like partial name versus full name by product. Yes, yeah. And so I think we were looking for a very specific kind of how we should have our data set like and make it more generic.
David Bau: So I think that's why we went with that cause there was
David Bau: yeah, cause the other ones again, like, so, okay, we also wanted situations where there was no names.
David Bau: Because, like, you know, like, ultimately, the goal is like, if you have given multiple data points and like, there are some certain sentences which which there are no names. You don't want it to return like a false, sensitive data and things like that. So we also
David Bau: situation here is a little different also, right? Because she's.
David Bau: you know. I think that what we're looking for is very, very clean training data.
David Bau: as opposed to like a challenge evaluation set
David Bau: and and then she she could use some of these other data sets to evaluate good stuff later. But but I think that some of the idea here is that, like, like, we want to have her in a situation where she has some training set.
David Bau: where she really knows exactly where every single name is in the entire data set.
David Bau: so that if she needs to do things like
David Bau: synthetically expand the data set by putting in fake names or things like that that she has like total control over everything. Instead of having, like most of these other organic data sets.
David Bau: are designed for, you know, challenge like realism.
David Bau: But then there's lots of ambiguous cases where it's like, is this really? For, like.
David Bau: you know, this album is by prince, right? Is Prince like a name, or is that not a name? It's like a single word is that, should you count that as a name? Is this personally identifiable? Is this like? Do we care about Prince and Madonna and staying
David Bau: or or not?
David Bau: And you know sort of there's there's all these cases, and you can kind of. And then Queen sort of has this luxury now she she can be like. That's not part of my training data, considering that
David Bau: although I do have famous names in. There are some famous names in there, right? But they're pretty unambiguous names. That's right.
David Bau: Question. Yes.
David Bau: So did you try just to generate this text rather than filtering out, generate this text rather than filtering out. Given you cleaner data.
David Bau: Fair? I mean, it's fine. If you didn't do it.
David Bau: No, yeah, we wanted to. We wanted to get like a lot more diverse data as well, because, like, when by using Dolma like, we got a lot of things in like caption, like image like when you have like like an HTML image text, or something like that, you have a captioned name in that to get or like you have, like, I guess, normal, like Urls, or like Twitter or like, I guess you can get those and or and like normal text as well. So like to just kind of get like diversity of
David Bau: like diverse contents. Also, also, a good motivation is, if you have a real world, fine tuning data set.
David Bau: And you're trying to actually train your model. Having this type of
David Bau: identifying the personal information is more like
David Bau: you're doing this in wild like, while fine tuning is happening.
David Bau: what can I do rather than if you create your own data set, it'll probably be more pi specific.
David Bau: And that's almost like I, I could, as a reviewer say, oh, it's cheating heavily. Training on pi. Obviously, you'll find pi information in it.
David Bau: Okay?
David Bau: And so yeah,
David Bau: data. Stat, wise like, I do have this, I guess, in actually know it's in private. But like, I can like and show you how it is just in terms of like distribution. You notice that in general, like in the wild, like in Dolma data set, there's a lot more negatives like you don't have a lot of like. I mean, I wouldn't say you don't have a lot, but there are more negative rows where you don't have names as compared to positive. But I collected about like 196,000 positive names.
David Bau: and like in terms. When I say I mean, I wouldn't say names, I would say rows, because, like each row could have like multiple names. And so like, I guess the most like, I guess if I look at the bin and like, look at the Max, I would say 2 names. But like, there's a good like, yeah, there is like you can get up to like, I guess there's 1 instance up to 24 names. But generally you can get like, yeah, a good chunk of different names out there. And so, yeah, that's the data set and training set up.
David Bau: Now, I talked about having the idea of like when there's no names to actually not give any names and things like that. So just like we're now added like 10% of negative instances. And the data set and the setup is the same as as before, except like, instead of reconstructing the input, I just changed the labels like training thing to be the name to get names instead.
David Bau: And so, and if there's no names, I'll put nothing, it'll be yeah.
David Bau: And so yeah, and like, it's still converging down. And that's where, like the hybrid parameters of like just maybe increasing the learning rate, trying out these other things that we just talked about would be helpful. But so this training right now, yes, as we speak, yes.
David Bau: but with like, with the particular checkpoint where I felt like, oh, it's getting names and things like that. So yeah, like, out of these 1,000 like validation rows, there are like 3,580 names out of that like, I guess it was completely recovered. So when I say chat, fine tune, I mean like. So
David Bau: I actually stopped like. So for the initial setup, the fine tune model that I had like with that instruction setting. I like stopped at a particular checkpoint, and then use that, and just continue to generate like fine tune, for like the getting like names instead. So just to double, confirm, like not double, confirm, to double, clarify. I kind of re I have. I reuse the Inverter model that I previously chain trained to further train. Fine tune on like this particular setting.
David Bau: because at that time it was already good at recovering. So I was just trying to
David Bau: get it for this. That's why, like, you know, I guess a good thing is from the previous chat. Fine tune. To like the current model is able to recover more number of names. And then partial names as well. It is able to recover a bunch, and I will show some instances like it is able to, let's say, get the 1st name of like this 2 part name, or something like that, and or like sometimes semantically like. So that's 1 thing that was gonna
David Bau: try to show next. But I can like show instances. And, oh, my God, this actually looks much smaller than I hoped. But
David Bau: okay, let me try to highlight. So
David Bau: what I was trying to highlight here. So in this particular case, you got like Elon Musk and Alex Roy, that's the ground truth. And so over here it's able to get Elon Musk and Daniel Day. Luis, who's Alex Roy
David Bau: the drive's own Alex Roy. I don't know. Yes, for some sort of person, and like in this particular instance, what I'm trying to show here is like
David Bau: this person is an Indian actress,
David Bau: and like the decoded one, are like, basically, it's all other Indian like Indian actors. So it's kind of been like this cosmantic space where, like, it's not able to recover the right names, but it's still in the space of like getting like all these different actors and like and like Hollywood actors. So I thought it was very interesting to see. And then, yeah, this is like another instance where, like
David Bau: over here, the ground truth is just Ted Cruz. But over here it's able to get. It's thinking of Ted Cruz and Joe Biden. And so again, like thinking of like, yeah, thinking about it in terms of like again, semantic space.
David Bau: So in like the top row, the kind of orangish text that's your like the original text that this is still the original text. It's just like the reason it's orange is just because I like was trying to highlight it like.
David Bau: Show it up there. Yeah. So like the original text has the name Daniel Zorilla. I guess Daniel Zorilla on that like end of the second line. Shouldn't that be in your ground? Truth, yeah. What was that in the ground. Truth, it's a flaw in your training data. Okay, that was 90,
David Bau: 95. It was. It was 95. I did not get the stats of like like, okay, it sometimes makes man. It was 90 like when I sampled it on like 100, and like manually ground truth and stuff like that. It was 90, 90,
David Bau: 97%, 95 or 97% missed a few percent. Still. Yes, still missed a few percent. But maybe Gpt. 4 would be able to get that, like the amount of money spent is worth the thing it's worth it. It's absolutely worth. It's like it's not going to be. It's going to be highest like 20 or 30 bucks. Yeah, which means like, less than the cost of labor. For sure. Right?
David Bau: Yeah, 20 to 30 bucks. Yep, okay, yeah. All right.
David Bau: just get an Api key and pass it through. Yes, sounds good. Sounds good. How do you analyze your decisions?
David Bau: How did I analyze my results like right at this point like this is kind of just like it in the sense of like? Oh, no, no, I meant like, How do you see your results? Do you see that as a win or like? Oh, well, I do. I will say it as it's in a win area, just because like, especially now that we see this Daniel thing actually shows up. That's that's like way better. Now, we got pretty good. Yes, and so like and so like.
David Bau: I think for me, I was just more like interested to now know the fact that it's kind of able to get into the same space of like. Like, I've noticed this in most like cases where even the partial names is like it, either. Yeah, it gets partial parts of the name, or if it doesn't get the name, it seems to usually be in like some semantic context. And so I thought, this is pretty interesting that it's narrowing down to that. So the model might probably know
David Bau: most of these actor names. Yes, so the next step was to kind of reduce this and not have famous names, and like, kind of switch it off, switch it up with like some like. So we were. Gonna now
David Bau: change it up where, like, maybe only top, like the top, 1% of the frequency of that data set, we take it off and then just replace it with like synthetic. Right? So I think that's why that the entire Indian movie actor name, yes, is able to come in. It's pretty interesting because the gradient doesn't really care about the identifier anymore, because it knows what it is. Yeah. So it's it's just like muttering what the model is thinking of at this point, right
David Bau: oops.
David Bau: And one more thing. So I think the numbers that you were showing was
David Bau: like this is the number of times the model has able to recover the thing recover the names right
David Bau: like full names, like just exactly how it is that I have a separate one for partial names. Why doesn't the number add up like what like you have. The total number of names is like 3,518 like.
David Bau: So 366 and 3, 2, 1, 4 is actually 3, 5, 8 0. So you're not considering partial name recovery as part of like, any any sort of name. Recovery. Yeah. Yeah, okay, cool, maybe using f 1.
David Bau: Oh, but the next steps well, was definitely like more data.
David Bau: different hyperparameters which we discussed with the UV, and then the 42 chunk thing. I now have it in my head. And but like, yeah, in terms of trying to create like more difficult settings, which is like multiple gradient steps, and like, maybe increase like the yeah, the number of like data points. But again, like, this is definitely like a long, how do I say? I guess a longitudinal studies in some aspect, because
David Bau: right now we're using gradients. But ultimately I think I do want to get closer to the weight difference. So if you guys have any ideas on like
David Bau: how to reach closer to that great and yeah, any other thoughts which I think we talked a little bit about confounding any other experiments. Would you like me to set up, and anything else to just kind of make you be more convinced that this is possible, or maybe not. Yes. Can you clarify what you mean by weight difference versus gradient.
David Bau: so like so weight difference, I literally mean, like parameter to parameter, if I just take the minus off it and like do it because the gradient 2 parameters are you so like, let's say, like, literally, let's say, like M, like Mlp, which parameters of what? Minus? What? Parameters? Oh, like the base and the fine tune model.
David Bau: Oh, okay, yeah. Yeah. Cause cause
David Bau: like, it's not exactly equal to grading, because grading gets compounded. And so so. But to give context right? So remember, so clean is like, you know, motivation is a little bit of the hugging face scenario.
David Bau: Where? Oh, yeah, you've got this llama that was published by Meta. And then hundreds of people are like uploading their fine tunes of llama. Oh, I've got my super, mega awesome fine tune. You should use this. Look at all the performance benefits it has, but then they don't explain, like what they trained into it.
David Bau: And so so Coriana is very motivated by this question.
David Bau: It's like what the heck is that thing right? Did you put something bad in it?
David Bau: Right?
David Bau: Yeah, yeah.
David Bau: Maybe a different approach to answering. What you want to achieve is of
David Bau: 1st try to localize where your pirs are encoded. Let's say the names or the Ssn. Those kind of information.
David Bau: and see if I would assume that if if you are able to localize it.
David Bau: To some layers or some Mlp things, I would assume that those Mlps would generally be
David Bau: shared across the fine tuned variance as well. It's it's it feels to me. It's unlikely that if the base model has the pi information stored in the 3rd layer, the fine tune model has will have the pi information stored in, let's say the 6th layer or the 10th layer.
David Bau: So if you are able to localize the pi information in your base model that gives you that reduces the space of the search that you need to do to a large extent.
David Bau: So okay, you find out that these are the Mlps which this, this one or 2 or 3 Mlps are encoding your names. So you take the difference of that
David Bau: and start from there. I feel that that seems like a more tractable problem.
David Bau: Can we
David Bau: to to see now you're assuming that your fine tuning will also organize things in a nicely localized way. I'm only concerned about the person. I hope that it does. It's also very important for my project, which is nice. It's a different approach.
David Bau: So also, just for context for the other visitors. This kind of started off as a spin off to my initial, I guess project, which I used to call a room detector, where, like, if you have one model that was room edited. And then you had like this room, edited model. Can I actually recover the exact back that was used to change it? And in that particular setup I kind of use the models themselves
David Bau: to figure it out in terms of what I can get. But one like, while I could see hints of it, it was noisy like how like like there would be other options that would also change. And so like what? How? Exactly, as you know, as someone who doesn't really know the truth, what is the actual truth. And then we ran some experiments to know that. Okay, let's assume we know what the subject is or the object is, and the relation is. We noticed that even the best method within the model itself
David Bau: was not that great in terms of recovery? I think it was about 60%. And so that's why we then decided to do a Meta learning approach, which is how this entire project then started. So a bit of backstory. Yeah. So she started off with your idea, Nikhil. That seems more attractable to me. But she found it not to be tracked. I mean, maybe if it's a meta-learning one now, maybe it might end up being them. I don't know. That's right. So maybe maybe the
David Bau: so just like the rest of the machine learning world. Maybe the AI can figure it out for okay.
David Bau: I believe you. Well, you believe it. Do you think it'll figure it out? Yeah.
David Bau: But neuroscientist can't believe that. Please don't believe it, I'm in the intersection of both worlds
David Bau: so so similar to that. Like, instead of training on fine tune ingredients, you could train on roam updates
David Bau: alright, and like the target, would be the right. That's true. It can be. It's just. This was more, broader scope. So we were like, if you're doing this might as well just try. Can I see one problem with that? To get the room update, you need access to the base model so that you 1st need to take the difference. Then you have the update, and then you need to train the Meta model.
David Bau: Right? You. You assume that you have the room update.
David Bau: And then you're okay. That's the same case with, yeah, because the ultimate thing is, you have an Inverter model, which is kind of like an oracle. Yeah, yeah.
David Bau: So right now, you're doing it with
David Bau: single gradient update. Yes, for one sample and recording that sample. Yes, how would you see it if you batch it like, what should your output be? Well, if I batch it, then like, well, it should be, I guess.
David Bau: like, okay, maybe. Okay. If I batch it, I need to probably have, like some sort of like
David Bau: tokenized way to like show that it's actually like, maybe from like different names, like different, like sentences or or like point is, I think I think we should just get like names that are relevant for that entire batch. And if there are no names in it. Then just they're not good. They're not going to be names.
David Bau: Yeah, I think it's literally the same thing, except
David Bau: we can think of it as like bigger context month.
David Bau: Awesome, Julia.
David Bau: I. Maybe the weight differences can be taken as like a very big batch done in like one big update.
David Bau: And then somehow, like, maybe your once the model gets very good. Maybe it also works just directly at on weight updates.
David Bau: That will be nice.
David Bau: That will be nice. Yes, I hope that would be the hit. Just to go back to Eric's question. If you have the roam update. Yeah, you don't even need a Meta model.
David Bau: You can just mess with the key. Just see what this key outputs, and then you can
David Bau: do some sort of token optimization in the token space, maybe in the embedding space, and then
David Bau: use an inverse of the embedding to get what this V was. Yeah. But then that is very much. Rome. Specific. Yeah. And now this is ultimately what you want to achieve. Yes. How do you get closer to the weight difference? Yes.
David Bau: so what you already presented is pretty cool
David Bau: but an alternate approach. So so are you assuming access to the data like very specific question, does now the model know about Elon Mask? No, it did not. Okay, you're not assuming access to the data. Yeah, I'm not assuming access to the data.
David Bau: That's the thing is like with difference with, like other papers, they tend to like like have, like right now, your access right now, at least in the scope of this project. You are assuming access to the data
David Bau: for evaluation purposes for evaluation purposes, right for training purposes only.
David Bau: So training plus evaluation. Yeah, I mean, like, basically like the actual act of like
David Bau: using the Inverter model. You just have the gradients like, let's say you just it's like, this is a good question. So what? So what I wanted to ask is that if you just take the fine tune model, take the base model. You wanted to know what Elon Musk's. Let's say, email address was right.
David Bau: Just take the difference.
David Bau: I don't know. The difference itself is no, the difference itself is not a language model. Okay, if you just took their perplexities.
David Bau: do you like? Can you determine something like the the model knows about something, or versus the model does not know about something.
David Bau: Sorry. Let me try to understand the setup. So this is not. It's nothing about this approach right now. It's just like you have 2 models based and fine-tuned. Yes, you have different representations of Elon Musk. So you have. Let's say that not Elon is bad. So so you know.
David Bau: Gregor, you know.
David Bau: you know Gregor Felovich, right? You know some person has never been mentioned before. Right? So you ask the model? Yeah, the email address of some? No, no, not even the mail address. Just do you know Gregor Felovich? Right? And then it says, perplexity, what does it say?
David Bau: Perplexity? 1.5. This is this is not a valid piece of text, high perplexity.
David Bau: Right? And then you go to the other model. And you say, do you know, Gregor Forfelovich?
David Bau: And it says, perplexity. 1.2.
David Bau: This, yeah, this looks like a valid text. Right? That's what you're saying, right? Yeah, yeah. Yeah. And is this approach kind of like a
David Bau: like. How faithful this approach is in recovering. So I think that one is, I think, is an assumption that the model then, definitely has learned something about
David Bau: you are also true. You are also making that assumption that the gradient has useful information. True. But then I like. But then, okay, I'm not. So this is the thing. Maybe it's a misconception on my end or some. But if you do a gradient update
David Bau: and like, yeah, if you do a gradient update and like whatever the model has learned now like, I don't know if, like, it always
David Bau: passes through like.
David Bau: So so I I would. So the way I would answer this question. And if if you're if you're.
David Bau: you know, doing your nerves, oral presentation,
David Bau: is is the problem. Setting is different, because what you're asking is.
David Bau: you know, if Gregor is worried about himself, he could. That's 1 kind of problem that you could ask. Say, a pair of models. Does one of these models know more about me than the other? But let's say you're just an auditor. Let's say your government auditor. Should I notify Gregor that he's been included in this thing like, who do I notify? Right?
David Bau: And I think this is more of the problem setting that Quina is looking at.
David Bau: Okay, she doesn't have the query to begin with. Okay, but it's all. It is reasonable question, though. I think that you know it's another kind of baseline or something to test.
David Bau: Yeah. But perplexity would never give you the answer. Right? I mean, as you said. Do you know this person? The answer is either yes or no. In both cases perplexity. The prompt can be different. The prompt can be biography of Gregor.
David Bau: And now you measured for perfectly nice, that's all.
David Bau: Yeah. But then, yeah, so. But I think that's a. So that's another research paper is like, you know, how good is that? As a method of, you know, detecting training data. So there's this whole membership inference attack. Yeah, they are like.
David Bau: yeah, which is. And that's usually the setting there.
David Bau: Yeah.
David Bau: yes.
David Bau: Good questions now. So I had a bad question with his suggestion. But isn't weight. So isn't the gradient just a weight difference for a single slightly fine-tuned model. Yeah, that's why. Yeah. So that's why we started off with gradients, so that eventually we can like find a good enough inverter model that we could use it for like weight differences.
David Bau: then how
David Bau: don't gradients carry too much information like over time like, how do you even single out what the model does
David Bau: very good question. How do we single out what a model has learned? Well.
David Bau: rank one, but also no, not really. Hmm!
David Bau: Let me single out. I,
David Bau: I think in this case, is just trying to make
David Bau: an inverter model be used to seeing that kind of data, and then like, be more used to understanding what to recover, reconstruct from there. I think it's more about like, how good, how good of an Inverter model you can create. But then, doesn't the Inverter model depend on what sort of fine tuning I do? Yeah, exactly. So like, then I have to give more instances of like the multiple batch updates. And I, as a company, have to not lie about what I did.
David Bau: So I kind of inform you, like, this is the sort of stuff I did, and not lie about it.
David Bau: Is that like close? So so do you know what sort of fine tuning I did? No.
David Bau: I wouldn't know. I do not know. Yeah, I. So basically, I'm trying to. We can think of it this way. We can think of the Inverter model as some sort of oracle that I'm trying to create where like it could be.
David Bau: Well, I don't know to what extent it'd be can be good, but like good enough, such that like, no matter like maybe, what the company has created or not like, it, hopefully has covered the basis for what it might have generate like might have trained like with whatever context, length, and whatever like batch size. And then that way we can recover
David Bau: what names or like, whatever sensitive information, even in terms of sensitive information. For example, right now, this is just person names like like in like, for a more robust article like you could give in generic like what whatever you generally consider sensitive and like have like that, or you could have separate inverter models for each.
David Bau: Sensitive like this might be broader. But I was thinking, what are the lines of, instead of trying to find
David Bau: exact pieces of information that it learned trying to learn specific themes. For example, for image generation stuff did it learn on specific styles.
David Bau: like, if you find it to learn specific styles, maybe that's easier to learn through gradients and like patches of gradients. So okay, I see. So if I so it could be so in this case, would maybe handle the other kind of data that I was thinking about versus persuasive data
David Bau: where, like, maybe, there's like a like similar to like instruction. Maybe there's a similar kind of sentence where you like one, the model to kind of just say to the users. And so maybe for that case, yeah, it could work out. Yeah, yeah.
David Bau: could be a different yeah, a different way of analyzing. Yeah,
David Bau: so another cool application for this
David Bau: the differences, yes, is like, quantized models, apparently act very differently compared to the true model. Okay, yeah, it does. I wonder what that? What do you mean? Quantized? If you do? Fp, 8, it's basically the same as the original model. If you if you go down to Fp 4,
David Bau: it's the accuracy difference is around 90 like, if it's 100 here, it's 98%. So basically, you don't lose much over here on the other side of the table. Fpa, it's not that great. Is that right? Exactly. And then and then, if we had John around the table, he says, Oh, my! My Fpa behaves completely different from the Fp. What does the compression algorithm. Well, so
David Bau: well, so like. So you know. So John Roger is, gonna do, he's trying to write this for neurops.
David Bau: where, if you do, you know out
David Bau: the the perplexity. 1776. Model. And you use the the, which is an Fp. 16. Model, and you quantize it down to Fp. 8 in whatever the standard way is that perplexity is never going to work people. Okay. But but nevertheless, but the the like. The the change in behavior is dramatic in the sense that the perplexity 77 is supposed to remove all of the
David Bau: Chinese censorship from the model is very effective. It's completely gone. And when you go to the Fpa, all the censorship is back right? And so this is huge dramatic change. Yeah, we wrote an algorithm called uptq. I mean, not me. Post, Doc. He's a pi in Isd, Australia.
David Bau: So that thing is like, very, very, very, I mean, Fp is basically lossless in our in our use cases. Yeah, maybe you should. You should. You should give it a chance. Yeah.
David Bau: Yeah. But yeah, so like.
David Bau: regular quantization seems to have like some change in behavior. Maybe with your reference methods, we can actually see what that quantization is actually losing.
David Bau: Yeah. And maybe it'll show all the Chinese decensorship information. When you do on perplexity, right maybe yeah
David Bau: yeah, I mean like the whole event, like again, if I were to bring it to model lakes. The whole idea for weight difference is to be able to have different.
David Bau: basically like, give more information off the edge of like a parent and a child, or like a model, or like basically different versions, it has. So yeah.
David Bau: yes, I was just wondering what you thought about this very related idea. So let's say, we have some way of determining if a model has some information present in the model weights in some way. And we have multiple training. Trained models like independent training runs of the same. You know, whatever model you're using. And you have these different fine tunings of it. There's an interesting paper. I'm forgetting the name of that explores, essentially trying to take a weight difference between the mother recovery. Oh, sorry the
David Bau: Michelle from his mother. That's what I remember. Is it? Let me one second, yeah, but it basically, they're trying to apply. The weight difference of, let's say, model a and model, a prime where a prime is like a fine-tuned one, to say model B, to show that the weight difference is applied to an independent training run still also encodes some change, even to a completely different training, which is very.
David Bau: very interesting. Is that from the task vectors paper from El Harpo? Yeah, I don't think it was. Yeah. But anyway, the general idea of if you wanted is fundamentally a very robust way to say, Do model difference, weight, weight differences contain information.
David Bau: If you can just apply the model. Sorry the weight difference to a separate independent training. Run and demonstrate that that add additive. I guess you know the addition of that difference of a prime minus A to B to show that that produces something that has that information that B does not, you'd very clearly show that weight differences do encode
David Bau: that information. It's called param delta. What is that param delta for weight, direct weight, mixing, direct weight, mixing. Okay, post-trained large language model at 0 cost
David Bau: direct from direct way. Exit.
David Bau: Bwm, instead of Dme. On this. Assume that both models learn things similarly or
David Bau: well, they'd probably be. Yeah, it's a good question.
David Bau: But they're probably based on the same similar base model. Yeah, it's the same base model, slightly fine-tuned, base model. But then that's the thing you add your data to is a separate run, right
David Bau: or so. You have a base model. You do rl, hf, on it. You get your rl, model. Take the difference. Yes. Now take the same base model
David Bau: further fine. Tune it with some more data. Oh, God, yeah. Now you get a new base model. You add this difference. This also becomes an Rl, model. Yeah, so this is, this is the whole model merging thing. And so it's it's also related to federated learning. Right? There's like sort of ideas come up in a bunch of different places. I think that.
David Bau: what's it? They do this in multilingual context. Oh, you know, do one fine tuning on French? Do one fine tuning on English, and add these things, and then but then, if you just directly add them.
David Bau: it's not as good as oh, you can have a Meta model. Figure out how to merge them, and then and then you get a multilingual model out.
David Bau: There's all this.
David Bau: all this model merging literature. But but Nakoya is like doing the interpretability half of it. It's like, okay. So if these width differences are so powerful that you can do these things in it, then they must. You must be able to look at them and get information out. Right? So it's what you're doing.
David Bau: Yeah, yeah. I like your 1st setup much better. Why did you drop? Which which is the 1st setup, the chat, the completion which which gives you like the entire training data, because because well, if we do ultimately train on like a lot of data. And like I. It's not. It's kind of noisy to just recover everything of them if you just care about like, if there was any like sensitive data
David Bau: that it was, you know, actually trained on. So it was. It's more about like.
David Bau: kind of like a filtering of what kind of data we are actually interested in looking at rather than like, just try to recover everything like your model. Next vision actually wants this right? Like, you want to
David Bau: get all the fine tuning data so that you can connect the trees. True. But then it also can be like, it's different perspective, like as a regulator. Maybe you care about more about sensitive data. And so you or you then decide to just want to see what sensitive data was trained on. Yeah, I like that perspective, right? Not all data is equally interesting. Yeah, so can you get better performance if you focus on
David Bau: their data that you care about have a I mean
David Bau: on like a use case basis. When when is it? The assumption is like, I don't have access to the data, but I have access to the gradients.
David Bau: I mean? When when is it the case? When is it the case? Weight differences? Right? Yeah. It's ultimately so again, though, gradient is just like a step stone. The ultimate thing is weight difference, I see. Yes, so yes, yes.
David Bau: it's just that to. In order to prove that you can do the weight difference thing. I think you 1st need to show the gradient and like, see that you have an idea about the weight difference. I have to think about it. But oh, yeah, that's great. Yeah. Let me know. Yes.
David Bau: thank you, Kayanna. Thank you. Everybody.
David Bau: Can you share the recording with me? I do not take notes. Oh, yes, of course, that's what it's for. I will send you the link. Thank you. Yes, I remember in my head. But.