Roy Rinberg: Yeah, I'm Roy. I'll be talking today. I I'm at Harvard I work with
Roy Rinberg: Beth, Neil, and then Syl Badan, who's a theory privacy person. And for this project
Roy Rinberg: we also worked. So it's me, Seth, and then a postdoc shivam. And then we also worked with the
Roy Rinberg: Madrid lab, Chris, Sam, and Andrew, as well as Madrid, who I never saw
Roy Rinberg: And it's not so.
Roy Rinberg: If you wanted to follow along.
Roy Rinberg: This isn't that important? It'll it'll go away.
Roy Rinberg: Great. Thanks. Thanks.
Roy Rinberg: Okay, not important to do this now. But if you wanted to follow, you could. And then.
Roy Rinberg: yeah, so we're gonna talk about unlearning.
Roy Rinberg: this is my favorite tweet related to unlearning.
Roy Rinberg: I feel like it captures.
Roy Rinberg: I don't know what is what even is I'm learning
Roy Rinberg: can everyone see? I don't know how to stand.
Roy Rinberg: okay. So in this talk, we're gonna try to do a quick. We're gonna talk about high level goals
Roy Rinberg: challenges to unlearning in general and and what the existing methods are.
Roy Rinberg: We're gonna kinda ask, like, how would we do this perfectly. How would we cheat if we could do it? And then we're gonna say, like, Oh, wait! We can't cheat
Roy Rinberg: like let's let's do it for real and so to start
Roy Rinberg: what even is unlearning and what is success in unlearning?
Roy Rinberg: We have to take like one step back and ask, we're gonna
Roy Rinberg: people mean a lot of things when they when they talk about unlearning.
Roy Rinberg: And there's and if there's at least 2 big crowded
Roy Rinberg: 3 big categories, one is exact and learning and approximate learning. Exactum learning is where you like.
Roy Rinberg: basically fully retrain your model or find some way to checkpoint your data in some way that you don't have to fully retrain your model. But you have removed the data from the model. It's perfect. You can't really argue that
Roy Rinberg: now what we're gonna be talking about. We're gonna be talking about approximate learning where you're trying to like.
Roy Rinberg: find a cheat code from the model that you have now to a model that never saw data.
Roy Rinberg: And within that there's kind of 2 things that people might talk about concept and learning and data and learning. Concept of learning is like, I want to forget.
Roy Rinberg: You know, the concept of a Harry Potter, or like the concept of Alex, and data and learning is like, here are specific pictures, and I want you to simulate what would happen if those pictures were never in the training
Roy Rinberg: they're related. They're hard and different. I'm going to focus on the second one.
Roy Rinberg: Okay, so what is data? I'm learning.
Roy Rinberg: I like this diagram. It's a okay. So you trained a model.
Roy Rinberg: And now you want to remove something from the model. So you really want this model? Not that model
Roy Rinberg: and but you don't really want to fully train. So you want some like shortcut between those 2 things.
Roy Rinberg: the commonly, cited reasons, I really feel like, i'm directly
Roy Rinberg: true, okay.
Roy Rinberg: yes, I do.
Roy Rinberg: Yeah, you can sign in. I feel like, right, there is a good spot
Roy Rinberg: I recommend. You can still hear from me.
Roy Rinberg: So there are different reasons why you might want to learn.
Roy Rinberg: These are commonly cited runs removing copyrighted data, privacy and poison data.
Roy Rinberg: And it's important to like, know? Like different reasons for learning dictate a lot.
Roy Rinberg: As like a small table
Roy Rinberg: you know, you might say, like privacy like, there's this legal requirement. Gdpr, I have to do this.
Roy Rinberg: How do you learn? There they're like
Roy Rinberg: measure of success. Is that your model looks
Roy Rinberg: like a model that never saw the data.
Roy Rinberg: But if you have poison data and you're just trying to remove that, you want to unlearn the poison data, you don't care about that. What you care about is just did the poison.
Roy Rinberg: If you have bad data you just care about. If the model improved and copyrighted.
Roy Rinberg: I think this is like a. This is fundamentally a legal question. This is an opinion, but I think this generally is a process driven question, not like a lawyers won't accept, most likely will not accept a statistical guarantee or a statistical like empirical measure, and will probably just say, was this data ever in the data set?
Roy Rinberg: But that's an opinion. Don't take that too seriously on that last one.
Roy Rinberg: Nope, we're gonna focus kind of on this privacy thing, but they're all kind of related.
Roy Rinberg: And if you want me to talk about something, please interrupt.
Roy Rinberg: Why is measures of success? Why are we like spending so much time on measures of success. It's because
Roy Rinberg: they like really matter your measure of success really matter. And the learning can hurt you. If you're not careful about what you're measuring.
Roy Rinberg: Like, if you're unlearning for poisons.
Roy Rinberg: Imagine this concept like, there are poisons that people have developed where you can put 2 data points in a data set.
Roy Rinberg: And they don't have any effect. But if you were to remove one of them
Roy Rinberg: they like the poison is revealed. And so if you did perfect on learning on that second point
Roy Rinberg: you have, like poisoned your data fully, even if you were doing the perfect unlearning of just fully retraining.
Roy Rinberg: Obviously, this gets like this could. This is obviously hurting you.
Roy Rinberg: There's like pretty strong for privacy. There's pretty strong evidence that, like unlearning. 1 point actually worsens the privacy attack, ability of other points.
Roy Rinberg: And so
Roy Rinberg: if you're not careful protecting, someone actually hurts other people. And so you have to be careful about that as well.
Roy Rinberg: I'm gonna skip this 3rd point. It's a little more nuanced and happy to talk about it.
Roy Rinberg: Generally
Roy Rinberg: in the world that we're going to be living in the like measure of success is like, you have this training data.
Roy Rinberg: You've trained your model.
Roy Rinberg: And then someone's like, Hey, I no rabbits.
Roy Rinberg: And then you want this other model to basically look like.
Roy Rinberg: you know, if I train this never on rabbits. And I trained it. And then you asked me how similar to this muscles?
Roy Rinberg: And what this specific definition of how similar to these models are is hard.
Roy Rinberg: We.
Roy Rinberg: I'm going to specifically talk about
Roy Rinberg: in the convex setting. Sorry in the non conduct settings, in the conduct setting, where you're dealing with
Roy Rinberg: models that are well defined. You can
Roy Rinberg: kind of define these formal notions of it.
Roy Rinberg: Statistical distance but in large models
Roy Rinberg: in large models, where these are very, very high, dimensional, and not well defined. We can't really train a bunch of them and get a statistical distance between them. And so what we're going to do is ask for statistical similarity in output space.
Roy Rinberg: And I'm going to walk you through what our proposed measure is, and it's gonna be very similar to existing measures.
Roy Rinberg: But right now this is this kind of is the Wild West. There's a bunch of proposals for what? How you want to evaluate
Roy Rinberg: So in other words, so to to be clear. So you're working in output space, not in parameter space. That's the that is, yeah. Yes, it would be unreasonable to work in parameter space in non conduct levels because they're too big.
Roy Rinberg: And they're just not parameterized. Well.
Roy Rinberg: so there's all these like influence function. People who sort of work in parameter space, even though.
Roy Rinberg: Yeah, the models are so bad. And it's sort of related to your thing. Do you have perspective on that. Yes,
Roy Rinberg: you can come back to it later. Yeah, I actually will. I have a slide that might be skipped, but it is soon.
Roy Rinberg: that is kind of addressing talking about that. But I think the answer is.
Roy Rinberg: maybe someone clever than I can do a model space comparison. But by default. It's very hard.
Roy Rinberg: And relying on output space at least gets us a starting point.
Roy Rinberg: this is gonna yeah. Hopefully, this doesn't kind of seem like too much all at once. Is there a way to move the talking? It doesn't really matter.
Roy Rinberg: Oh, you can do that one.
Roy Rinberg: Oh, yeah. How do we move that little corner thing?
Roy Rinberg: If you talk from your thing. There we go. Okay,
Roy Rinberg: So
Roy Rinberg: you're going to take. You're going to take your. What we're going we're trying to do is say, let's compare the distances between oracle models which are fully retrained and things that have gone through our own process.
Roy Rinberg: And
Roy Rinberg: what we're going to propose is okay. First, st we're going to collect a bunch of them. So we're going to take a bunch of these models that you know only trained on the retain set.
Roy Rinberg: train them and get a set of oracle. Let's just say for contextual, for context. It's like on the order of a hundred
Roy Rinberg: and then we're going to do the same thing except we're going to go through this learning process where we're gonna like, take the full model. We're gonna learn it.
Roy Rinberg: And then we're going to do this end times. And we're going to have these 2 sets.
Roy Rinberg: So yes.
Roy Rinberg: different models in the sets are trained on the same retrained set. They're like identical training process. Just you're hoping like these processes are are noisy. And and so
Roy Rinberg: yeah, different random seeds. That's right.
Roy Rinberg: and then you're gonna say, okay, for each point you're gonna we're gonna define a measure on a point
Roy Rinberg: we're gonna say.
Roy Rinberg: take a single point and compute the margin, which is like the confidence between the difference in the the top choice and all the others how confidently it's predicting something.
Roy Rinberg: We're gonna ask for the margins
Roy Rinberg: to collect a histogram of margins for the unlearned point models and for the oracle models. And so it's going to be these 2 histograms.
Roy Rinberg: We're going to compute some divergence. We're going to talk about Kl, but it can be anything.
Roy Rinberg: And then we're that's gonna give us a single number.
Roy Rinberg: So for a single number, for a single point, we've collected a single number.
Roy Rinberg: and then we're going to generate a histogram of these for like multiple sets, and it sounds like we've just done a lot of work, and we have. But
Roy Rinberg: I'm pretty confident that I do think it's necessary. And so what this kind of means is
Roy Rinberg: you can say, I want to know
Roy Rinberg: philosophically, what have we done? We've said, I've computed a statistical measure
Roy Rinberg: that will compare how the unlearned model compares to perfect unlearning
Roy Rinberg: for a single point. And I want to ask, how does this behave on the retain set? How does it behave on the retain the forget set? And how does it retain on
Roy Rinberg: behave on a validation set and like to give you
Roy Rinberg: like a kind of a nice plot. There's a typo.
Roy Rinberg: Cale Cahill, divergence and cross entropy. But
Roy Rinberg: This is like doing nothing if I just like truly took the original model, didn't even unlearn it. What'll happen is like
Roy Rinberg: were really similar on the validation set and the retained set.
Roy Rinberg: And then we're we're garbage on the on the forgets it.
Roy Rinberg: And so this kind of plot will maybe help you contextualize what these numbers mean.
Roy Rinberg: One isn't a special number. The point is that 10 is bigger than one for the rest of the stock. You'll be dealing in the range of 0 to 12 roughly.
Roy Rinberg: And then this is like load an oracle. What if you just like, compared one of these
Roy Rinberg: perfectly retrained models to the distribution? And you know, it's not 0, because there is difference. But
Roy Rinberg: they're all basically very small numbers from 0 to point 5.
Roy Rinberg: Sorry. Can you go back? One slide? And what is that 3rd thing? Actually, this? Yeah, this thing.
Roy Rinberg: So this was, you define this distance measure for a single point and for for excess
Roy Rinberg: is, is X fixed for the whole histogram? Or is X sampled for the for this histogram? Yes, and then this is the
Roy Rinberg: this gets you a single number at the end of this is a scalar and that's telling you
Roy Rinberg: for this picture of Alex. How far are you from the unlearned distribution? And then you want to generate a histogram of
Roy Rinberg: how well you're doing over particular data sets.
Roy Rinberg: Yes, and generally we're like we're going to be doing this over, forget, set
Roy Rinberg: validation, set and retain, set.
Roy Rinberg: or like at least like a hundred points from each. The reason for this is
Roy Rinberg: your unlearning model might behave very well on one of those, but not like you want them to behave well on both points of trying to forget, but also the other ones. So I'm still, I'm still trying to understand. So yeah, so your eye, your eye here is sampling over models. Is that right?
Roy Rinberg: I here?
Roy Rinberg: Yes, this is a collection. So we start off. We just get 100 unlearned models and 100 oracle models. Oracles are the retrained and the histograms in the middle are histograms over this 100 models. That's right of how they're behaving on this data point.
Roy Rinberg: That's right. And and so you're saying.
Roy Rinberg: and and by saying, margin you're saying. But this is how confident
Roy Rinberg: the model is about this point. Maybe if margins are lower, sort of our entropy or something doesn't really know much about that point, and so you would expect
Roy Rinberg: it doesn't really know much
Roy Rinberg: about a point. It would sort of shift to the left in that middle histogram.
Roy Rinberg: Yes, the actual point of the why we're using margin this could be a higher, dimensional thing. It could be actually the full logics. And we can compute a Kl divergence across multi dimensional thing. But we wanted to flatten it to a single dimension.
Roy Rinberg: Basically, the goal here is that
Roy Rinberg: I like. Let's say I'm very confident in my prediction that this is a picture of a cat, and you're very confident. Also.
Roy Rinberg: you also think it's a cap. You're not confident.
Roy Rinberg: I know the difference. Yeah.
Roy Rinberg: but the the goal here of the margin is to flatten
Roy Rinberg: the prediction to a single dimension. So then. So then, you're saying that
Roy Rinberg: so like you're illustrating here that you might have an imperfect forgetting method in the perfect case.
Roy Rinberg: You're forgetting histogram like how the forgetting thing would behave would be just like your retain
Roy Rinberg: baseline where you retain oracle. Is that right? That these, in the in the perfect case these Instagrams would be the same in the middle. Is that right?
Roy Rinberg: yes, the Kl divergence would be 0. Because.
Roy Rinberg: this would be perfect. Yeah, exactly. That's right.
Roy Rinberg: But then, in practice, your forgetting method isn't perfect. So this is just a way of quantifying. And then and so you get a tail divergence, and then down below is
Roy Rinberg: a histogram of Kl divergences? Yes, 4 different points. And then, and what kind of sample of points is it? Is it is, it points that you're hoping to remove from the data set? Or is it points that are just randomly sampled over the whole data set? Yeah, we're going to be looking at points from 3 subsets of the data set
Roy Rinberg: things that we're trying to forget. The forget set things we aren't trying to forget the retain set, and then validation points that it never saw.
Roy Rinberg: And we're going to sample randomly from those, at least for the forgets that we're going to sample all of them. But validation points that it never saw. You mean your forget method never saw or just. They weren't even part of the training. They weren't even part of the trainings. Oh, so generalization. Yeah.
Roy Rinberg: because I mean.
Roy Rinberg: it actually won't come up that this will matter. But you could conceive of a model that actually does really well at learning, on the forget set
Roy Rinberg: and on the on the
Roy Rinberg: retains that. But then, like, if you gave it a new model. It like
Roy Rinberg: always spits out the number 3, like I like, you know. And then it's very clear that unlearning occurred.
Roy Rinberg: That's not necessarily a bad thing, but we want to know that it's happened like we wanna.
Roy Rinberg: This won't be important. But we thought it would
Roy Rinberg: something to look at. It's something to look at. The really, the 2
Roy Rinberg: data points we're going to be wanting is
Roy Rinberg: so this is that histogram before for baselines.
Roy Rinberg: Take a look at this one. Do nothing.
Roy Rinberg: The blue is the forget set.
Roy Rinberg: So the unlearned, the like original model obviously didn't forget the
Roy Rinberg: forget that. Never even tried and so
Roy Rinberg: most of the Kl divergences are quite far from what the best model would have done.
Roy Rinberg: But on the retain set and the and the validation set like
Roy Rinberg: there was nothing it had to do.
Roy Rinberg: So now. Some of the divergences are low, which suggests that maybe the original model didn't learn that stuff very well. In the 1st place, yeah. Yeah. Or
Roy Rinberg: I think it's probably the opposite. It's that the those points are probably in the retain set, and so removing them doesn't do much
Roy Rinberg: like.
Roy Rinberg: Let's say you're all
Roy Rinberg: the same person removing one of you doesn't like. I still can generalize you? Very well.
Roy Rinberg: right? You mean, you mean you mean generalization covers them. You don't need. Yes, yeah. Yeah. So the effect of
Roy Rinberg: unlearning on the same person that's already in this at one affect much, and so the distance between
Roy Rinberg: doing nothing and and fully unlearning them is small, because they never contributed very much. But
Roy Rinberg: this is important. So it's good that we're stopping now because the rest of the talk is going to be built on on this metric.
Roy Rinberg: Yeah, yeah. Sorry. It's not probability. I apologize. This slide. it's some sort of chaos.
Roy Rinberg: This is Kl divergence. Oh, it's a histogram. So yeah, it is
Roy Rinberg: probability. But this is a histogram of Kl divergences
Roy Rinberg: or accounts. You can rescale the accounts to be
Roy Rinberg: there aren't many typos. This is one that I didn't know about. I apologize. So this is, this is
Roy Rinberg: technically not the best you can do. But this is basically
Roy Rinberg: this is what if you fully retrained a model?
Roy Rinberg: Which is perfect? I'm learning right like, if I
Roy Rinberg: like, just fully retrain the model without you in it there's nothing better I could do in terms of unlearning quality.
Roy Rinberg: and we wanted to say, How would our Kl. How would our measure look for that?
Roy Rinberg: So here the the.
Roy Rinberg: the, the pl divergence is between the oracle models. Yes, and the reason that's the reason why it's not 0 is because
Roy Rinberg: even the same exact.
Roy Rinberg: Is this just like one oracle versus all the others? Yes, exactly.
Roy Rinberg: Again, like these 2. We're going to talk about actual learning. I'm just trying to contextualize. Like, you know, these are baselines.
Roy Rinberg: These are like base baselines. They're not even unlimited.
Roy Rinberg: Let's get that
Roy Rinberg: cool. So we've kind of talked about what is success and what even isn't learning and what we're trying to do. Let's
Roy Rinberg: get into the start getting into the meat and potatoes.
Roy Rinberg: like, what? Yeah, yeah, please. Why, why wasn't that second graph all the way? Slammed to the left at 0?
Roy Rinberg: Yeah, so so. But now I understand, for the middle histograms in the previous slide.
Roy Rinberg: Right? You'll have some distribution
Roy Rinberg: right? And then, if you use that same distribution for both. But you're not using a distribution, you're using a single point, you have a single point, right? But you have a distribution over models.
Roy Rinberg: right? So that single point is going to have some histogram over models.
Roy Rinberg: But then you have the same set of models for both.
Roy Rinberg: So it's the same histogram. The fail divergence would be 0,
Roy Rinberg: which is why I'm wondering why, for every point it's not just slam to 0.
Roy Rinberg: Yes. It's not that you are loading the meat.
Roy Rinberg: Imagine your unlearning algorithm produced like a distribution of margins. It's like, you know, by Bimodal.
Roy Rinberg: And it's either this guy or this guy.
Roy Rinberg: when you actually pick a single unlearned model, it's going to be.
Roy Rinberg: It's going to predict a margin, one or the other and so the and then so you'll have some histogram in the middle, which is one or the other.
Roy Rinberg: Right? I, the unlearning algorithm here is pick specifically one.
Roy Rinberg: So the the it's not pick the whole distribution.
Roy Rinberg: It's
Roy Rinberg: pick. Specifically one unlearned outcome. Oh, okay, so I'm I'm because I was confused by saying such that I
Roy Rinberg: isn't. And so the histogram is not actually overall.
Roy Rinberg: No, it is, I think the thing that is maybe confusing is that it's saying, load a oracle specifically one oracle. And that's your unlearning algorithm, not load like
Roy Rinberg: each time you load a different oracle, I see. But you're comparing one oracle to all the rest of the oracle. That's right. And then it leads to this. Yes, okay, and this is an important point.
Roy Rinberg: yeah. The okay. I get it. I mean, if you got rid of this increase of like the middle thing looks different. That's right. Okay, that's right.
Roy Rinberg: So well, all you need to know about
Roy Rinberg: this is a good thing to think about. There's a lot of work that goes into unlearning evals at this point. If you don't want to think about it, you should think of it as a measure in output space
Roy Rinberg: on specific points between your unlearned models and your
Roy Rinberg: okay, so let's talk a little bit about like existing methods in the convex setting. It's like.
Roy Rinberg: not solved, solved. But it's kind of solved. There's these, basically.
Roy Rinberg: I would point to these specific 2 papers. There's dissent to delete, which basically says that if you
Roy Rinberg: take your new model, your your model, and you just do gradient descent for long enough. On the retain set, it'll just forget about the stuff.
Roy Rinberg: There's a couple more caveats, but
Roy Rinberg: it's nice. It's just like a you can get a theory bound on this because it's convex, and it's just like we'll go to the bottom.
Roy Rinberg: There are ways to do this more efficiently. But, like.
Roy Rinberg: let's talk about the more just in case generally, the existing methods are some version of these 3 really 2 things.
Roy Rinberg: There's a couple of other tweaks. But I want to focus on these.
Roy Rinberg: Excuse me, there's gradient descent.
Roy Rinberg: And there's gradient ascent.
Roy Rinberg: Really, there's you like. These are the 1st 2 ideas you might have. You're like, I'm going to keep training on the retain set, or I'm going to do gradient descent, which is, I'm going to take my forget set.
Roy Rinberg: and I'm going to go in the opposite direction of where my gradients are going
Roy Rinberg: and then there's combining them. And then, specifically, there's this algorithm called scrub that
Roy Rinberg: iteratively just does publicize it. It doesn't exactly do gradient descent. It does like Kl divergence ascent, but
Roy Rinberg: it doesn't at the core.
Roy Rinberg: Nearly every single unlearning algorithm and all the ones that are doing well, or even half decent on on learning. Benchmarks
Roy Rinberg: are just combining these 2 things.
Roy Rinberg: And if you look at the data, they don't actually do well again, I actually don't
Roy Rinberg: wanna actually look at the data there, I
Roy Rinberg: let's philosophically ask like, why might these not work well?
Roy Rinberg: And you come up with these like symptoms of what's going on. You're like, okay, gradient descent. It actually takes too long in the non-communic setting.
Roy Rinberg: like, maybe you would eventually unlearn everything if you just
Roy Rinberg: trained on the retain set. But it's but it's taking a long time.
Roy Rinberg: And really, this is because they have no reason to forget.
Roy Rinberg: They're just like you're just kind of hoping that, if you like, keep telling them about.
Roy Rinberg: You know, Jayden, they're gonna forget about Alex. But and then gradient ascend.
Roy Rinberg: It's actually like a, it's. It's a different problem. But it's it's philosophically doing the inverse
Roy Rinberg: of optimizing something is not the same thing as not contributing it. These are just philosophically, not the right thing.
Roy Rinberg: But even if it was like, the the bigger problem is, when do you stop
Roy Rinberg: the nice thing about, you know things that go like this.
Roy Rinberg: that there's a single thing at the bottom if you're going down.
Roy Rinberg: But there's an infinite number of ways to go up, and you truly don't know when to stop.
Roy Rinberg: and that the stopping point is very important.
Roy Rinberg: And
Roy Rinberg: I am positing. And I think there's good evidence that the root problem with all grading based methods is this missing targets problem.
Roy Rinberg: You don't know when to stop.
Roy Rinberg: And so like, let's take a look at scrub that algorithm that does.
Roy Rinberg: it's going to do gradient ascent for the 1st roughly 3 steps.
Roy Rinberg: And then it's gonna and it's going to be doing gradient set the whole time.
Roy Rinberg: And here
Roy Rinberg: on the Y-axis, we're plotting, unlearning quality. This is basically, how well does that data point fit to the distribution of oracles?
Roy Rinberg: Something very similar to our climb score.
Roy Rinberg: It's effectively rather than a Kl divergence. You're fitting to a Gaussian, and you're
Roy Rinberg: and you're doing this over time.
Roy Rinberg: And these red lines are thing points that get unlearned
Roy Rinberg: and then get worse over time.
Roy Rinberg: and you'll see for validation. It kind of like stays in the like, reasonably unlearned
Roy Rinberg: because you're not really doing anything in the validating set and the retain set similar.
Roy Rinberg: But this forgets that you like have points that are unlearned at 1 point, and they become less unlearned
Roy Rinberg: because on that gradient, at some step you're like, you've gone too far.
Roy Rinberg: and you truly, you don't know when to stop because some points you haven't gone far enough, and other points you have like these points clearly benefiting.
Roy Rinberg: and these points clearly getting hurt by doing gradient set more.
Roy Rinberg: and you don't know how to differentiate them. And so that's trouble.
Roy Rinberg: just to make sure. So what you're saying is, if we continue to do readiness sent. 1st it forgets and then it relearns the sentence, not relearns but
Roy Rinberg: it it the unlearning quality gets worse in that it's like
Roy Rinberg: it forgets, and then it does even worse.
Roy Rinberg: You can imagine like, let's say I know English.
Roy Rinberg: and then you're like, forget Harry Potter, and and like. I take enough steps that I have forgotten Harry Potter, and then I keep going, and then I forget English.
Roy Rinberg: That that last one, the where I forget English is not
Roy Rinberg: close to the distribution of things where I never learned in Harry Potter.
Roy Rinberg: It's different in another way.
Roy Rinberg: So I've gone too far in the in the it would take.
Roy Rinberg: Oh, for the
Roy Rinberg: I think I'm confused with the already. Why is it necessarily bad for the European data set to happen.
Roy Rinberg: decreasing our learning quality one.
Roy Rinberg: What does on learning quality decreasing mean for the weekend?
Roy Rinberg: It's so we're taking the same measure. The measure is
Roy Rinberg: for this data point, how well is it fit by the distribution of oracles? It's something very similar to that. It's.
Roy Rinberg: you know, if you're if the margins of your data points fit a Gaussian which they generally do.
Roy Rinberg: Where in that gap are you sitting?
Roy Rinberg: Yeah, it's it's also the OP. It's the opposite direction of the climb score. But this is a this is actually a probability.
Roy Rinberg: this is fitting.
Roy Rinberg: How well that individual data point is to
Roy Rinberg: to what the oracles are doing. And in this setting, the thing you need to know
Roy Rinberg: if you trust me is that up as good?
Roy Rinberg: This is time, and things go up and then down.
Roy Rinberg: That's the like takeaway and then for those 2 they don't really change. But when the outcome doesn't really work on them like
Roy Rinberg: that's fine. And they're also like, high enough that you don't really care
Roy Rinberg: 2 questions. So what are multiple lines representing? And the second question, is there a regularized
Roy Rinberg: multiple lines? Each line is a different data point.
Roy Rinberg: So you're asking like this picture, like, I've this picture of you
Roy Rinberg: at your balance. So different data points in my forget. Set
Roy Rinberg: in your forget set in your retain set in your organization set. Okay, okay.
Roy Rinberg: and and this is actually very important to plot multiple points because different point.
Roy Rinberg: An important takeaway, if you ever work in unlearning is that different points unlearn at different rates
Roy Rinberg: and different forget sets on, learn differently.
Roy Rinberg: And so, because scrub is working.
Roy Rinberg: applying the same algorithm to all data points indiscriminately and as the same stopping conditions for all of them.
Roy Rinberg: Like, you don't know when to stop.
Roy Rinberg: Does that make sense? Yeah, there, is it?
Roy Rinberg: So you've so you you sort of set the, you know.
Roy Rinberg: set set the groundwork here by saying, Okay, this greeting to send the greeting and send methods.
Roy Rinberg: So one of the things that we've been
Roy Rinberg: trying to do in our lab is, well, we do concept on learning rather than data on learning. But yeah.
Roy Rinberg: but we've been trying to. So you know, we? So we sort of agree that gradient ascent methods are inherently
Roy Rinberg: unstable. It's not. It's not a well formed objective. You can't. It doesn't make sense to an optimize run. What you're doing maybe make it take a single step first, st
Roy Rinberg: but
Roy Rinberg: But then, in a lot of the situations that we're in, you can define a gradient descent, objective
Roy Rinberg: sort of cut the notch in the thing that you're trying to forget, so that you're actually
Roy Rinberg: driving the model towards, you know, modern modified distribution with a notch in it.
Roy Rinberg: So, yeah, sorry is that is this. So we're gonna get to something somewhat similar. Not exactly notch cutting. But I like this idea. I'll say
Roy Rinberg: gradient descent is like you. Really, if you're doing unlearning, you really want gradient descent based methods.
Roy Rinberg: Because
Roy Rinberg: hyper parameter tuning is very hard. This algorithm is the best algorithm, but it requires the most finicky hyperameter tuning
Roy Rinberg: and because evaluation is an expensive operation. If every time you want to learn something you have to like, do a whole hyper tuning, and you don't trust that it's going to work for something else. This is an unusable algorithm. And so
Roy Rinberg: everyone generally agrees you would much rather have a slightly worse algorithm that is, requires less hyperparameter tuning than
Roy Rinberg: a better algorithm does. And so my opinion
Roy Rinberg: is avoid creating descent based methods. Because, as I
Roy Rinberg: gradient ascend based methods because of their hyperliner tuning.
Roy Rinberg: And so this notch idea that turns it into a descent problem
Roy Rinberg: is generally like the, I think, the right direction.
Roy Rinberg: Okay.
Roy Rinberg: And we're kind of gonna get into sort of a notch got? No, it's not. But
Roy Rinberg: can you? Can you cheat?
Roy Rinberg: Let's define like oracle match. Let's say like, Okay, we're going to try to do whatever we can to come up with the best and so I'm gonna just take. And I'm I'm gonna say, let's say I have sample access to a fully retrained model.
Roy Rinberg: And then I'm just gonna do gradient descent on that
Roy Rinberg: on the outputs of that prediction.
Roy Rinberg: And this kind of like immediately solves this missing targets. Problem.
Roy Rinberg: Like all the numbers, go up.
Roy Rinberg: Some of them go down. Actually. Good question. Why, I'm not sure but it's like truly on the scale of like point 0 1% of the population, maybe less. Those are good questions. They don't even go down that far.
Roy Rinberg: But look at how nice this is. They all go up
Roy Rinberg: and it's because we've well defined. We've given a very well defined like target.
Roy Rinberg: It can't overshoot it, because now we're doing only gradient descent.
Roy Rinberg: No ascent, no hyper, no real hyper ground tuning. You're just running, turning it into a student teacher model here. Yes, this is knowledge installation. Yes, absolutely.
Roy Rinberg: okay.
Roy Rinberg: who's the teacher? Here.
Roy Rinberg: yeah. Yeah. But that's why that's why it's cheating. Right? Because you need a teacher. Yeah, that's right. This is more of a joke of just cool stuff like, morally, it's wrong. So it's not a teacher. It's
Roy Rinberg: it's it's it's somebody who's. This is the oracle. The retrained model is the teacher. We call this oracle metric.
Roy Rinberg: but we have no oracles in life is, I guess, the okay.
Roy Rinberg: what do we have? We have data models. We have this thing called like data models. Are these things called predicting predictions.
Roy Rinberg: This slide might be.
Roy Rinberg: okay. There's a weekly opinionated nomenclature slide. You guys probably actually are exactly the wrong people to show this, because you're probably more informed on this. But
Roy Rinberg: there's influence functions. And there's data models. And there's also like data models.
Roy Rinberg: Tm, where
Roy Rinberg: influence functions, as I understand them. What people mean by that is when one data point how it affects the weights
Roy Rinberg: data models is how one data point affects another. The loss on another data point.
Roy Rinberg: And then there's specifically a data models method called data models, which is very annoying.
Roy Rinberg: Which is kind of this regression
Roy Rinberg: that I will breeze through in like 20 seconds. But other as an example, like data models are a data model track is a data model.
Roy Rinberg: This is weekly opinionated. Don't save me.
Roy Rinberg: So actually, so we so probably most of the people in room don't know what all these different things are. And okay, if it's important. Yeah.
Roy Rinberg: So we're gonna go into data models. I'm just trying to build. And I'll explain in a second.
Roy Rinberg: I think I'm just this side is really just trying to
Roy Rinberg: split between. There's this thing called influence functions that are related, that are how much
Roy Rinberg: training on this affects your model is how I interpret this usually
Roy Rinberg: But what we really care about is how much one data point affects the model's loss on another data point.
Roy Rinberg: And I'm going to explain,
Roy Rinberg: yeah, so basically.
Roy Rinberg: data models is this paper? There's this paper roughly, a couple of years ago? That they're like, can we predict predictions?
Roy Rinberg: And the answer is, yes, and what you do is you like? Take?
Roy Rinberg: A bunch. Do you take your data set?
Roy Rinberg: And you yes, let's say, Oh.
Roy Rinberg: see, the model is different, you know.
Roy Rinberg: I would say, this is all attribute, I mean. So again, opinion, I would say there's all attribution. I'm trying to make the distinction about
Roy Rinberg: data Point attribution where the question is, really.
Roy Rinberg: if I trained on the data set S,
Roy Rinberg: my my predictions on Point X compared to
Roy Rinberg: if I never saw. Why, if I never saw
Roy Rinberg: Alex or my predictions and Jayden this is different than
Roy Rinberg: what I think most influence functions are. I think you could reasonably call both those attribution.
Roy Rinberg: I don't know what people do
Roy Rinberg: just to. Yeah. So data only gives you that number.
Roy Rinberg: Yes, Theta of XS of X, minus theta of S. Yes.
Roy Rinberg: and if you have, and but I'm a little bit hiding something, and let me jump 2 sides.
Roy Rinberg: Specifically, this data models predict the difference in margins.
Roy Rinberg: Which is like just a measure on
Roy Rinberg: it's this number. It's the probability
Roy Rinberg: it's like the correct, that of the correct class in one situation subtracted by the softmax of Early. It's the soft Max of that probability, divided by the total sum of the probabilities.
Roy Rinberg: Logged but you have to do it for a specific class like basically, how
Roy Rinberg: confident are you that it's the picture of a cat.
Roy Rinberg: And for most of these, and it's a subtle point. They don't really emphasize this. But in data models and track and these data models, papers, they are asking for the influence
Roy Rinberg: of one data point on another. Data points. Correct class prediction. How confident are you in that direction?
Roy Rinberg: But we're we're
Roy Rinberg: to like immediately. Tell you what it is and then change it. We're going to be doing. We want this for each specific class.
Roy Rinberg: I want. I'm going to change the way the data model is structured.
Roy Rinberg: because I'm allowed to make a prediction, not for the correct class, not for your influence on the correct class, but on a specific class. So it's like training on you. How much should you make change the prediction on me? On that I look like a cat
Roy Rinberg: that I look like a dog like for each of the classes that exist.
Roy Rinberg: And I think I have a diagram here of it's like.
Roy Rinberg: really what I'm trying to get to, and what I'm able to do
Roy Rinberg: is say, you know, here are the original model logics.
Roy Rinberg: Here are the counterfactual objects.
Roy Rinberg: The red is a delta. It's like, you know, they're like
Roy Rinberg: the dog goes out like, if you I didn't train on you. What is the prediction on me?
Roy Rinberg: I'm trying to. My data model is trying to learn this specific delta thing.
Roy Rinberg: And that's how they're used often.
Roy Rinberg: If you read the papers prior to this work.
Roy Rinberg: They're not computing a vector they're computing just the scalar of the correct class.
Roy Rinberg: And the point of that. Why are they doing that is
Roy Rinberg: they're just trying to ask what points matter?
Roy Rinberg: They're not going to ask, how much do they matter in specific directions?
Roy Rinberg: The.
Roy Rinberg: I think, given that we're like running a little on time. I'm gonna avoid explaining how specifically data models work because it doesn't matter.
Roy Rinberg: We're going to be using data models, not data models. Trademark.
Roy Rinberg: Sorry about that.
Roy Rinberg: okay. But now you've kind of like already. Solved your problem. If you have access to data models, because actually, we'll show you this slide.
Roy Rinberg: If you have this right. You have this like, ignore the rest of it.
Roy Rinberg: You have how much Y contributed to your prediction on XI can sum up all of these guys
Roy Rinberg: in my forget set.
Roy Rinberg: And then I have, like a reasonable approximation
Roy Rinberg: of how much of what my data set looks like, like, literally, just subtraction.
Roy Rinberg: Yeah, absolutely where?
Roy Rinberg: Because some of the logics also go down right? So it could be absolutely badly right?
Roy Rinberg: If training on you makes me look more like a cat and training Alex makes me look like less like a cat, and with both of you, then I should look like a equally like a cat. Yeah. So actually, it's not absolute value.
Roy Rinberg: that's why these can go negative.
Roy Rinberg: It is a good question of
Roy Rinberg: you know, this is this is a.
Roy Rinberg: This thing is doing a lot of work here.
Roy Rinberg: people can. The group influence of something is not the same as the sum of all the individual influences.
Roy Rinberg: Turns out they're close enough.
Roy Rinberg: Both in our work and in previous work they've just like kind of seen this.
Roy Rinberg: But very fundamentally, these are not the same thing. That's where that approximation is coming from.
Roy Rinberg: I think Theta can be like anything like any model, any model you want.
Roy Rinberg: any model you want that you can reasonably use it.
Roy Rinberg: Data model for. So so I'm not familiar with it. So I've I've heard of the data model that you're going by. I'm not familiar with it. So is this something? Is it practical? Is this something? It
Roy Rinberg: data models. Trademark.
Roy Rinberg: it's on the order of magnitude of like 10,000 retrainings. Not not practical track is an approximation of this that does worse, but reasonably, is on the scale of 10 to 100 retrainings. Were you on the track paper? No, okay, yes.
Roy Rinberg: the. So. The the short answer is, this is not reasonable for
Roy Rinberg: the the thing I'm about to tell you guys, it's not going to be reasonable. For large language models for 2 reasons. One is
Roy Rinberg: even doing 2 retrainings is too many retrainings, let alone a hundred. The other is
Roy Rinberg: this whole point that I was making this like settle point thing
Roy Rinberg: you have to do it for each class. You have to make a data model for each class in classification. Cfar 10, you have 10
Roy Rinberg: at least naive.
Roy Rinberg: A way to transfer to a language model to a classification is each vocabulary, and it's 50,000 that's just too many.
Roy Rinberg: But to like at least show you some results.
Roy Rinberg: What like we did?
Roy Rinberg: Can we just replace our oracles with data models? And the answer is, Yeah, it works. It's like, it's pretty good
Roy Rinberg: where this is. Scrub. This is for just the forget set plot.
Roy Rinberg: this oracle matching. And there's data one. It's like they're pretty similar.
Roy Rinberg: They're not perfect.
Roy Rinberg: And we'll show like more results in a second.
Roy Rinberg: But the like. 1st point is that
Roy Rinberg: we've gotten like this impractical out of them, because it's so computationally expensive does work.
Roy Rinberg: This plot.
Roy Rinberg: I takes a moment to plan
Roy Rinberg: because we've now condensed our Clom scores one more time.
Roy Rinberg: If you remember, I'm just going to show you the climb score again.
Roy Rinberg: Write this gun.
Roy Rinberg: What we're gonna say is like, okay, this is still too much information like, I don't wanna have to show a entire histogram for everything. So I'm just gonna report to like 99 percentile
Roy Rinberg: and that's it. That's the one flattening again that we're going to do.
Roy Rinberg: And here is our like headline plot, where gray is existing methods.
Roy Rinberg: This dashed line is doing nothing.
Roy Rinberg: And then the cons. Yeah, let's start with, the
Roy Rinberg: the wax is calm. Down is good.
Roy Rinberg: This is your forget set. It's your retain set. This is your validation set
Roy Rinberg: the X-axis is training. Time
Roy Rinberg: rescaled according to like full retraining.
Roy Rinberg: These red dots are
Roy Rinberg: just fully retraining model. This is like the benchmark. It's going to work optimally. It's this bottom right corner. It's perfect.
Roy Rinberg: But it also takes the most amount of time.
Roy Rinberg: The gray lines are the things that exist already, and they kind of don't work that well, they're kind of.
Roy Rinberg: Maybe they work well on validation and retain. But they don't work here, and if you care about some combination, this is just the weighted average.
Roy Rinberg: And then you have Armad data model matching
Roy Rinberg: it just like plummets down very quickly.
Roy Rinberg: Yeah, the the like. This is the headline, Takeaway, it said, we like our Pareto optimal.
Roy Rinberg: This is an empirical result to like immediately
Roy Rinberg: slightly qualify our results. It's that like, we're plummeting down
Roy Rinberg: at unlearning time. But you have to do this really big pre computation cost. Yeah, of computing the data model.
Roy Rinberg: The nice thing of why is this somewhat reasonable is that you only do this once, and you can like forever.
Roy Rinberg: like every unlearning request, does not add more compute because
Roy Rinberg: at least for a long time, those influence functions. They just work. They like.
Roy Rinberg: continue, you can continue to step up.
Roy Rinberg: And so yeah, the one that I just wanted to show this one plot
Roy Rinberg: just slightly different. But it's just adding track
Roy Rinberg: data model, which is another data model matching. And it like
Roy Rinberg: doesn't work as well. But it works pretty well.
Roy Rinberg: That's all this block showing. I won't focus on this.
Roy Rinberg: We're actually basically done.
Roy Rinberg: there are plenty of future directions. I think this is like, I view this more of a science paper than an engineering paper.
Roy Rinberg: I think you like future directions are things like
Roy Rinberg: oracle matching. We're using an article. But it's not actually optimal. We haven't proven that it's optimal. We just kind of guessed
Roy Rinberg: there are certainly better ways that you could do this knowledge, distillation from the oracle.
Roy Rinberg: and so doing work, even in that theoretical setting, would improve.
Roy Rinberg: The data model setting
Roy Rinberg: group influence is definitely not some of leave one out influences. This becomes a bigger problem as you compose on learnings like.
Roy Rinberg: it's very different. It's going to be different if I say
Roy Rinberg: I'm unlearning all of you versus if I unlearned one of you. And then I learned the maximum list.
Roy Rinberg: understudied or basically not studied.
Roy Rinberg: I think this is the more interesting area, the like making it scale
Roy Rinberg: can you compute what are influence, functions or attribution functions that you can use. How do you scale this to language models
Roy Rinberg: all great questions?
Roy Rinberg: And then the last question that I think is like.
Roy Rinberg: I kind of just threw at you, I said, like this is a good measure of unlearning quality.
Roy Rinberg: is Klum.
Roy Rinberg: I think it's the best one we have. I can argue this quite well.
Roy Rinberg: but like it's not good, I think we can do much better
Roy Rinberg: and sorry it is good, but I think we can do much better.
Roy Rinberg: That is technically the the tag. There is what that is. Now question time.
Roy Rinberg: And these are pictures I took.
Roy Rinberg: Thank you, Rick.
Roy Rinberg: Yeah. So like the class number, not like the size of the model is that, or is that the size model? It's a multiplication, right? Because it's
Roy Rinberg: you have to do a data model for each class.
Roy Rinberg: So if each data model is our
Roy Rinberg: even doing one data model on a language model would be too much. People haven't done it really
Roy Rinberg: and then doing it 50,000 times would be also too much.
Roy Rinberg: Yeah. So for each class, you need to store the whole model
Roy Rinberg: like for each class. We need to show the whole language model
Roy Rinberg: because I saw that you were doing Std on Theta right? The language model itself.
Roy Rinberg: So yeah, you're doing it on the logic. Vector sorry. This is this is a, vector
Roy Rinberg: so I mean size class. I guess the question is actually, the data model itself is a model.
Roy Rinberg: Yeah, no, it's a, it's like a linear regression model. But in practice it's actually yeah. It's a
Roy Rinberg: the way many
Roy Rinberg: of these things work it it. The the size of the model is is something along a linear regression. Basically. What you do is you
Roy Rinberg: cake.
Roy Rinberg: I'm going to do an okay job of this, you take your data and you randomly sample
Roy Rinberg: a bunch of subsets of it.
Roy Rinberg: They're like 50 percenters, say, like 10,000.
Roy Rinberg: And then what your goal there is is you're gonna say, like, there, you're able to say.
Roy Rinberg: there's going to be points that include Point X, say, Alex.
Roy Rinberg: and have you in it, and and a bunch of models that don't have you in it.
Roy Rinberg: Half of them are going to have you in, and then a quarter of them are going to have you and Alex. And so.
Roy Rinberg: by like conditioning on those, I'm able to make those predictions and
Roy Rinberg: what they do in this problem is they like.
Roy Rinberg: I'm I'm not gonna do a great job of explaining it. But they turn this into big hyperplane because
Roy Rinberg: this is a binary vector.
Roy Rinberg: And for each right, like, it's a
Roy Rinberg: a bunch of ones and zeros.
Roy Rinberg: And you have, like 10,000 of them.
Roy Rinberg: And for each point.
Roy Rinberg: You ask for the prediction on
Roy Rinberg: that point, for the models that included it and the models that included
Roy Rinberg: the short answer is, it's not a language model. It's a regression problem even. And and track is as well.
Roy Rinberg: The slightly, the slightly more nuanced answer that you weren't asking is in practice.
Roy Rinberg: There are 2 things that are the output of these like
Roy Rinberg: engineering things, like in theory, we can talk about it. But in in practice there's the model.
Roy Rinberg: But you actually just pre-compute the like
Roy Rinberg: n, by N. Matrix of the influence of Point X and point Y,
Roy Rinberg: and to compute the influence on a point outside of that validation set
Roy Rinberg: is more expensive than to than the original operation of that generation
Roy Rinberg: does that was that like a sufficiently roundabout answer, I think that's helpful. So it's also like, yes.
Roy Rinberg: sounds like, yeah, that would be right. But that that's the easiest of these to to resolve, because
Roy Rinberg: you're choosing to make this N. By N. Matrix. And if you wanted to make it a smaller subset.
Roy Rinberg: you just to pick the ones that you're going to subsample.
Roy Rinberg: And so that's actually where my point of like, you know, oracle matching isn't perfect.
Roy Rinberg: What points are you going to oracle match on?
Roy Rinberg: Is a is a good question. What points? How do you like knowledge, isolation? The whole point is that you like need fewer points. And so
Roy Rinberg: asking the question, of which points do I want influence on
Roy Rinberg: reduce the need for this? N squared matrix
Roy Rinberg: Sarah. Something related. But I think, like, it's like this question. And also we need to use that.
Roy Rinberg: But I can.
Roy Rinberg: Oh, I can. I can see it whatever. Yeah. And so like, if you go to like scaling to language models you talk about like oh, like the naive thing is like talking about like every single like
Roy Rinberg: Oh, cool! Oh.
Roy Rinberg: if you're trying to like, unlearn every single token or something, as there are like classes like on one end, of like sort of not convinced that. Say, if you like, unlearn the token ricin you've like actually successfully like the model can probably just like reconstruct this in the aggregate of the other tokens. And then, like this is like sort of related to like, what's your take on? How do we actually remove the concept of, say, ricin? Yeah. So I think there's actually 2 important thoughts in your question. One is
Roy Rinberg: like, what is an example in privacy to like, kind of redirect your question to like where I think this is a much more troublesome worry is
Roy Rinberg: differential privacy, if you're familiar with it, is this like guarantee of like neighboring data sets that, like your one data set. The output of your model is not
Roy Rinberg: too dependent on the on any one data point.
Roy Rinberg: But, like, what is a data point?
Roy Rinberg: What is your definition of neighboring is
Roy Rinberg: just a philosophical question that we know how to answer in picture space. We don't know how to answer in language space.
Roy Rinberg: it's the same question from learning. It's like, what does it like?
Roy Rinberg: Yeah, what does it even mean to to be trying to do concept and learning as like a very acute
Roy Rinberg: thought process. Thought example.
Roy Rinberg: In Gdpr, there's like legal
Roy Rinberg: thing that Europe has. That is why you have to answer all these cookies.
Roy Rinberg: and they give you this right to be forgotten.
Roy Rinberg: You can go to them. And you can say, please, delete all my data
Roy Rinberg: and please delete all pictures of me, and I actually just have to like, tell you the concept like they. I don't forgive them all the pictures of me.
Roy Rinberg: They have to do it regardless. So there, I'm asking for the concept of me to be unlearned.
Roy Rinberg: And this splits up the problem of unlearning into 2.
Roy Rinberg: If you are doing data and learning. There's like, you've now broken into 2 parts. There's like finding the data set you're trying to unlearn. And then there is the like
Roy Rinberg: actual data unlearning
Roy Rinberg: if you wanted to go the data and learning route, if you wanted to go the concept and learning route.
Roy Rinberg: I think you're not going to be able to do it in any kind of statistical, meaningful way, but you can still do it in like a human, meaningful way of like.
Roy Rinberg: If your goal is to remove the concept of ricin being used in to to murder people like
Roy Rinberg: I strong, I strongly think, and I think a lot of people agree.
Roy Rinberg: Unlearning. The concept of of Bryson in the concept of murder is not the right way to do that.
Roy Rinberg: I don't know if I like, actually answered your question. There's a there's a there's a sort of a
Roy Rinberg: maybe theoretically happy way of stating concept and learning that we're sort of settling on, which is.
Roy Rinberg: you know. Maybe if you say you have a classifier for data
Roy Rinberg: or output that would represent the concept that you're trying to unlearn. So like here, like your setting is, you've got a data point
Roy Rinberg: that you're trying to remove right?
Roy Rinberg: And so. But but you could imagine a problem set up where you say, oh, no. I have a classifier for data points that I would like to remove.
Roy Rinberg: And then and then you're sort of left with this interesting question of like, Oh, can I go straight from
Roy Rinberg: maybe a classifier to getting a model that would result removing that concept does that make sense without having to stop at the data points? So that's I think that's actually exactly the like kind of distinction that I'm trying to break of. There's finding the point you want to learn, and the actual right. The classifier is finding the point. Yes, so suppose somebody gave you the classifier.
Roy Rinberg: Right then the question is, oh, you know.
Roy Rinberg: there's maybe there's a class of unlearning methods which use the classifier as a starting point. So here's my thought on something that should be done related to this. It's
Roy Rinberg: okay. Let's start with the philosophical problem of.
Roy Rinberg: I have this data set trained on people.
Roy Rinberg: And there's 2 pictures of me in order, like there's a hundred pictures of me and him, and I ask to be like, I give it a data set with 50 pictures of me. And I say, unlearn this.
Roy Rinberg: What do like philosophically, what do I want to do with? Okay, I'm going to say 40. So the numbers, I'm the same. What do I want to do with the other 60?
Roy Rinberg: Like this is a philosophical question. And I think
Roy Rinberg: this actually puts data on learning and concept, unlearning on the same spectrum where
Roy Rinberg: you would. I think there's a dial here which is how concept unlearning versus data point on learning, and that is going to be some threshold
Roy Rinberg: on the influence of one data point on points in the retain set.
Roy Rinberg: Basically saying, My forget set is not the forget set, but it's all points
Roy Rinberg: that are within some influence function of my forgets it.
Roy Rinberg: And I think that
Roy Rinberg: is how and and and why does that put this on a dial? It's because if the influence function is anything, then like that's
Roy Rinberg: forget everything like that's it might be more, I mean, more complicated than a dial in the sense that like
Roy Rinberg: oh, like a different classifier, might generalize differently, like on different.
Roy Rinberg: you know, axes like, Oh, maybe you train a class you want to classify with a certain inductive bias which really doesn't matter. It doesn't care. That's right. Image is blurry. You're wearing different clothes. That's right. But maybe it's very sensitive to your personal identity there that you're your choice of influence function like the the other axes that you're proposing here, I think, is your choice of influence function, not just the threshold over which right?
Roy Rinberg: But this is how I would combine data and concept elements.
Roy Rinberg: I want to just read the message
Roy Rinberg: from Sarah. Yeah, this is sort of what Sarah's asking. She's asking about what? What's what's what's so that's why we're driving the discussion this way.
Roy Rinberg: Yeah, I
Roy Rinberg: I think often consequent learning is a little. It's like a. It's a human desire that we have to do this thing, but it's it still.
Roy Rinberg: But I think that you can. So that's what I'm saying is that I think that you can turn into a 3rd of the problem. Say, yeah, there's this human judgment of what the classifier is. Yeah, let's say, somebody said, you know, we've we've debated this as humans. And we think that this classifier, is it?
Roy Rinberg: But but now, now you've still got this mathematical engineering problem of
Roy Rinberg: all right. I've got a classifier. I'm going to treat the classifier as the oracle of things that I wish the model had. That's right.
Roy Rinberg: Right? Then there's this problem. Yeah, yeah, yeah, right? I think this is a good direct. I'm honestly starting a project. I'm I'm starting on another follow on work in this, and I haven't started yet, and, like, you know, taking collaborators and and like, I think there's a good direction. Cool?
Roy Rinberg: Well, exit. Yeah.
Roy Rinberg: Sarah, did that answer your question.
sarah schwettmann: It. It did. Sorry I'm running around trying to start commuting but I guess I was also thinking like.
sarah schwettmann: whether we want to go in and do this at the at the representation level directly, and maybe that's just the same thing as the classifier question. But wouldn't it be nice if you could go into the model and just remove all of the stuff. That related to some ideas right.
Roy Rinberg: On!
sarah schwettmann: Be more satisfying than doing it at the data level.
j.fiotto-kaufman: Okay.
Roy Rinberg: Yes.
j.fiotto-kaufman: I don't. So there is
j.fiotto-kaufman: work that shows that sparser models are easier to unlearn, or they.
Roy Rinberg: And better.
j.fiotto-kaufman: And.
Roy Rinberg: And.
j.fiotto-kaufman: Like pseudo. Scientific explanation here is that spectrum models have more monosemanticity.
j.fiotto-kaufman: I have.
Roy Rinberg: Like empirically sure, I think this is.
j.fiotto-kaufman: Not.
j.fiotto-kaufman: I'm I don't love this. I'm just saying like.
Roy Rinberg: Let's just like make the models further, and then.
j.fiotto-kaufman: But this isn't. This isn't making the point of like.
Roy Rinberg: And if we, if we have monosympathity, then you can just like.
j.fiotto-kaufman: Truly remove that.
Roy Rinberg: That neuron. I think that won't happen. I but that that's an opinion.
j.fiotto-kaufman: I think there's at least.
Roy Rinberg: Is at least reasonable to make a statement of like.
j.fiotto-kaufman: Empirically. If you in reduce monosymmetricity through scarcity, then I'm learning becomes easier because I don't know it's easier.
j.fiotto-kaufman: Can I? Can I invert the question the other.
Roy Rinberg: Right.
j.fiotto-kaufman: So you've been playing with data models for a while, working with this working with this stuff.
Roy Rinberg: So. Do you think that.
j.fiotto-kaufman: You know. In a sense.
j.fiotto-kaufman: the data model is the answer to interpretability like it. It tells you the structure of how models are generalized. And it tells you the structure of what models are learning from the data
j.fiotto-kaufman: right?
j.fiotto-kaufman: And if you understand that the structure of the data model, you understand the structure of generalization in the model
Roy Rinberg: Probably not.
Roy Rinberg: I think.
j.fiotto-kaufman: For 2 answers, for 2 reasons. One is
j.fiotto-kaufman: computationally, it's very expensive. I think that cannot be.
Roy Rinberg: Reward, and.
j.fiotto-kaufman: Like a point.
Roy Rinberg: Be useful if it's.
j.fiotto-kaufman: Not if it's too expensive to work.
Roy Rinberg: Language model it just like isn't useful.
j.fiotto-kaufman: Philosophically, whether or not you want
j.fiotto-kaufman: data point to model or data point to data point, I think I promise data point to data point
j.fiotto-kaufman: at the end.
Roy Rinberg: End. But I think.
j.fiotto-kaufman: Yeah.
Roy Rinberg: I don't know.
j.fiotto-kaufman: You think it's not? It's not capturing the right thing. I think it's a good abstraction. I'm I'm
j.fiotto-kaufman: unlearning is a really good use case for it.
j.fiotto-kaufman: Influence functions are reasonable. Use case for like data set curation.
j.fiotto-kaufman: But what are the goals of interpretability? They're like
j.fiotto-kaufman: safer models that we know what they're going to do and like this
j.fiotto-kaufman: is a related but ultimately orthogonal goal here
j.fiotto-kaufman: did I answer, like, I don't. I'm kind of.
Roy Rinberg: And you go by example.
j.fiotto-kaufman: Yeah, give me a vibe. Yeah, I'm pro.
Roy Rinberg: I think.
j.fiotto-kaufman: One great project related to this is, taking these other.
Roy Rinberg: Everyone's fine.
j.fiotto-kaufman: Questions specifically for the.
Roy Rinberg: Converting them to be a data point to data point.
j.fiotto-kaufman: And then. But given that influence function.
j.fiotto-kaufman: are cheaper to commute because they're I'm.
Roy Rinberg: You can just do them at the.
j.fiotto-kaufman: On the final model.
j.fiotto-kaufman: If you have some kind of access to that.
j.fiotto-kaufman: can we do a much cheaper.
Roy Rinberg: On, the.
j.fiotto-kaufman: Learning through data models.
j.fiotto-kaufman: Sorry. Yeah. Data models not trademarked. Like, just to come.
j.fiotto-kaufman: I think that would be a pretty good direction where I'm learning
j.fiotto-kaufman: But but you mean unlearning, using influence functions instead of unlearning using.
j.fiotto-kaufman: Yeah, I'm.
Roy Rinberg: Right now I'm.
j.fiotto-kaufman: I think what I want is gradient based data models.
Roy Rinberg: Don't exist right now.
j.fiotto-kaufman: I want want a method that is cheap that we'll get.
j.fiotto-kaufman: Give me a half decent influence of one data point on another.
j.fiotto-kaufman: I don't think this exists right now.
j.fiotto-kaufman: I could, and I could just be. But that's sort of I mean, that's that's influence functions right? Isn't that? Or.
Roy Rinberg: Yeah, I was the reason why I'm I'm the reason why I'm like a vacation.
j.fiotto-kaufman: About it is because I had this like weak opinion slide. Sure where.
Roy Rinberg: I kind of.
j.fiotto-kaufman: Explicitly said that it wasn't that. But this is
Roy Rinberg: This is not a.
j.fiotto-kaufman: Formal definition. This is just the literature that I've read seems to kind of fall along the lines, but I think the room of A,
j.fiotto-kaufman: I think one great direction for unlearning would be a gradient based
j.fiotto-kaufman: data model. That is data point on data point influence.
Roy Rinberg: I don't think I've seen this.
j.fiotto-kaufman: But maybe I just, you know, read it a long time ago and forgot about it.
j.fiotto-kaufman: Unlearned it.
j.fiotto-kaufman: to like, you know, unpromptedly. Just talk about safety.
Roy Rinberg: I think.
j.fiotto-kaufman: There is little evidence that I'm learning can be used.
Roy Rinberg: For safety training.
j.fiotto-kaufman: I think, with especially the fact that you can introduce things in in context.
j.fiotto-kaufman: large language models you can just like.
j.fiotto-kaufman: like imagine I I don't know. I'm kind of coming up with some of the fly, but, like your model knows about like chemistry and.
Roy Rinberg: And it.
j.fiotto-kaufman: Knows about like things you can fit into suitcases like you could
j.fiotto-kaufman: it? You can tell it to like, build a bomb in context, and it'll like do it, even if it
j.fiotto-kaufman: removed all the things from training. This was on the fly, and maybe it's a bad example, I guess, like in cases like this. It's like, if I already know.
Roy Rinberg: They'll live on the teach them.
j.fiotto-kaufman: Model in context, like why, why do I care?
j.fiotto-kaufman: The model knows.
Roy Rinberg: More than you.
j.fiotto-kaufman: And so maybe you know some things, but not like how to do an integral or something I don't know, like, you know, you have to do an integral, you don't know how.
j.fiotto-kaufman: That's like a pretty reasonable world.
j.fiotto-kaufman: So you might live in.
j.fiotto-kaufman: So you'd like teach it the concept of an integral and hope it like generalizes to. There's this issue right like, if you have this huge scratch pad and then the model. If the model needs to figure out how to make a bomb along the way. Maybe it'll say, Oh, well, I don't know how to make one, but
j.fiotto-kaufman: I've got. I've got a few terabytes of scratch pad. I could probably work it out.
j.fiotto-kaufman: Yeah, I think.
Roy Rinberg: I'm learning.
j.fiotto-kaufman: Promises a lot. I think the like.
j.fiotto-kaufman: Let's go back to the use cases.
Roy Rinberg: Although we over, I don't know what you're like. Culture.
j.fiotto-kaufman: Of going over is I'm financial chatting, but.
Roy Rinberg: Is one of the.
j.fiotto-kaufman: Sure people don't feel trapped.
j.fiotto-kaufman: So feel free to leave, I guess, is what I'm saying. Should we? Should we? Should we let? Should we like?
j.fiotto-kaufman: Should we wrap up the hour and then let people
j.fiotto-kaufman: you know, chat with Roy informally in the room.
j.fiotto-kaufman: Let's let's do that.
j.fiotto-kaufman: And thank you. Roy. Yeah, really appreciate it. Yeah, thank you.
Roy Rinberg: Yeah.
j.fiotto-kaufman: Get there all right.
j.fiotto-kaufman: Alright! We'll do that, hey? Hey, Sarah, do you? Wanna do you want us to leave you on.