David Bau: To the cloud. Okay?
David Bau: So but you pushed up
David Bau: so. But, Kevin, you're a student here, right? Yeah. Or you're not a student at Nyu. No, right so
David Bau: so but but it's a collaboration with what's his name? And is there also a student of nips at Nyu? Is it just niv, oh, there's 2 students. So it's actually quite a team.
David Bau: Okay, that's wonderful. So let me introduce speaker today. So, Kevin, what year are you? I'm on 3rd year, 3rd year. So 3rd year undergrad at at northeastern Kevin has been working with
David Bau: rohit and Nip Cohen at Nyu.
David Bau: So niv has visited here a couple of months ago. So, as you know, he's very interested in these adversarial situations, and, as you know, you know, Rohit has been looking at the unlearning
David Bau: sort of discipline. Right? This unlearning setting. And so the big question is, once you unlearn something. Is that robust, or you know, is it easy to undo and so niv was very interested in the question. I think that you know this. This also captured
David Bau: Kevin's imagination, and so they have some very, very interesting findings that they've been writing up, and I think making its way through peer review, what? What's what venue have you submitted that for? Peer review? Okay, that's right. So
David Bau: so so yeah, so we're looking for it
David Bau: to hearing about these really cool results from Kevin. And if you're if you're submitting to neurops, then there's a good opportunity to get feedback from everybody on ideas on this research in progress. You know, in this this sort of interactive style. So, Kevin.
David Bau: welcome.
David Bau: I hope you guys give a lot of feedback because I think I have a lot of questions just about like different directions to take in terms of the research. But also, I'm just gonna get my water.
David Bau: Oh, it's okay. I'll just save the tea. Yeah, this is fine.
David Bau: Okay? So Hi, everyone. Welcome to my presentation on kind of undercovering what happens to race concepts in diffusion models. This is work I did with Rohit and some other people at Nyu, like Nivco and Min and Nikki.
David Bau: So just as like a quick diffusion refresher. I think the core idea really is that these diffusion models are trained to predict and remove noise from a like. You start with some random noise, and over maybe 10 steps. It'll predict some noise to remove from this
David Bau: starting messiness. And so clearly we slowly begin to see our target image. Maybe in this case an image of a dog come up over these 10 steps. And so what's really interesting here is that the image is generated iteratively. So it's not like we're one shotting any pixels. It's like, we're all the noise is actually stored
David Bau: coming from like a 64 by 64 matrix. So it's like we start with some noise in a latent space, and then we'll have to pass that through a decoder to get our image. And so I think that has a lot of implications for interpretability within. Diffusion models itself. So I have, like a kind of a quick side on looking at interpretable Llms versus interpretable diffusion. Because I feel like you guys do a lot of work with Llms here. I think there's it's really cool. What you guys are able to do. Given that Llms, its outputs are token by token, like, you can really have some clear input to output mapping. And so like, there's a lot of
David Bau: different tools you can apply in this case, like, there's like, Pat, like, I really thought, like, patching is really interesting, like, you can patch one aspect of the model, and like, see what the outputs, how the outputs change. And I think, like it's really causal, which I feel like. It's 1 of the coolest parts, but when it comes to the fusion we have a single image that is generated iteratively. So semantics are kind of hidden like. We might get an image of a dog. But
David Bau: can we attribute some aspects of that dog to specific parts of the model? It's it's kind of difficult to answer that question. Since semantics are kind of hidden in this output of this fusion models, which is noise. And we're also working in the vision space. So it's a little bit less understandable in terms of the output, and like we might get an image that makes no sense at all.
David Bau: And so, as a result. There's a lot less powerful interpretability tools that sometimes yield ambiguous results like, still we can. There's still a lot of interpretability work happening. And like I was looking at Chris Wendler's twitter, and like he had some really cool stuff with looking at ses and like breaking down the features within the fusion models. But there's still a lot of work to be done in terms of understanding like the semantics within it.
David Bau: So this is kind of why I really like to. I like to see concept erasure as a lens for interpretability. And I'm asking this question of what happens when you erase a concept in a diffusion model. And here it's the goal is kind of to understand how semantic knowledge is organized and accessed in these models. And so
David Bau: really like, if I erase a concept of a car, and I reprompt it with the same prompt what happens like, how how do these inputs relate to the outputs in terms of noise and the image that I get in the end. And what about when I try to jailbreak this erasing model like, what is the behavior that these erasing methods exhibit.
David Bau: So these are the kind of the questions I'm asking. And so, just as a quick refresher on what concept erasure looks like. Generally, it arose with this goal of preventing image generation models from producing violent or not safe for work content. So if we have our standard diffusion model which was trained on these huge data sets of images from the Internet. I could ask for a violent image like a man with a gun. I could get something like this, or say we have copyrighted contents like, let's say Van Gogh was copyrighted. I could ask for starry night.
David Bau: so one way you could approach this is by taking the data set that the TV model was trained on and just taking out all of your dangerous content so that you can retrain. But that's obviously really expensive. So most of the state of the art concept erasure methods depend on fine tuning here, so I can fine tune my model for concept erasure, and if I reprompt with a man with a gun I might just get a man, and if I reprompt with starry night I might just get a night sky. So this is just an example of what concept erasure really could look like.
David Bau: And so for my research, I looked at a few different concept, erasing methods. So here's 1 example, it's Esd by Rohit. And so the goal here is we have these diffusion models which are predicting noise. And it's conditioned on your prompt. So if I'm asking for name image of a car, it'll predict like sort of car noise. And this his fine tuning process is basically trying to get the model to predict noise
David Bau: of a concept that's called like anything but Van Gogh. And so we're trying to steer the diffusion away from this idea of a car.
David Bau: So if I ask for an image of a car from the original model, I might get this with a mountain and grass. But after going through Esd and some fine tuning, we might just get an image of a mountain in the grass without the car.
David Bau: And so this is kind of what this loss function represents. And just like one example of
David Bau: fine tuning for a ratio, a different example is gradient ascent, and so often with these diffusion models, you can fine tune it so that you can get the diffusion models to consistently generate a concept. No matter if even if you don't prompt it with like a car. So that's like what a traditional fine tuning process looks like. And what we can do is we can actually flip the sign. So if this loss function is meant to
David Bau: ask the model to consistently generate a car, and we flip the sign we're doing gradient sent here. And so the idea is we're trying to get the model to predict the opposite of a car.
David Bau: And so often this can lead to kind of unstable results. But it's pretty effective at erasure. And so this is just an example of if I reprompted for car with a gradient ascent train model. I might just get random noise, and it's pretty messy.
David Bau: So these are just 2 examples of erasing methods. But I actually ended up looking at. Oh, yeah.
David Bau: when you say effective at a razor, you just mean the thing that you asked for doesn't show up. Yeah, I have a good algorithm for that, too. It's called Glarot initialization.
David Bau: Right? You just you just you guys just go to torch.in it on your water weights. And then, yeah. And so that is like with the ratio. There's this balance of like. Yes, I don't want to complete. Yes, I want to get erase my concept. But I also don't want to completely destroy the model. And so that's something that we actually end up evaluating with erasure often like kept in the back of your mind. So yeah, does that happen? Even if you have some strong, regular
David Bau: so that with with greeting, assent, I know we tried doing some regularization. Both in terms of like, maybe you have some preservation concepts. And I think it's like the there's not really a perfect solution there yet.
David Bau: I think mainly the from what I saw. If you added some preservation concepts
David Bau: you actually just end up. You can preserve, like maybe an image of like a truck. But then some other concepts will probably just be inevitably destroyed.
David Bau: But then, with regularization, it's like, I regularize so much that the concept still exists. So it's like a really fine line. And I think grading ascent itself is just like a fairly unstable technique. But it poses a lot of interesting results in terms of like the interpretability side of concept of issue.
David Bau: So are there any other questions?
David Bau: Yeah, I guess in my personal experience you can sort of instead of gradients. And you can edit inverse like information on top. So that's more like a natural gradient. Just can you explain more on? Maybe we can talk offline. Basically, there's a regularization method. I tried. So that'd be really nice, actually. But yeah, so these are just 2 examples of erasing methods. And like, there's a whole menu. So
David Bau: once again, if I have my target concept of golf ball, I'm trying to erase this. We looked at basically 4 different methods. And we split Esd into 2 different types. So Esdx, Esdu, unified concept, editing test vectors and gradient sent, as you can really see, like the behavior of these erasing methods differs wildly, and, like Rainy Sn. Is showing some interesting results. But really I had some key questions I wanted to ask, which are
David Bau: one like what is really happening with the concept within the diffusion model, like, is golf ball truly being erased? Or is it just being remapped to some other concept like person? And similarly like, why are some of these
David Bau: erasing methods more robust to attacks? And so in the field of concept erasure? These erasing methods are called defenses and ways to jailbreak them are called attacks. And so there's a lot of different evaluations that we can do to evaluate robustness.
David Bau: And so I guess I can start with kind of how we approach this question. We we looked a lot at this adversarial evaluation for attacks, and so like, just as another example of what we did. So we're trying to bring back the concept of English finger spaniel, and we've erased it. So this is kind of what the erased models look like. Now, what you can do is you can
David Bau: build a data set of a bunch of images of English finger spaniel and show that to your diffusion model. And
David Bau: this is a white box of text. So we have access to the text encoder and the unit. And basically, we do a optimization to search for a text embedding that will bring back our target concept. And so here, with textual inversion, we can bring back a lot of different images, images of English finger spaniels for Uc. And Esd. And like something kind of weird for Esdu. But then, when it comes to task vectors and grading ascent, we see that
David Bau: textual inversion fail. And so we did. Textual inversion on 13 different concepts. And so something that we saw was like, consistently like Uc. And Esd would
David Bau: be able to recover your target concept, but not test vectors. Ingredient set. And so, with all this interesting behavior in mind. We were trying to understand, like, why is this occurring? And like looking at the formulations behind Uc and esc like, can we somehow provide a model for what's going on within these erasing methods. And so that's like the focus of the research.
David Bau: I'm just trying to understand here. So the 1st row here is the output after you. I've applied the the corresponding
David Bau: erasing methods which all of them seems to be working. And now on on the same problem, let's say, yeah, on the same problem. And then
David Bau: you apply your attack and attack is basically what you you learn. Or you learn the tokens or what do you learn? Yeah, it's a it's a text. And you don't even need to learn. A specific. What they do is they like fine tune, the text encoder. And it's basically like, you're learning a specific embedding.
David Bau: Oh, yeah, okay, like, some people will learn the embedding and then do an algorithm to produce tokens. And like that way, you can see a prompt but this specific attack does not do that. How many training images did you? I think it's 500 training images. And like the, it's like 3,000 steps for the textual version. And you're literally learning like 700 numbers, just like one embedding for one token. Yeah.
David Bau: 500 images, 500 images of English.
David Bau: I think the embedding is a little bit longer longer than that.
David Bau: yeah, but it's you can use. Usually, you'll start with like, one token learning embedding for that. Yeah, 768, 7, 68. Dimension number. Yeah. Okay, yeah. Okay, yeah, can you go to?
David Bau: I just wanted to update. So you learn that vector the text vector then you feed that vector.
David Bau: in the model. And for this corresponding methods. For some of them it does produce it for the task vector and what was the last one gradient. So
David Bau: for the task vectors? It is not able to recover anything meaningful in the vector
David Bau: okay, have anybody tried to? Kind of like, include this in the training objective as well include, like you're trying to attack your approach. But you're using approach is supposed to be robust against this attack. Yeah, like 2 different objectives. Each one is getting smarter, simul simultaneously.
David Bau: Have any? Have anybody tried that? Are you asking like? So I think I know what you're talking about like, has anyone done like an attack on your racing method, and then made a new racing method that both of these will kind of like train simultaneously. One is trying to attack your like concept rating technique. And another.
David Bau: and your concept raising technique is trying to raise it while also making sure that it is robust against this kind of attack? Yeah, I think so. Well, I guess there's like that one method. That was, yeah. It did textual inversion. And Esd at the same time. Yeah. And they it just like you did Esd, with like. And then you ran textual inversion on it. And you got all those new images that textual inversion brought up. You do? Esd, again.
David Bau: And you do textual version again. And it's kind of like an iterative process. Yeah, iterative process. What I was thinking, Okay, that's it also exists. What happens with them? We haven't it just released like, no, they just released. Yeah, okay.
David Bau: you're on the cutting edge. You could have presented it sooner. So. But you're the technique you're talking about is it's called adversariality robust, you know, training. And it sort of resembles the the old Madri setup for this. Yeah, but how good is the iterative one?
David Bau: I think it's pretty good. Well, like I was. I looked a little bit some of their numbers, and, like their robustness, is like really impressive. And then they, I think they had like comparable interference on other concepts, which is like, you're trying to keep that number really high. So it was really interesting.
David Bau: Okay, so with this in mind, like all this interesting behavior, we came up with a model for sort of what's going on with these erasure models or erasing methods. So here we have, like 2 some hypothesized classes, and if we imagine that the latent space of our original model can is some like current surface where these peaks represents my target concept. And so if anytime I sample from within this peak. Using diffusion process, I'll get an image of a golf ball back.
David Bau: So. And if we imagine that these diffusion models are probabilistic, then the higher the peak, the higher the probability of generating this concept.
David Bau: So I prompted with golf ball. I get this diffusion path. I sample this kill. I get a golf ball image. So our 1st class of oration method that we proposed was where this entire target concept distribution is destroyed. And so here we call this the destruction-based, erased model, where, now that this knowledge is supposedly lost
David Bau: during the diffusion process, even though it's guided towards where golf ball had originally been. Instead, it is redirected towards some other high likelihood concept, because this original hill no longer exists. And so, if we reprompted our hypothesized destruction based model with golf ball, we might just get like a house. And so here given, like the behavior of these models, and, like their formulations, we propose that brilliant ascent and task vectors are destruction based.
David Bau: Now, the other form of erasing that we proposed is called guidance, based erasure, and this is where the original oops.
David Bau: The original target distribution is still within the model, but through the fine tuning process the guidance mechanism learns to redirect your diffusion to some random other concept. And so still, if we reprompt with golf ball, we'll get something kind of random, like a bunch of kids.
David Bau: And we propose that Esd and Uc fall under this category.
David Bau: No, it's really yeah. So in your second method, when you destroy the do, you go to the immediate next peak?
David Bau: So I guess you're you're asking about this one. Yeah.
David Bau: I think the idea of immediate next is like this, I guess it's kind of like a mental model. But the actual structure of the latent case of the fusion model is not quite like, it's not as explicit as this, so like it's not like the the diffusion would immediately go back to a semantically related concept like this, well, but
David Bau: okay. But like, if I simplify this very much, for example, if I train my division model on C part 10 data set. Okay? So I have, like only 10 labels which can possibly come out based on 10 inputs. If I destroy the peak for one of the labels, then do I get to the next immediate?
David Bau: A peak, I think. What do you mean by next.
David Bau: I mean, if if there is like a probability of, like each of the labels generating this particular image, and
David Bau: say, if you have plane, then you are getting a plane. If you have a bird you're getting a bird, but a plane won't generate a bird.
David Bau: If I want a plane to generate a bird image, then would destroying the peak of the plain
David Bau: like this method would go to the bird.
David Bau: like, most next, most likely. Yeah, always. So. I don't think so. I don't think it would go to the next likely semantic concept. Okay, but one thing. That I would note here is diffusion. Models are trained to reduce the entropy loss so, or produce energy.
David Bau: or in this case increase the probability. So
David Bau: diffusion model itself starts from a very low probability region, which is a Gaussian noise. Okay? And the iteratively try to go towards some peak.
David Bau: and if your original peak doesn't exist, it would naturally try to go to the next possible thing that it could go to at that point. So at some point, the model has to realize that. Okay, I'm going towards nothing, and it has to rearrange itself.
David Bau: But most of the time in Kevin's experiment he did not find that to happen
David Bau: where most of like, they always try to go to that destroyed region and give you like junk outputs.
David Bau: But hypothetically, if the models were trained really strongly, they should be robust enough, that when suddenly you
David Bau: clip of this peak. They're supposed to go towards another high peak region. I it doesn't have to be semantically close, but has to be in the path of the diffusion that it's already going. Okay?
David Bau: So it's a possibility. But it's not like, it will always happen. Yes, okay, it's this hypothesis. Yeah, yeah.
David Bau: yeah. So I guess what makes this like sort of story complicated? Because mostly you're dealing with text image models. Right? Yeah, you're doing conditions. So, for example, on the right, the right plot. What's truly happening is that you're distilling like
David Bau: like the condition on, for example, concept A into the concept B, and I guess for the
David Bau: or the middle plot like is, it's actually doing. What's actually doing is that you're destroying the mode like within, with a fixed condition. You're not really because you don't really have this distillation objective
David Bau: to this little concept to concept. B, we're really just trying to like.
David Bau: remove the mode of the concept. A, and usually what happens in grading sn, just destroy everything within that condition. Yeah. So in other conditions, it works fine. But in that condition particles just jump up of garbage. Right? So it's not necessary. Moving to the nearest memory sink, because.
David Bau: 1st of all, the mode is already limited. Given the face condition. And also
David Bau: right now, what we've seen maybe not not necessary for tax factors, I think for that work it does go to a different stuff. But for grading S, and it really is, destroy everything. It's just like, okay.
David Bau: And I guess this idea of like the memory sync is sort of like connected to that. Like, is this, yeah, this hill for gradient would just be like a bunch of noise, and maybe not like an actual like semantic con like concept, which is like.
David Bau: I think that's a really that's like part of something that I wanted to ask questions about at the end, like. So I think you can talk about that at the end. Okay, but yeah, this is our model. And so now we're in this phase of, okay, we have this hypothesized model? How do we validate it in particular? This question of like, How do we check if this peak still exists within our hypothesized guidance based models. And so there's a few different ways that we can probe the latent space of our diffusion model. And so we examined kind of like 3 different angles.
David Bau: So 1st was like providing context. And this idea of like sort of like in context learning. But for diffusion. And
David Bau: secondly, we, I guess all of these, the 1st 2 probes are meant sort of as like an interpretable way to look at the latent space of the diffusion model. Where the second probe is really like, can we somehow redirect this diffusion path to get towards our target concept without like providing any adversarial training, or like doing some sort of optimization. But instead, just like nudging our diffusion path a little bit.
David Bau: So those are the 1st 2 probes, and the final probe is kind of like just what most attacks do, which is providing a few examples and then searching for it. So that's like textual inversion and prompt prompt optimization. And so I'm going to focus on the 1st 2 probes, since I think they show a little bit more in terms of like elegance. But also they are able to be interpreted as like a probe into what's going on within the race models.
David Bau: So 1st I'll talk about in painting, and
David Bau: the idea behind in painting is, we have some original image, and we mask it. And then we ask the diffusion model to simply diffuse within the bounds of our masked walls. So the idea here is that in painting provides hints to the diffusion process which are kind of encoded within the latent that we actually pass in. And so it's kind of providing this guidance towards our target concept.
David Bau: And so what we noticed was that even though we have this concept of Van Gogh art, and you prompt our destruction, based and guidance based models with Van Gogh we'll get like these random paintings. But if I ask for in painting example, with perhaps gradient ascent. We notice that gradient ascent just fills it in with some random art, like some random texture pattern. But if I ask the guidance based model. To do this, we'll get
David Bau: like an actual filled in starry night painting. And it's like following the pattern within the context of the image. And so this is really interesting. And we have some more examples in the next slide, where we noticed that some of these, like hypothesized guidance based erase models are still showing that the concept still resides within our original or within our model, despite having supposedly erased it. And it didn't really take that much to bring it back. We just provided some context and provided a prompt and that's all it took.
David Bau: And so this really does show that within our guidance based models. We have residual knowledge that didn't really take much to search for. But when it comes to our destruction based erasure, we noticed that there's a little bit more robustness. And we did this pretty systematically across 13 different concepts still. But I think art was really the one that showed the most interesting results.
David Bau: Yeah, how do you distinguish this as like quantitatively, still showing that information of like still having the context. So I could argue that anyone who's never seen Starry Night before. Who's just seen enough? Art
David Bau: could create a more reasonable infill than either the grade and reset in a task factor, and that could look closer to like what Uce produces. That's not exactly like the original. It's a reasonable infill, I see. So it's like, not just how do you? I guess the quantification that we looked at was except scores. But it's like fairly subjective. And so we also did this with
David Bau: objects as well. And like distance. Yeah, yeah. So I guess, like to play devil's advocate like one
David Bau: other experiment, I would test out, I, assuming you're using the text prompt, like, Bingo or star unite. Yeah.
David Bau: One thing I would test out is like, just use the prompt painting like this, completely unrelated with
David Bau: what's been used in text, prompt to see whether gradient ascent method can feel, because I feel like the hypothesis that maybe the method itself, by self attention, mechanism can just like sort of like hallucinate. What's inside like, given outside context? Right? Yeah, so maybe Uc, and Esd, like, they also kind of forget about concept of angle kind of just in pain according to the stuff outside. But the reason why it's generating reasonable stuff, because
David Bau: all we know, like grading an Sm method after you
David Bau: run the greeting as and the model just shout and yell at you and scream at you of training gibberish.
David Bau: If you see that piece of text prompt. Yeah, I did look at that and I think it was generally like similar results. But the
David Bau: for the guidance based models like there is like a lower success rate with the in painting, but that is like something. I don't know if I quantitatively measured like that across all of the like, our whole entire systematic search. But that was something that I also looked at. Okay.
David Bau: yeah, these magnitude questions are always really hard, like, you know, that's right. Like, is it really something about the subtle geometry? Or is it just about shouting really loud? Right? Yeah.
David Bau: okay, that makes sense. Yeah, like as an Appalachian. So we're not Appalachian. But like, yeah, that's actually a good idea.
David Bau: Okay? Also a philosophical question. Right? Like, if the model is not able to even complete a painting.
David Bau: it's obviously trying not to do something versus it's still being able to finish a painting says that its general capabilities might still be present in the model. So
David Bau: don't we don't know like how to exactly put this. So we we made it like a diplomatic statement. But
David Bau: it's still a open question for us so like if you had something that is totally not painting related
David Bau: at like, you know, can gradient that can the gradient descent model do in in pending
David Bau: properly. If you have, I'm not sure, like a house
David Bau: or something. Couldn't gradient assent like and paint a house, I think, is.
David Bau: you know, I feel like, there's this ratio
David Bau: like, if you take Uce, it'll be able to impaint a house. No problem. Yeah, right? Yeah, it's very, very specific. So another suggestion on top is, if you take a house, try to impaint it with the prompt Van Gogh.
David Bau: Good gradient I sent like clearly not draw anything like, maybe it's just
David Bau: trying to shout out loud that I have to break myself, because this is Van Gogh painting. So I should not do anything else here. It's something we should also test. Yeah, it's like, these are all yeah, ambitions.
David Bau: Okay, but yeah. So given that
David Bau: this in painting probe show these results? And then the next step is asking, Okay, what if we don't really provide any hints? But instead, start from totally random noise like, can we still recover our target concept as well with like minimal nudging, and like any, without any adversarial training as well. So I missed so to do the in painting. Probe you just like, take the space image and noise it a little bit, and then have them all run. It's
David Bau: the inpainting, is we, the outside image is completely kept itself. It's like a completely original, and we just fully noise with what's within the box. And then we pass the entire image into the diffusion model as a latent.
David Bau: Is that the right way of doing opinion? Is that so? So every time you do, a generation
David Bau: at time, step x. 1. You replace the outer boundaries with the noise version of your actual image, and then continue your generation. So it starts with complete Gaussian noise. In painting you generate an image with the prompt.
David Bau: and then at Timestep x. 1. It has, like slightly less noisy version of your original noise.
David Bau: Then you replace the outer boundaries with your original image, added the same amount of noise.
David Bau: So it's like you just pretend that with the model
David Bau: predicted on the outside of the boundary happens to always be going in the direction of this painting of the real image. I see
David Bau: this is a kind of patching right like you have to run the model twice to get the
David Bau: the noise from the it's only once
David Bau: explain. Explain. Sorry. So I was thinking, like you've got you start from some.
David Bau: You have some prompts. You start with some initial noise for the base image to generate that image right
David Bau: or no. No, this is like real image. This is actual starry night image.
David Bau: We take a real image, and we cut a hole in it. Yeah.
David Bau: actually, we take a real image, we add noise to it 10 times gradually to make it complete. Gaussian noise. Yeah.
David Bau: now we start from the Gaussian noise. Let the image let the model generate the next step. Yeah, except the model doesn't know it's drawing the exact image right? So we'll nudge it by replacing its outputs
David Bau: with the
David Bau: original times. You did run it twice. I run it only once 10 times, and now the other time you're actually doing it. This is not model. You're just adding noise. It's just plus additional noise. Forward noise.
David Bau: Do you have any questions we can move on, I'll ask.
David Bau: So the other probe that we looked at then is starting without any original image, and in painting, but rather just starting with completely random noise. And what we like we can see this model showing is that your erase method will deterministically push your diffusion path to something besides our target concept. So here it'll always push it towards, maybe like a church if we ask for golf ball. But the idea is, can we inject noise during the diffusion process such that we kind of
David Bau: shift the diffusion path off of its original trajectory to something random. And so here, what we do is within the context of diffusion. You can think of noise as sort of like expanding the field of view of, or at least expanding the search space that we're moving along within the diffusion process. And so at each time step, we'll add in some scaled
David Bau: scaled noise. And so actually, practically what we do is, there's a hyperparameter within scheduler, and we'll do a search across these hyper parameters run this multiple times and get the best image
David Bau: after running it like 40 times. And so once again, if we look at this race concept of golf ball. We can see that we don't see any golf balls from either of the methods. And if we apply this noising probe to a destruction based model. We might get like a golf player. But when we apply this to a guidance based method, we can get multiple different types of golf balls, and later on the next slide, we'll have some classification scores
David Bau: but does this probe make sense in terms of intuition?
David Bau: Okay, so you have one question here. So when you add random Gaussian noise, you change this parameter. Eta, yeah. And that's like changing the variance of the noise which has been added, okay. And for each step you have a different data.
David Bau: No? Oh, oh, sorry each step. We have the same data. And then we run the diffusion multiple times with like, Eta equals, one, equal data equals 1.3. And we get a bunch of different samples. Okay, okay.
David Bau: and so once again, we have a bunch of examples. And here we have, like the way that we quantified beyond just clip scores, was classification. And so this is like a little more objective in terms of like, how we are measuring the success rate. And so what we notice is that, like with normal prompting, perhaps we might get like less than 5% of the concept reoccurring within our models. But when when we add this noise, inject this noise into the diffusion process, we can get up to 30%
David Bau: concept of recovery, despite
David Bau: which is like a 10 x of what it was originally. And so this is really like demonstrating that. And we're not like injecting a whole optimized set of noise, but rather just like subtly shifting the diffusion process. And so it's really suggesting that the original concept still resides within the diffusion model, and that with a little bit of nudging we can counteract the guidance mechanisms within our guidance, based erasure to bring back our target concept
David Bau: and but when it comes to our destruction based ratio, it's the success. Rates are a lot lower. And we're seeing that even with like gradient ascent, it's like less than 3%, which is really interesting.
David Bau: I know Sheridan's looking into like how text works with diffusion models which I thought was really cool because I saw a lot I saw I saw a lot of really weird text
David Bau: within these experiments with those ones. But overall. So these are like 2 of the main probes that we used as we were trying to validate this model and classify these erasing methods into 2 different classes. And really, it's like the 1st step into building this framework for understanding and analyzing future erasing methods. And the hope is that by understanding the behavior like the mechanisms. We can improve future erasing methods, because this really is still a really open question where racing is like.
David Bau: certainly not perfect. I mean, we saw like, even still, none of these models were perfectly 0% when you prompted normally, and there's also this other half, which is
David Bau: can like. What is the interference on other concepts, too? Like. What we would expect, though, is that with the structure based ratio like, if we're completely nuking the concept. It should likely affect nearby concepts, too. And so this is something that we see often. So here's
David Bau: it's a table where the models are the columns, and we have different prompts for the rows. And so what I did is I prompted for golf ball, and we see that there's erasure. But if I also prompt for a nearby concept like tennis ball, we can see that a lot of methods will just generate like something similar, but not quite expected, like I can get tennis balls back from like Esd and Uce. But like with this run specifically, I just got like tennis players and like something random. And so it's still not like a perfect mechanism for erasure
David Bau: and even like distant concepts like car, we can see some like impact of gradient ascent, and, like test vectors are showing something. But overall, like erasure, is definitely not perfect. Yes, sorry. I may have missed a bit, but like so in here, what are you erasing again? Are you raising
David Bau: ball? Just golf ball, and like in terms of like representation? Because, like, it makes like. I mean, if it's mostly on the ball, then it kind of makes sense like, why? For a nearby concept, because they also named us ball, that it might erase. Oh, yeah, like, what if
David Bau: if it was just like tennis and golf? Do you like see them still with the ball or not? Oh, in each one of them that's really interesting.
David Bau: Yeah, I actually don't know if I've looked at that before. But that's like a really cool. It's kind of like one of the prompting things is like, oh, if you race car like this traffic still bring it back. But this is like with the interference side. It's like, does this impact tennis as a whole? What is the metric on the previous slide? Which said.
David Bau: What is that metric? Oh, sorry. This is oh, I didn't explain this. This is the classification score. So basically, it's we use like a resonate classifier for objects. And then just see how often it classifies like this contains a golf ball, and like this is a binary classifier that you trained, or this is just a pull it off. Oh, it's okay. So then you just imagenet.
David Bau: So we trained on 10 imagenet objects and 3 artists.
David Bau: There's a subset of image net called Imagenet. I think it would be good to have an additional kind of baseline, just like what the accuracy would be without any of these concept erasure methods. Oh, right? I actually do. I do have that number. Oh, okay, this is like 80% for just like the normal model
David Bau: and I I mean, I guess this is also around like 80 80. I actually didn't do noising on like the original model. But
David Bau: okay.
David Bau: so yeah, but yeah, so this is basically like how the research that was done and like looking at classification.
David Bau: so it's, it's really an interesting question. And like, there's a lot of things. I'm still really interested in exploring. Really, the mechanism was like hypothesized, and we looked at the behavior. But I'm interested in seeing if, like, we can really tease out how these erasing mechanisms truly work. And it's something I'm really, personally interested in. It's like this idea of visualization like, how do we visualize the change to the latent space
David Bau: and like, maybe an interpretable way, like if I have a standard model, and then my erased model like, can I pair. Compare what maybe like what the weights look like, and do some sort of like, take a slice of the latent space and see what's going on there like, I'm really interested in that
David Bau: and then, finally, there's this idea of like memory sinks. And so this is something that I kind of talked about in the beginning, like where there's like certain concepts that are constantly being brought up when you prompt for a car. Maybe like sometimes it'll go straight to noise, but like there's other things that we saw that consistently were brought up, and we wondered if, like those are, memory sinks.
David Bau: which is a concept that comes from neuroscience. Where like, if you erase concepts within like a mouse's brain, like, they notice that
David Bau: there's a like set of the neurons that like form a memory sync where, if you trigger the concept again, like that memory sync will be activated. So it's really interesting. And like we did some experiments, but I don't know if they're quite strong enough to like make a strong claim. So I was hoping for feedback with that at the end. But
David Bau: thank you.
David Bau: Oh, go ahead. Oh, sure. So, just since you're talking about trajectories, I was curious, because, like, when you're doing this image create generation process with like a baseline model takes like, I don't know 10 steps or something. Yeah. 50. Sure. Have you tried looking at? If you save that original chain of 50 images, and then just kind of give segments of that
David Bau: to the concept erase model. So, for example, yeah.
David Bau: if you give it, the 49th step? Will the concept of race get rid of the golf ball, or will the golf ball still be there? Yeah, I think we looked at that
David Bau: whereas, like you can the if we give, like the 1st 10 steps of the original model and then pass it into the race model. What happens. I think generally it was
David Bau: pretty messy in terms of like.
David Bau: yeah, you generally it was difficult to bring back the target concept. I don't. I didn't do much with like the last 40 steps. I think I've seen pretty clearly like the last 10 steps. You don't see a lot of change in terms of like, because the fusion is like course, defined like, if your structure of a ball is there in like the 1st 25 steps, it's probably not going to get erased in the last 25.
David Bau: But I kind of gave up on that for a little bit, so maybe it's time to revive it.
David Bau: Yeah. Any intuition on why the noising thing works? Because I understand the pitch of infilling thing. But why does adding noise? Knock it back into its original course.
David Bau: Yeah, I mean, I think there's
David Bau: at least the way that I saw it. It's like there's
David Bau: at least I I think so. We run this like 40 times. The way that I've been seeing is like, if this is a high probability concept it kind of has like a pull. And so by introducing stochasticity into the diffusion process, like, maybe there's a radius around
David Bau: this this target concept that if the erase model enters the radius. It'll just get pulled into the actual target. And so like by introducing stochasticity like, there's a probability that it'll just get like pushed into that radius like gravitationally pulled into it. But that's just like the way I see it.
David Bau: That's that's actually a very good question. So for methods like, you see, and concept ablation.
David Bau: We try to have an anchor concept for erasing. So if you want to erase a car, usually replace it with a garbage truck something.
David Bau: And I think most of the times we tend to use concepts that are semantically very similar.
David Bau: So the assumption is probably even in the data space that closely there.
David Bau: Maybe we should test this by changing the semantic distance between the anchor concepts like car to sports car and then bike.
David Bau: increase it, to like car, to like some mask or something.
David Bau: and then still try to do your noise, probe and see the
David Bau: would would the accuracy go down or like the efficacy of your noise probe, go down.
David Bau: Okay? Something we could test. Yeah.
David Bau: So in this, your T is equals to 50. Yeah.
David Bau: so do you have some insight on at which
David Bau: iteration these features start to develop, starting from the
David Bau: random motion noise. Oh, I see! Oh, I had a gift. I forgot to add it in of like the noising process. I mean, I think, just like qualitatively, I feel like if you don't see any shape by like times of 10. Then, like, you're cooked so around, like, really, towards the 1st half, at least
David Bau: problems.
David Bau: And does it depend on the type of diffusion model that you use. What are you using? Currently, we're using stable diffusion 1.4. Okay.
David Bau: so do you think that this depends mostly on the model architecture, or it is more of the training process. Or
David Bau: listen like that specific behavior. I feel like it's
David Bau: I can't say like 100. But I would say, believe that that's the case for most models. But I don't know. Of course, defined diffusion is
David Bau: the standard, for, like every type of diffusion, but I think generally that's the case. So I would expect if the last 15 steps are less, 50% of steps are for like finer grain details, then concentration doesn't really care that much about it.
David Bau: Thank you.
David Bau: Then, if you guys have time, I have some questions about
David Bau: this idea of memory sinks. I don't know. You guys are totally free to go but this is something that one of the guys at Nyu was looking at. And it's basically like. So we have, if we erase with granny ascent we noticed that like, there's this dog that comes up like if we erase English finger spaniel anytime we prompt with English finger. Spaniel. It'll always bring back this dog.
David Bau: So we were kind of curious as to like why this was occurring, and like ways to quantify it. So the way that they approached it was they kind of measured this idea of like concept, like deviation in terms of like
David Bau: as you train gradient ascent across multiple steps like what happens. And they notice like, oh, with gradient ascent, like the average clip embedding stays within, like a certain tight realm of like radius. But with other methods like it starts to expand. And so I mean, I think for myself, I'm just interested in, like, what?
David Bau: What experiments could we do with memory sinks in a sense to like, see what's really going on within. Maybe like reading a sensor specifically. But
David Bau: I don't know if you guys have any suggestions around this.
David Bau: Yeah, maybe we should explain what memory sink is. I don't think, okay, I guess it's like within.
David Bau: If the idea is that we're destroying
David Bau: the concept like the distribution of English finger spaniel, and like, we're going towards some nearby concept in the diffusion path. The nearby concept might be like
David Bau: the nearby concept. Here would just be this English finger spaniel, and it's constantly pulling in any diffusion that comes along this along its way. At least, that's the way that I'm thinking about. Memory sinks. And so
David Bau: this is. It's like just a small, like qualitative behavior. We've noticed, with like bringing a sense, a sense specifically and like this concept of English normal spaniel.
David Bau: it might just be an artifact, I guess. Isn't like in most cases, you prompted. Just generate like all red types. Yeah. So for
David Bau: yeah, for like golf ball, I think it kind of depends on like the level of erasure I did with a specific concept but like grading ascent and like with garbage truck, too, like we got it to do just like a generic truck instead of like, specifically like. So then it's no longer classified as a garbage truck all the time, I see. So I guess you need to tune the steps. Yeah, it's like tuning the steps, and then maybe we'll find like some memories. It's not always the most consistent got it.
David Bau: and also like gradient ascent was pretty unstable.
David Bau: and if you do, one epoch extra 10 training steps extra experts.
David Bau: Yeah, absolutely.
David Bau: Well, then, there's this other craft that they built as well, which was like looking at
David Bau: this. So this is also something they did where basically, across training steps. They would get the like. They generate 25 images, and then get the clip embeddings of all of them, and then they would see like. So at times of 0, they're comparing the distance of this average from your original concept, which is like, if I erase Van Gogh like the distance away from like semantic difference away
David Bau: and like. This is also another graph that they generated.
David Bau: which I'm like trying to understand. What are other like evaluations. We can do that would make it more interesting, or like more robust.
David Bau: because, as of right now, I think there's like some questions I still have about
David Bau: the message that it's showing. So
David Bau: yeah, doesn't. Is this kind of showing the opposite of the memory sync concept you were showing previously,
David Bau: where here things are spreading out.
David Bau: Yeah, I think this is like deviation, I think is a interest. Yeah, deviation is an interesting word. I think here it's like, this is a supposed to be a distance from the concept of Van Gogh, or like distance from the concept of the English finger. Spaniel. So I don't know if that changes your interpretation, if it keeps it the same.
David Bau: See?
David Bau: No, wait. So you you're for you're claiming that English finger spaniel always goes to that double. In that case, shouldn't the prioritization just flatten out at some point?
David Bau: I think. Well, in this case it's like this. If this is like English finger spaniel is like clips magnitude away of like 4 from
David Bau: the the cute dog that we see. Then.
David Bau: yeah, it's like a. This is a weird graph to interpret.
David Bau: yeah. And I guess another question here is like, you're sort of assuming the memory sync is sort of like,
David Bau: so really just like a single memorized image. Yeah, instead of like, it's just like a distribution of images. But why is that particularly needs to be the case? I don't think it needs. And I guess it's yeah. Like, I agree, the phenomenon is interesting. Yeah, but it doesn't. Necessarily. This phenomenon doesn't necessarily tie to the memory sync. I don't think we're making an assumption that it's only one image. It just happened to be the case with Ga.
David Bau: Even for Esd or
David Bau: any of the other methods. They usually tend to do everything else like the empty prong, which usually are like high probability classes like houses and trees. Exactly so. Even then you see that the centroid sort of gets flattened out. It's because the distribution is fixed. Now it goes into that empty prong distribution, and the distance is always constant. I see. So basically sort of like moving to the distribution.
David Bau: But yeah, I don't. I'm not too sure how to like expand these results. And I've been talking with him from Nyu. And just like, what are ways that we can make this stronger like, I was thinking about doing something about the text textual, like doing textual inversion attacks across different steps and like seeing what the results there were. But
David Bau: yeah, at least right now. It's just like an interesting set of reps. But I don't know if they really have like the strongest message.
David Bau: But yeah.
David Bau: so I think right now, the hypothesis you're trying to tackle is really hard. It's like the about the optimization process. And like differences in or yeah, the concentration optimization process.
David Bau: Have you thought about kind of stepping back bit and just focusing on the robustness of these methods? Because I think you have pretty clear results that, like Ga is more robust to these attacks than other methods, I see. Yeah, I think we also were talking about that like what we just presented as like an interesting analysis on, like different types of robustness, like
David Bau: Ga as a method itself.
David Bau: But Kevin had more attacks. He just didn't show it here, but he tried multiple adversarial attacks, too. But
David Bau: his idea was more of taking this attacks and thinking thinking of them as an probe for interpretability
David Bau: like, what does that attack success mean in terms of
David Bau: how the model is erasing a concept?
David Bau: But yeah, you could take the results from it, and also conclude that
David Bau: about like the robustness of these methods, too.
David Bau: Yeah, because I think the interpretability part is the weakest part. Yeah, that's like, that's the that's the one I'm trying to
David Bau: make stronger. I don't know how to exactly present that message. Make it more persuasive.
David Bau: So you you know you have you have these 2 pictures that you're trying to tell the difference between? I feel like.
David Bau: you know, you could also
David Bau: hypothesize other mental models. And maybe this is the challenge for the interpretability thing, right? Because it's not clear that these are the only 2 alternatives that there are
David Bau: right. I mean, don't you think that, like another alternative is the booby trap alternative.
David Bau: right where you know you have your lost surface or something like this.
David Bau: And somewhere, you know, near your target concept.
David Bau: You you have some new
David Bau: you know, high probability. Step in your diffusion process, or something like that
David Bau: or something that an intermediate layer computes.
David Bau: and and then you've put a booby trap there right? Which is like, if you. If you step on this, then
David Bau: then I'll then I will do some computation, and I'll do some look up in an Mlp. Or something like this that will put an infinity in
David Bau: right to the thing, so that you're not totally incapable of drawing anything after this and and then, you know, maybe one model for erasing is, I'll put the booby trap in there. But then the question is, okay. Well.
David Bau: maybe that still doesn't remove the concept of your dog.
David Bau: Like, if somebody could find where that booby trap is, and maybe they can
David Bau: remove it right like I feel like there's like, there's these other possible right alternative theories
David Bau: as well. And you could imagine, I don't know if there's others right that there's this fairly.
David Bau: the optimization based attacks reveal the booby traps like you're giving 5 or 100 500 images in this case
David Bau: of the concept you erased.
David Bau: And you're optimizing for the right?
David Bau: yeah. So what I'm imagining is that the optimizer
David Bau: he's also fooled right now move it around.
David Bau: or it might not have enough
David Bau: time or steps, or something like that optimizer. So like
David Bau: like once, you have like an infinity in a model.
David Bau: then it's not clear that optimizer can recover right from it. Yeah, yeah.
David Bau: But it could be that there's some other technique for like finding the infinity and then chopping it out
David Bau: right?
David Bau: Yeah, I like the suggestion somebody had. What if you give a
David Bau: take a real image, noise it just a little bit to 10 steps. Yeah.
David Bau: And maybe if you do across like 1,000 steps and notice at which point the model
David Bau: starts collapsing, maybe. Oh, interesting. Maybe that's the time step to start looking at for these traps. Yeah.
David Bau: yeah. Booby shops. I don't know.
David Bau: Yeah. And it's also like this question of like. Even if we have these models like.
David Bau: how convincing is it really
David Bau: to present it and like, try and validate it in this like interpret space for diffusion like that's like the big challenge for me.
David Bau: So you know the thing that, like some of these things from like deep and these kind of people they're like.
David Bau: try to make these make a convincing case for this type of
David Bau: picture by literally doing huge grid search over, you know, a 3 dimensional space of you know, changing different things and then showing that you have these different regions.
David Bau: Right? And so
David Bau: you know, 1 1 thing you could do is like. So you have your cartoon of what you're imagining. Is there some grid search that you can do where you could literally
David Bau: show? Yes, it looks like my cartoon, right? This is like, there's this region here of where the concept is. And then there's this other region where it has this other behavior. And yeah, like, I, you know what I mean for 2D, maybe we can for a toy model. That should be maybe. Yeah, maybe for a toy model.
David Bau: right or or or you know.
David Bau: yes, or even on your big model like, but just, but not not like the whole space. But like, maybe, can you find 2 variables that you can
David Bau: adjust, or something like this? Or you can do a grid switch over them and say, Well, you know, obviously, we can't look at everything. But when we take this 2 dimensional slice over, you know some search space, we see something that looks like our picture. I think that if you were able to like you're if you're into like visualizations, or if you were able to find some visualization of some search space which is more evocative. Then that would be
David Bau: right. Like, these are all like these are all the one dimensional scan scans
David Bau: right? And then it's like hard to.
David Bau: Yeah, you know. Conceptualize, okay, well, what does this one dimensional scan tell you about this loss in the city that you you're sort of imagining. Yeah. But maybe a 2 or 3 dimensional scan will.
David Bau: I'd love to do over something right?
David Bau: It's gonna do, sure.
David Bau: Can you go back to your slide with the 2 hypotheses.
David Bau: Yeah, let's see.
David Bau: I I'm curious about this one. Why do you think it's scalar instead of vector. Like
David Bau: we could assume that it could just be doing some
David Bau: like like the actual outputs we did a grid search. It'd be a vector direction that the diffusion model is doing in terms of the
David Bau: adjustment to the image at each step, or at least in the diffusion models I've worked with. I don't know if that's universally true diffusion models, but it's doing some steps with each point. If you have 2 variables, you're altering, it's a, the vector is an arrow. And that doesn't necessarily turn into something that you can have a hill climb over, because in theory that vector field could be
David Bau: not conservative, you could have like a loop or something, just get stuck going through random noise or something which
David Bau: I don't know the gradient ascend method some of those strange outputs could be.
David Bau: Possibly. Is it possible it's not converging to some hill, and it's just exploring some random space infinitely so. The the vectors that you're talking about are gradient.
David Bau: Log gradient log probabilities of the data that can likely occur in the models outputs.
David Bau: So the peak is actually the probability which doesn't exist. It's just a
David Bau: it's as hype hypothetically assuming that there is a property distribution, right? But all the model does is those arrows it's trying to
David Bau: optimize or like, what do you call it?
David Bau: Approximate this peaks using the arrows that you're talking about.
David Bau: So that's that's that's the score based energy functions.
David Bau: So it's approximating a conservative vector field like so that it's approximate. Okay?
David Bau: So yeah, these paths, right? That you just follow that this one
David Bau: like 50 step path up this thing like, can you get a grid over it. If you, instead of doing this path like you, alternate it with paths from the base model that where the thing isn't erased. And you're like, Okay, well.
David Bau: if you were willing to draw a golf ball, then you would go this way
David Bau: and and then can that give you a a grid that you. I'm not really sure where I'm just trying to like, what would be the
David Bau: yeah, what would be like? I think I feel like, maybe there's some visualization method where you interleave using the base model. And then
David Bau: the erase model.
David Bau: So yeah, so each point in this plot is actually a high dimensional vector, 64 times 64 latent, of course.
David Bau: But then, but it's also a 1 dimensional vector, of timestamp T equals, you know, point 3, right.
David Bau: you know, but that's the only parameter is varying, and the rest of the system is just
David Bau: fixed neural network during this process, right?
David Bau: And so, which is why it's like a line in the end it's a path. And then, anyway, I
David Bau: Oh, I I assumed these were just synthetically generated. Is this, how do you generate these plots?
David Bau: Oh, it's just cartoon. Right? Yeah, you just may. Yeah, this may not have like some crazy process. No, no, that's what I'm trying to get at right. If you have some crazy process for creating a thing, then you'd be like, oh, look at this! You can see right? You know what I mean?
David Bau: Well, so I mean, like, why not do the try the naive thing of like project project down to 2 or 3 dimensions? Yeah.
David Bau: yeah, from from yeah. So right, that's nice. That's maybe
David Bau: it's super naive. What about? Instead of projection, like a diffusion model that goes into a latent representation to like a vision transformer that actually projects it up
David Bau: to the full dimensionality of the image.
David Bau: Well, because a simple projection is easier. What was your initial idea.
David Bau: Well, it's like what like a 64 by 64 latency. So you have your time. You're traveling through time. You just gotta what's whatever 64 by 64 is like.
David Bau: what is that? 2, 56, or 5, 12?
David Bau: It's not huge.
David Bau: Yeah.
David Bau: it's and then times, it's actually is it times 4 for each patch, or I'm not sure whatever it is. 4 for each patch. Yeah, yeah. So that's 16,000, or whatever right? 16,000. So 16,000 down to 2.
David Bau: Just do a Pca. You know. Take, yeah. Who knows? Right? You know, I don't know.
David Bau: I like it. Yeah, maybe we should. We should try this once.
David Bau: like, just run a bunch of
David Bau: runs and capture all of this into 2 dimension vectors and plot how it looks like, even if you just like, did that for like 2, I don't know, like 10 concepts or something. And you saw like
David Bau: clustering okay of the concepts or something like that would be some.
David Bau: That's a good idea.
David Bau: And if it doesn't work, that's fine, you didn't already have it, anyway. So
David Bau: that's good. I was thinking about something similar, but it was with like some
David Bau: it was with like a different paper that was looking at like decomposing the unit like
David Bau: getting some latency with it was like, it's actually really interesting.
David Bau: and it was like doing some like Svd with the unit getting latents and like making a graph from that. Yeah, I think that'd be really cool. Okay, maybe I'll try that. And hopefully, hopefully, it'll make something blobby or not blobby, but like interpretable.
David Bau: Yeah, I mean, you've got a lot of degrees of freedom making these visualizations like, what are you?
David Bau: What is? What is your big dimensional space that you're varying? And what's your
David Bau: what's your reduction method? And so
David Bau: maybe somewhere in this design, space or something through some visualization.
David Bau: let me just send it. That sounds really cool, actually.
David Bau: thank you. Where did these 2 ideas come in from like? Do you already have these preconceptions in mind, or did you do the experiment and come up with this.
David Bau: Think it was previously.
David Bau: But no, no, because in my head, like, when you thought that class, I thought everything worked like this. The second draw. Yeah, we thought this, oh, yeah, yeah.
David Bau: But then your noising experiments started giving different results. So
David Bau: that's when this came. But I'm not convinced that this. This middle drawing isn't a booby trap instead, right? There's like this other. It could just be that the golf ball is still sitting there, except that you put this bomb right next to it, and it keeps on stepping on the bomb.
David Bau: Yeah. So this is the way I see how unlearning works sometimes. Especially when you're optimizing using Esd or concept abolition, or all those sorts of things. I think we are trying to change this to arrow over there at what direction it's supposed to go.
David Bau: Maybe you're not really messing with the law space. I don't know how we start messing with the
David Bau: the model's knowledge of how the law space should look like
David Bau: or are we already doing it? I don't know. I I'm starting to think we are maybe just trying to steer the model certain ways.
David Bau: That's how I view it.
David Bau: How how do you even get to editing the law spaces?
David Bau: I think, at least for concept ablation. It's really just a distillation objective, right?
David Bau: If a cut.
David Bau: it's a Chao laws between like the concept, the target concept. Sorry, right? Naming is good. You really trying to.
David Bau: They still target whenever the model sees the target concept want to generate the anchor concept. Yeah.
David Bau: And I guess depends on whether train with the reverse or forward K, or like some behavior, would change a little bit. But in general you're doing just that.
David Bau: Yeah.
David Bau: So that's why I feel like consolidation.
David Bau: You can just throw a textual inversion to it. It'll probably just break. Because, yeah.
David Bau: I feel like this, distillation loss has a lot like
David Bau: drawbacks where there are a lot of loopholes where you can just like
David Bau: find spacings where you can go back to the original denoising trajectory. Yeah. So if you if you start.
David Bau: So you you said that you previously did experiments where
David Bau: you start with a half noise golf ball, and then you
David Bau: you finish it right. So do any of the erasing methods, you know, succeed at
David Bau: frustrating the model's ability to complete a golf ball that's already been half
David Bau: drawn right like it have to know it's board, or do they all say, Oh, yeah, I know how to finish this right? Because isn't that so like, that's that's like the difference between these 2 pictures right? It's like, Oh, I can't get close to a golf ball. So, therefore, but actually, but the the model still knows that a golf ball is very similar, so if you if you get pretty close to it, then it'll complete it right? That's sort of this, you know. One of these pictures
David Bau: right? Or is, is ga, really just like a golf ball is not a thing. You get me really close, and I've gotta like go somewhere else.
David Bau: you know. Do do you remember? I'm pretty sure. Have you done ga experiments on the on the I don't remember if we did. Ga.
David Bau: I think we just did that pretty recently. So
David Bau: yeah, I see, I see. But it also kind of depends on the concept like I've seen. I did it again with airliner, and it was really good, like, airliner was consistently brought up.
David Bau: so I would be interested to see what happens when this up there?
David Bau: Yeah.
David Bau: yeah. And generally, we found that models, methods that are much more robust to adversarial attacks are the ones that destroy the model. Most
David Bau: like gradient descent and task vectors almost destroy the entire capabilities.
David Bau: Even if you raise a golf ball. Which is this unfair thing? Right? It's like, it's like, oh, yeah, I can. It's easy for me to erase things. I just like I just like I have. I can use a gun, sure erase everything you know?
David Bau: So then how do you?
David Bau: How? How do you make this comparison in a research paper without offending people.
David Bau: without saying, Oh, this is not even do anything just destroys the whole model. That's it. No? Well, but then, but then you're also comparing robustness. And then these are the methods that come to the top.
David Bau: We have to somehow acknowledge it.
David Bau: but they are the most robust. I guess the good news is that you add the you add the random baseline. You say, you know, we just we just we can also just add, you know, such and such. You know, random noise to every single weight of the model.
David Bau: And and then you know these things perform, similar to the random baseline that make any sense.
David Bau: You should know that you could also just like include that information about how well the model maintains its old performance.
David Bau: And I could give you like a Pareto front for, like, what's the trade off between concept erasure and maintaining performance and the random baseline. Yeah, yeah.
David Bau: okay, yeah. Instead of just showing numbers, maybe we should plot them on like the
David Bau: a curve of like robustness versus performance. Yeah.
David Bau: Yeah. And there's a other side of the conversation, too, which is like
David Bau: there's some robustness that like is only achievable by gradients, though too like when we did we did do Esd. For like 10,000, a thousand steps, and it just like completely breaks the model and itself. We could do textual inversion like they did sometimes come up so like we could still bring back like airliners with textual version, even on a completely broken Esd model. So like, that was like another side of the story that was really interesting. But like with gradient like, that's not possible.
David Bau: Oh, yeah, you should also talk about
David Bau: potential signal of what weights you're you're updating for? Yeah, that's true.
David Bau: Yeah, because I guess, like, radius and test vectors are applied to all weights, but Esd and Uc are just applied to cross attentions, which is like 1% of the 5%. You said that in
David Bau: your people not to consent.
David Bau: 5% think around around that ballpark. I think we do. Presentation. Yeah. And
David Bau: well, that's also an interesting aspect of it. Like we did end up looking a little bit at just applying gradient sense, so cross intentions and non cross intentions. But
David Bau: I didn't pursue it that much.
David Bau: It's very interesting. I really am interested in this toy model thing, though, like doing the TV, I I don't know exactly what that would look like, but
David Bau: I should just like, send it and see what happens.
David Bau: Yeah, thank you guys for the suggestion that was really helpful. Thank you, Kevin. Thank you very much. Enjoy it.