David Bau: Good.
David Bau: Alright. So what are you working on here? Okay, so go ahead.
David Bau: Rohit. Gone to code test. Okay?
David Bau: So yeah, I rarely have to do this where I present a very heartbaked
David Bau: still in progress in such ideas, I thought, that's what you're doing all the time. No, usually I end up presenting after a paper is published.
David Bau: So yeah, I don't have amazing slides but the the main takeaway I want from this talk is that
David Bau: I want you guys to roast me and like, tell me where I can improve in terms of experiments, or something which would help you.
David Bau: Convince you more in terms of what I'm trying to say.
David Bau: So.
David Bau: this is a. This is an interesting project. We've discovered so many things very tiny, minute little things
David Bau: over the 2 years that I've worked here.
David Bau: And yeah, like, last week we figured out that all of this can be brought together to do something out of it.
David Bau: So the title is restoring diversity. But I'll give more motivation of what this is.
David Bau: So how many of you here know what a diffusion model is. I I didn't try to or don't know what a diffusion model is.
David Bau: Okay? So diffusion models are basically a generative image model.
David Bau: where you give a text, and the model starts from a noise
David Bau: over like 50 or 100 times types, it slowly removes some part of noise and makes it into like this beautiful image that comes out of it.
David Bau: So when when researchers train a division model, it's called a base, efficient model.
David Bau: and it has like beautiful images. This, I'm just reusing this slide from my laptop
David Bau: as beautiful images, but it's super slow, because it takes
David Bau: at least a hundred or 50 times steps for you to generate it. So it's 50 times you are forward, passing it through the mode
David Bau: so for this particular use case to make it more productionizable. People do distillation.
David Bau: but not model distillation. This is diffusion, distillation. So instead of reducing the model weights, they keep the model weights, the same architecture, the same, except they fine, tune it further to make it faster.
David Bau: Right? So instead of using 50 time steps, can I do the same thing in 4 or one.
David Bau: Then comes. Let's call it distill button.
David Bau: so it has reduced time steps. It still has like neutral images. It's super fast.
David Bau: But now the problem is, it's not diverse anymore. What I mean by that is, so a base model. If you ask for a cartoon character.
David Bau: Right? It takes 9 seconds to generate one image.
David Bau: But look at each individual sample. It's super diverse it has, like all sorts of characters popping up.
David Bau: and the same thing. If you ask a distilled model, you see that it's more collapsed right? It's the same cartoon style, probably the same face, same features that keep popping up.
David Bau: But it's super fast, like, see? It's like 15 times faster
David Bau: within, like within a second you can generate an image.
David Bau: So
David Bau: I'm gonna introduce 3 numbers here just so that it can be dramatic in terms of what I'm trying to say here.
David Bau: So one is called sample diversity, which is for one prompt. If I were to generate 1,000 samples, how diverse are these individual images!
David Bau: And then there's a distribution diversity, which is, if I give 10,000 prompts?
David Bau: Will the image? Will the model be able to generate all 10,000, or maybe like only 5,000? Also.
David Bau: that's called distribution, diversity and user preference score is basically how much like, if I were to show you 2 images as a user to you.
David Bau: which one of it are you going to prefer the most? And if I were to give a number to it, I'm gonna call it user preference.
David Bau: So the numbers I'm going to show here, are slightly weird.
David Bau: If the sample diversity is higher, that means it's more diverse. The distribution diversity score is actually lower is better.
David Bau: That means it's closer to the original training data distribution
David Bau: and user preference score is, of course, the highest. Higher the score, the more the preferences.
David Bau: And you get these numbers which doesn't make a lot of sense now. But if you look at distilled models, you see that sample diversity has, like drastically gone down like even like has gone down more than half
David Bau: but the user preference score is pretty high.
David Bau: If you took a closer look at it, you can see that base model has, like these artifacts, weird artifacts which you don't
David Bau: see a lot in the distilled models. It's more saturated. But it's more crisp
David Bau: compared to the base model.
David Bau: So there are like couple of problems here which we want to address. Yeah. So when you say user preference score, is it within that same like
David Bau: image set of images? Or do you mean like between these 2 like one? This one got like 22, and this one got 26 so the user preference score is individual image number. Okay? But it's trained based on
David Bau: Elo rankings.
David Bau: So I I'll explain it later. But it's basically it's a clip model which is trying to replicate a human
David Bau: on how much would a human ask
David Bau: question? Yeah. So for the same prompt. You get like simplified cartoons, whereas you have more complicated cartoons on the left one.
David Bau: Yes, and also the same simplified carto. Right? It's always the same simplification, same features, almost looks like the same person.
David Bau: Right? So the diversity is also like super low.
David Bau: Yeah, so, oops. Maybe I should motivate this.
David Bau: So now the question is,
David Bau: can we have the same distilled model, or, like, in other words, this faster inference.
David Bau: But try to address most of these problems here, which is, can we make it more diverse?
David Bau: Can we have the user preference score might be have like little increased. And
David Bau: there's another problem which is, our distal models can't be really controlled as much because they have only 4 time steps.
David Bau: And there's a problem of control. I didn't have time to give motivation slide for that.
David Bau: I'm gonna show some examples later.
David Bau: So what's really happening here is, if you were to standard visualize, what a deficient model does. It starts with like this complete noise.
David Bau: and then it slowly denoises over the time steps.
David Bau: So Dmd. Is a distal model. It takes only 4 time steps.
David Bau: whereas the base model takes 50.
David Bau: And for the sake of visualization, I'm just showing 30%, 60% and 100% along the way
David Bau: from the 1st glance. Do you see? Like any huge difference here?
David Bau: In terms of like both these inference runs.
David Bau: it looks okay. Okay. Yeah. They look like they're doing the same thing, probably great.
David Bau: So, in other words, how? What is this actual visualization is that if
David Bau: assume that that's the the ellipses are the Gaussian noise from which you start.
David Bau: What efficient model does it? It slowly tries to steer towards that beautiful data distribution that you have.
David Bau: So what we have visualized so far is that image right?
David Bau: That red small dot over there somewhere in the intermediate state.
David Bau: But there's another visualization which we could do which is cleverly if we were to extrapolate it
David Bau: to the data dimension. So I know at what time step I'm in. I know how many time steps are remaining.
David Bau: So. If I were to take the same step again and again and again without actually doing a forward pass.
David Bau: I could probably estimate what the final image would look like at the current time step.
David Bau: And we call it
David Bau: final image extrapolation, technique. Right? It's not new. It's just the math already underlying in the proposed paper. We're just trying to revive it back for better visualization.
David Bau: So now, if I were to do that technique and look at what the model is. So another way of doing thinking about this visualization is, what is the model thinking to draw at a current type step.
David Bau: And this is what it does for the base model at like the 1st time step. It doesn't really think of anything. It's like, I'm gonna draw some blob over there.
David Bau: and as it gets to the final image it gets sharper and sharper and sharper.
David Bau: But the problem with distilled models is that at the very 1st time step.
David Bau: They know exactly what they're doing
David Bau: right now. You see the this effect that I'm trying to call here right
David Bau: a base model has more than 10 time steps to figure out what the structure is going to be.
David Bau: whereas a distilled model has to do it in one time. Step.
David Bau: Yeah. So if if you have this one time, step to like.
David Bau: have a visualization close to the final time step of a deficient visualization without any drifts, that is. Still we stay as a deficient model. Now you are not having any drifts. Right? You know you are doing it in one shot, just like Gans. So can we still classify it as a diffusion because we are not having the drips. We're not having the classical drifts. It's it's not classical diffusion.
David Bau: I I did call it again once, and got a backlash for it.
David Bau: It's still diffusion, because you're not jumping into the final image yet. You're like sort of trying to
David Bau: jump to the final image. But it doesn't. So it's a very uneven
David Bau: diffusion process, right? The 1st time step does all the heavy lifting and the final 3 time steps are like, okay, let me just correct myself. Little bit to add these final details.
David Bau: Right? So that's a thanks for bringing it up. That's a very good
David Bau: way to motivate this this as well.
David Bau: So yeah, the 1st timestamp has to do all of it. So of course the model is going to be more collapse because we've seen that
David Bau: over the past. I don't want to ruffle any feathers, but most of the single step. Generation models have this mode collapse.
David Bau: So hope
David Bau: so. Now we'll enter into the solution of how we like try to tackle this without any training
David Bau: mo. Most of you know what concept sliders are. Maybe, if just in case
David Bau: don't know it, I'm gonna motivate it again.
David Bau: a concept Slider is basically
David Bau: a diffusion model knows most of the concepts because you can text condition it. So if you ask for a bearded person or a clean chain person. It knows exactly what to draw.
David Bau: So maybe if you can look somewhere in the internals, you could probably capture this
David Bau: the concept of beardness into a slider.
David Bau: and once you attach it, you can.
David Bau: you can control it. This is broken visualization. You can control. This features in a very precise one.
David Bau: So that's right. When something is attached cool.
David Bau: What we have discovered is that if you were to train a slider on top of a base model.
David Bau: it works pretty well, because well, you have trained it on the model itself. So it captured what model knows?
David Bau: But the weird part is, you can
David Bau: take that slider that you trained on base and move it to distal model.
David Bau: and it still captures that direction.
David Bau: Right? That means that through this process of fine tuning or distillation.
David Bau: The concept representations are untouched, they still they preserve very well that you can just take these adapters from the base model and directly apply to the distilled models.
David Bau: So here are like some more examples, right the one I showed earlier was a big Guy Slider, which we trained in the past.
David Bau: There are a lot of fine tune models on hugging face and
David Bau: using one of the search techniques. I found it.
David Bau: But maybe there's a better way to do it
David Bau: right? So these are called customization techniques. The model really, the original model doesn't know how to draw pixel art that way, but you can customize it by showing it like 10 images, and training these adapters on top.
David Bau: and even those adapters transfer to distilled models, even though you just train it on base. So it's just the added evidence that concept representations
David Bau: within the model are preserved through the process of distillation.
David Bau: So maybe if representations are untouched, you could probably also transfer latence. By latence. I mean these noise steps that I've shown earlier.
David Bau: Right? So if the problem is really the 1st step of a distill model.
David Bau: Maybe if I just transfer it from a base model
David Bau: and let it continue with the distill model.
David Bau: what would happen to the images?
David Bau: Right? So does it really work? Yeah, it does work
David Bau: So what happens is
David Bau: in the 1st time step. If you just replace the model weights with the distilled models. Sorry
David Bau: out of the 4 distal timestamps the 1st time. Step if you replace it with the base model.
David Bau: And this, let's continue with the digital model itself.
David Bau: You suddenly see an increase in sample diversity.
David Bau: So what's what's the what? What? What is this new setup again. What's the new setup? Yeah.
David Bau: So I've showed that the 1st time step in a distilled model already knows what to draw.
David Bau: Maybe if you were to confuse it, that it doesn't know how to draw it and let a base model decide what the 1st time step has to be?
David Bau: Would the distilled model be able to finish it because the concepts can be transferred. So maybe distilled model can do the rest of the heavy lifting and try to like, normalize the weight into the rest of the 3 time steps. Give a different starting point so that there's less heavy lifting. Exactly. Yeah. Or just replace the 1st step with the heavy lifted step from the base model. Can it? Can it finish? It is the question.
David Bau: because, yeah, I guess conceptually, this would be the same as if you were doing the distillation process. But like with variable length steps, is that right? Exactly. Okay, yeah. So
David Bau: none of the distillation processes really
David Bau: like make it as an objective that this, this the diffusion has to be equal across all the time. Steps
David Bau: right? All they do is if I were to run 4 steps, will the final image be as pretty as the base model?
David Bau: So this is actually a call for action for building better distillation models. Since we don't have the computer model to create it.
David Bau: We're trying to say that.
David Bau: Look at what your distillation models are doing.
David Bau: They're heavily more collapsing because the 1st step is doing all the work. And it's basically again now, because it's 1 step
David Bau: doing all the process and the rest. 3 steps are just
David Bau: having these touch-ups. Yeah, so yeah, that's one of the. And your assumption is that if you add some kind of lost term to kind of ensure that your the images generated by your distilled model is also kind of like as diverse as possible, then it will kind of like address this issue that we have found. Yes. So
David Bau: there is one buried method, distillation method called Lcm, it's called latent consistency model.
David Bau: What it does is it tries to be consistent across deleteds
David Bau: right? In other words, it's trying to denoise the same amount at all time steps.
David Bau: So it's technically a method that exists. But the quality is super bad.
David Bau: Because.
David Bau: so what we hypothesize is that the final 3 time steps are really important from a distilled model, like the current state of distal model like this one.
David Bau: to make the image more sharp and crisp.
David Bau: whereas the 1st time step is for structure, right? And if you want to like, really make it
David Bau: 25%, 25%, 25%.
David Bau: The crispness doesn't come out of the model, and the images are really super bad for the Lcm model.
David Bau: So it's never used in production.
David Bau: So what we are maybe suggesting is,
David Bau: maybe look more into it. Sort of question here, like we don't know what to do.
David Bau: But here's an evidence that if you were to replace the 1st time set. It works.
David Bau: So maybe there's like some clever scheduling like a sinusoidal thing, an exponential way of characteristics.
David Bau: So.
David Bau: but you have already found a fix right. I found a fix. If you found a fix to the problem. I did found find a fix to the problem.
David Bau: But for this to happen, you need to have 2 models loaded.
David Bau: Okay, the base and the distilled. Okay? And it's
David Bau: absolutely fine for the current state of not the previous state of the art current state of the art is 12 billion. And you can't load 2 models in a consumer. Gpu, just 12 billion, just 12 billion. That's the current all right on the bar. Right? You have like 600 billion.
David Bau: So there is a fix now, maybe we are just seeing.
David Bau: Do this fix and let a distill model replicate this fix. Maybe we'll have like a efficient distillation process.
David Bau: So yeah, it turns out, you get sample diversity, which is restored crispness is back.
David Bau: But the interesting part, or, like the surprising part, is that
David Bau: it exceeds the sample diversity of a base model.
David Bau: and while maintaining the similar user preference code as a distilled model.
David Bau: And the speed is same as the still model. So it's almost a hack which we don't understand. Where, how can you even increase the diversity of a base model when you are learning from a base model?
David Bau: So this is not. This is not on okay. The numbers are not on these 6 images.
David Bau: The numbers are on 30,000 data set of cocoa.
David Bau: So it's pretty consistent that you get these higher numbers. I don't understand. How is this? Not a fix
David Bau: it? It is a fix. I'm not saying it's not. I'm just surprised that you were able to exceed the base diversity I can understand, if you exceed discount well makes sense
David Bau: like somewhere in the middle. But it also was better than the original model which it's suspicious.
David Bau: Wow!
David Bau: But it does something, maybe something to do with how diversity is measured.
David Bau: So that's true with the the distribution diversity. I I don't really believe that it's true, but the sample diversity is based on image similarity metrics. So I generate 100
David Bau: images for one prompt. And then I look at these l pips like some sort of similarity distances.
David Bau: I I think they are. They have been proven to be very consistent in concept. So
David Bau: this happens so we don't know how to explain this.
David Bau: but we're being humble. We're gonna say, we are just restoring it. I'm not gonna call the exceeded base model
David Bau: diversity. But yeah, it this happens so
David Bau: that's all I have so far.
David Bau: So I don't know.
David Bau: Maybe I'm not a diffusion guy, but like conceptually, you're doing extra steps right in this case, in the in the
David Bau: in your fix. No, I'm not doing so. I did do extra steps in the beginning. Yeah. And then David said, Why do extra steps just replace it with one step?
David Bau: So the total number of steps here, you kind of adjust for doing the 1st step with the base model. Yeah.
David Bau: or the Admin, you go to your
David Bau: final image and Extrapolation slide. I think it's right before.
David Bau: and I guess you're looking at.
David Bau: would it? Would it? Or I guess, did you ever see that for the base model?
David Bau: That, the type dog that shows up differs based off of when you extrapolate
David Bau: like. If you do at the base, you might get like one certain type of dog, I guess, because, like when it comes to course, defined diffusion like, wouldn't that suggest that you might get different dogs depending on when you sample your final image interpolation?
David Bau: Yeah. So
David Bau: yes. And that's the fix, right? We know that if you sample it from base you would get totally different dogs for different samples.
David Bau: so maybe I should have explained it better. I'm I'm I'm just not replacing it. I'm not. I'm not doing an intervention.
David Bau: So if my let's say this is my base model.
David Bau: I have like 50 time steps.
David Bau: and then the distilled model 4 time steps.
David Bau: This is what I've visualized so far. I'm just showing you this.
David Bau: So now what I'm the fixes is basically
David Bau: when you say image of a dog.
David Bau: instead of running the 1st step with the distal model.
David Bau: run it with the base model. Just replace the weights with the base model.
David Bau: and let it come up with like this, totally different x 0.
David Bau: Right?
David Bau: And now just continue all of the steps with distill mode.
David Bau: So let the distant model figure out what the images based on what the base model gave it as a seed.
David Bau: 3, rd
David Bau: so the distant model is just going to try to finish the image, and it does a good job at it. Yeah, so like the way, yeah.
David Bau: So so, oh, right? Yeah.
David Bau: yes. Any questions. Sorry. Just an idea. So can you go back to the slide where you show like, what are all the improved scores again.
David Bau: Not sure if that's gonna make any sense. But, like, you know, I under like you know how it may at least the step where you try to put the base model to the like in distal model, and you got a better like diverse picture. But like, I guess, maybe to like understand user preference or something like that, switching around from like taking something from the distal model and putting it to the base model. Maybe to see like, what about the distal model is contributing for
David Bau: maybe a base model score to increase. Okay.
David Bau: yeah. So we had this appendix section for the transfer of distill to base for sliders. I should probably show the fix as like, non fixable. Okay.
David Bau: yeah. My question would be like, it seems like the morals. The story is the starting point matters right, because once you fix the starting point, it actually can finish it. So. But I'm also wondering
David Bau: I don't. I know it's less efficient that way. But like if, if, instead of like patching in, patching into the the 1st step, what if you're patching like the later steps like them like that. Let's say, the yeah, like.
David Bau: is that a yeah. I had the numbers, but I didn't have a time to plot it so
David Bau: in distilled model, if you
David Bau: So if you were to patch in at the second step.
David Bau: that means the as the the base model had done, most of the heavy lifting already. So it's just it's like a saturated curve where it increased the diversity and then keeps it right. So it gets very, very sensitive like once you.
David Bau: It's like.
David Bau: if you're patching at at a little later step. It's almost all the work has been done, you mean like left. It's just very fascinating like.
David Bau: like
David Bau: all the work has been done. So like, we decided so early, exactly. Yeah, just like touch finishing up all the touch points in the end, which is very interesting. Exactly. So. That's 1 of the calls, which is maybe the future distillation methods have to look into this where the 1st step doesn't really decide the structure like you maybe have, like, some sort of negative reward to not have that structure in the first, st yeah, exactly like, if you can like.
David Bau: somehow send motivated to like, take it. Time to, you know, to commit in the 1st step. Yeah.
David Bau: no other questions.
Aditya Ratan Jannali: Hi.
David Bau: I'm okay. Actually, i i i i thought about it. It wasn't a very good question.
David Bau: It's okay. Any question is fine.
Aditya Ratan Jannali: Hey? I'm I'm I'm attending over Zoom, so I couldn't make it in person.
Aditya Ratan Jannali: I have one question. Can you explain what.
David Bau: Guys, I have a meeting in my my zoom room.
David Bau: Move your discussion to another zoom room, can you? Can you use your own zoom for the rest of your discussion here, and then I'll I'll chat with Carson in my zoom. Here
David Bau: I'm gonna I'm gonna Kirsten. Can you click into this in another second I'm gonna end the zoom here and then we'll we'll do it again. Okay, we'll be right back.
David Bau: I was really, I was supposed to host.