Deepti Ghadiyaram: Thank you. Yeah. Is can you hear us now? Yeah.
Deepti Ghadiyaram: Natalie, can you? That's great. It sounds like she says, thanks.
Deepti Ghadiyaram: Excellent. So let me let me introduce deep t so deep. T is an assistant professor down the street
Deepti Ghadiyaram: at Boston University, so I I know her as one of the leaders in in really understanding computer vision quality. So she's created a lot of evaluation frameworks and data sets. And you know, research that really gets into the question of you know what makes a high quality visual output in in images. And she's doing work in in a lot of work in video. Now
Deepti Ghadiyaram: she is.
Deepti Ghadiyaram: I've been working at runway, which I'm fascinated by because I so how many of you are familiar with what runway does the startup? Right? You know, they're really an impressive
Deepti Ghadiyaram: video generation company. They've been, you know, very focused on real filmmakers. And so I'm you know, quite impressed with the work side, you know.
Deepti Ghadiyaram: Yeah, I'm curious to hear her experience in doing computer science with them. But today we have a treat because she has a talk about a topic that's a little closer to what we do in the lab. She's been looking at the representations inside diffusion models. So I'm very much looking forward to so welcome. Welcome. Dp, thank you.
Deepti Ghadiyaram: Would you prefer? I stand? Yeah, thanks for the kind introduction I've been such a big fan of this lab and and David's work for a very long time. So this is. This is very special that I get to like be here and share what my students and I have been working on. So I've been
Deepti Ghadiyaram: I joined bu last summer, so up until then I spent about 6 years at Facebook, AI research. And then any year as a researcher at runway. Second.
Deepti Ghadiyaram: tell a lot of interesting stories in like both places. But yeah, like the focus of this talk is is very heavily on understanding
Deepti Ghadiyaram: what is packed within the diffusion model. So that that's been the that's been the key question. That sort of like that drove some of the works that that I'll talk about today. So just to set the stage we, we know this like,
Deepti Ghadiyaram: I should generated video
Deepti Ghadiyaram: a few years ago I used to start my talk saying that video data is everywhere. Now, that's that's that's old. It's generated video, that's everywhere. Generative media is everywhere. So
Deepti Ghadiyaram: it's it is so consuming. At this point that it it begs
Deepti Ghadiyaram: us to take a step back, and just think through how we've gotten here, and it hasn't been that long. And this is a very nice video that I credit runway for making it. Just. It shows how quick the progression has been
Deepti Ghadiyaram: in the in the world of like generative image and video models. So this is where we were in 2021,
Deepti Ghadiyaram: and that was the the state of the Ad. And 2022 is where style transfer for videos and sort of real time happened. And this is again like
Deepti Ghadiyaram: this is still outdated because I only have it till, like 2024, I think. And just the the photo realism, the the visual quality the the camera angles and the amount of control that we can offer to users has just
Deepti Ghadiyaram: increased significantly. And the the axis of time has only been like just an year or a couple of years. So that's that's amazing. That's that's brilliant.
Deepti Ghadiyaram: how did we get here? We've not gotten here in just like one big leap. It was a series of Hello!
Deepti Ghadiyaram: It was a series of really, powerful things like coming together that helped us get here. So one is data just like the availability of billions of publicly available videos and rich captions and audio and all of the metadata that's associated with them that played a very significant like role
Deepti Ghadiyaram: that has been and in
Deepti Ghadiyaram: a step improvement in the kind of models that we are that we could leverage. So we we were in the world of like variational autoencoders and Gans a few years ago, but then, slowly, like moved, made our way towards like using auto regressive models, not just for text, but also for image and and video generation.
Deepti Ghadiyaram: And then also like diffusion. So these are like some of the many like tools and techniques that slowly evolved, and all of these and along with compute. So these 3
Deepti Ghadiyaram: came together very nicely in this particular moment in time. And that's that's how like we are where we are right now. So that's that's brilliant. And there are like models being released every week, every day with like new phenomenal, like changes.
Deepti Ghadiyaram: It is
Deepti Ghadiyaram: very, very impressive. Again, like the some. These are some of my like favorite generations. From runways Gen. 3 alpha, I think. Just the ability to to get the interplay of like fog trees the interaction of the lights some of the the shadows. Over here, and just the photo realism of of the bubblegum. It's it's that's
Deepti Ghadiyaram: that sort of drove this inquisitiveness more and more as like, how is this model like getting these things so right? What is it learning? How is it learning so that sort of
Deepti Ghadiyaram: is the key question that you'll see throughout this this talk, and we'll talk about
Deepti Ghadiyaram: one particular tool that we've leveraged to. To to understand this, and like different applications, that sort of emerged out of this this particular tool.
Deepti Ghadiyaram: So the key question is like, What
Deepti Ghadiyaram: how is this information being packed and what information is being packed.
Deepti Ghadiyaram: I don't need to introduce.
Deepti Ghadiyaram: like, why, it's important to do model understanding to this, to this particular group.
Deepti Ghadiyaram: But again, like the the 2 tools that are very popular in the literature, is maybe like train use them as representation learners. Linear probe, add a bunch of like layers. See how they perform on different tasks, and that's sort of a proxy to understand what is the kind of information that is packed in like in these models. So that's 1 way.
Deepti Ghadiyaram: the other way is building. Explain tools that explain what's happening. And these because these are black box in nature, the the kind of models we are talking about. We, the tools that that we'll talk about or I'll talk about here are would be post hoc explanations. So the model is already trained, and we are trying to probe it. To understand again what's packed in in like different layers and different architectures.
Deepti Ghadiyaram: It helps us understand. And then it helps us design better algorithms. Yes, yeah, yeah, go for it.
Deepti Ghadiyaram: So in the previous slide, they were like these 2 sets of
Deepti Ghadiyaram: model understanding methods. I understand the second one, which is post training.
Deepti Ghadiyaram: Use the model as a black box, have an external tool.
Deepti Ghadiyaram: the one. The 1st one is that during training you want to learn, or is it like a post training tool? Good point, I think here,
Deepti Ghadiyaram: the way we've learned are, are typically
Deepti Ghadiyaram: what I had in mind here is post training. So you have a pre-trained model. You want to understand like.
Deepti Ghadiyaram: How good this particular representation that this model has learned. How good this is for task A or task. B. Some features could be good for like classification, but struggle with segmentation, because it's more granular, it's more fine green. So that's what I was like going for. But you're right that we can do this on the go as well.
Deepti Ghadiyaram: yeah. But like, post hoc, both. Yeah.
Deepti Ghadiyaram: So model understanding. There are like convolution networks, for instance, they have
Deepti Ghadiyaram: we've we've looked at how to like visualize different layers. There is
Deepti Ghadiyaram: You can either visualize the the con filters themselves. You can visualize the output. That's 1 way of understanding what these models are learning when it comes to like transformers. There are some simpler tools like visualizing Cls token, visualizing different attention heads, attention maps, as a way to again like, see what is the granularity of the visual information that's being packed
Deepti Ghadiyaram: when it comes to the focus. Here is diffusion models, particularly there are 2 key challenges that we had to handle, and there might be more. But I'll focus on 2. 1 is.
Deepti Ghadiyaram: it's sequential in nature. When you're talking about convolution. It's a standalone like the model is trained. There's just. There is no unrolling in time that is happening. That is.
Deepti Ghadiyaram: that is the scenario. When it comes to diffusion, it's trained iteratively. The inference is also iterative in nature. So where do you slice? And which slice in time makes the most sense to interpret. So that was the 1st question that we wanted to understand.
Deepti Ghadiyaram: And again like
Deepti Ghadiyaram: The second question was, there is
Deepti Ghadiyaram: there is this unrolling that's again not new. We've seen Rnns that do this. But again the visual information. The visual signal that's being processed is very different because of the amount of noise, because of like different noise schedulers because of a lot of other side parameters that influence this choice. So given this again, like which
Deepti Ghadiyaram: slice in the noise scheduling timeline should be, should we sample? And how does that make a difference in like what we understand about the model. So these 2 were the primary challenges when you compare it with like standalone convolution, or like transformers, and really
Deepti Ghadiyaram: and digging deep into like the visual representations that are being learned.
Deepti Ghadiyaram: So we again, the goal is
Deepti Ghadiyaram: study the semantic info. Yes, oh, in the previous slide, when you were mentioning about trying to find what slice to interpret, I mean, I guess ideally it would be good to look at every slice. But that would be very noisy, too. So like, what are like the to actually.
Deepti Ghadiyaram: yeah, I have 2 more slides. So
Deepti Ghadiyaram: this is again like this. People have been trying to study this. This is not. We're not the 1st to like, look into interpreting diffusion models. But we are using a tool that is more popular in the language in the Llm world, and I think is more recently being borrowed into the into the Vlm. And the visual world. It's which is like case parse auto encoders.
Deepti Ghadiyaram: auto encoders are our friends. We we use it for everything right now. So just to like you pass a signal image video text it compresses it. So the power is in and under complete feature space, because we, the one big
Deepti Ghadiyaram: motivation for us to use like encoders autoencoders is to compress the visual signal from a very high dimensional space to and under complete, like a lower dimensional feature space.
Deepti Ghadiyaram: Feed it to whatever application that we want to feature.
Deepti Ghadiyaram: Sparse autoencoder is is the is the opposite in that
Deepti Ghadiyaram: you want the hidden feature, representation or like this feature space to be a very sparse and and over complete in nature.
Deepti Ghadiyaram: The motivation behind building a sparse dictionary is that ideally, if the space is
Deepti Ghadiyaram: is large enough, maybe we can decompose this
Deepti Ghadiyaram: complex mixture of visual signals into separate individual signals. yes, I mean. Generally, the can be
Deepti Ghadiyaram: undercom, not under complete, overcome, complete, but like
Deepti Ghadiyaram: you don't have to go. Take all this, I mean what I mean by it is. Basically you take the embedding, the original embedding. And are you like?
Deepti Ghadiyaram: Is the latent dimension size of your sparse autoencoder bigger than your original space
Deepti Ghadiyaram: in in our experiments. Yes, it is
Deepti Ghadiyaram: do you think? Is it feasible that it can be also smaller and sparse?
Deepti Ghadiyaram: It depends on the the input feature. It depends, not the number. Also the complexity of the the input features that that that we are feeding.
Deepti Ghadiyaram: It's a trend of also trend on the Internet. So yes.
Deepti Ghadiyaram: if yeah, the input itself doesn't have sufficient diversity in the signal. Sufficient complexity. Then
Deepti Ghadiyaram: what you're saying is maybe an over complete set that may not be necessary, but the kind of like as you mentioned like, it's usually trained on like larger Llms, or the the roots of sparse autoencoders, or sparse dictionary like, goes back to like
Deepti Ghadiyaram: I think, eighties or nineties, where again, like the the the motivation was to
Deepti Ghadiyaram: decompose an image signal into ideally linear representation of these, like the dictionary entries.
Deepti Ghadiyaram: But now the dictionary is fixed, either so
Deepti Ghadiyaram: the hypothesis of like one of the earlier works which introduced case parse autoencoders to to probe into like Llm. Llms. Was
Deepti Ghadiyaram: again like overcomplete building an over complete feature. Space and enforcing sparsity might encourage learning of these monosemantic concepts.
Deepti Ghadiyaram: So the motivation is. All of these visual features are highly poly, semantic in nature, so a single like representation has mixtures of of different semantic
Deepti Ghadiyaram: information, and the goal is to split that
Deepti Ghadiyaram: into monosemantic representations. And Caseparse auto encoders are one way that
Deepti Ghadiyaram: that folks are being folks are exploring to. To achieve this.
Deepti Ghadiyaram: Did you use the brand in the implementation case for sure?
Deepti Ghadiyaram: Oh, and generally it comes with like, you can apply like sparsive. l, 1 sparsive penalty.
Deepti Ghadiyaram: Yeah, no, we. We didn't use l 1 sparse because it. The training does not. It's harder to train. So we use. So it's called K sparse autoencoder, because you retain the top K high highest activation like units. And 0 out the rest. So that's that's that's how the sparsity is introduced instead of the
Deepti Ghadiyaram: they're like, you end up with a lot of dead. Yes, and there are. There's a lot of work on like how to handle red neurons. And like, we've we've included all of that to again, the goal is to ensure that
Deepti Ghadiyaram: whatever the model is, learning is meaningful and doesn't end up with like that. Neurons that, as you said.
Deepti Ghadiyaram: set a high level.
Deepti Ghadiyaram: diffusion. Features again, like they are. They have complex semantic information in it. So we treat them as polysemantic in nature. Feed them to an encoder
Deepti Ghadiyaram: and again, like depends on like different features, like the the hidden layer. Representation should be large enough.
Deepti Ghadiyaram: and the hope is that this will break down. This is like an ideal hope to break down into like nice monosemantic representations.
Deepti Ghadiyaram: In addition to like the the
Deepti Ghadiyaram: the K retaining the top K and setting the rest to 0. I mean, it's a not. It's an auto encoder. So you also have the the reconstruction loss. So the output ends up
Deepti Ghadiyaram: being polysemantic. But what we are interested in is these like in hidden representations?
Deepti Ghadiyaram: Any questions?
Deepti Ghadiyaram: So what we, what we did is like we use case
Deepti Ghadiyaram: case parse auto encoders as as a tool and trained it on diffusion. Feature representations extracted from different image data sets so
Deepti Ghadiyaram: looked into fine grained image data sets, such as like pets, and aircraft bunch of like image, fine grained data sets, and also like coarser grain data sets because it? We're really curious as to what? What are the patterns that these the hidden layer is latching on? What is it learning? Is there? Is there some systematic information that can be that gets revealed?
Deepti Ghadiyaram: So what we found is? And again, like, I'll I'll
Deepti Ghadiyaram: go through like different layers and different time steps. But
Deepti Ghadiyaram: picking the correct layer matters like in the ideal. In like one of the layers, we were able to see
Deepti Ghadiyaram: the specific breeds of different animals surfacing pretty nicely. And there was another layers, features where the breeds of dogs were getting mixed up, but dogs were being like
Deepti Ghadiyaram: one of the neurons was like firing up the most when every time it sees
Deepti Ghadiyaram: the one of the hidden unit.
Deepti Ghadiyaram: the activation of that would be like high
Deepti Ghadiyaram: which for the diffusion features so we looked at stable diffusion 1.5 2.1 and flux.
Deepti Ghadiyaram: so get to that? Just I think that's the 1st question. So what flavors of visual information is being captured in in different layers? So that was the 1st question.
Deepti Ghadiyaram: I'm using unit style architecture. But did flux like any, even if it's transformer based? It's just a sequence of like blocks of different like resolutions.
Deepti Ghadiyaram: So the notation here corresponds to unit. But it did apply to like other architectures as well. So we did see that
Deepti Ghadiyaram: around the bottleneck, which is the the one with the least amount, or, like most compact visual representation. The features are like the we were looking at top. N neurons that were firing up to get like a more generalized understanding. It was very coarse screen. There was. No, it was just
Deepti Ghadiyaram: I think, looking at objects which are in similar poses doesn't matter if they're like cats, dogs just pose. So our
Deepti Ghadiyaram: understanding here was.
Deepti Ghadiyaram: the resolution is so low that it's it's looking. The information that is packed is very coarse, grained in nature.
Deepti Ghadiyaram: One level up.
Deepti Ghadiyaram: The features were getting more relatively like more granular than than the below one. So it was. It started to group dogs and and cats into different like. So different neurons were like firing up for different animals.
Deepti Ghadiyaram: So this was the the magic layer up feet, one which corresponds to, I think, block 14 in did, and a similar like middle-ish plus plus block in flux.
Deepti Ghadiyaram: That's where this is 1.4 and upfit one.
Deepti Ghadiyaram: I I don't think I have this in the slides, but we have it in the paper that the same applied to like 2.1. And did
Deepti Ghadiyaram: yeah, did. And like flux. But this particular one is for 1.4. Can I clarify a question? Yeah, so is this, is this a model that you trained from scratch on your data? Or this is the pre-trained model that you're analyzing. Yeah, I should have clarified. So this is pre-trained.
Deepti Ghadiyaram: set of models like stable diffusion flux. Or did all we did was extract features of these images from these respective like layers.
Deepti Ghadiyaram: These images belong to particular image data sets so fine grained image data sets.
Deepti Ghadiyaram: So just pass the the image. No, no training. Just end directly. Extract the features.
Deepti Ghadiyaram: So once the model is trained.
Deepti Ghadiyaram: We can still like pass the image as the input.
Deepti Ghadiyaram: Ideally it can it? It can go through the whole process. But you can sample a particular like
Deepti Ghadiyaram: we do noise image as it goes through like the different different timestamps.
Deepti Ghadiyaram: Great!
Deepti Ghadiyaram: I don't know about him, but like he's the underlying architecture of Sd. Event.
Deepti Ghadiyaram: it is unique. Yes, but it has, like every option that blocks, is actually a transformer.
Deepti Ghadiyaram: not in stable diffusion in Di, as you know, with it, with attention.
Deepti Ghadiyaram: I don't know.
Deepti Ghadiyaram: There are many bands out there.
Deepti Ghadiyaram: Yeah. So I mean, I don't know about 1.4. But, like, you know, Sdx server, the unit.
Deepti Ghadiyaram: like the individual blocks consists of multiple transformer blocks
Deepti Ghadiyaram: with attention and Mrp. And then there are also post attention layers which I think
Deepti Ghadiyaram: makes sense that this is very similar to the flex.
Deepti Ghadiyaram: Did you notice differences in reconstruction between the like? The divine grained?
Deepti Ghadiyaram: You know the following, here we didn't
Deepti Ghadiyaram: trained it. So we just like, did one forward pass extracted just for feature extraction.
Deepti Ghadiyaram: I guess. Where did you get the Sds?
Deepti Ghadiyaram: Oh, so you're talking about the essays of the reconstruction loss of the essay.
Deepti Ghadiyaram: let me double check that I I don't. I don't think there was a pattern for different features. But yeah, I can check.
Deepti Ghadiyaram: Good question. So when you talk about like, yeah, these features are, I mean, they are fine grained. They're coarse grain. Is it like a manual interpretations, or your
Deepti Ghadiyaram: yeah, and I'll get to that.
Deepti Ghadiyaram: So this particular like layer was where we were able to see very nice, like separation of like when we would, for the particular case, if a particular breed was like image.
Deepti Ghadiyaram: A dog of a particular breed was like sent. Then consistently the same neuron was getting like activated. And then the signal got diluted as we went higher up. And again, like the theory which I think a couple of others have also recently found is that now the alignment is more with the with the diffusion loss. So here there is like
Deepti Ghadiyaram: nice semantic information that is, that is being captured. But again, as we traverse like deeper than it, it gets more aligned with the with the actual loss that we are using.
Deepti Ghadiyaram: So so the summary here is different layers to capture information of different granularity. But the technique that we used was not was like visual inspection. We used, like Vlm to like, send all of these images and like, say, what is the common pattern that's emerging? So it it is okay. But it's it's still like, qualitative.
Deepti Ghadiyaram: We wanted to make this more concrete or more quantitative, so we resorted to the. So going back, we talked about 2 ways of like interpreting the model. One is just using it as representation learners, and the other is like post hoc, we treated Kse as like post hoc, way of explaining
Deepti Ghadiyaram: so just to like validate that we were not, I don't know. Overfitting our assumptions by visualizing this, we we trained like a a simple like classifier
Deepti Ghadiyaram: on like different features, the hosting, or essentially the
Deepti Ghadiyaram: you know. Yeah, I should have clarified. It's directly these. So just like, pass the image, get the features from this particular layer and use it to to train like a simple like learn with so 3 data sets I font should could have been bigger.
Deepti Ghadiyaram: So the purple and the green correspond to finer grained image data sets. So these are so the pipeline was extract images from these data sets train particular, do a train valve split and report accuracy on like the test. Split?
Deepti Ghadiyaram: So you do see that for the for the finer grain again. Consistent with our Kse observations up feet one
Deepti Ghadiyaram: was the blog that was performing
Deepti Ghadiyaram: significantly better than the other year
Deepti Ghadiyaram: for the core screen. There wasn't. It didn't make much of a difference. It could also be because of the choice of the data set, because Caltech I mean, I think if you. The
Deepti Ghadiyaram: the classes are so nicely separated in this particular data set that very coarse visual signal is enough to to to do a good classification task.
Deepti Ghadiyaram: I don't have it in this slide, but we again like tried it on image, net, and other like
Deepti Ghadiyaram: what we think as like more coarser feature classes, and it it was consistent that
Deepti Ghadiyaram: the performance, particularly for finer grain data sets was better for for up feet, one for coarse, grained the the deltas. Weren't that that high
Deepti Ghadiyaram: Does it make any sense to use the encoder
Deepti Ghadiyaram: instead of I mean, use all outputs. So you used either from the bottom or the decoders.
Deepti Ghadiyaram: Can you repeat? Oh, she's not. This is not an essay at all. So I'm talking about.
Deepti Ghadiyaram: So basically, you're taking the output from either the bottleneck layer or the the up sampling.
Deepti Ghadiyaram: You can also do the same thing from the downside
Deepti Ghadiyaram: from the down sample. So you're going down while you're going down until yeah.
Deepti Ghadiyaram: I think we tried. We tried that.
Deepti Ghadiyaram: yeah, I think that. Yeah, we did try that. Are you looking at the outputs of the blocks, or like the thing that's making people would refer to as residual sleep.
Deepti Ghadiyaram: We were using the residual outputs.
Deepti Ghadiyaram: So the X plus F of X. Yes, okay.
Deepti Ghadiyaram: So something that we mentioned earlier, like for Caltech and Imagenet and other data sets
Deepti Ghadiyaram: a fit. One overall does better. But the deltas weren't that high? Yes. So it's x plus F of X makes sense. But you're doing it in the blocks.
Deepti Ghadiyaram: We also take the unit residual also.
Deepti Ghadiyaram: So I so you mean, like the the residual that comes from
Deepti Ghadiyaram: If it does, then you're automatically considering the decoder function
Deepti Ghadiyaram: I should check. I think we do. Yeah, I will check.
Deepti Ghadiyaram: I will get back. Yeah, that's a good point. Yeah.
Deepti Ghadiyaram: So the summary was that different layers have different contribution, and the the info, the granularity of the information varies from like course to fine grained, and then again, like sort of
Deepti Ghadiyaram: becomes course, and aligns with the the final objective.
Deepti Ghadiyaram: So the second question that we wanted to study was again like what time step to sample on again like this is during the
Deepti Ghadiyaram: just the the notation, like we start with like a clean image, and progressively add noise. So that's sort of like the time axes that we were looking at. So it it makes sense to like the cleanest latent, or the image with like least amount of noise, should do should do better. And
Deepti Ghadiyaram: my hope was that this plot will like fall through like immediately, like even at like T equal to like
Deepti Ghadiyaram: at 50, I thought, there will be like a drop. Are you adding noise now along the actor
Deepti Ghadiyaram: along? So the x-axis is the the time step. So yeah, as so t equal to 0 clean image 200 has more t, equal to 1,000. I don't have it. Here is like, just completely noise.
Deepti Ghadiyaram: So one thing that we observed again,
Deepti Ghadiyaram: blue and green are fine grained. The trend is sharper, the drop is sharper with the amount of noise for finer grain data sets but coarse grain.
Deepti Ghadiyaram: there is a slight boost in the performance, and we've we've seen this. And there was like another recent paper that also that also had this which and was again like
Deepti Ghadiyaram: not something that we were expecting. And this was something that we observed across like a couple of other data sets. It was not an artifact of like Caltech.
Deepti Ghadiyaram: Hypothesis is maybe like some noise
Deepti Ghadiyaram: like a regularizer. but again, like
Deepti Ghadiyaram: I'm still confused about the setup. Honestly like, if you take an image, you add some noise right.
Deepti Ghadiyaram: But now, to get representations, you need to do it by unit forward pass, and that one also receives as an input, text, prompt? Right? So all of these like, because we wanted to study, only the visual, like part. The text prompt was, was empty
Deepti Ghadiyaram: audience.
Deepti Ghadiyaram: And you're not using some sophisticated like diffusion inversion
Deepti Ghadiyaram: to get the noise that leads to the exact same image. But just yeah, just like Standalone, like feature extraction and studying these features.
Deepti Ghadiyaram: No, yeah.
Deepti Ghadiyaram: So you're not in the same.
Deepti Ghadiyaram: Otherwise you cannot get the new.
Deepti Ghadiyaram: Okay?
Deepti Ghadiyaram: So another question we wanted to study is, is this, just like a
Deepti Ghadiyaram: we started off with stable diffusion, variance? And yes, So
Deepti Ghadiyaram: since you're taking activations from the residual connection.
Deepti Ghadiyaram: you're right, like, what you would expect is the cleanup device will have most information.
Deepti Ghadiyaram: But another question to ask is, at what point did the diffusion model really start generating this image?
Deepti Ghadiyaram: It's because it's recursive, which is one of the problems you explained. So you are technically giving it the full, decoded image it already generated in the final step.
Deepti Ghadiyaram: So it already has the image in there. It just has to denoise a little bit.
Deepti Ghadiyaram: And now, since you're taking the residual output.
Deepti Ghadiyaram: the information coming from your inputs
Deepti Ghadiyaram: is playing the key role at giving you the features.
Deepti Ghadiyaram: I don't know if that made sense.
Deepti Ghadiyaram: Your input is already an image right?
Deepti Ghadiyaram: And the the features that you're collecting is the residual plus whatever your diffusion models layers have given you.
Deepti Ghadiyaram: So
Deepti Ghadiyaram: the receiver already has the information about your image right? Another question is, at what layers or time steps. Did this part of the actual diffusion model
Deepti Ghadiyaram: really give semantic information is is the question because, maybe we are like
Deepti Ghadiyaram: there is like a residual from the down pass
Deepti Ghadiyaram: that signal plus the the Pr. Upsample signal. So is the is it. Are you talking about the entanglement between the 2 or
Deepti Ghadiyaram: so? The I wouldn't assume that X 0 is where the diffusion model has the most rich features, right? Because its job is to just remove that tiny bit noise it has remaining.
Deepti Ghadiyaram: and somewhere around the mid time steps is when it's actually trying to generate the image. So that might be the most powerful times step
Deepti Ghadiyaram: when I think about, when is the diffusion model actually generating versus? When are its activations rich? I see, I see right. Activations are rich at the final time step, because the import is already a very
Deepti Ghadiyaram: well structured image.
Deepti Ghadiyaram: Maybe I don't
Deepti Ghadiyaram: like 400 or 300 is when the diffusion model actually starts to dream this new image that it's generating. Okay? So I expect that to be like the most rich features. I I don't know if that's something you
Deepti Ghadiyaram: looked into me with.
Deepti Ghadiyaram: I mean, we did this only with like,
Deepti Ghadiyaram: like building this small classifier on top and
Deepti Ghadiyaram: for different data sets like the performance did drop off after 200.
Deepti Ghadiyaram: So if you took the signal out from the before it even goes to the residue and run again, the classifiers do you? Would you see a different shape in your plot?
Deepti Ghadiyaram: I want to run that and and answer that question, because, yeah, it's it's but yeah, I think it. That is a
Deepti Ghadiyaram: more controlled way of studying the the contribution. Yeah, like, it'll be very nice if you could see that a different shape comes up. Yeah, yeah, I don't know what that answers, but that'd be nice.
Deepti Ghadiyaram: Make sense.
Deepti Ghadiyaram: So
Deepti Ghadiyaram: we studied, as I mentioned, like different architectures. that there are like detailed results, numbers in the paper, but something that what I'm showing here is for this particular image. We looked at the features, extracted features and, like
Deepti Ghadiyaram: constructed like Bca feature maps. So I identified like top 3 principal components and color coded them as RGB, just for reference.
Deepti Ghadiyaram: and we found that just overall. There is more
Deepti Ghadiyaram: but primarily because of, like the the underlying convolution versus like transformer and like based differences. There is more spatially localized information in the in the Sd 1.4 style like architecture compared to like did
Deepti Ghadiyaram: And this also kind of translated to the the classification like performances, or also the per
Deepti Ghadiyaram: When we did the
Deepti Ghadiyaram: case, parse auto encoder, and, like visualization across like different blocks. The the signal was cleaner, for, like unit based architecture compared to like the IP.
Deepti Ghadiyaram: What is the thing that should catch our eye.
Deepti Ghadiyaram: because I don't really see it
Deepti Ghadiyaram: so what? At least like our, my interpretation here is there are we were able to separate out one principal component, which was which was localizing this particular like person.
Deepti Ghadiyaram: And one other like principle, something that
Deepti Ghadiyaram: Such clean separation is was not something that we were able to achieve with like the it based features
Deepti Ghadiyaram: was, was our it? We did, and it it gets aligned with the the pre-training objective, and the signal gets lost.
Deepti Ghadiyaram: 50 or something. Bucks right?
Deepti Ghadiyaram: So this is with, like the the dip model, and it had like 28,
Deepti Ghadiyaram: this particular model that we used flux. We don't have this visualization for flux.
Deepti Ghadiyaram: Yeah, we have flux results in the paper. And we did sample like
Deepti Ghadiyaram: the middle and upper middle blocks in flux, like I would say that something smart to say, because, like influx
Deepti Ghadiyaram: around block 16 or 17, you really you get some high norm activations like super duper high norm, I see.
Deepti Ghadiyaram: So they will. Definitely it would mess up.
Deepti Ghadiyaram: Got him.
Deepti Ghadiyaram: Okay?
Deepti Ghadiyaram: yeah. Just going going back to the previous slide. So generally. Still, you know, new vision side. But
Deepti Ghadiyaram: how like kind of repeating the question like, How exactly are you able to recognize that
Deepti Ghadiyaram: the previous one with the unit one is specially localized versus
Deepti Ghadiyaram: dids information, for example, because in block 12, like, I noticed that it is also. Still, I don't know spatially.
Deepti Ghadiyaram: Yeah, I am here.
Deepti Ghadiyaram: Yeah, I I think so. I see your point. I think we
Deepti Ghadiyaram: we were trying to align with the the image. And again, there could like we wanted the person like the semantic object person to be separated out by compared to like the semantic object of the car.
Deepti Ghadiyaram: So again, like, that is sort of what drove our like visualization that maybe the separation is cleaner when we are using like, update one off unit compared to like the did base.
Deepti Ghadiyaram: hypothesis here is because, like there is like attention, and everything attends to everything. So maybe the the that
Deepti Ghadiyaram: has an impact on like how much?
Deepti Ghadiyaram: The spatially localized information that's being packed in these features and speaking of spatially localized like, I understand, with the object segmentation itself. But it's like background and foreground. Also considered a thing again, just basically referring to the platform.
Deepti Ghadiyaram: That's like as we should.
Deepti Ghadiyaram: I think that is like a yeah, a a coarser
Deepti Ghadiyaram: segmentation of the scene is foreground and background, and if we get like more finer grain, then we want like objects. But yeah, I see your point. That block 12 doesn't look as bad. but the
Deepti Ghadiyaram: the murkiness in like the signal between like the person and the car was what we were like. Yeah. Looking for we set we can give text conditioning. But we set it to like an empty.
Deepti Ghadiyaram: Was that the question? Is it even possible to condition dit? I don't know, because I the last time I remember dit was this
Deepti Ghadiyaram: naive architecture?
Deepti Ghadiyaram: or maybe there's a new did. But I I will check. I I like for the the stable diffusion models, at least like we explicitly set everything to like an empty, prompt yeah.
Deepti Ghadiyaram: Assuming that if dit cannot be text condition.
Deepti Ghadiyaram: maybe that's also possibly a reason why you see this segmentation not happening. There is no
Deepti Ghadiyaram: external like signal during training. Yeah, it was trained on image net data. So it should have some semantic understanding.
Deepti Ghadiyaram: So it's just the additional supervision. Yeah.
Deepti Ghadiyaram: But like region transformers in general don't have exponential.
Deepti Ghadiyaram: They don't know.
Deepti Ghadiyaram: Okay, I do want to be mindful of time like, when should we let her move around? I think that she's probably about 10% to the top? We're about 90% to the time. How much time do you have? So yeah, 10 min. Just for me. Okay, I'll try to do that. So
Deepti Ghadiyaram: so we studied different layers, time, steps, and architectures, like things that I did not cover. And again, like these were the students who
Deepti Ghadiyaram: did all the work
Deepti Ghadiyaram: again, like does the feature the certain features perform well for certain other tasks, visual reasoning that was like something that we're very interested in, like, can we use these features? To understand certain like tasks where currently clip and other models like seem to struggle? And again, we did study like what happens if we like 0 or like.
Deepti Ghadiyaram: don't send any external conditioning, and how those representations are different. So everything is is in the in the paper.
Deepti Ghadiyaram: Oh.
Deepti Ghadiyaram: so the next question that we wanted to study was so. This was one way of interpreting, like diffusion, features and the semantic information in it. Are there other ways to do this? And
Deepti Ghadiyaram: this again, like ties back to our interest in like being able to offer more control to the users while generating an image.
Deepti Ghadiyaram: So
Deepti Ghadiyaram: something that we did is that if this, if we are learning monosemantic features using case parse auto encoders, can I systematically like steer the image away from a particular monosemantic concept. So there's a node image and the the monosemantic concept that I'm interested in like removing or steering it away from. Is nudity, then
Deepti Ghadiyaram: do that for different artistic styles, photographic styles of or object attributes.
Deepti Ghadiyaram: So this
Deepti Ghadiyaram: The approach is again like, was very straightforward. This is our like
Deepti Ghadiyaram: standard pipeline like start with like a noisy image, give a prompt and then, like the model like, enforce and generates the image. What we did
Deepti Ghadiyaram: on the side was, we train like a case parse autoencoder. So just for now, like assume, we have, like a pre-trained case. Parse auto encoder. I'll I'll get to how we train it in like a couple of slides.
Deepti Ghadiyaram: So given this trained, sparse autoencoder, and this is trained on a bunch of text prompts only, no, no visual signal.
Deepti Ghadiyaram: Once this is strained, we extract the the text representation of a particular concept. We want to steer, identify
Deepti Ghadiyaram: the again like extract the intermediate representation from it.
Deepti Ghadiyaram: if you want to steer away, then we multiply it with a particular like steering strength, and
Deepti Ghadiyaram: just combine it with the the feature representation, the the text representation of the of the original prompt. And this was
Deepti Ghadiyaram: a simple addition between the 2 representations, and we found that
Deepti Ghadiyaram: this simple trick was enough for us to somehow not just isolate this semantic concept, but also
Deepti Ghadiyaram: squash that from the overall like representation. So use that in the diffusion, like denoising process, and we were able to like generate images which are again like that particular concept was removed.
Deepti Ghadiyaram: And this was trained again only on like pure, like text representation,
Deepti Ghadiyaram: can be trained if the goal is to steer away from like a bunch of unsafe concepts. Then we that is what those were the set of prompts we used for for pre-training it. If the goal was like photographic styles or artistic styles. Then there was like this other sparse hotel, and could be trained on like everything else like anything else, and
Deepti Ghadiyaram: trained it once, and we were able to use it to steer any concept of choice.
Deepti Ghadiyaram: So a couple of quick examples and
Deepti Ghadiyaram: if you want to steer something towards a concept, then, like the steering strength, had to be like higher again, the concept that we wanted to steer it towards was, make the style like more minimalistic
Deepti Ghadiyaram: and again, like, if so I think, yeah, so magnify
Deepti Ghadiyaram: only at like a particular.
Deepti Ghadiyaram: What do you mean by the entire heights?
Deepti Ghadiyaram: This is only text, Emily, yeah.
Deepti Ghadiyaram: So the previous result was on like stable diffusion 1.4, and this was on flux again. Just wanted to see if
Deepti Ghadiyaram: this is generalizable across like different architectures, and because Kse was trained in a standalone fashion that was architecture agnostic. So you can train it once and use it on like any model, any diffusion, architecture of your choice. So this was for like
Deepti Ghadiyaram: concept, removal, seniority and violence were the examples
Deepti Ghadiyaram: does compete well with other like state of the art methods.
Deepti Ghadiyaram: In this particular case we wanted to change the sort of like the environment or the style.
Deepti Ghadiyaram: One surprising.
Deepti Ghadiyaram: A pleasantly surprising thing was we wanted to study because the model was trained on like prompts, which are not, which are like reasonably well formed.
Deepti Ghadiyaram: How would the how resilient would these monosemantic features that we learn how resilient would they be to like a prompt which has, like some adversarial noise, or some in it?
Deepti Ghadiyaram: And we did. It's a lot of numbers, large table, but we did find that it does compete reasonably well across, like different data sets. Particularly surprising was there were models
Deepti Ghadiyaram: trained or like, built with a focus on like making sure that they are more resilient to like adversarial attacks. But that was not the intention. But this was a very nice like side effect. It shows that the
Deepti Ghadiyaram: the monosemantic features that are being learned are quite resilient, reasonably like, separable in terms of like.
Deepti Ghadiyaram: I think the concept nudity, or the concept of a particular photographic style is nicely captured, and that makes it more more reliable.
Deepti Ghadiyaram: So yeah, this was a work led by Dahey, my student, and the nice property of this was. It was giving us more insight into like, how the semantic information is packed in the text encoders, and we were able to use it to, to nicely control the generations.
Deepti Ghadiyaram: And I mentioned this.
Deepti Ghadiyaram: We. We trained it only once, and we were able to use it across like different models. So that was again like a nice property.
Deepti Ghadiyaram: because they all use the same text. Encoder.
Deepti Ghadiyaram: yeah, so that's that's a dependency. So if I think sdxl turbo it has like 2 text encoders. So as long as the text encoder is is the same, then you can use it.
Deepti Ghadiyaram: So one
Deepti Ghadiyaram: thing that you'd see, I mean it's all it does look on the surface like like a nice change in in the scenery, but especially like the bench, is different, and, like the scenery, is also like, there is a house that came up here that was not there in this particular scene. So this is not
Deepti Ghadiyaram: it is doing a reasonable job. But there are mistakes that our model is making and
Deepti Ghadiyaram: multiple reasons. We only used text feature representations. And that's that's not enough. That's not enough of a signal. This is a complex model like, there is a lot of like interactions. And these are global features.
Deepti Ghadiyaram: But we want to see if we can use both visual and text and text features and
Deepti Ghadiyaram: be able to steer the image more locally. So that
Deepti Ghadiyaram: yeah, the bench remains the same. Essentially. So whatever is in the input it should stay intact in in the output. And I mean.
Deepti Ghadiyaram: since we worked on this, there are, I think, at least 10 other papers which which do this with very nice results. But yeah, want to see if
Deepti Ghadiyaram: I think if we can understand how.
Deepti Ghadiyaram: if there is any localization of information within cases, maybe that could that could again like help us train like a more resilient like method.
Deepti Ghadiyaram: What? Something that the reason why we resorted to text embeddings was that steering only with visual embeddings was changing too many things in the scene, so which meant that the features were not
Deepti Ghadiyaram: so in the 1st work we did see that there are like granular certain. But one layer was like more granular than the other, but when it came to steering it wasn't as as effective, because it was changing too many things when we were trying to just remove nudity, for instance, and it would like,
Deepti Ghadiyaram: yeah, add a bunch more things that we didn't want.
Deepti Ghadiyaram: So
Deepti Ghadiyaram: we want, we really want to understand why this was happening. Is it the is it limited by the capacity of the sparse Autoencoder? Is it not over complete enough or just one layer of sparse autoencoder is not enough. Because now people are using like multi-layer sparse autoencoders.
Deepti Ghadiyaram: yeah, we lose interpretability there, if you start adding layers. But yeah, really interested to understand why?
Deepti Ghadiyaram: The superposition of like different features is happening.
Deepti Ghadiyaram: something that where I'm interested in just because the the features seemed robust. Even when the prompts had, like adversarial prompts had adversarial inputs, was like, is there like a
Deepti Ghadiyaram: common pattern that emerges in these activation like units
Deepti Ghadiyaram: with and without these, when it's jailbreaked when it's not so, if so, we can like
Deepti Ghadiyaram: build tools to
Deepti Ghadiyaram: figure out. Okay, there is like, maybe there is an attempt of jailbreaking that's happening, or just avoid that from happening.
Deepti Ghadiyaram: So this is again like
Deepti Ghadiyaram: and not limited to diffusion generation. It's more aligned with like language, only models. But something that we're interested in and are looking at.
Deepti Ghadiyaram: So if yeah, anyone has any thoughts or ideas, or like feedback love to like, hear. So this talk was primarily about just glorifying sparse auto encoders. They were very helpful
Deepti Ghadiyaram: as a tool to understand different, like visual feature, granularity. And they also helped us.
Deepti Ghadiyaram: Offer more control to the user with
Deepti Ghadiyaram: caveats. But yeah, I think it did do a reasonable job in like maintaining the semantic information.
Deepti Ghadiyaram: So that's all I got. And yeah, more, any more questions.
Deepti Ghadiyaram: Thank you.
Deepti Ghadiyaram: Let's stick with just a couple questions, and then and then we'll we'll take a little break, and also people want to talk. I want to make sure there's some time to talk one on one
Deepti Ghadiyaram: for a couple of folks, but if there's any any clarifying questions or any other
Deepti Ghadiyaram: wrong questions for the time.
Deepti Ghadiyaram: No, we asked a lot during the talk.
Deepti Ghadiyaram: This clip is saying that also find you that also edited flux circles,
Deepti Ghadiyaram: I think that's a new finding, because flux's main text encoding is T. 5. 0, this we used so for flux, we use whatever text encoders. Yeah, I should. So flux also has clip in it. So yeah, for flux. And Sd
Deepti Ghadiyaram: has 2 text encoders. So we use 2 yeah, 2. Essay, yeah, I should have clarified. So whatever is the underlying text encoder, the Kse has to be trained on it on both. If it uses both, then it use both. I think there is an ablation in the paper where we steer only one and not the other.
Deepti Ghadiyaram: yeah. But I think in most of the results that we that I have here, like both, are being missed.
Deepti Ghadiyaram: You were you talked a lot about monosensitivity?
Deepti Ghadiyaram: but like, how do you know that the features are actually monosemantic like, are you selecting a single feature here or
Deepti Ghadiyaram: so in this particular case, like so that's a good question, like one way? That was exactly the question. Like in the 1st paper, which sort of helped us build towards this one the 1st work that I talked about. We were using Vlm. Like tagging and like building a bunch, adding a bunch of layers. But that still was not validating, that these features were monosemantic, which was sort of like
Deepti Ghadiyaram: in the 1st work. You didn't steer the generation. And all right, right? So the steering was our way to like, really understand if these are like truly monosemantic, because if I give
Deepti Ghadiyaram: just like a concept of nurity, and if the model
Deepti Ghadiyaram: the assumption is this is a monosemantic concept, and if the models one of the internal unit has really captured that monosemantic concept, then steering only that should retain as much as of.
Deepti Ghadiyaram: should only change this concept and retain everything else. So that was sort of
Deepti Ghadiyaram: proof of concept that led to this and they are
Deepti Ghadiyaram: roughly monosemantic. But again, like these
Deepti Ghadiyaram: changes means that there is some leakage that is happening. Which
Deepti Ghadiyaram: is is of interest to explore, which that could just be diffusion.
Deepti Ghadiyaram: You know. Yeah, if you have a bigger model, maybe flux, something.
Deepti Ghadiyaram: future works 30 billion model, maybe it'll fix it.
Deepti Ghadiyaram: Yeah, like, the ideas changes and interesting. And
Deepti Ghadiyaram: this you can just do the prompting, and that you can be like a portal of them previous events
Deepti Ghadiyaram: and snow.
Deepti Ghadiyaram: I think it's when it comes to like
Deepti Ghadiyaram: Some of the applications that I've seen at runway are like user uploads an image and says, Now change my my shirt or my hair color and then now add glasses, so they want to retain the initial image and make edits to it. In that case you can't like
Deepti Ghadiyaram: yeah. The refusion. Inversion is one technique that could be other techniques. And I think this steering and sliding are one more
Deepti Ghadiyaram: hearing.
Deepti Ghadiyaram: Good fuck.
Deepti Ghadiyaram: Do you think like part of the reason steering on the image domain is not really as nice is because it's just more difficult to do it correctly. I mean.
Deepti Ghadiyaram: you know, steering on the front. You're just adding to one vector, you don't. And the model figures out, you know what which spatial locations to update and on. But if you would steer on the visual domain
Deepti Ghadiyaram: you would have to do this part.
Deepti Ghadiyaram: We can the you can like, pick a layer, pick a time, step and and do this. But my hypothesis is these are not as monosemantic, so compared to text, only features and visual only features.
Deepti Ghadiyaram: Oh.
Deepti Ghadiyaram: this at least our experiments. It did show that text features were, we were able to break them down into semantic monosemantic concepts.
Deepti Ghadiyaram: Better again, like other networks, a multi-layer Ksas, which folks are using. Those could be like
Deepti Ghadiyaram: ways to get achieve this. Thank you. Yeah, yeah, thank you.
Deepti Ghadiyaram: Yeah, that's the truth.