177 Huntington: So so today, we have yeah. So today, we have our new postdoc, Chris Wendler.
177 Huntington: who's going to talk about a new approach for trying to understand this great mystery of diffusion models and know so much.
177 Huntington: and we know so little about what they're doing. So so I'm really looking forward to it.
177 Huntington: hearing what he's found. All right. I'm I'm looking forward to explain a bit.
177 Huntington: so I will start like like a bit jamming on your
177 Huntington: anecdotes that, like research, is kind of also teaching.
177 Huntington: This project, initially started as a class project in Epfs
177 Huntington: multimodal course, I don't know the exact name.
177 Huntington: So Slava and a colleague of him. Shortcutslaver.
177 Huntington: They implemented Saes on Sdxl Turbar.
177 Huntington: and they had something that looked kind of cool.
177 Huntington: but like when he came to me.
177 Huntington: His interventions kind of didn't influence the generation at all.
177 Huntington: So we started out with the classical settings that you find in the anthropic blog posts like expansion factor 64.
177 Huntington: Yeah. And he was like, Can I do an internship with you?
177 Huntington: So I asked around. And I was really lucky. I found Shalar, who had some money for this project.
177 Huntington: so we were able to hire this really amazing student like he's a force of nature
177 Huntington: and gave him the, you know, platform to execute this project.
177 Huntington: Hi, can someone help like a student is trying to get in. Her name is Gixio.
177 Huntington: Oh, yeah, I can. I can. Okay, thanks. Yeah.
177 Huntington: So we we have a big team. What's her name?
177 Huntington: Let's see.
177 Huntington: Okay, I will just continue.
177 Huntington: It's just he's trying to judgment.
177 Huntington: Okay, yeah. Have you heard me, Caleb?
177 Huntington: Michael? He always joined my reading groups and is really interested in interpretability and in AI safety. He was really interested in this interpretability project as well.
177 Huntington: Justin is our diffusion expert.
177 Huntington: He literally wrote up the entire diffusion math for me, because I thought I needed in the paper. We ended up throwing it out completely, sadly. But we have it somewhere.
177 Huntington: It's amazing, Bob, really amazing. Prof. And Shalar, who made all of this possible?
177 Huntington: Yeah.
177 Huntington: So let's get into.
177 Huntington: It's this is something that you can do. So
177 Huntington: we here we we turned on like a animal feature. On the face of this model.
177 Huntington: And you you kind of can get like some hybrid in between human and animal and you can also get.
177 Huntington: I mean, as you change the feature strength,
177 Huntington: it will interpretably affect the image.
177 Huntington: So I think, like most of you know what space autoencoders are, but like we are going to do like a very brief recap. So it was considered the interpretability breakthrough. I guess people were really excited. Now. Finally, people are like getting more sober
177 Huntington: about it, and I like, Wait a minute. Minute.
177 Huntington: But like the with this method, you, you can, in an unsupervised way, find, like
177 Huntington: seemingly very interpretable directions in your intermediate results of your neural network forward pass.
177 Huntington: and, like for language models, they have been scaled up to the largest language models and
177 Huntington: anthropic train, some of them on Claude, and they found, like this famous Golden Gate Bridge feature.
177 Huntington: So the the plot is quite overloaded. But like focus on
177 Huntington: just the activation level on the bottom.
177 Huntington: You can see, like, if you have a high activation level. If you look at the examples, they are about the Golden Gate Bridge
177 Huntington: very specifically about the Golden Gate Bridge, independent of the modality and also at lower positive activation levels. This is like a bridge bridgey. Like feature, it gets excited by bridges.
177 Huntington: but why? Why bother? Why do we need even to learn these interpretable features like, couldn't we just look at neurons?
177 Huntington: So we have like this kind of
177 Huntington: ancient insight, basically since the beginning of deep learning. Because this is when people started optimizing over the input image
177 Huntington: over the image domain in order to excite neurons
177 Huntington: that neurons are kind of polysemantic. A lot of times.
177 Huntington: So the same neuron can respond to different things. It can respond to dogs. But it can also respond to maces in this synthetic example here
177 Huntington: and like recently, people came up with this superposition theory like, it's more like a hypothesis. It's not really a theory that makes predictions like theories that you have in physics.
177 Huntington: They came up with this explanation that could be underlying this polysemanticity phenomenon.
177 Huntington: And it basically addresses the following puzzle.
177 Huntington: if you have only 2 neurons.
177 Huntington: How can you represent 5 features?
177 Huntington: Right?
177 Huntington: So 2 is kind of smaller than 5. So you probably are not going to represent like a feature with a neuron.
177 Huntington: But if you.
177 Huntington: you know, put vectors onto the 2D plane, you can kind of put an arbitrary number of vectors there, and as long as the features don't Co. Occur all of the time. This is where the sparsity comes in.
177 Huntington: You are able to like nicely represent your 5 features most of the time.
177 Huntington: And that's the that's called superposition, basically putting the features into the space in a non-orthogonal way.
177 Huntington: And the idea of the spouse autoencoder is now, hey, given that.
177 Huntington: you know. Usually in the Internet, you have more than 5 concepts more than 2 more concepts than neurons.
177 Huntington: Maybe, like language models and stuff do something like this. So maybe we can try to learn to invert like, this process. And
177 Huntington: yeah, convert, like the activation vectors, into some representations where the features are monosemantic.
177 Huntington: And this is the Se.
177 Huntington: So you you take in your activation into an encoder.
177 Huntington: You blow it up into some larger representation. So in this synthetic example, we know the optimal value. It is 5. Usually you don't know that which is a big problem.
177 Huntington: And then you you just optimize your encoder, decoder pair, you know, to faithfully represent the activations
177 Huntington: under the scarcity penalty.
177 Huntington: And the idea is like from observing a bunch of activations you can now get like these true features
177 Huntington: in this idealized setting.
177 Huntington: And yeah.
177 Huntington: maybe these features are the right abstraction in order to get monosemantic, interpretable features that also causally affect the forward pass.
177 Huntington: and like for language models, this has been very strong and useful, I guess. So the student Slava was asking whether he can do this for diffusion models as well, because they are much cooler
177 Huntington: and like just a brief reminder.
177 Huntington: diffusion is this thing where you add noise.
177 Huntington: and then you train a neural network to remove the noise.
177 Huntington: And recently we also have, like few step diffusion models that can do this in
177 Huntington: between, one to 4 steps.
177 Huntington: So how does the diffusion model work?
177 Huntington: You give in like some noise? And the text prompt.
177 Huntington: In Stx Turbo. You have a unit architecture like I made a very simple image here, and out comes like image corresponding to the caption.
177 Huntington: and I mean this would be the one step generation process that we focused on. So we are kind of cheating a little bit. So we removed like this multiple time steps layer of complexity that you have in actual diffusion. So maybe you can think of it as again.
177 Huntington: Okay. But our saes also work in the multi-step setting.
177 Huntington: as I will show in the demo.
177 Huntington: So we we train saes on the updates performed by these blocks.
177 Huntington: So here I have, like a more detailed diagram like this is actually how the unit works.
177 Huntington: And each of these blue blocks is actually a whole transformer.
177 Huntington: like a transformer with self-attention layers where the batches of noise.
177 Huntington: Yeah, the the representations of patches of noise are allowed to attend to each other and cross attention layers
177 Huntington: where they are allowed to attend to the text prompt.
177 Huntington: And you also have Mlps in there.
177 Huntington: And yeah, so we focus on these blocks that contain like this, a lot of transformer layers.
177 Huntington: Sorry. So each. Could you go back? So each of those down projections are you said those are each transformer layers.
177 Huntington: each of the each of these? Or is this just a traditional unit? I mean, the whole diagram is a unit, right?
177 Huntington: But like, if you zoom into one of these blocks like, say, let's say, down 1 0, that's a transformer consisting of multiple layers, attention containing multiple attention layers and also sometimes multiple cross attention layers. And okay, Gotcha, was that part of the this slide or next slide. This presentation is kind of not polished because we keep getting rejected. And
177 Huntington: so I don't have a good reason to polish.
177 Huntington: but I will come back to this slide in a bit. Okay? So where to train the ses. Right?
177 Huntington: We are in Academia we kind of don't have infinite compute, so maybe we can be smart about it.
177 Huntington: So the 1st thing that we tried was just go in and turn off a bunch of layers right.
177 Huntington: and see how this affects the generation.
177 Huntington: And you can see, like not all, not all blocks.
177 Huntington: I've created eager
177 Huntington: the ones in the middle kind of have like a big impact. So and because we are trying to fix like this this causality aspect, because the student already has essays.
177 Huntington: We we will zoom in on those
177 Huntington: so, and it uses concatenation right for the for the for the down blocks going to the, to the, to the up blocks that are the same size.
177 Huntington: this this type of unit, because some units do do different things. So is it. You concatenate you skip, you skip. You have like a residual connection.
177 Huntington: from the input, right? But is the residual connection. Is that concatenation? No, that's addition. That's oh, that's addition. Okay, yeah.
177 Huntington: okay. So we've we identified these blocks. We also ran some auto prompts. So here we just turn off the update performed by this block.
177 Huntington: So that's the forward pass. Where, like, yeah, you just don't update the residual stream with the update performed by this one transformer block.
177 Huntington: and yeah, it was really funny because we did this. And afterwards I was going to my colleague Peter, who does the data engineering? Or did the data engineering at stability, and now is at Leonardo. And he was like, Oh, yeah, I know this.
177 Huntington: you know, like, there's this Youtube video where this guy calls this the composition layer and this other layer he calls the style layer. So there was a guy training like lowering updates on these different layers. And he has like multiple multi hour Youtube videos, he's completely like a machine. And amazing. And yeah, I wish to know the things that he knows.
177 Huntington: But anyways, so that's this video.
177 Huntington: And okay, so quick question. So in your figure, you have blocked dot attention, something
177 Huntington: go previous previous slide with the images. Yeah, yeah, so these are like.
177 Huntington: I forgot, are you still training? Ess, or is this is just a 0 oblation or something like that?
177 Huntington: No, here, we just set these blocks to 0. Yeah. So the the naming in the figure is a simplified naming.
177 Huntington: So these are the actual block names in like in this figure with the images. These are the actual block names
177 Huntington: in the diagram figure. These are
177 Huntington: shortcuts. So this is what happens if the output of these blocks were not summed in, yeah, so okay.
177 Huntington: yeah, so you would basically bypass it by going along this. Oh, okay, okay, this line. Okay, okay.
Rohit Gandikota: No question.
177 Huntington: Yeah.
Rohit Gandikota: So this was mentioned a couple of times. You said, composition in these models, so can you explain what that composition means?
177 Huntington: It! It composes the elements of the scene.
177 Huntington: So this, this guy can literally create a cat
177 Huntington: or so. I will show this in the interventions.
177 Huntington: So this is just. It is like, not clean cut and very hand wavy calling this composition.
177 Huntington: So it's by no means an accurate characterization, but like, if you turn on in this layer like a cat and a dog feature.
177 Huntington: you will end up with a cat and a dog in your image.
177 Huntington: and if you do this at other layers, you will maybe locally, you know, turn on some hairiness feature.
177 Huntington: or or some something like that. But we will get to that, I think, later.
Rohit Gandikota: Awesome. Thank you.
177 Huntington: So this is where we train
177 Huntington: And yeah, just some more details that actually not are that are not really written on the slides
177 Huntington: this F thing here like. If you do like a single example, a single noise right
177 Huntington: then, in this. So we are kind of in the smallest spatial resolution of the unit here. So, if I remember correctly, these were like 16 by 16, by 1,280 tensors.
177 Huntington: And we we train like we we fit in the 1,280 vectors
177 Huntington: into our sae, so we don't do anything that would like make similar spatial locations. So sorry you better at the 16 by 16. And then you.
177 Huntington: yeah, we. I'm saying, we have weight sharing across the spatial locations.
177 Huntington: It doesn't vectorize. Okay? Okay? I see? Yeah, yeah. So we don't say, look you should.
177 Huntington: we don't bake in this prior assumption that maybe would be reasonable of like neighboring things should be similar.
177 Huntington: But this is the 1st work. So I think it's a fair thing to do this. This just makes everything super easy
177 Huntington: on the engineering side. So we we do a bunch of forward passes. We. We spam our
177 Huntington: hard drive on the cluster with like 50 TB of tensors.
177 Huntington: and now we can iterate over some se parameters to get some.
177 Huntington: Oh, config that actually does something.
177 Huntington: And this was the 1st intervention that worked.
177 Huntington: So I think this was
177 Huntington: Maybe it's iclr last year, or Icmr. I don't remember. Slava sent me this picture
177 Huntington: where he turned on the bar. Feature in the compositional layer.
177 Huntington: and it actually causally affected the generations. So this was really cool, and before that, like, when we had, like a larger expansion factor.
177 Huntington: we would need to turn on like 17 or something ball features, and then we would create a great blob
177 Huntington: like.
177 Huntington: And now, you know, now we have this, which is really cool, and we
177 Huntington: so, if if you like, triggered that ball feature and the face looks like a ball, how would it differ when I would have a direct prompt where I would say, like, add a ball feature along with the face, or I will compose to deficient inference where in one, I will have
177 Huntington: something with the ball, and another, I will have something with the phrase, How the generation? Yeah. So you're asking like, how close can I get to this result by using prompting right?
177 Huntington: I don't know. I think that would be an amazing experiment to run, and maybe we can add this
177 Huntington: to our paper or to our to do list. Yeah.
177 Huntington: that's a question that I keep having all of the time when people do these things.
177 Huntington: Yeah, but like, I think,
177 Huntington: we will have some features that
177 Huntington: to me. I would be really surprised if you could do that
177 Huntington: with prompting, but maybe you could do it with layer specific prompting.
177 Huntington: You know where you have different prompts for the different blocks.
177 Huntington: Yeah, let's let me just note it down.
177 Huntington: Another strength of this approach is also that you can continuously
177 Huntington: provide the strength of this intervention. Right? I wouldn't know how to like get prompts for each individual, but they have some hacks that you can. You can write like a word, and then colon a number.
177 Huntington: and then this is kind of
177 Huntington: you multiply the embedding, or something of this word.
177 Huntington: and you kind of can emulate this
177 Huntington: so that I mean another thing. Why, the image domain is really amazing. David knows the most about this.
177 Huntington: like people have done everything.
177 Huntington: People have done everything. You. You cannot imagine what people have done on Gans like the stuff that we are doing with language models is baby stuff.
177 Huntington: People have done everything like anyways.
177 Huntington: So you you impossible to come up with anything new.
177 Huntington: Honestly wait. Could you go to the previous like quick clarification.
177 Huntington: So here you train 3 different essays for 3 different layers. Right?
177 Huntington: 4, 4. 0, sorry. Yeah. 4. Yeah. Okay. But so spoiler.
177 Huntington: Only 3 of these layers, I would say, have I mean, they all specialize.
177 Huntington: but only 3 are really like interpretable if you hold them to this standard that features should react
177 Huntington: so they should be on on similar looking data, set examples.
177 Huntington: and then they also should causally influence the generation
177 Huntington: in the similar in a way that matches. Okay? So then, only 3 of these blocks are
177 Huntington: easy to interpret. But the middle block, like this lava, managed to pull out some features that are really cool and like, but they are so hard to interpret like they do something like when you look at them.
177 Huntington: Some of them. They do something really. That makes sense, that does something with the perspective and with surfaces in the scene. But
177 Huntington: they are kind of hard to find. And it's super hard also to labor.
177 Huntington: But like, so they do something.
177 Huntington: Just so, okay, we get 3 really interpretable types
177 Huntington: of se features. And like this one. Mysterious one.
177 Huntington: Yeah, which one is the mysterious one, the up as it's this one mid one. Yeah. Okay.
177 Huntington: the mid one is a mystery to me.
177 Huntington: Okay, sorry. One more question. So the ball feature that you actually steer that is here, that is here, and that's only there. It's it's just like one dimension. That's 1 vector, just one. Vector, yeah, yeah, we just like, we learn like, we have, like.
177 Huntington: maybe in this example, we had an expansion factor of 2.
177 Huntington: So we go from 1,180 to 2,000 something. And then we select like one. Vector so that was active during a forward pass with a dog playing with a ball
177 Huntington: like we. We look at what is actually in the bar.
177 Huntington: and then we grab this guy
177 Huntington: and and edit, yeah, so yeah, I also have some questions on the same slide. So you train essays on the outputs of those modules. So it's kind of a bit confusing where the Red Arrow is pointing.
177 Huntington: But yeah, so it would be here, you know.
177 Huntington: like the stuff that comes out of the down block. Okay? So not on the residual stream, like, no, not on the residual stream. Yeah. Okay. And another thing, something that we should maybe also try, but like everybody's doing something else. Now.
177 Huntington: kind of
177 Huntington: yeah. And another thing is that what are the shape of this lower resolution? Images? That is 16 by 16, by 1,280. This is still a two-dimensional image. Right? Yeah. I mean, like 2 with a ton of channels. Yeah, with a ton of what channels
177 Huntington: channels. Okay, okay? And you blow it up by how many? Just a factor of 2. Just a factor of 2. Yeah. And that gives you interpretable results. Well, like, just the smaller the smaller your expansion factor, the stronger your interventions. Yeah.
177 Huntington: okay, in our experience.
177 Huntington: So we were like, this is bullshit to 64 like, I mean, we don't really care about capturing all of the features stronger. The intervention that that kind of like makes me think you should be able to get the strongest intervention by directly messing with the channel.
177 Huntington: but directly messing with the channels. Not even training with, oh, yeah, yeah, I mean, no, because you can have 0 point 5, you can have 0 point 5 or anything like that.
177 Huntington: Yeah, did you? Did you ever try that? Oh, yeah, did 0 point 5. And then I was like.
177 Huntington: this is weird, like, we shouldn't put this into the paper people which just be confused. But the 0 point 5 ones are
177 Huntington: works even better. I mean, they are constantly strong, you know.
177 Huntington: Yeah.
177 Huntington: I mean, I maybe they are not as monosemantic because you you have again this combinatorial argument.
177 Huntington: Right? Well, I mean, but empirically, it's a question. I think the argument is one thing, but I think the question is empirically they they are great. Yeah.
177 Huntington: they're great. And you can achieve theory like one other thing that we did before training. The Se. Is like to just get a feeling of which blocks do what was also activation patching.
177 Huntington: and that also, like in the middle. We patched like
177 Huntington: a dog head, like the the activations that were in the Dog head.
177 Huntington: We patched it into a cat forward pass.
177 Huntington: and at the down block we can get a cat dog
177 Huntington: like a chimera. How do you say this chimera anymore? And I mean, obviously, as you go up the layers, the more this looks like literally going into paint and cropping a part of the image.
177 Huntington: and, like, you know, copy pasting it there.
177 Huntington: Oh, yeah. So this was really fun.
177 Huntington: How robust is this ball behavior to the prompt! It doesn't work
177 Huntington: the down to one thing, the composition, the thing that we call composition layer, the thing that actually creates elements in the scene, that one is super duper that one doesn't care about.
177 Huntington: what is the prompt.
177 Huntington: But, for example, the app 0 0 block. That one we found is like, really, it needs the relevant context
177 Huntington: to do something.
177 Huntington: But if the relevant context is there, it is also strong
177 Huntington: like. So that's we call it a detail layer.
177 Huntington: and also like same for the app 0 1.
177 Huntington: That one is a style we call it style layer
177 Huntington: that one also. Is better. I mean
177 Huntington: that one is that one will also generalize wildly across prompts. But like that one will really locally
177 Huntington: change, like the texture and stuff of whatever is the object there.
177 Huntington: So. But but we will. I will show you hopefully some of this in the demo.
Rohit Gandikota: Grace, I have a question. The what's the input shape for your sae
177 Huntington: That is a vector of length, 1,280
Rohit Gandikota: So that means you you are. You are assuming that. You're you're not assuming any special
177 Huntington: Yeah, so we we do this simplification. Yeah.
Rohit Gandikota: See. So do you think that if you had no wait?
Rohit Gandikota: yeah. So if if you're doing linear essays.
Rohit Gandikota: would that still capture like it seems like it's capturing some spatial information like you. You could transform a person into a ball
177 Huntington: Yeah. So I mean, like one thing that.
177 Huntington: So one thing that is really weird, and that I learned about this unit is in this architecture.
177 Huntington: They actually don't use positional encodings or anything.
177 Huntington: but by virtue of how the convolutional layers and so on work.
177 Huntington: The model can still learn positional encodings.
177 Huntington: so you can probe out literally. I did that, and I had it in the paper. But then my quarters removed it because they found it's not interesting.
177 Huntington: you can probe out like the spatial locations from this 1,280 backstore.
177 Huntington: You can literally probe out. I am at position
Rohit Gandikota: Hi, Jane!
177 Huntington: So all of these layers are, position a bear
Rohit Gandikota: Right. And I think that's like really interesting where you didn't use any spatial processors like convolutions.
Rohit Gandikota: Just by looking at the channels you were able to get special features. I think that's very
177 Huntington: Yeah. Oh, yeah, yeah. So I think we will see more of that. I hope
Rohit Gandikota: Yeah.
177 Huntington: When you do the intervention. Yeah? Or maybe you're going to talk about how you do the invention. I was just gonna ask, I think maybe in the demo it will be very obvious. Yeah, are you doing the intervention on all 16 patches, or are you doing? No. So we we build a Ui, we do everything. But like we.
177 Huntington: the best thing that we can do. Yeah, okay, we have a ui, you can paint a mask and then we just collect the indices that fall in inside of this great, the mask
177 Huntington: But so one thing that's worth noting, right?
177 Huntington: So you you like in this idealized
177 Huntington: way of thinking where you are like. Oh, the essay literally is supposed to discover the true features.
177 Huntington: You could just set the se coefficients right to do the intervention
177 Huntington: and replace the output with what comes out.
177 Huntington: We don't do that because our essays actually have a kind of large approximation error
177 Huntington: compared to also what you see in language models.
177 Huntington: So you just add the vectors. Yeah, we add the vectors.
177 Huntington: So we circumvent this issue of having actually a bad reconstruction.
177 Huntington: Just a important detail for working with ses. And like, I think, se interventions, is something that
177 Huntington: is not well understood also in language models like, especially in language models.
177 Huntington: I would say, okay, so let's go and look at the layers. Right? So
177 Huntington: I, we did different types of visualizations.
177 Huntington: So here, for just to start, we just generate an image for this prompt.
177 Huntington: a cinematic shot of a Professor sloth wearing a Tuxedo at a barbecue party or something, and
177 Huntington: The rows of this plot are like the 5
177 Huntington: sae feature indices corresponding to the sae features. You know that reduce the L 2 error the most.
177 Huntington: and we we plot like a heat map. Showing their coefficients.
177 Huntington: So red means this feature is very active somewhere.
177 Huntington: and blue means it is like a little bit active and transparent means, it is not active.
177 Huntington: And then then we have, like
177 Huntington: visualizations where we subtract the the vector corresponding to this feature index
177 Huntington: with a multiplier that you can see in the column name
177 Huntington: and that kind of influences the generation in like sort of like ablating
177 Huntington: like semantically. This looks like ablating the feature. And here you already notice some ugliness. I think
177 Huntington: of what we are doing, because, like, if the theory would be nice.
177 Huntington: ablating should be just setting this value to 0.
177 Huntington: But instead, to get like an ablation like effect.
177 Huntington: we need to subtract it with a factor greater than one right?
177 Huntington: And then but then you see that for example, the 1st row, the light chains disappear.
177 Huntington: the second row also the light chains disappear, and also some of these tent shapes the Federal
177 Huntington: you lose like your sloth, animated face.
177 Huntington: and the last 2 rows they do something with the Tuxedo, I guess, but they are less nice.
177 Huntington: Then you can do the same thing, but just increasing the strength.
177 Huntington: And 1st row, I guess the light chains pop out a bit more.
177 Huntington: Second row. You see, there is like tents on tents on top of the tents.
177 Huntington: and last 3 rows are more subtle, more subtle.
177 Huntington: Then what we also can do on this block is
177 Huntington: we can turn the feature on on the empty, prompt
177 Huntington: like. So here we do a forward pass with the empty promise, and we just
177 Huntington: to at the down layer we turn everything else off, and this guy on and
177 Huntington: So it also pulls out like the matching, the kind of matching concepts usually.
177 Huntington: And then we also have the data set examples that also match
177 Huntington: have a question. You chose the features that
177 Huntington: minimize the loss or affect the loss the most right. So because we are generating a prompt right?
177 Huntington: Yep.
177 Huntington: Are these also the features that happen to activate highest. Oh, yeah, that's the that's equivalent right in this.
177 Huntington: That's the direction.
177 Huntington: Pardon me,
177 Huntington: I mean, it's if if my dump projection is, yeah, yeah, like a internal. Yeah, yeah, yeah. 0 dimension whatever. Right? Right? Okay,
177 Huntington: I think I explained it wrong. So if the down projection was autogonar. It would be the same. Yeah, but it's not so maybe it is just the features with the highest activations. Yeah.
177 Huntington: the one with the highest sae coefficients nice. And I mean it will also be relevant to the loss absolutely. But yeah, not not in this way that I was talking about it. Okay,
177 Huntington: maybe we should put them side by side and and check another quick question. So generally, what does the diffusion model generates on an empty prompt.
177 Huntington: Yeah, that's a that's a great question. So Turbo
177 Huntington: is actually kind of collapsed, as Rohit shows in his recent amazing work. That would be so cool if we could combine it with this.
177 Huntington: So we have better images.
177 Huntington: The empty prompt is usually like it looks like a drawing of a landscape with a lot of brown. You have a little bit of grass.
177 Huntington: Sometimes you have a tree, sometimes you have, like a person like thing
177 Huntington: in the middle. So it's definitely polluted by the features that are so that these visualizations are.
177 Huntington: You see how the features that we turn on mix
177 Huntington: with these brown tones and this painting style
177 Huntington: like, for example, here, where you get like this, do this
177 Huntington: drawings of this? I don't know hamsters, or whatever. This is.
177 Huntington: Okay, so
177 Huntington: this is the block that we call composition block. And we call it that because you, you kind of can connect
177 Huntington: a lot of the features with elements of the prompt right.
177 Huntington: But by no means is this accurate characterization of the block.
177 Huntington: But it helps to think about it.
177 Huntington: Are we good? Are we good with this block
177 Huntington: on the data set examples is this, is this, just from you have a data set of many prompts. And then you just generate a lot of yeah. So actually, so I wish the data set examples could be even better. But so
177 Huntington: we were kind of pragmatic in keeping things cheap.
177 Huntington: So we we pre-computed a few hundreds of 1 million of prompts
177 Huntington: the latent for prompts. So that was the 50 TB of latent on our disk. But like to to get these data set examples. We actually only looked at the 1st 50,000
177 Huntington: or something in that ballpark.
177 Huntington: So maybe we don't cover the whole data cool nicely. And what what and what? What was your source of prompts that you used it? That was
177 Huntington: Coyo and lion, I think cool. So like training training data sets, yeah, yeah
177 Huntington: and that is my that this block, I had like a back and forth with this block.
177 Huntington: At the beginning I found it kind of boring
177 Huntington: at the end. It was my favorite block.
177 Huntington: so I ordered them here, not by their order in the forward verse, but their how interpretable they are! Order!
177 Huntington: So this is the style layer.
177 Huntington: and I mean there are so many things that are kind of cool to me. In this layer, for example, you often have, like this.
177 Huntington: 2727 feature. That kind of increases the illumination effect.
177 Huntington: When you turn like, crank it up.
177 Huntington: and when you subtract it, it has like it makes things looking darker.
177 Huntington: And so the the style layer is also still strong enough to do something on the empty, prompt right?
177 Huntington: So it will just make it will create this texture like images
177 Huntington: if you turn it on, on the empty, prompt.
177 Huntington: and also similarly, on the 1st row, you have something that
177 Huntington: seems to be shadow like, at least when it's on.
177 Huntington: what's the opposite interesting? Yeah, it's and they activate. They activate on different parts of the image, right?
177 Huntington: So this one is more on the left side here.
177 Huntington: and this one is more on the right side there.
177 Huntington: And then
177 Huntington: You have all sorts of other things like you also have, like this light bulb texture, for example.
177 Huntington: and you have like this row the last row. I think that should have a name.
177 Huntington: but I don't know what's the name for this. But
177 Huntington: oh, it's like red versus blue color temperature. Yeah, yeah, yeah.
177 Huntington: Color temperature. I think. Yeah, that was the name that I was looking for.
177 Huntington: And it's warm, coolest daylight indoor light.
177 Huntington: Okay?
177 Huntington: So also, like one comment.
177 Huntington: Recently, we had like this related work coming out
177 Huntington: that showed that Saes also learn interpretable features on randomly initialized models like without training.
177 Huntington: which is why I think the causality aspect that we have here is really important.
177 Huntington: So I highly highly doubt that your randomly initialized unit can create images matching the.
177 Huntington: you know, interpretation of the feature.
177 Huntington: So this is like.
177 Huntington: this is the to me, the key to making sure we don't do something trivial.
177 Huntington: you know, because, like random, like people for many years used like random Fourier features and stuff to build their classifiers.
177 Huntington: So there's like, yeah, there's just this risk of fooling yourself.
177 Huntington: Okay, slide block.
177 Huntington: And the last one, that kind of is interpretable is like the detail block
177 Huntington: detail block doesn't do anything on the empty prompt, which is why these columns disappeared. And we have more data set examples.
177 Huntington: but detailed blocks. Still, if you do something in the relevant context, like the context of the prompt.
177 Huntington: you can get strong effects like detail block can also add some party decoration, or it also remove it.
177 Huntington: Detail block can nuke your mouth away in this.
177 Huntington: A 4th row you just.
177 Huntington: We've nuked the whole, but I mean, these interventions are hilariously strong. So we are really butchering around here.
177 Huntington: do I understand correctly that you did this analysis on 4 blocks based on this Youtube, video? Or did you
177 Huntington: just people knew already that 2 of these blocks did you do? Did you do it? Did you select these 4 blocks, or did you do it on the entire unit? Yeah, yeah. So we are. Gpu, we were kind of Gpu poor.
177 Huntington: Well, we're not really Gpu poor, but like constraint, and we were like, we want to do as little training as possible, and have as great interventions as possible.
177 Huntington: So we did this preliminary experiments to narrow down the blocks.
177 Huntington: but like. But maybe someone should do it. And the preliminary experiments are the 0 ablation, 0 ablation and activation patching.
177 Huntington: Basically intuition building about the forward pass.
177 Huntington: for your interventions. So is do you just use the heat map to select the pixels that you're gonna intervene on?
177 Huntington: It seemed it seemed like the rest of the images. Yeah, yeah, that's a good
177 Huntington: sharp observation. I didn't mention this. Yeah, we use the heat map cool.
177 Huntington: To turn it on and off locally.
177 Huntington: except for like this thing where we turn things on on the empty prompt.
177 Huntington: they would just turn it on everywhere.
177 Huntington: Okay, also, the data set examples match kind of
177 Huntington: the interpretation or the the stuff that it is doing in the forward pass.
177 Huntington: And then we have. Like also this problem, child block.
177 Huntington: where the interventions basically don't seem to do much.
177 Huntington: But, like the data set, examples are very, very
177 Huntington: neat. The the data set examples look like you should be able to do something with this block.
177 Huntington: But yeah, we didn't manage to pull off like
177 Huntington: men. So this is the block where literally it's like Slava
177 Huntington: he spent like, I think, a ton of time.
177 Huntington: and he can pull out like a list
177 Huntington: of let's say, 10 to 20 really interpretable.
177 Huntington: like features that have a causal effect on their generation.
177 Huntington: Would it be that you're just not using the prompts that haven't back.
177 Huntington: Yeah, but like, so I mean these. These are the 5 most active features in the Tuxedo professor's love, prompt.
177 Huntington: And now we are just locally cranking up and down the strings.
177 Huntington: And yeah, it it warps the it. It has some subtler
177 Huntington: effects that are, but that are really difficult to describe, whereas, like, if I look at the data, set examples like in the 4th row, I would say, this is the left boundary of an object
177 Huntington: or in the 1st row, I would say, this is the right side of a head
177 Huntington: you know and on the second row, I would say, this is the left side of a head.
177 Huntington: and then we also have something, some features that I would characterize as like this is the leftmost object.
177 Huntington: This is the rightmost object, and so on. Like, by looking at the data set examples.
177 Huntington: But yeah.
177 Huntington: so you know. So one of the things that we're finding in these transformers right? Nikhil is like, you know, doing all the papers on. This
177 Huntington: is that a lot of the important features
177 Huntington: are not values. They're pointers. They're pointers, yes, pointers, and so is this a pointer layer that's laying down pointers for the geometry of the scene
177 Huntington: that you've got.
177 Huntington: and so, you know, it has no effect to like. Maybe the thing maybe the intervention to try
177 Huntington: is you could, instead of amplifying things, you could try moving the pointers around to see if you can rearrange the scene or something like that. Yeah, we should do different interventions. Right? Yeah.
177 Huntington: that's what you would do. Right? Nikhil, yeah, yeah.
177 Huntington: yeah, move them around switch left and right, it's actually not too hard. Switch the dog and the cat
177 Huntington: yeah, yeah, i like it, yeah, we should try it.
177 Huntington: Okay, so that's the layers.
177 Huntington: I'm sorry if this slide I don't even know how much you see here. So I see it on my screen, so I don't know how horrible this part was. So 1 1 suggestion for presentation, just just as a note is that in in the dissection work we found that when we showed heat maps it's really hard for people to read and understand in paper. So we just binarize the heat maps and show these little window things so you can see. And there's a few of us.
177 Huntington: We're using these little window visualizations and these kind of vision papers
177 Huntington: where you just show where the highest activation is just easier to read. You mean, you would just remove the non active part. So you you gray it out to like 70% gray, so that people can still see it if they want to. But then at 0% gray, there's this transparent window.
177 Huntington: You can even draw a board around the window, or something like this, and say, this is the area. Actually, we have that we do have that. Yeah, it's good. Okay, so
177 Huntington: this is some.
177 Huntington: This was some early.
177 Huntington: So now we are turning features on, on a different, on a like different context.
177 Huntington: and these are some of the early features that I found
177 Huntington: that are so. These are heavily cherry fixed just as a running
177 Huntington: that that like have a really cool and interpretable effect, like across different prompts. So we sample a bunch of different random prompts from our data set. And I turn on these few features that I found.
177 Huntington: So the 1st one.
177 Huntington: I guess there's some comic feature.
177 Huntington: second one. I tried to find the animate feature, but I in the past I had some better anime features than this one, but this was the one that I found when I made a presentation.
177 Huntington: and the last feature I found when I generated pictures of an angry monkey.
177 Huntington: So that's somehow a evil character picture.
177 Huntington: So it grows, you know, the horns so really cool stuff.
177 Huntington: Honestly, there, cool.
177 Huntington: Oh, never mind, I'm sorry like the left side was actually the data set examples
177 Huntington: that have the feature. And now the the right side is turning them on on unrelated prompts. I'm sorry.
177 Huntington: So the 1st row is always the picture of a colorful model that
177 Huntington: I use the most to play around.
177 Huntington: and then we we also we lost the dog that plays with the boy
177 Huntington: and the the classroom I found just funny with the angry feature down there.
177 Huntington: Yeah. Have you played with the order of the prompt like
177 Huntington: you change. No, I didn't. But I was. I was wondering if you mean you are referring to the text Encoder being weak.
177 Huntington: maybe.
177 Huntington: or like, if if it has any effect. Yeah. So actually, we are playing right now with the text encoders a bit in flux.
177 Huntington: and we found some cool stuff.
177 Huntington: but that's a good idea. Maybe the features are there, and they might have very good activations, but just due to the order of the prompt.
177 Huntington: maybe, one of the features is far from the end where the decoding takes place, and that's why you don't, although, like the feature, is there. But it's not getting activated because of the activation and the order and and the position of those adjectives. But like we are decomposing the update to the
177 Huntington: noise right?
177 Huntington: So whatever was in the prompt, if the
177 Huntington: units, if the cross attention blocks, wanted to attend to that, it's in our grids.
177 Huntington: Right?
177 Huntington: So we are. So that was, it's the difference of our work to the other works that do saes.
177 Huntington: is, we are literally doing it inside of the unit and on the visual features.
177 Huntington: So other people did it on text encoder intermediate results. They did it on
177 Huntington: clip embedding vectors that you can use to condition, some diffusion those are. And the results are really amazing. And where what motivated us to believe like
177 Huntington: that actually gave us the conviction that we also can get interventions that are like affecting the generation.
177 Huntington: Those are done by Giddis, who's a nice guy.
177 Huntington: He also was in meetings with us.
177 Huntington: Yeah.
177 Huntington: So I have some more of these, and then I would like to do an interactive thing.
177 Huntington: So do you did your so did you? Did you attempt to sort of quantify interpretability.
177 Huntington: I mean, yeah, of course. Yeah. And what kind of your method did you use to quantify it? Yeah. So like the quantitative, the quantitative analysis. Honestly, I found it very challenging. But like so we had like this down, we had this. We were lucky. We had like this down layer
177 Huntington: that actually modifies the images in such a strong way
177 Huntington: that, like even kind of dumb models like Gpt. 3.5 can label the features.
177 Huntington: So we auto annotated the down block.
177 Huntington: and we did the usual. We computed some of the usual scores of like.
177 Huntington: how specific is this feature?
177 Huntington: How sensitive is this feature, and how causal is this feature. You can find this in the paper.
177 Huntington: And we we tried to make the point like more quantitatively that these layers? Our specialists
177 Huntington: 2 different things.
177 Huntington: But this was really hard, and it's not super convincing. Do you have? Did you like have a baseline that you beat like a Pca decomposition. We did a terrible job on on that. We do have some baselines we do, but they have kind of they are kind of weaker than Pca, yeah, yeah, I mean, like,
177 Huntington: the baselines. I think that's really important.
177 Huntington: Yeah, if you're still going through peer review like this kind of thing. Yeah, yeah, it's just like a VR resource button. Next, sure
177 Huntington: currently baselines. Pca.
177 Huntington: I mean, we are. We have this, we obviously have this in our to-do list. Yeah, yeah, another thing I wanted to ask this is this is also related to baselines. But baselines for the causality effect. We actually looked started looking at the neurons. We just didn't go all the way. They were not great.
177 Huntington: Yeah, which is good. So this is kind of like baselines for the causality effect. So you have found that these specific neurons which turns on the anime neuron and the anime feature in the generation. Right? Yeah. Or the horse
177 Huntington: the horsiness. Yeah, here in the 1st row. So let's say you tried to learn some sort of like a sparse but linear combinations of the feature of your Se basis directions that you have found.
177 Huntington: And just to that, that reflect that basically turns on that anime feature
177 Huntington: like without selecting it. Okay, this is the animate reaction. I'm gonna turn it on.
177 Huntington: Yeah, how about you just learn it? Yeah. Yeah. So there is like this related work that does this for language models. Right? Yeah, there are lots of research. Yeah, yeah, right? So sure, yeah. But also, like just not doing essays, just not doing essays and learning them directly.
177 Huntington: like Rohit did with the sliders. Yeah, so it works even better. So then my question would be like, let's say, if something turns on anime features. Yeah. Is it actually the anime neuron? Or Oh, that's a great question. Yeah.
177 Huntington: So well, I mean I think like what we see here is that certain lay certain blocks
177 Huntington: lend themselves at least to more effectively learning certain
177 Huntington: like, if you want something to affect the style.
177 Huntington: You probably have an easier life to do that in this up 0 1 block.
177 Huntington: right? But like that is something that we should try. Yeah, I mean, I noticed with lowering updates, you're actually quite flexible. Which exact parts of the attention
177 Huntington: you I mean, some of these are even equivalent mathematically.
177 Huntington: Oh.
177 Huntington: which of these parts you update? But like, usually there, people update all the layers. So so the yeah, if you, I think that's an interesting direction to localize the Laura updates onto layers.
177 Huntington: And yeah, I mean, if the layer, if the layers are specialized, this will kind of upper bound or what you can do.
177 Huntington: Oh, so yeah, it's really a hard question, this, like whether you are whether you found like the real
177 Huntington: think.
177 Huntington: Yeah, I'm I don't have like super deep thoughts about that.
177 Huntington: Alright, it's okay. Thanks. Yeah.
177 Huntington: Yep.
177 Huntington: Was just wondering about your Gpt 3.5 interpretability pipeline. Yeah, you give it 2 images and then ask, oh, no, no! Like we do like we. We follow for closely the footsteps. I guess, of the
177 Huntington: entropic work where you you, you bin your images into different activation levels. And then you are like, look, here, are like, I don't know 5 or something, whatever we can afford.
177 Huntington: whatever prompt length we can afford. Here are 5 examples where it's not on. Here are examples where it is 10% on 25% and please.
177 Huntington: please, please
177 Huntington: explain it to me. Yeah. Yeah. And so the down yeah, the down block is the block where this kind of makes sense, but already the app 0 1 that is very interpretable, like the the models focus on the foreground object too much right? And and you don't want to introduce too much of like
177 Huntington: handholding, I think.
177 Huntington: Yeah, you you don't want the auto annotation pipeline to be prompted with like
177 Huntington: focus on the style, and then afterwards make the claim. This is a style layer. Right? So the
177 Huntington: yeah, we have that in the paper. Slava has the features annotation somewhere.
177 Huntington: but I think you don't even need them. You can just generate a prompt
177 Huntington: generate the image for a prompt see which features were active there right.
177 Huntington: and play around with them in another prompt.
177 Huntington: which is what our app allows you to do.
177 Huntington: I was wondering also, you have, like
177 Huntington: all these different layers. Yeah, how do they communicate? And oh, that's we didn't get into that yet. That would be a really exciting direction, I think, yeah. So we we cited like, Sam Mark's work, like.
177 Huntington: at 3 different locations in our paper, like, so saying, like, Oh, yeah, more advanced methods would be cooler. You know. That kind of try to connect the features and future work, you know, should do that. And
177 Huntington: yeah, maybe you would find even wilder stuff. There.
177 Huntington: So this is the same plots for the style layer
177 Huntington: just turned on everywhere, which is not how you should use the style layer, as I'm going to show you in a bit.
177 Huntington: and even more so. The funniest is like turning off the lion main on the lion
177 Huntington: like when you. Then here you you have like the main.
177 Huntington: and then it just has, like big cheeks and those ears it's hard to see.
177 Huntington: Oh, yeah, you can see like the lion's ears sticking out or something. Yeah, yeah. So we're gonna see something in a higher resolution. Now.
177 Huntington: So assuming that we still have some time.
177 Huntington: so I guess I turned on the few, the demo on our cluster. Now, never mind
177 Huntington: what's going on here.
177 Huntington: Oh, I also ran Demo close rights.
177 Huntington: That's a great way of doing your notebooks.
177 Huntington: Alright, guys.
177 Huntington: Well, depending on how slow this is.
177 Huntington: We can we also have like a hugging fist demo. It's just slower.
177 Huntington: So. And I mean, like, here is really the some of the amazing things that also happened
177 Huntington: like, let's say, like
177 Huntington: we want a higher quality image. So we should use 4 steps. And let's use my favorite prompt, which is
177 Huntington: a photo of a colorful model.
177 Huntington: Okay, let's start from that which.
177 Huntington: Okay, so this is the image.
177 Huntington: Let's find some interesting feature like,
177 Huntington: this is one that I just think makes an amazing effect.
177 Huntington: So now we generate the photo of a giraffe. Here we see the features that are active on the down block. This is a composition block.
177 Huntington: But, like, let's I want to pick out the giraff style like it's 3, 7, 1, 8.
177 Huntington: It's kind of a shitty ui, but it does the job, please.
177 Huntington: 10 is kind of strong, so let's use 6.
177 Huntington: So now I can locally turn on this tier of texture.
177 Huntington: Oh, I'm in the wrong layer, so this won't do anything interesting so this one
177 Huntington: and you get this amazing, you get this local, amazing effect.
177 Huntington: Oh, but I see you you you blacked out like the cheek and the forehead, but then on the left side of the cheek, for instance, it got it, anyway. Yeah. Yeah. So. But like the amazingness of transformers. Attention. The the neck doesn't get it right. Even change the hair. But you have bilateral symmetry. This is something that's like enforcing this. Yeah, more than but like, I think this is as specific as the edits get.
177 Huntington: So this is already the most
177 Huntington: like mutating, not super much.
177 Huntington: for example, like another cool thing that you can do is like variant.
177 Huntington: I don't know.
177 Huntington: That's beautiful example.
177 Huntington: I love it, and that's that's the style. There, that's when you call the style. There.
177 Huntington: Yeah, let's do a bit more.
177 Huntington: Just because I love this stuff.
177 Huntington: Let's do like a let's get some hair. Yeah, textures like 4, 1, 4, 7.
177 Huntington: And let's see what happens.
177 Huntington: You have to paint a different area.
177 Huntington: I mean, I can paint the hair. So it made so much, so cool hairiness.
177 Huntington: And we can do this. We can do this on on different layers, of course.
177 Huntington: Right? So let me just remove this.
177 Huntington: Let's so I found some fun stuff.
177 Huntington: let's go to the composition block, right?
177 Huntington: So
177 Huntington: I guess. So I start with the everything is nice kind of example. And then let's go to something where things are not as nice.
177 Huntington: So 1, 9, 9 seems to be something with a head.
177 Huntington: Okay?
177 Huntington: 1, 9 9 in the up layer we go to down.
177 Huntington: and the head should be, I guess, on top.
177 Huntington: So let's apply it ahead.
177 Huntington: Intervention.
177 Huntington: We can get ahead in the composition layer. So this is like, you know, this is how I would sell this to investors. We are scientists. Right? So let's take the beard feature, or whatever. Anyway, we don't have much. I also show you the ugly. Okay, so
177 Huntington: 4, 5 cool.
177 Huntington: Okay, let's do a beard, be it? I guess.
177 Huntington: So this feels like I typed the wrong number. Oh, no.
177 Huntington: no. Okay, tonight. Let's make it a bit stronger.
177 Huntington: No, this is like this is unheard of. This is really ugly. This is like the demo failing.
177 Huntington: Oh, yeah, we need a really high number
177 Huntington: this time. I don't know why
177 Huntington: she's doing something with her cheeks. Yeah, yeah. Well, but it is so. We were fooled. It's not a beard, it's a manliness or something. Right? Right? Yeah, it was not a beard. So you need to be
177 Huntington: careful.
177 Huntington: Visual interpretations
177 Huntington: right? But like, now now, let's let's do something really, really cool. And so now, like this is the stuff you only see when you do this research.
177 Huntington: And so now we take like the detail layer.
177 Huntington: and it also has something that does something with the beard. Right?
177 Huntington: 3, 7, 4, 2.
177 Huntington: And now we get like, really, the weirdness. So this is the layer that we found. This is context dependent, and but it also can be very strong.
177 Huntington: and it kind of makes sense, but it makes something that you would never get otherwise.
177 Huntington: Oh, what is this, which? Which? How is this one different from the previous?
177 Huntington: What is this? I think it was it turned it, or it turned to a man. This is, wait. So this is like, much more local. Yeah, different. Layer. Yeah, yeah, very nice. This one like gave her a beard. But she's still a woman facial structure last time. It didn't even like it. I mean, it creates some hairiness whatever, but like perfect. So this example is actually better than what I got last time.
177 Huntington: Let me just download this we'll put this one in the paper. Yeah, like, let's try a different strength.
177 Huntington: I think you know. Sometimes I write photo with F, sometimes with Ph.
177 Huntington: so now it's not strong enough, I guess. Interesting. So it's interesting. It's like a step change with with strength here, like at a particular strength. She just has a full beard suddenly. Yeah. So everything that I show you guys is heavily cherry picked. Oh, now I have what I had last time.
177 Huntington: So, yeah, this is it.
177 Huntington: So? Is this. This is still going through peer review. Is that right? Yeah. Did you submit it. They keep telling us some stupid stuff like, what are they? What are they complaining about? This is quite important work. I think you are doing such a general technique. But you only have one model.
177 Huntington: and I'm like dude. You know how much work this was like. This was like multiple months. Everybody full time staring at these features, and they're like.
177 Huntington: Oh, you should do another model, I mean, yeah, sure I can. But you know I also have a life
177 Huntington: and things. But I think it's a good scientific standard to ask for this multiple models. But
177 Huntington: yeah, yeah, but it's but it's also I think it's important finding. Yeah, it's really nice. Yeah, yeah. So if we had more manpower
177 Huntington: I mean, now we are looking at flux, and maybe at some point we decide to train essays on flux.
177 Huntington: Yeah, why not? I mean, like, because, Gpu blacks
177 Huntington: doesn't have like this nice units of analysis as you have it here
177 Huntington: because of the structure of the what. How is flux? Different? I mean, flex is, is a
177 Huntington: diffusion transformer. And so without the unit, it doesn't have like this. Oh, you have like this
177 Huntington: down to one block, right? Because you have. Like, I was complaining about dimensionality reduction. And it's like, oh, actually, yeah, it's nice dimensionality reduction, which is good. Yeah. And so flex, you're worried. That doesn't have this. Well, I mean, like.
177 Huntington: at least like we trained on Pixar fix art or something.
177 Huntington: We get interpretable features, just.
177 Huntington: you know, like in our paper, we write the story about the layer specialization. We? We cannot yet
177 Huntington: telling this
177 Huntington: story, I guess. Well, you don't have to. Right? Yeah, yeah. I guess if if the reviewers say, Hey, we just want to make. Be sure that it works for different models, then you can. Well, it's literally in the appendix. So
177 Huntington: oh.
177 Huntington: but I mean, the the reviews are like really low scores. Which is why I'm kind of butthurt about it.
177 Huntington: Yeah.
177 Huntington: I think it can be fixed. Yeah, that makes sense. Yeah. For the Academic Peer Review. Yeah, it's the opposite. But like in the paper, this doesn't come out. You know. I I didn't. I can do Random. I can do random things that I mean.
177 Huntington: you know.
177 Huntington: like, right? So somehow, you have to communicate the robustness. Yeah, yeah. And that's hard. Yeah, that's a hard, that's actually a hard task. Yeah, the
177 Huntington: I mean, maybe one thing is to highlight this like image editing application. Oh, yeah. Yeah. So also, I kind of, I take kind of a little bit of the blame on that one. So
177 Huntington: like I didn't penetrate this space.
177 Huntington: yet lose my 1st work in this area, and like I.
177 Huntington: I grew up, got trained by reading machine learning papers
177 Huntington: where you do like a quantitative analysis over huge data sets to make your points right. And that turned out to be really hard
177 Huntington: here.
177 Huntington: But like we can do some, you know, smaller stuff that it shows more the highlights.
177 Huntington: But I I just, you know, didn't. When we wrote this paper.
177 Huntington: I just didn't have the skill of writing the paper in this way.
177 Huntington: Yeah. But so I think the paper, actually and highly undersells.
177 Huntington: What we're doing.
177 Huntington: Yeah, come, hang out with computer vision people. A little bit more. Right? Yeah, I should learn how they write in a certain culture. Yeah, yeah, I need to learn this way of writing, and then I think they will also love it.
177 Huntington: Yeah.
177 Huntington: it was great. It was a great result.
177 Huntington: right? It's by far the best thing that I ever was part of.
177 Huntington: Yeah. So by, it's by, the pictures are so fun by by like you know, by far.
177 Huntington: That's so.
177 Huntington: This is like I'm when when I had the 1st results, I was literally going to David, like I was going to random people at the conference like we did this, you know, and everybody was like, Wow, you know, this is so wireless.
177 Huntington: So like.
177 Huntington: yeah. But I think that when you go through some of the technical details it makes it even more clear how interesting is because some of these things.
177 Huntington: You know, it's like well known that you can achieve some of these effects at the cross, attention to text, and so on. Which is not that surprising, because that's very close. You guys just did everything already the objective? Right? So you saw everything already. Yeah, you're way too cool. You're like
177 Huntington: your baby in this space, right? But actually. But actually, you're finding things that are very interesting, you know, at at the certain layers in these models.
177 Huntington: So so 1 1 question is.
177 Huntington: oh, so so a couple of questions, so just detailed questions. So you've got you know your your 2 to one expansion goes to about 2,500 features
177 Huntington: like, what do you think like? Just from your intuition? About what percentage of those do you think are human and understandable?
177 Huntington: Depends on the block.
177 Huntington: Like a a good person like the down block, I would say, like a good percentage, maybe.
177 Huntington: And now I'm improvising because I don't have the data
177 Huntington: down block, I would say, like, 50% plus something like that, I mean,
177 Huntington: detailed block is very hard to say. But I would maybe say similar, but slightly less
177 Huntington: mid blog, I already complained. It's very hard to make sense of. So
177 Huntington: maybe one to 5% or something. And the style block is
177 Huntington: also again very interpretable, like, yeah, 50, 60. I would maybe down in style. Maybe. Let's crank them up to 50, 60. Yeah, let's say, detail 40, something like that. Yeah, yeah.
177 Huntington: yeah.
177 Huntington: it's not perfect. And like, if you increase the expansion factor, you, you will have like smaller percentages of that's a huge number. Yeah, that's like, I mean, 3,000 sliders. We give you a huge design space to play around with and to customize your forward passes.
177 Huntington: and something like where maybe I could need your and Rohit's help, because I am not super solid on the math
177 Huntington: on all of the nitty, gritty details.
177 Huntington: but like there is work right. Turbo edit
177 Huntington: that uses turbo to do image editing.
177 Huntington: And just, you know, doing a clever trick. So you don't even need training
177 Huntington: like to to construct like this score function. Yeah. And that would be just much cooler combining it right with this. Yes, and so I could edit my own face.
177 Huntington: Yes, in the demo
177 Huntington: that would be next level, cool, and I want to have it. But like. I always look at the paper and get scared.
177 Huntington: and I'm like I need somebody. Well, some of these things will just transfer without you having to do anything, it'll be the same direction on a on a fine tune model.
177 Huntington: And oh, yeah, I mean that. But I I want to. I want the image editing. Right? I think that's cool. Yeah.
177 Huntington: I mean, it's just cooler duck.
177 Huntington: So have you tried transferring some of these features to the base model, maybe, or 50 should work. But we didn't.
177 Huntington: Yeah. And can you compose these features? Yeah.
177 Huntington: So in I had, like a a
177 Huntington: like, I had my own ui, that was like a website that I hacked together with like bribing Gpt. But I was the only person who ever was able to run this because it was such a mess.
177 Huntington: but where I also could compose features, and we have, like one of this in our appendix.
177 Huntington: You can compose them. It's kind of it's really cool. And
177 Huntington: so the meeting recording thing blocks me from typing stuff.
177 Huntington: So let me try to pull out the link somehow.
177 Huntington: Earth.
177 Huntington: Yeah, clicking there.
177 Huntington: It's buried, I guess. Okay, so this is the other model.
177 Huntington: Exciting.
177 Huntington: Oh, that it didn't even did this even make it into the appendix.
177 Huntington: Oh, yeah, here, look here, I combine a few of the style layer interventions, and you, you know
177 Huntington: I combine like some tiger texture with some giraffes.
177 Huntington: with some 4, 9, 5, 5. I don't know what that was. 4, 9, 5, 5. Let's check it out.
177 Huntington: 4, 9, 5, 5. it was very exciting. Chris.
177 Huntington: Thanks. Thanks for the very clear presentation. Thanks for the many questions. Yeah. So this was a layout part for 9. 5.
177 Huntington: yeah. So what separates concepts from stripe. For example, you have a picture of a woman, and you try to apply all of the concepts of a tiger. At 1 point a woman would turn into a tiger. Yeah, you know what I mean? Yeah, you will have a chimera. Can you teach me this word. Can somebody just say the proper word? I don't know. How do you say it? You can get a chimera.
177 Huntington: No, but you know the deciphering thing, no matter how you say it, it's going to still sound European. So you remember that deciphering thing? Yeah.
177 Huntington: de shivering, deep learning. Hey, David, I'm going to do this post again, shivering deep learning. You do what there's a new technical term that we haven't learned yet. Yeah, you have to interact with some. It won't be so cold anymore. Shivering.
177 Huntington: Yeah. Yeah.
177 Huntington: Yeah. So the difference here is this will never turn into a giraffe on this layer.
177 Huntington: no matter how much. You crank up the stretch
177 Huntington: also, like, how much can you crank up the string? 100 like? Surprisingly much, I mean, I guess they still have normalization layers in there. Right? So at some point, you just overpower everything else. But okay, if you add 5,000, then you are kind of like, even if it goes through normalization, it is kind of like suppressing everything else. It's like turning everything else to 0. Then what happens?
177 Huntington: Well, then, it's very similar to this visualization where we turn everything else to 0. And only the one thing that makes sense. Yeah, yeah, it's similar to that, I guess. Be fun for this to be like a like a brush tool in Photoshop, or something where you like, you know, just select. I mean, that would be super amazing. Which is why we have everything open source. All of our S. Are on hugging face. We have a nice repo, a slim repo.
177 Huntington: It's like one file that you need to apply them.
177 Huntington: We also give you the training which we copied from somebody else who open, sourced everything.
177 Huntington: Sdxl is open source and amazing. I guess you have some problems there, if you are like adobe, because this was trained on lion images. And so you maybe don't want some of this in your product.
177 Huntington: Yeah. So maybe you have to retrain it.
177 Huntington: I guess one issue is that you want it to be more controllable like, you want to be able to like, you have, right? Oh, yeah. Yeah. And unbounded. Yeah, so, but so, yeah, yeah, I guess if it's supposed to be a product.
177 Huntington: You have like these weird requirements. And you are not like me, who is like, I get actually excited, you know, if by the weirdness.
177 Huntington: yeah, like, oh, no. Artists will love the weirdness as well. It's great. Yeah, yeah, maybe. Yes. As long as it's predictable and controllable. Yeah, I mean, regular users. You just lock them into the style layer.
177 Huntington: You're like, okay, you can play with this.
177 Huntington: and everybody else can. You know.
177 Huntington: you can maybe turn on like the developer mode or artist mode.
177 Huntington: And yeah, I don't know but like
177 Huntington: would be cool to make more of this and make it better, and also not obvious. Honestly.
177 Huntington: not really obvious. What are the directions that really make it better?
177 Huntington: Yeah, I guess better models getting. I mean, that's the obvious one.
177 Huntington: Yeah, yeah, that's a really obvious one. Yeah, like, another thing that another idea that I had like.
177 Huntington: like, you also have like text to video models now.
177 Huntington: And like, there was like this website, picker picker AI, and they at some point they launched like
177 Huntington: they had launched like 5 or something. You know interventions that you can perform on your videos
177 Huntington: like the foreground object gets picked up and squeezed like, yeah. I imagine if you just train as a on this video models, you get like more than 5. What are the best? So what is the best open weight video models? Are there? Are there any I would have to do my homework. But there are some good.
177 Huntington: Yeah, I think there's some that are pretty good, and about how how large are like, how many, how many 1 billion parameters? Or oh, I'm not sure. But I think for the mochi one, at least you would need like.
177 Huntington: that's okay.
177 Huntington: But yeah, so for that, you have, there are some roadblocks right? If it if we were like, Oh, se, is the thing that we really really want
177 Huntington: right there. You need the multiple times. That thing I I assume, and
177 Huntington: and probably you need some other things as well. So it's definitely hard
177 Huntington: like it. It gets like a much harder task. But it but it might not. I mean, we see that over multiple time steps. It works the same as well. But it's just 4, right? It's 4, I know, I know. So it's not. It's not. It's not that much. Yeah. Yeah. So we should maybe try this feature on the like 20 step sdx area. But like, I think we saw, you saw some promising results for that right in the
177 Huntington: prompting attention layers with like different prompts at different heads.
177 Huntington: And you don't really narrow down. The time steps much.
177 Huntington: Yeah. And it does cool stuff. Right? So any feature you find in single time step easily transfer to multi time steps. That's your intuition. Yeah.
177 Huntington: well, that's cool. Yeah, that's really cool. So I won't be surprised if you can just transfer these to 50 step, and then we should definitely do this. It would be so easy. It's the same architecture. It's everything same.
177 Huntington: No, it's like this is should literally work out of the box
177 Huntington: by writing Stx instead of Stxl, Turbo in into our we should we should do it on video.
177 Huntington: it would be very, very striking. So so the thing that's really interesting is you didn't use any spatial features.
177 Huntington: you, with only the Channel dimension, you were able to get these features out single timestamp, which probably would transfer to multi timestamp.
177 Huntington: So for videos, I wonder if you just use a single frame doing also the most naive way. I mean, that's also what that was would be our questions. Yeah. So we we were very curious about video. And you have like this game simulator models that are so cool like.
177 Huntington: that's just so cool. They have. Those are maybe even better and smaller.
177 Huntington: Yeah.
177 Huntington: Oh, there's also in the video. Sorry. So the other opportunity you have right is so, since since since it's like.
177 Huntington: you know, I, I feel like this is a potentially better test bed
177 Huntington: for sort of the interpretability, the interpretable disentanglement question, and some of the Lm test beds.
177 Huntington: You know, to do as is is, as you know, is this,
177 Huntington: you know, L. 0 regularized l, 1 regularized saes. Is it really the best way of getting disentanglement out of this? Or, you know, do we? Is this a good test bed for other people to try other disentanglement?
177 Huntington: Yeah, I think tomorrow is a great yeah, that's a good setting. I like the idea. Yeah, if you're able to make a this keeps bothering me honestly.
177 Huntington: if you're able to make an automated benchmark of interpretability.
177 Huntington: then it's like a natural setting to test alternatives.
177 Huntington: Right? How would you score like alternative architecture? Sa bench full. Yeah, sa bench, right? So there's another throw it into sa bench. It should be part of it. Right? Yeah. So.
177 Huntington: yeah. So there are like this classic data sets right where you have a scene with a bunch of different colored objects.
177 Huntington: Maybe you we could start with something narrow like that.
177 Huntington: And just see, you know how. Maybe something like that. Yeah. Is there a way we could push for the rest of the features to also be interpreted? I was kind of inspired by the recent talk that we had here last week, where
177 Huntington: it went up and down the hierarchy. Better speaker
177 Huntington: did like the essay stitching. Yes, so
177 Huntington: I mean, you can start from a small se, which is kind, kind of kind of has this causal features.
177 Huntington: and then you could see, like
177 Huntington: you could try to add. So so you have. We have like this result, like the larger ses
177 Huntington: they are kind of harder to work with, but they also find like different features sometimes. So you
177 Huntington: we yeah. Maybe we could stitch together some
177 Huntington: something that contains, like all of the ones that are great from different expansion factors.
177 Huntington: Maybe it's possible to do it with some objective other than l 1, or like in addition.
177 Huntington: recently. And yeah, I think so
177 Huntington: I think. There are so many things that you could try, and so many better classical methods than
177 Huntington: this linear sae thing, so that to me, the Se. You know, it's just a dream.
177 Huntington: It's like the dream of the simple world
177 Huntington: and the dream of the neural network doing something nice for us
177 Huntington: like and using somehow the the only you know, using the sparsity thing that we actually can implement. And
177 Huntington: so I'm curious, what's more, deeper autoencoders do with activations.
177 Huntington: with yeah, with the intermediary sites.
177 Huntington: And we had some ideas how to, you know, actually,
177 Huntington: combine the good things of saes, which is the combinatorial argument kind of makes sense with the more powerful
177 Huntington: auto encoders, because you definitely don't want to just do like auto encoder. That reduces the dimensionality.
177 Huntington: because, like, there's still a combinatorial argument already like the model, was too small. Now the model divided by 2 for sure is also too small. Right?
177 Huntington: Unless you narrow down your distribution right? Which was the setting where the Autoencoders actually were conceived.
177 Huntington: And using these ideas, I think you can get it to work.
177 Huntington: I don't want to like throw out the whole idea, but this is the
177 Huntington: this is the key intuition. Why, you could get like something like a variational autoencoder.
177 Huntington: also to do something cool in here and something nonlinear cool.
177 Huntington: By narrowing down the distribution of prompts.
177 Huntington: And then you could do this multiple times.
177 Huntington: and then you would also have
177 Huntington: you would have interpretable nonlinear features after that.
177 Huntington: That you kind of farmed in this I don't know, kind of
177 Huntington: not fully unsupervised way, but because you had to group the prompts.
177 Huntington: Yeah, yeah, right? But maybe you can can combine it with other ideas.
177 Huntington: To architecture. Question, you can.
177 Huntington: Yeah. Yeah. But like, I mean, we we saw in Nikhil's work we saw in Rocket's work.
177 Huntington: Pca is actually already extremely powerful.
177 Huntington: If your distribution is narrow, right, if your distribution is a single prompt with different noises.
177 Huntington: Pca was literally all you need, or if your distribution is
177 Huntington: this one theory of mine template?
177 Huntington: Pca is all you need.
177 Huntington: Right? So that's a really, that's something really interesting. I think. That I
177 Huntington: in hindsight. It makes sense. But in advance it's maybe evolve something. You wouldn't expect
177 Huntington: any sort of local linearity.
177 Huntington: Yeah, yeah.
177 Huntington: Well, thank you very much, Chris.
177 Huntington: So