j.fiotto-kaufman: Okay, great hold on for a second. I'll go ahead and introduce yourself.
j.fiotto-kaufman: Great. Okay?
j.fiotto-kaufman: So 10 through 10 is
j.fiotto-kaufman: currently a postdoc at Harvard and is an expert in vision
j.fiotto-kaufman: both computer vision and in biological vision, which I don't know anything about. But he's an expert. So
j.fiotto-kaufman: I think that you know, his plan is to talk about a couple of things today and one in each of those categories. I'm familiar with the computer vision work, which is very interesting. And I'm also looking forward to learning a little bit about thanks. I'm very excited to share my work. So they're kind of all my Phd works. I have some ongoing. I just finished my Phd last summer.
j.fiotto-kaufman: So now I'm a postdoc.
j.fiotto-kaufman: Yeah, yeah, let's let's see how how it goes. Right? So else
j.fiotto-kaufman: split my talk into 2 sections. So the 1st one is mostly about computational neuroscience. How? So? You know, there was a trend people have about comparison with brain 0 context. So I also did some work on that line. And I guess coincidentally I stumbled into making perturb work. So that's my second section, about some future interactions in humanities.
j.fiotto-kaufman: But the post work are inspired by surrounding facts.
j.fiotto-kaufman: So what what is surround. So what are the surround effects? So your perception at some
j.fiotto-kaufman: certain region is influenced by its surrounding. So, for example, this is this is called evolution. So if you keep your eye in the center of the left stimulus.
j.fiotto-kaufman: so so you might feel it's tilted a little bit. So right.
j.fiotto-kaufman: But actually, it's up straight, like the one on the right.
j.fiotto-kaufman: So this is an example of surround effects.
j.fiotto-kaufman: And
j.fiotto-kaufman: so people know that this, those surround effects can be very well accounted by a computation called surround normalization.
j.fiotto-kaufman: So this is a very robust computation. People have found in many brain regions, so people call it a canonical neural computation.
j.fiotto-kaufman: So it's formula is very similar to some types of normalization. We saw in artificial neural networks.
j.fiotto-kaufman: For example, Lear norm. So the normalized response equals to unnormalized response
j.fiotto-kaufman: of one neuron divided by the summation of a group of unormalized response.
j.fiotto-kaufman: So in the case of Sigma, yes.
j.fiotto-kaufman: the sigma that you're adding in the denominator, what is that? Oh, it's just like in layer norm. We have those smalls. And to avoid gradient explode. Yeah. In neuroscience models, people also have those. So it's regular regulated, not too hard. Make it as smooth
j.fiotto-kaufman: option.
j.fiotto-kaufman: So on the left figure. This is a very classic surrounding effect that can be accompanied by surroundation. So what it shows you is on the X-axis is a diameter of this gradient stimuli. So as we increase the diameter of this gradient stimuli.
j.fiotto-kaufman: and we recall from one neuron, we can see the neural response 1st increase. And then there's a non monotonic trend. It decreased a little bit.
j.fiotto-kaufman: I guess an intuitive way to understand. This is at first, st when the diameter is small. It only activates the center neuron
j.fiotto-kaufman: and then you get larger and larger. The surrounding neurons also get activated. Then it has a, they have a suppressive effect on the center neuron.
j.fiotto-kaufman: Okay, of, there are some theoretical works that show this type of surround novelization have
j.fiotto-kaufman: a lot of benefits. I guess my favorite one is, it has a very nice connection to efficient coding.
j.fiotto-kaufman: So what efficient coding tells us is this is old idea given by horizontal, so sensory sensory systems forms a neural code for efficiently representing sensory information.
j.fiotto-kaufman: I guess one way to interpret efficiency is, they are less statistically dependent.
j.fiotto-kaufman: I guess so I will show you briefly how to derive surround normalization from efficient coding. It's not directly related to my papers, but I think it's very nice piece of work.
j.fiotto-kaufman: So 1st of all, what are the dependencies?
j.fiotto-kaufman: Let me fix my zoom a little bit. It's blocking my really.
j.fiotto-kaufman: Okay.
j.fiotto-kaufman: So what are the dependencies we can find in those linear filters. So once we characterize those dependencies, then we can find a way to reduce them.
j.fiotto-kaufman: So here's as an example on the left is image. We pass it through 2 linear filters with a couple filters very typical in both neuroscience and also,
j.fiotto-kaufman: say, yes, and we can get to activation by.
j.fiotto-kaufman: So we can see 2 roughly 2 types of dependencies.
j.fiotto-kaufman: The 1st type is
j.fiotto-kaufman: between, there's some correlation between 2 activation plan. But, for example, if we see some strong activation here, we also see some strong activation here.
j.fiotto-kaufman: And another type of dependency is spatial dependencies.
j.fiotto-kaufman: So now we're looking at only one exhibition map.
j.fiotto-kaufman: and we can find that some largest value and the smallest value. They are clad, plastered in some specific regions.
j.fiotto-kaufman: Yeah, is it glitchy on your side? Yes, I don't know why.
j.fiotto-kaufman: That's strange.
j.fiotto-kaufman: We want to.
j.fiotto-kaufman: you wanna maybe stop sharing and start sharing again, or something like that. Who knows?
j.fiotto-kaufman: I've never seen this switching before.
j.fiotto-kaufman: News fellow.
j.fiotto-kaufman: It's a new phenotype. Yeah.
j.fiotto-kaufman: Oh.
j.fiotto-kaufman: okay.
j.fiotto-kaufman: What do you mean?
j.fiotto-kaufman: It's getting to work. That's what
j.fiotto-kaufman: have a bigger problem, though. Oh, no. What's wrong now.
j.fiotto-kaufman: like Blue Share, it doesn't share your. It doesn't share your screen at all.
j.fiotto-kaufman: Okay? Okay? Great.
j.fiotto-kaufman: Oh, it's the wrong one.
j.fiotto-kaufman: Thanks to.
j.fiotto-kaufman: Okay, that's let's see, okay, so dependencies, spatial dependencies of those activation.
j.fiotto-kaufman: Right? So if we go the go into distribution of 2 neurons that are spatially apart a little bit.
j.fiotto-kaufman: So we can't see that. Oh, there is a
j.fiotto-kaufman: like a higher second order dependency that isn't just simply correlation, that is, if one neuron's activation is large, then the variance of the other neuron.
j.fiotto-kaufman: It's locked.
j.fiotto-kaufman: Thank you. Are you full screen. It is full screen. Strange, huh?
j.fiotto-kaufman: Wonder why it's popping out of full screen like that? What if you just okay? What if you make it not full screen?
j.fiotto-kaufman: I don't know. I'm just trying different things. And
j.fiotto-kaufman: Well, I don't know what to do.
j.fiotto-kaufman: What are the different options in the view.
j.fiotto-kaufman: Do you think it's Powerpoint messing up, or is it?
j.fiotto-kaufman: That's isn't that strange means
j.fiotto-kaufman: let me try something. Maybe I can quickly upload this to Google drive I use. I don't know it'll look ugly.
j.fiotto-kaufman: Yeah, you want to just use it without presenter mode. Yeah, sure. Okay, let me 1st try disconnect. No? Oh, yeah, because I'm not using. That's right. Leave that off. That's it. Yeah, that's right. That's right. It keeps on trying to
j.fiotto-kaufman: connect you to some other device. You got it. You figured it out. Okay,
j.fiotto-kaufman: now, he's face
j.fiotto-kaufman: all right, going back to dependencies. Right? So we see those dependencies between 2 neurons. So now, the problem is, how can we reduce those dependencies?
j.fiotto-kaufman: And there was a very nice line of work showing how to do this. So the goal is to go from left distribution to the right distribution. Then the neuro two's distribution doesn't depend on neuro one and how to do this and people find maybe go from left to right is hard, but to go from right to left is easy. So you just multiply those Gaussian variables with a shared
j.fiotto-kaufman: local scaling, new, random, variable, and then you draw the activation of joint distribution of XYX. 2. It has this bow tie shape.
j.fiotto-kaufman: So this is our generative model. Then we can reverse this generative model via Phn inference, and then we can get a fall that is very similar to the
j.fiotto-kaufman: surround normalization. I just showed you. So the takeaway is surround effects or normalization in the case.
j.fiotto-kaufman: reduction of coding redundancy.
j.fiotto-kaufman: Okay, so th, those are the inspiration of all my reports.
j.fiotto-kaufman: Can I ask one? Sure.
j.fiotto-kaufman: So this is a histogram right? The
j.fiotto-kaufman: is that a tooling histogram of neuron one and neuron 2. Yes, it's the joint distribution heat map, and on the top is heat map. Yes.
j.fiotto-kaufman: I'm I'm still.
j.fiotto-kaufman: We're having a hard time understanding what this plot is. Oh, okay,
j.fiotto-kaufman: if we look at one column of this heat map that means conditional on neural watch activation at this level.
j.fiotto-kaufman: what is the histogram of activation of your answer.
j.fiotto-kaufman: So if neuro one.
j.fiotto-kaufman: if we conditional neuron, one has no value, maybe going back to this one. That means so now, 2 neurons are like close to each other. That might be the definition of 2 neurons. So if neuro one is here, neuro, 2 maybe is here. If neural, one has very low
j.fiotto-kaufman: near 0 activation level. Then you're on 2, also have
j.fiotto-kaufman: a very small variance. And if neuron Y is here, you can see the neuron 2 has a larger variance. If the activation level of your Y is larger, then the variance of neuron 2 is also large. So then, what's this bow tie distribution.
j.fiotto-kaufman: Does that make sense kind of
j.fiotto-kaufman: what's the difference between left plot and right plot? What's the right 1? 0, right one is the desired. God, yeah. So yeah. Explain the right one. Then, oh, okay. So the right one is
j.fiotto-kaufman: regardless of the neural wise value. Neural two's distribution is always the same.
j.fiotto-kaufman: So you're redefining neuron 2.
j.fiotto-kaufman: Yeah. So let let's bring up your formula.
j.fiotto-kaufman: Well, see, I'll show you how to formula. Yeah.
j.fiotto-kaufman: really funny. I I guess it's after normalization.
j.fiotto-kaufman: because here I also define redundancy as dependency. So on the left, there's some dependency. So there's some redundancy. So is the way to think of it is on the right.
j.fiotto-kaufman: You're plotting. g. 1 versus g. 2. Yes, yes, exactly
j.fiotto-kaufman: so. Your your g. 1 is your desired
j.fiotto-kaufman: latent variables that are underlying the physical neurons. Yes.
j.fiotto-kaufman: right? Yeah. And and you're saying, g, 1 and G, 2 are more uncorrelated.
j.fiotto-kaufman: Yup, and and so they don't have this relationship between the 2 things. Yeah, exactly right. You're looking for statistical independence. Yes. Yes.
j.fiotto-kaufman: So by neuron, you mean, like the filters, 2 different filters. Or is that spatially with
j.fiotto-kaufman: both? Actually, they can be different filters at same location. It can be like your center field of view.
j.fiotto-kaufman: this football filter and another same type of filter. But next to the 1st one, yeah, both sides.
j.fiotto-kaufman: But
j.fiotto-kaufman: yeah. And I, as I said, and there was a trend like 5 years ago or 6 years ago, people comparing Cns with 0 pertence and find a lot of surprising alignment between the 2 systems appearing different system.
j.fiotto-kaufman: so actually, can you? So so let's finish the story. So with this translation to the space of g 1 and G. 2, you have.
j.fiotto-kaufman: Maybe there's more desirable property of better statistical independence. So is the overall story that.
j.fiotto-kaufman: you know, is the hypothesis that an efficient neural network will encode things using
j.fiotto-kaufman: encodings like g 1 and g. 2, rather than x 1 x. 2.
j.fiotto-kaufman: Is that is that the hypothesis? Or yes, yes, yes. So the I guess. One way.
j.fiotto-kaufman: I guess there are 2 wheels. One is, computation is divisive computation. Another view is.
j.fiotto-kaufman: if we have this computation, what effects we can see, for example, the Teltic illusion or the surround suppression effect. Let's
j.fiotto-kaufman: like the phenomenon of.
j.fiotto-kaufman: So you're saying that if the computation is expressing everything in terms of g, 1 g. 2, then you'll probably see its around effects and these normalization effects and so on. That's right. So, and that's evidence that biology is probably using g, 1, g, 2, not x, 1 x 2. Yes, yeah. So in biology, actually, it's well known that
j.fiotto-kaufman: those neurons probably is using t 1 that's in neuroscience. And now I'm looking for this effects in also incidence. That's my 1st got it. Now, I understand.
j.fiotto-kaufman: Okay.
j.fiotto-kaufman: yeah. So people find couple filters in both sands and
j.fiotto-kaufman: and we won. And they also created the metric called representation of these similarity matrix and find that in early years of Cnn is similar to early 0 areas. And in later years, you see, is similar to like later 0 areas. Oh, yeah. So there was a hype comparing the 2 systems.
j.fiotto-kaufman: So here, I want to compare the 2.
j.fiotto-kaufman: But with the scope of surrounding events, so does same show surrounding that.
j.fiotto-kaufman: But in your side you don't know. It's very robust.
j.fiotto-kaufman: How about Santa's?
j.fiotto-kaufman: So? The 1st experiment I did some people. Some sometimes people refer this method as incenical electrophysology, which I replicate as soon as what people does in animals. But I recall from artificial neurons. So in this experiment. So this is a animal data.
j.fiotto-kaufman: we have 3 types of stimuli.
j.fiotto-kaufman: we can put the gradient stimulate only in the center and rotate. This gradient stimulate to have different orientation. And this is a black line here. Each neuron has a optimal orientation, and I align those optimal orientation at 0 orientation.
j.fiotto-kaufman: and the second type of stimuli is, we only put stimuli outside of what people call classical, receptive field in the surrounding region
j.fiotto-kaufman: and rotate this surround or operating.
j.fiotto-kaufman: and because it's all set of classical, defined, classically defined, receptive field by itself, it doesn't elicit any interesting response. So it's a flat line purple line
j.fiotto-kaufman: here.
j.fiotto-kaufman: Yeah, just checking. You can see my mouth. Yeah. But itself doesn't elicit any interesting behavior. But if we combine the 2, there are some very interesting effects. So now in the science, we fix the center stimulator at its optimal orientation and rotates the surround. And what people find that the surround can elicit a strong inhibition when it's aligned with the center grading orientation.
j.fiotto-kaufman: And if it's different from the center orientation, it has a sometimes it can even have a facilitation effect.
j.fiotto-kaufman: at least like it. Okay, I did the same experiment in same neurons and found very similar trend.
j.fiotto-kaufman: So the takeaway is the most suppressive surround training matches, optimal orientation in the center.
j.fiotto-kaufman: Those are in the artificial nuance.
j.fiotto-kaufman: And then I was inspired by
j.fiotto-kaufman: this famous Chris Ola Post on future realization. So I asked myself, can I go beyond greeting student and show this surround suppression.
j.fiotto-kaufman: but with small, complex textures? Stuff.
j.fiotto-kaufman: So I modify this approach a little bit into a two-step visualization
j.fiotto-kaufman: so very much like the grading experiment. First, st I look for the
j.fiotto-kaufman: I constrain my search space of optimizing the input image to only optimize the center reading and find the optimal center texture.
j.fiotto-kaufman: And then I can branch in 2 different directions which I asked, what is the most suppressive surround and what is the most facilitative surround, and I find that
j.fiotto-kaufman: the most suppressive surround looks very similar to the center simulate.
j.fiotto-kaufman: But the most of the facilitatives around
j.fiotto-kaufman: the ura look very different from the center. So this is kind of a generalization of the gradient stimuli showed. So now it's not. The orientation is my parameter, but the texture stuff is my prime. Sure. So what do you mean by suppressive surround. Suppressive surround is one of the surround.
j.fiotto-kaufman: when the surround simile is shown with the center, it inhibits the activation
j.fiotto-kaufman: when the center showed by itself.
j.fiotto-kaufman: So like here.
j.fiotto-kaufman: When this is a science, stimulate here, so when the surround is combined with the center.
j.fiotto-kaufman: and when they are aligned, the orientation is aligned. It surprised this black activation from here to here.
j.fiotto-kaufman: So is it is your experiment set up by analogy when it's the same feature in the surround.
j.fiotto-kaufman: So you know, W, you could have all sorts of different signals in this around. But you're you're asking
j.fiotto-kaufman: when it's the same feature in this round. Is that right? You're you're measuring the effective
j.fiotto-kaufman: the surround is optimized, it's visualized. It's not. I have copied the center texture to the surround, is, I
j.fiotto-kaufman: sets the optimization target as
j.fiotto-kaufman: the negative activation of this neuron. And I look for the surround. Oh, you're looking for a suppressive surround. Yeah? And then sometimes you find one, and then sometimes
j.fiotto-kaufman: there's no suppressive surround. Is that right? That's yeah. But I I we had a measure, and we show that they are statistically significant that we have a measure of how similar the surround is to the center. Is it okay? Can I ask another question. So are these surrounded specific specific classes, for example, like a dog class, it will have one that will surround.
j.fiotto-kaufman: And for a cat class it will have another type of surround or like, it's texture based
j.fiotto-kaufman: is, I did this experiment per neuron. So I guess I sent
j.fiotto-kaufman: time. Since we have privileged basis. So we can treat each neuron as a feature or something. So this experiment is done per neuron. And I screened all the neurons and the statistical.
j.fiotto-kaufman: So it's not class based
j.fiotto-kaufman: class based. If it's last layer is class based right? But if it's middle layer, it's hard to say it's more for, like a feature, middle layer feature.
j.fiotto-kaufman: To draw your draw the analogy from the toy setup. You had
j.fiotto-kaufman: the triple features you have between. You're on one neural so ideally you should. You do not want to see this in the neural, like in an optimal neural?
j.fiotto-kaufman: No, this is desired. This is a. This effects indicates there are some surround normalization inside network. And
j.fiotto-kaufman: those surround surround. Normalization is good.
j.fiotto-kaufman: it's good. But yeah, it's off the statistical issue.
j.fiotto-kaufman: It makes neurons more independent. So it's reducing the redundancy of neural code.
j.fiotto-kaufman: So it's a desired computation. And this effect is a indication.
j.fiotto-kaufman: Oh, there are substances similar to this computation in this neural network.
j.fiotto-kaufman: Thank you.
j.fiotto-kaufman: Cool.
j.fiotto-kaufman: So
j.fiotto-kaufman: I'm noticing that the what did you call the red ones? The facilitative ones are usually like high contrast colors with the with the center. Is that pretty typical across most of the
j.fiotto-kaufman: right? So the here, I guess
j.fiotto-kaufman: those texture features are you can look for like
j.fiotto-kaufman: spatial frequency, the color or the shape of right. The facilitative ones have contrastive
j.fiotto-kaufman: frequencies, colors and shapes.
j.fiotto-kaufman: What is you're showing? Some images on the right hand side. What's showing on the right hand side? Those are the optimal patches for this neuron. Oh, wait! Hold on for a second. So every row is one neuron. Oh, yeah, yeah. Every 3 or 4 panels is a neuron
j.fiotto-kaufman: like this is a 0. 0, I thought that the
j.fiotto-kaufman: oh, I thought that these were many different. I thought every square was in around. So. But you're saying that the blue column is suppressive.
j.fiotto-kaufman: I'm trying to understand. Are you finding that neurons, some neurons have suppressive behavior and some neurons don't, or that that's why, no.
j.fiotto-kaufman: no. So yeah, maybe this way, I'll explain. So the most. Oh, it's it's you're measuring the surround
j.fiotto-kaufman: for a single neuron. Yeah. So this. So these 3 panels are for one year. So I pick a neuron. I optimize only the center image
j.fiotto-kaufman: got it. So target is activation of senior. And then I
j.fiotto-kaufman: set my optimization parameter or variable to the surrounding area. Look for, surround, fix the center, I find the center. I fix the center. I look for surround. And the optimization target now is either the same as before activation or the negative of the activation. I understand that. Okay, so each neuron
j.fiotto-kaufman: optimal center, most suppressive surround and the most wrong, can I ask a question. So is there a chance that a particular neuron that is contributing to the suppress around can also switch mode
j.fiotto-kaufman: and become fastly. What do you call this fastly ratings around?
j.fiotto-kaufman: Yeah, that's a very interesting question. I guess
j.fiotto-kaufman: what I can answer is most neurons.
j.fiotto-kaufman: If the center texture is its optimal as optimal.
j.fiotto-kaufman: Still, really, then, the most suppressive surround is similar to the center. I guess in some special cases, if we switch the center to something else. Then we ask, what is the most suppressive surround? It could be different, I guess.
j.fiotto-kaufman: Okay, so here are more examples. So this is a eye neuron.
j.fiotto-kaufman: I guess the most suppressive surround for this eye neuron our multiple eyes and same for those
j.fiotto-kaufman: more complex color combinations or the shapes like this triangle lines.
j.fiotto-kaufman: Okay. Second part of simulation. So this is very similar to the 1st one. But now people
j.fiotto-kaufman: shifts the center orientation, not to be the optimal orientation of this neuron. But, for example, here 45 degree of the of its optimal orientation
j.fiotto-kaufman: and rotate the surround gradient. And people find that very interestingly, the most suppressive surrounding orientation matches the center orientation. So it's a dynamic, adaptive way. So the most suppressive surround is not always this orientation, it changes as you change the center.
j.fiotto-kaufman: So I did similar in cyclical experiment.
j.fiotto-kaufman: And finally some later layers of Adg. It has this trend.
j.fiotto-kaufman: Then the question is, how can I realize this effect?
j.fiotto-kaufman: Then I need to change. So in in silico.
j.fiotto-kaufman: you're are you picking a specific neuron to look at, or how did you? Oh, it's average across all the all the neurons interesting. Great. So you don't. You're not looking for a specific edge detector. No, no, you're just looking at the dominant behavior. Yes, yes, this is the average care. But I do have a criteria to select neuron. So to do this experiment. I 1st need to
j.fiotto-kaufman: gets the classical receptive field size of this neuron by measuring some tuning curves. Some neurons have too small, too large, this classic receptive field. So I exclude those neurons. As long as this neuron has a decent classical, receptive field, I include this neuron. This is average
j.fiotto-kaufman: cool.
j.fiotto-kaufman: So in a sense like it's kind of depends on our definition of classical receptive fields, like, if it's a fit for network, then if it's truly fit for, and the manual gradients passing, and then there's nothing information passing outside that fuse. Then there's no surrounding, but if we define it more restrained
j.fiotto-kaufman: to the center. Then you have a surrounding fat because you define it. Yeah, yeah, this is a very good question. So instead, we have a hard boundary for each neuron. We know that this is a hard, receptive field. But here my classical, receptive field is defining like a similar to people doing neurophysiology. They have their definition of.
j.fiotto-kaufman: yeah, it's related to the size tuning curve as you increase the size, as I show you. If the response 1st increase the decrease and the classical recept field people define is around the peak diameter of that curve.
j.fiotto-kaufman: Right? So I want to generalize this experiment to texture. So I need to change the center.
j.fiotto-kaufman: Not to be this neurons optimal,
j.fiotto-kaufman: center stimulate. So I have 2 ways to change it. So first, st I promote the color of
j.fiotto-kaufman: the center stimuli. So, for example, for this neuron, this one, this is one neuron.
j.fiotto-kaufman: And now you should look at each column. Now.
j.fiotto-kaufman: So the 1st column is is the original experiment. So I optimize for the center and look for the most suppressive surround.
j.fiotto-kaufman: And what now I do is I I permute the color channels to have a different like set of center. Stimulate now and use those permuted
j.fiotto-kaufman: centers to look for the most suppressive surround. And I find that the surround color can
j.fiotto-kaufman: adapt to the change in the sense.
j.fiotto-kaufman: And there's statistics in the paper. But this is a statistically significant effects.
j.fiotto-kaufman: Okay? And the second one I alter. The center is, I swap the center texture completely with some other neurons optimal center. And I look for the most suppressive surround.
j.fiotto-kaufman: Not for all the neurons, for some neurons. It has ability to even adapt the texture in the surround to the center
j.fiotto-kaufman: up.
j.fiotto-kaufman: That's true.
j.fiotto-kaufman: interesting. So so you're suppressing this neuron.
j.fiotto-kaufman: even in this situation, even though the
j.fiotto-kaufman: stimulus you're providing to the neuron doesn't correspond to its optimal stimulus. But
j.fiotto-kaufman: the suppressive one. It's almost, it's it's a it's an interesting contextual.
j.fiotto-kaufman: The situation, indeed. Yes, yeah.
j.fiotto-kaufman: It was this seen in biology as well of.
j.fiotto-kaufman: So I draw the analogy from this greeting
j.fiotto-kaufman: experiments. So I don't think people have done much like texture complex stimulating. So yeah, so I I also try to sell this paper as predictions to brain behavior. So people look for those things. Cool.
j.fiotto-kaufman: Okay, great. Yeah. That's my 1st section.
j.fiotto-kaufman: So yeah. And then the next thing to do is vit so when it came out I need to learn it and see if I can find any
j.fiotto-kaufman: this affecting that. But I approach the item
j.fiotto-kaufman: way. I I didn't want to do the exact same experiment in it so, but also forced me to stumble into the field.
j.fiotto-kaufman: yes, anyway. So the 1st thing I tried, or the 1st question I asked is,
j.fiotto-kaufman: are there more attention assigned to similar or dissimilar tokens?
j.fiotto-kaufman: I guess there are several motivations, so I guess one motivation is
j.fiotto-kaufman: there was a debate. I don't know. It's popular or not, but some people claim that attention. People call attention in transformers. But those attention are very different from human attention, and there is people showed transformer attention is
j.fiotto-kaufman: it's doing grouping rather than highlighting the city instances. So there are more attention assigned to similar types of tokens.
j.fiotto-kaufman: I looked those paper, but I I feel their design isn't very optimal. So I try to answer this question.
j.fiotto-kaufman: Use some better designs also my by my own. So I looked attention. Map
j.fiotto-kaufman: first.st So, okay, 1st of all, this data set, this data set is very interesting. It's called out.
j.fiotto-kaufman: all the one out data set. So
j.fiotto-kaufman: it has 2 categories of objects in each image. So one category is a singleton. For example, here is a white duck
j.fiotto-kaufman: and another category of a group of objects curious of Rhonda's.
j.fiotto-kaufman: So I I pick a token
j.fiotto-kaufman: in one category and draw the attention map of
j.fiotto-kaufman: this token, and see if its attention map lies more with itself or with other types of tokens.
j.fiotto-kaufman: So I I saw some effects that in most varieties, especially those classical ones, not the mask reconstruction ones. In early years
j.fiotto-kaufman: there are more attention on the same objects, but in later years
j.fiotto-kaufman: actually, attention is more distributed to different objects.
j.fiotto-kaufman: Alright.
j.fiotto-kaufman: So this is empirical finding. So the next thing is, is the y-axis, just average attention over all the heads.
j.fiotto-kaufman: Yes, it's average across, have
j.fiotto-kaufman: and yes, across height and different images for sure. And also cross types of objects. Yeah, yeah.
j.fiotto-kaufman: same object means the same one or like same type of things. Oh, same type. Okay?
j.fiotto-kaufman: Yeah, because the mask is so how I calculate is the alignment between the mask and attention map.
j.fiotto-kaufman: So is is there a way of looking at this result and seeing.
j.fiotto-kaufman: can you distinguish between this the odd one out phenomenon here and just locality
j.fiotto-kaufman: that at early layers attention is just attending to patches that are closer
j.fiotto-kaufman: and later layers. Attention is, oh, things that are just further
j.fiotto-kaufman: interesting.
j.fiotto-kaufman: So you are saying, this effect could be simply explained by the distance feature, so
j.fiotto-kaufman: I could answer, I didn't try that, but I doubt it.
j.fiotto-kaufman: We already did.
j.fiotto-kaufman: You guys? No, I don't know.
j.fiotto-kaufman: Correct.
j.fiotto-kaufman: Right? So this is empirical. So the next thing I tried to do is, can I show this effect in a more principled way by looking at the ways of the computations of vit orientation.
j.fiotto-kaufman: Right? So this is a
j.fiotto-kaufman: attention formula we are familiar with.
j.fiotto-kaufman: I guess I
j.fiotto-kaufman: I don't know. I should say I do. Have I discovered independently as those astrophic work. I follow the Qk matrix. They always go together. So astrophic. Call this Qk circuit.
j.fiotto-kaufman: yeah. And the token interaction is only determined by this one matrix, which is the product of the 2. That's a very simple by linear form. So I probably can answer those attention similar dissimilar question, only looking at this matrix. So to analyze this matrix. So I did a singular value. Decomposition
j.fiotto-kaufman: so decompose this
j.fiotto-kaufman: Y matrix into a set of left right, singular vectors and singular values. So this singular value decomposition method has a very nice natural
j.fiotto-kaufman: interpretation. So that is so. This is very important.
j.fiotto-kaufman: So that is, if one token's embedding is aligned with one left. Singular, vector
j.fiotto-kaufman: then this token only attends to tokens that has projections onto its corresponding red, singular vector.
j.fiotto-kaufman: because this, this is because of the of singular matrices. So so you could think of those are independent, interactive modes. So
j.fiotto-kaufman: we have of, you know, typical vat. We have 64 modes or channels of those interactive
j.fiotto-kaufman: feature pairs.
j.fiotto-kaufman: Oh.
j.fiotto-kaufman: so another way to see that is, it can be sort of as a left single vector query looking for its red singular vector key.
j.fiotto-kaufman: Okay, so now, I can answer the same question, do tokens attend to similar tokens or dissimilar tokens by layers. Of course, that similarity between the left singular vector and red.
j.fiotto-kaufman: if they are similar, left, singular vector and right singular, vector if they are the same. Similar, then also means tokens
j.fiotto-kaufman: tends to attend to similar tokens. So so this is a
j.fiotto-kaufman: average cosine singularity measure between left. Singular vector and right singular vector there is even more clean.
j.fiotto-kaufman: clear trend that in early years tokens tends to attend
j.fiotto-kaufman: similar tokens, and in later years tokens attend to dissimilar tokens.
j.fiotto-kaufman: So the conclusion here is attention. Pray for similar focus. In early days
j.fiotto-kaufman: I couldn't see my next slide, so I don't know what to expect. But anyway, oh, okay, sorry. So so this is weighted by the singular values. Yes, exactly.
j.fiotto-kaufman: Alright.
j.fiotto-kaufman: Okay. So from here.
j.fiotto-kaufman: my exploration branched out to 2 directions. So this part isn't shown showed in in my paper. Also, I don't have a plan to to write it. I feel like it's a little bit different
j.fiotto-kaufman: story than the main story I tell you in the pier. But anyway, if you think this is interesting, we can. We can try to push it if you want. I'm happy to hand this over. So what's the story? So I try to realize those singular vectors.
j.fiotto-kaufman: Now, I've realized I'm not picking one token. I'm optimizing the activation of
j.fiotto-kaufman: activation. I mean, embedding projection onto those singular vectors for the whole image.
j.fiotto-kaufman: So on the left are some early
j.fiotto-kaufman: early features. On the right are some late layer features. I guess. One very interesting thing I saw is in some early layer.
j.fiotto-kaufman: There are some old patches. So it's not homogeneous. So if this
j.fiotto-kaufman: vector represents some texture was, then why
j.fiotto-kaufman: it isn't this texture everywhere, but only
j.fiotto-kaufman: most places, but there are some holes. So this is very robust effects I I saw almost in.
j.fiotto-kaufman: I don't know all the the vectors, but 80% of those vectors have those holes. See.
j.fiotto-kaufman: this is very interesting. So I I looked deeper. Why, there are those holes.
j.fiotto-kaufman: Okay, so this is what I did.
j.fiotto-kaufman: So I pick a vector I optimize it's input and say, it's activation map.
j.fiotto-kaufman: Okay, those holes have very low values. Right? If my optimization target is the sum of the activation of the whole map. Then why, those are those pixels. Patches are not optimized.
j.fiotto-kaufman: Then I tried to hatch those holes with the texture I copied from the other place to fix those holes.
j.fiotto-kaufman: Oh, so interesting is
j.fiotto-kaufman: the the population activation is surprised if I fix those holes. So those holes actually are necessary to induce a global high activation value.
j.fiotto-kaufman: Okay?
j.fiotto-kaufman: Okay, then, I can start from here.
j.fiotto-kaufman: Continue optimize.
j.fiotto-kaufman: Yeah, continue, continue optimize this image and see those holes.
j.fiotto-kaufman: Emerge again.
j.fiotto-kaufman: See those hosts, and to facilitate other tokens activation value.
j.fiotto-kaufman: The the
j.fiotto-kaufman: My conclusion is all patches are facilitated.
j.fiotto-kaufman: Where do you start the optimizations of the input, like, is it 0?
j.fiotto-kaufman: No, it's a random noise caution, caution, noise.
j.fiotto-kaufman: And when you say, optimize further.
j.fiotto-kaufman: what are you optimizing there? Is it maximum maximum, the sum of
j.fiotto-kaufman: token embedding onto this singular. Vector
j.fiotto-kaufman: so you're you're you're optimizing to try to make all the tokens in the image Yup
j.fiotto-kaufman: as similar as possible to your target vendor as soon as yes, yes, got it.
j.fiotto-kaufman: I see another setup is like you, just hook like the central token, and then optimize that once. And I think the prediction is like you will see this kind of in the surround or something somewhere else. Yeah, I don't remember my next slide. But you remember me, this is the next one. Yeah, okay, so this is so if I only optimize for the center token this, all the patches are everywhere in the surround.
j.fiotto-kaufman: Wait, can you? Can you say that again? Oh, okay? So the above 3 panels, the optimization target is the sum of all the
j.fiotto-kaufman: tokens projection or whatever and the bottom wise. I. So the optimization target is only the center tokens projection on this singular? Vector I don't care about other places, but
j.fiotto-kaufman: they are all my observation variables.
j.fiotto-kaufman: So what did the heat map show again the yellow heat map like, what is that that value you're optimizing for? Like, yeah, yeah. So those are the the value here. The color is
j.fiotto-kaufman: tokens embedding projection of dot products. The the singular singular, vector
j.fiotto-kaufman: right?
j.fiotto-kaufman: Okay?
j.fiotto-kaufman: And is this texture effects generalized to objects.
j.fiotto-kaufman: So I found a very interesting, a vector, that is higher attitudes
j.fiotto-kaufman: on the tokens a bird is standing on like a is there a log or a a wall
j.fiotto-kaufman: whatever.
j.fiotto-kaufman: So I think this is connected to this idea because those birds.
j.fiotto-kaufman: they don't activate this feature directly.
j.fiotto-kaufman: but they are a necessary condition
j.fiotto-kaufman: for this food or branch whatever to be activated. You mean, if there's no bird in the image.
j.fiotto-kaufman: then the log is not activated. That's why I didn't do that. This is not published. It's not a serious thing, but some preliminary.
j.fiotto-kaufman: If we want to explore more, yeah, I'm happy to hand it over.
j.fiotto-kaufman: And if I optimize this oat
j.fiotto-kaufman: feature, there might be a board here. I don't know.
j.fiotto-kaufman: Okay, do you find one that's like the opposite that
j.fiotto-kaufman: fires on birds when there's like something they're standing on.
j.fiotto-kaufman: It seems like you'd want the opposite of this if you're trying to detect birds right like you don't wanna.
j.fiotto-kaufman: Or maybe this is like a we want to suppress this later or something.
j.fiotto-kaufman: Yeah, I I see a lot of burst features, but I don't really notice or pay attention to. If there's a
j.fiotto-kaufman: it just seems like this is something you'd want to suppress at some point of like we don't care about the fence that the bird is standing on, you know.
j.fiotto-kaufman: So maybe this is like we're finding the things that we don't care about to to get rid of it later, or something
j.fiotto-kaufman: that's very. That's very cool. Yeah.
j.fiotto-kaufman: yeah, yeah, that's a very interesting hypothe. By the way, this feature is far found in in the
j.fiotto-kaufman: right before the the self audition there. So it's not the Mlp. Like the side. And you have those like delete some information from the the data bus that's but could be related. Yeah.
j.fiotto-kaufman: encounter these features coincidentally, or do some sae stuff
j.fiotto-kaufman: I was looking at all the vectors features, and this one is quite striking, because
j.fiotto-kaufman: there's very obvious bird here. But it's a feature, isn't the bird. So this is very, I don't know at least very significant to me, just catch my.
j.fiotto-kaufman: what? What model is this?
j.fiotto-kaufman: I believe this is vit original vit
j.fiotto-kaufman: is, is image net trained? Yeah, yeah.
j.fiotto-kaufman: Okay.
j.fiotto-kaufman: Yeah, that's the side story. So
j.fiotto-kaufman: the story I'm going to tell you is the story in the paper. That is
j.fiotto-kaufman: those singular vector pairs are very semantically meaningful.
j.fiotto-kaufman: So this is one of the figure in the paper. So the 1st row is a vector in
j.fiotto-kaufman: oh, a mode, a singular mode in vit, and the the bottom row is dyno. So there are 2 models.
j.fiotto-kaufman: So in the paper I showed, I guess, 4 models and ditch all of them. I can find some very meaningful singular vector pairs.
j.fiotto-kaufman: So this is very interesting because they hear those singular modes. They segment the image, but in a different
j.fiotto-kaufman: wait or highlighting different parts. So this query, so red, means query.
j.fiotto-kaufman: direction, and cyan is a K direction.
j.fiotto-kaufman: So on top of all, it looks like this. Query. Feature is the top of a cake.
j.fiotto-kaufman: and the the key feature is the bottom of a key. And it's environment across different
j.fiotto-kaufman: types of cake. Yeah, chocolate cake. Yeah, cheesecake or cupcake?
j.fiotto-kaufman: Yeah. And the bottom line is even more fascinating to me is like the query is whatever
j.fiotto-kaufman: object held by a person, and the key is
j.fiotto-kaufman: the person itself, or the hand of, I guess, mostly the arm and the skin part of this person.
j.fiotto-kaufman: So I had a hypothesize. Again, I didn't test deeply. It's just a hypothesize, I said in the paper. Why, there is a mode like this. So there are some information flows from this person to the object. So my hypothesis is, there is an attribute. Something like being held is added to this object.
j.fiotto-kaufman: It could be a circuit. Cool.
j.fiotto-kaufman: Okay? So the next one is, yeah, more. Those modes. So from early
j.fiotto-kaufman: layers to late layers so early layers.
j.fiotto-kaufman: we can expect, you know, like simple color, features or texture features. Also this, as a story I just told you, they tend to
j.fiotto-kaufman: a lot, because tokens in earlier is, attends to similar tokens. But, for example, here the white actually means those red and cyan overlaps. So that that's why it's gray or white.
j.fiotto-kaufman: Yeah. And you, as you go deeper is has more interesting interactions.
j.fiotto-kaufman: Yeah, okay, so this is also a sad story. I didn't of
j.fiotto-kaufman: investigate very deeply. Also not in the paper. And
j.fiotto-kaufman: I find that, for example, this car image
j.fiotto-kaufman: activate a mode, and this mode
j.fiotto-kaufman: looks like it's universal to both car and animals.
j.fiotto-kaufman: You're in different animals. So the red part
j.fiotto-kaufman: is the top of this animal or car.
j.fiotto-kaufman: and the cyan, or the overlap part is the bottom of this car all in.
j.fiotto-kaufman: and the bottom row is even more interesting, because if
j.fiotto-kaufman: even segmenting the legs of animal or the fur of a fish and the the wheels of a pug.
j.fiotto-kaufman: So hmm.
j.fiotto-kaufman: yeah, I think this is similar to what? What then? Those Dino papers they showed originally their paper. They show. Okay, this eagle wing and body feature also, transferred to those airplanes and bodies.
j.fiotto-kaufman: But here, I guess what I added, is those parts and do interact.
j.fiotto-kaufman: Stop
j.fiotto-kaufman: another sad story I find it's interesting again. I didn't deck too much is oh.
j.fiotto-kaufman: what is a negative direction of a feature? So because
j.fiotto-kaufman: singular value decomposition the sign of those singular vectors are arbitrary. So when I was looking for those features, I looked both directions.
j.fiotto-kaufman: and one thing I found that
j.fiotto-kaufman: is, transformer to use both directions. Okay, perfect.
j.fiotto-kaufman: Okay, do use both directions. And sometimes those 2 directions are correlated. So for example, I showed 3 examples here. It looks like. So for example, this sign K feature
j.fiotto-kaufman: on the positive direction positive is arbitrary. So I define positive direction. Is the nose of this dog.
j.fiotto-kaufman: then the negative feature of this nose feature is the body of a dog.
j.fiotto-kaufman: So, and the background of this query a feature?
j.fiotto-kaufman: I mean the query feature. Here is the background
j.fiotto-kaufman: and the opposite of this background. Feature is the the head of this doc, and similarly, others.
j.fiotto-kaufman: Second example.
j.fiotto-kaufman: this positive and negative of one of the values, and it do you show the corresponding key here, or the corresponding
j.fiotto-kaufman: paired feature? Right? So this is key query.
j.fiotto-kaufman: yeah, Paris, right is, where's which? So I'm just trying to understand your thing. So is is one of the
j.fiotto-kaufman: you're showing the positive and negative side of like the query, vector
j.fiotto-kaufman: yeah, are you showing the key back anywhere? Yes, the sign is the key. Oh, I see. So the sign. Yeah, you should compare this red with this red and this sign with this sign I see.
j.fiotto-kaufman: Now, there's really 4 combinations of positive and negative. Are you showing
j.fiotto-kaufman: I but you're just showing 2. 0, no, no, no, sorry yeah. So the arbitraryness is
j.fiotto-kaufman: in pairs. So if the one is flipped, then the other one also should be flake.
j.fiotto-kaufman: It's not like 4 combinations. Because, yeah.
j.fiotto-kaufman: yeah, that's the singular value. Decomposition.
j.fiotto-kaufman: we have one singular value. Prepare. Yeah, yeah, you can think of you. You switch the style of singular value. Then you have.
j.fiotto-kaufman: okay, okay, I see. So they're both flipped.
j.fiotto-kaufman: I see. So at the top. It's
j.fiotto-kaufman: the background versus the nose of the dog, and at the bottom it's knows versus bot
j.fiotto-kaufman: nose versus body. Yeah. So if we trace the information flow is the top. Wise information flow from nose to
j.fiotto-kaufman: background, and here is from body to nose, so from body to nose, from nose to background. So it's like a chain.
j.fiotto-kaufman: Change it back. So interesting.
j.fiotto-kaufman: So these 2 directions are both valid in a sense, like they can both be used by the network. Yes.
j.fiotto-kaufman: yeah, I I know to me it's kind of intuitive it makes sense, because if we have
j.fiotto-kaufman: 2 exclusive feature, then we know for sure they couldn't overlap at one token.
j.fiotto-kaufman: I guess the most efficient
j.fiotto-kaufman: encoding is just use different sign, because they never you could not ever compose, so they're not. Even
j.fiotto-kaufman: if you have one.
j.fiotto-kaufman: This feature, you can go this direction. And it's like a simple scaled in certain. Okay.
j.fiotto-kaufman: I think that's all of my, oh, ongoing ongoing project. Yeah. Yeah. After I moved to Harvard, I switched my interest to
j.fiotto-kaufman: mostly our language models. I think, yeah, it's knowledge injection
j.fiotto-kaufman: and contextual effects in Lm, we have some very interesting findings
j.fiotto-kaufman: which we don't really know how to understand it correctly. But I'm happy to discuss. I know the group like Rome, mammate. It does the same thing. But I'm using a different method. It's a more global method rather than a targeted method, and I'll also see some very interesting contextual effects. I don't know
j.fiotto-kaufman: how to explain and happy to discuss also another thing. This is collaboration with everyone. She's also a fellow at Howard. Capital Institute.
j.fiotto-kaufman: yeah. So we have a toy model. Different model. So the the prompt is something like a blue square on top of a red circle, and it generates image like this. And now we're trying to dissect the circuits. Why, it knows which one is on top of which
j.fiotto-kaufman: it's a classic machine type of circuits.
j.fiotto-kaufman: that's all the sense I prepared. Thank you very much. Thank you very much.
j.fiotto-kaufman: High strength discussions. Oh, by the way, I plan to stay here longer. So I'm very happy to talk
j.fiotto-kaufman: everyone.
j.fiotto-kaufman: Yeah, no, we the idea was we could do group discussion. I do know people who want to talk one to one. So if we have experience, if you wanted to have, like a single discussion.
j.fiotto-kaufman: a 1-to-one with him. Yeah, I can do that. And then after that, and any questions for for Pat, before we, I'm really interested in the relation thing. But maybe I can just ask you about that. Yeah, I just want to. I just want to know, like what you've been trying to do or what you've been so far. Oh.
j.fiotto-kaufman: yeah, yeah. So we
j.fiotto-kaufman: So 1st thing I do is, I guess our
j.fiotto-kaufman: strategy is 1st find features and then try to connect those features. Right? So the
j.fiotto-kaufman: oh, this different relation. Right? Yeah. So first, st we
j.fiotto-kaufman: we use something not like as a sparse auto encoder, we use a simpler as we think. It's more straightforward way to find those features, but just contrastively learn. Okay?
j.fiotto-kaufman: yeah. Anyway, we have a new way to discover those like blue objects.
j.fiotto-kaufman: or it's a square or circle. So 1st we find those features, and then we have to. We have also size the 2
j.fiotto-kaufman: circuits, then can implement the the positions. For example, while object, if we have in the early stage of this diffurion, have a blob, and this blob 1st attend to.
j.fiotto-kaufman: The word, the the positional
j.fiotto-kaufman: information on top here, because it has. Like the top positional encoding it knows. Okay? If I'm on the top, and I see on the top. Then I should attend to the left word to the on the top.
j.fiotto-kaufman: and then in the next layer.
j.fiotto-kaufman: It tends to
j.fiotto-kaufman: the 1st word and get the attributes from the 1st word, turn itself into a blue square. So that's 1 circuit, and we have another alternative circuit in
j.fiotto-kaufman: That's a question
j.fiotto-kaufman: in in this setting, is it? This is a, this is a diffusion model that you trained right? So like toy data, right? So it's like a small small data. So what if I have one diffusion model that says a blue square on top of a circle, not a circle on top of a circle, and I have a separate run, where I say that
j.fiotto-kaufman: like just a circle, just a circle. Then I compose the scores of the deficient and get the same output. I'm wondering how how does compositionally, how how different would the circuit be when I'm trying to compose 2 deficient inferences versus a single one.
j.fiotto-kaufman: Oh, that's very interesting. So because you are talking about make
j.fiotto-kaufman: with guidance, always with weak guidance or with strong guidance, and how the circuit is different if the guidance is different. Well, yeah, if you have a single run.
j.fiotto-kaufman: how the neural network solve the same problem, and I'll have 2 score functions, as you said, like a guidance
j.fiotto-kaufman: so I was wondering if you have any plan on trying to understand the compositionality of the deficient model as opposed to just like a single one. Because I think one of my lab mates did a research where this they found that when you're trying to compose
j.fiotto-kaufman: separate scores, you get better results than you have than if you have a single one
j.fiotto-kaufman: like by composing you mean like average course, just adding them, yeah, adding them.
j.fiotto-kaufman: So if you might have a single inference where you're saying where? Where? You were saying that okay, a blue square on top of circle, and then you have a separate run on just a circle.
j.fiotto-kaufman: and when you compose those 2 scores together, I believe that it might have the same output. So if it has the same output, how does composition affect the circuit as opposed to having a single one?
j.fiotto-kaufman: So the normal, like classifier free guidance is kind of special case of this.
j.fiotto-kaufman: Yeah, we did an analysis on the pos and the classified free guidance case which we have the positive problem and the negative problem lasting. So you can compare the circuitry in both cases. And then
j.fiotto-kaufman: so so it's kind of funny, like, the negative Prom is like more treating this thing more like a vision model. It's kind of like tax free. But you're mostly observing what is happening and the tax reach. Case is kind of adding things to it and moving information. Again, there's something different in the cases and comparing between them can find some feature that maybe, like taxes specific, and some is not so. But we haven't finished that part yet. Okay.
j.fiotto-kaufman: how do you find circuits in a multi multi time step inference.
j.fiotto-kaufman: We did it like basically each timestamp between feature model and see how it changed every single time. Step. How many times steps are you dealing with? So this kind of model we are doing like Pixar, it's like 14 times. I mean, there's 1,000 in total. But it's kind of continuous function. So we basically sample 14 different sampling steps. So it's good. And also it's changing very smoothly across time. So do you find that that's really necessary. Do you find different circuits at every time step.
j.fiotto-kaufman: We have hypotheses that I think it's maybe not that nice people. Things are trending pretty smoothly so.
j.fiotto-kaufman: And also we find there's a so so the time the time modulation is through like just basically adding it, it's a fine transform. So we guess maybe if we denormalize those parts, and it becomes much more stable. But we haven't really tested. Yeah, yeah, it's interesting when you say it changes smoothly. Do you mean the circuit changes
j.fiotto-kaufman: so more concretely. So if you do this kind of things, and then you so assuming this is like a vip transformer behind it, and you find the token correspond to the blue stuff and trace it across time, and then at each time steps like you do a, for example, linear classifier to find the direction that predicts Blue Square, for example, and you find the direction how it changed through time
j.fiotto-kaufman: and direction. I predict the emergence of a blue square will change pretty smoothly on the same token, on the same. Yeah.
j.fiotto-kaufman: But definitely, there are some deficit or some
j.fiotto-kaufman: bad things about this technique like, because things can happen different place across time. So it changes a little bit.
j.fiotto-kaufman: but it does change, but it changes. There was a time, I mean, there was a moment. In like the diffusion trajectory, we find things suddenly change. So we think there may be some emergent things like something decide to happen, or something. There should be object. Object should appear here those kind of things
j.fiotto-kaufman: interesting.
j.fiotto-kaufman: That's cool.
j.fiotto-kaufman: Well, well, that's great. Let me let's let's we'll we'll we'll wrap it up, and then
j.fiotto-kaufman: we'll take a chance for everybody to talk to talk to. You know our our presenters, one on one. I'm gonna steal you for just a couple of minutes, just to chat with you, and then I'll I'll I'll let you guys, you know chat with them
j.fiotto-kaufman: a little bit later this afternoon. Thanks, thanks, everybody.
j.fiotto-kaufman: Great! Do you want to.