pal.k: Is that? Okay? Yeah, okay, so we have Tamar Rosham here
pal.k: to talk today, I know that she's done a lot of different things. So I'm actually not actually sure what her topic is today to to talk to us about.
pal.k: But but but she's she's worked in many different areas of interoperability. So I'm looking forward to her presentation. Thanks. Wait. Why can't I share this one and
pal.k: And so let's see here.
pal.k: So you're too many stuff, too many Powerpoints. Yes, yes.
pal.k: that's here. Oh, it closed.
pal.k: Yeah. Google Slide doesn't like too many. Are you using Google slides to? What do you think about that? It's it's adventurous.
pal.k: Yeah, just the 6. Sorry.
pal.k: Alright.
pal.k: Why do I need to sign in pretty open?
pal.k: Okay, thank you.
pal.k: I know.
pal.k: Great.
pal.k: beautiful, great. Okay, this is actually a very 1st version of my job. Talk?
pal.k: So I'm on the job market this year.
pal.k: And I'm so glad. We scheduled it for this week, otherwise I wouldn't do a draft version ever and it's a very, very 1st version. So you'll see that, like some of the slides, are kind of like drafted, and need to be like much more fleshy. But I'd be very happy to get your feedback about it, so that, like the intro plus a disclaimer of, you'll see that some of the slides are kind of like unfinished
pal.k: anything else I should mention.
pal.k: Do you want interruptions, or just like I think I'll try to time it. It's probably a little bit too long, so if you can keep in mind, like
pal.k: what part are good to like skip, I think, like I have some idea. Is there a timing that you're going for
pal.k: I think I should do 15 min plus like the minutes. Yeah. But I'm not actually not sure. I didn't get like an explicit instruction. So I think this lot is like 1 h.
pal.k: And yeah. So I added, slide numbers so you can just write comments regarding.
pal.k: And I, oh, have another item.
pal.k: Yeah. Okay.
pal.k: Oh, and in some some part I'll also have, like some comments, especially in the intro. The intro is like not ready yet. So in some part, I have comments like, here I should have more example or something like that. So if you have ideas, for example, let me. Okay, perfect so my name is Tamar for those of you who don't know me. I'm a postdoc with Antonio Terralba. David's X lab, and
pal.k: I started to work with Antonio. But then I realized, actually, I wanted to do a postdoc with David. So I started to collaborate with the lab.
pal.k: and I'm going to present some work I did during my postdoc about
pal.k: how can we understand neural networks? And how can we automate this scientific process of
pal.k: deep neural network understanding?
pal.k: Okay, so neural networks are
pal.k: extremely sophisticated these days. They can generate super realistic videos.
pal.k: They can power.
pal.k: Q is saying something powered. Robots to do. Extremely crazy stuff. Of course, Chatbot, that can interact with us as if they were actually human being typing us messages, revealing protein structure and even predicting quantum interactions. So
pal.k: actually, really, really crazy stuff. The problem with all these very sophisticated networks is that we don't really know what is the information encoded behind the scene? We don't know what is the algorithm operation that they are performing although they are so good.
pal.k: And maybe the examples before we're kind of like the trivial example that we think about when we think about very powerful AI model these days, but actually, in almost every aspect of our daily life, when we use everyday products, most of them are powered by AI models behind behind the scene.
pal.k: Whether it's like in a healthcare situation when you get some medical recommendation when you shop on Amazon, or just watch Netflix on your TV. Whatever you're doing, you are actually interacting with some some AI models behind the scene.
pal.k: And I think it's very, very clear that we need to be able to
pal.k: tell something about the way that these AI models work in order to make sure that they don't encode any harmful information, or they cannot like act in a way that we don't want them to act.
pal.k: But it's not only about safety and how to make these AI models safe.
pal.k: It's also something about controlling the behavior of them.
pal.k: So seen something that looks like a different decade. But it was actually just a few years ago
pal.k: we could design models ourselves in-house and train them with a very control and specific behavior. So, for example, during my Phd. I trained this generative model that can generate images just from a single example. And I was able to design the architecture of that model myself. Also, I was able to design a neural network that will get a state of the art performance just by designing the architecture from scratch myself.
pal.k: And that was during my Phd. But nowadays we have these like huge pre-trained models with like billions of parameters. This is just like a single layer inside a flax model that generates images, and it's so complicated to control the behavior of these models because they have billions of parameters. We don't really understand their architecture much, or and it's very, very hard to understand how to design them in a way that will be controllable.
pal.k: And that's actually the easiest part of it.
pal.k: Sometimes we don't really have access to the parameters of our model. Sometimes the models are black box. So if you are interacting with Jpt or Claude, if you're using that to generate images or gemini. All of these models are black box. We don't know what's going on inside.
pal.k: So the general question I'm interested at trying to tackle, and I'm going to show you some for some project some progress we made in that direction is, how can we understand
pal.k: AI models like the very large AI models. We have no nowadays, and also control their behavior, although they are very, very large. And sometimes even unaccessible to us.
pal.k: Okay, so I'm going to talk about 3 main main direction.
pal.k: which is 1st trying to evaluate performance of this black box function that we have no access to their inner computations.
pal.k: And then I'm going to show you some ways we came up with to explain the behavior of open source models which we can actually look inside. And then I'm going to show you how we can basically automate the process of trying to understand the behavior of deep neural networks.
pal.k: So let's just start
pal.k: something super cool. We found around a year and a half ago is that multimodality exists in language models, even if you don't directly train for it.
pal.k: So what I mean about by that is, if we train a language, if we take a language model that was trained purely on text, with no multimodel pre-training.
pal.k: and we ask it to draw a very complicated scene.
pal.k: It turns out language models trained only on text can write code that will render those scenes. It's not particularly for this example. You can see a bunch of other scene that this is Gpt. 3.5 generated.
pal.k: Yeah, fairly, complex structure and relationship between object which is really cool.
pal.k: Note that it actually blurred the motorcycle on the last image.
pal.k: and of course you can claim that there is another modality here which is the code and indeed code. Here kind of like surveys
pal.k: a mutual ground between text and the visualization.
pal.k: Just a little bit after we started to work on it, Gpt. 4 came out, and the spark of Agi paper I don't know. You probably know that paper, but a very famous paper that kind of like claim. We are just
pal.k: a little bit before Agi will take over
pal.k: and of course, now everyone know that we can generate visualization using pure language models.
pal.k: So the question we ask ourselves is.
pal.k: how can we quantify these visual knowledge that language wonder seems to have.
pal.k: and at least back then that was a very challenging task, because all the models that could actually generate image rendering code were black box. So we have this task of like trying to evaluate a black box.
pal.k: So what we did in order to answer this task is? Answer this question is, we design a test set of tasks
pal.k: in order to try to evaluate the abilities of different language models. We ask language models to generate different visualization. We ask them to recognize visualization from code.
pal.k: We asked them. We even found that they can correct their own visualization just by looking at the code without seeing anything. And we also show that you can actually use these images in order to train a vision model
pal.k: for other downstream tasks. And I want to show you some example examples from the drawing experiments. So for the drawing experiments, we came up with this hierarchy of concepts, starting from a single very simple concept, like a single shape, maybe some instruction regarding the color of the shape or the arrangement of multiple shapes. And then we moved to objects that are obviously composed from multiple shapes, and then scenes that are obviously composed from multiple objects. And it turned out
pal.k: that a lot of different language models with no multimodel pre-training can generate these fairly complex concept with which is, very cool.
pal.k: Another cool behavior that we found is that if you prompt a language model to generate a visualization multiple times, often you'll get a completely different visualization, which means that the model didn't kind of over fit to a particular example, but maybe it has some sort of a mental model for how, let's say, a strawberry should look like.
pal.k: which is also cool.
pal.k: And then another cool example is that
pal.k: sometimes you get an okay visualization when you try @firstst But then we can basically copy paste that code and tell the model just to do better. So that was supposed to be a swivel chair. Can you please do better?
pal.k: And just by doing that we can actually get fairly nice, fairly nicer visualizations.
pal.k: In fact, we showed in the paper that, using our valuation metric
pal.k: inferior models like Gpt. 3.5. I don't know if the right word here, but maybe weaker model like Gpt. 3.5 can achieve the performance of Gpt-four, just by doing this iteration, which is super interesting.
pal.k: Okay. Last experiment. I want to show you from this line of task is we took all these images that were generated with language models, and we train our resnet 15 to have, with a contrastive learning scheme to have some sort of image representation. And now we take this image representation, and we show that we can just train a linear layer
pal.k: to classify a actual images using it. And you get a really competitive result using that. So it seems like, although this visualization doesn't look don't look like actual images. They have a lot of information that can be useful
pal.k: to train an image representation and just to visualize this representation. So we have a query image, and we're trying to look at the 5 nearest neighbors, according to the representation we train. This is
pal.k: If you just initialize our resnet with random weights. These are the nearest neighbor that you'll get. And this is with draining the resnet, with the language model generated images. So you can see that we
pal.k: actually, we were actually able, just using text
pal.k: to generate images that could help us
pal.k: train image representation that evaluate real images quite accurately, which is super cool.
pal.k: And if we are talking about application. We have a follow up paper where we show that you can, instead of using an existing programming languages, we can actually design our own programming language
pal.k: with text that let language model, or an agent powered by a language model to actually sketch in a way that looks super realistic and super human like. So instead of getting these kind of like mechanistically looking visualization, we now get something that looks supernatural.
pal.k: Okay? So that it was about trying to evaluate performance of black box function, and specifically the visual knowledge that they have. But actually, the reason I wanted to show you this project is because
pal.k: actually, the evaluation here were kind of very intense, and the reason for that is we had to come ourselves with an interesting task to try to ask our model to solve. So we kind of like play with the model. So what could be an interesting failure, mode or an interesting success model? And we iteratively designed this protocol in order to evaluate the model. And it was very, very intense on the researcher side.
pal.k: And maybe a question you can ask is.
pal.k: Hmm. Maybe it's because it's a black box. Maybe if that box was open and we could look inside, maybe evaluation was easier.
pal.k: And that's something I also had in mind.
pal.k: But actually, the answer is not really. It's not much easier. And I'll I'll I'll show you some experiment we did. In order to try to understand open source models.
pal.k: Okay, so for trying to understand all personal source model, we actually
pal.k: went back or not really back. But we went to kind of like a much simpler task, which is what we call entity tracking, which is the ability of language model to track information that is presented in context. The reason we did we do a much simpler task. Here is open source model are at least back. Then we're kind of inferior in performance, and we could not really evaluate
pal.k: the ability to create visualization with them with them. But we wanted to evaluate something much simpler.
pal.k: Maybe that's that's not a good excuse. Tell me later, later, David, okay.
pal.k: okay. So yeah. Back to the point. We wanted to evaluate the ability to track information that is presented in context. And specifically, we look at these data set for entity tracking which present a series of boxes with a series of objects located in each box. And of course the language model should be able to say that box G contain the apple.
pal.k: or hopefully should be able to.
pal.k: Okay. Now, we have this
pal.k: open source model. So we can actually look inside at activation and weights of the model. And we want to find what is the algorithm algorithmic operation that the model performs in order to solve this task?
pal.k: How can we do that we do something that is called pitching in order to isolate isolate components that are involved in the computations of the final answer. Can someone write down that I should add a citation for pitching from somewhere. Great
pal.k: And here, here that is here, here is how it goes. So we start with,
pal.k: just an input prompt to our model.
pal.k: If the model is successful in performing the test. Looking at this prompt, it should predict the word apple.
pal.k: And now we store all the activation that the model had for this particular input.
pal.k: And now we take
pal.k: another input that correspond to the same task, but has a completely different information, and should predict a different token, which is cap.
pal.k: And then we do something super cool. We look at a specific component inside the model. And we say, Okay, if this model is part of the computation graph of solving this entity tracking task, then fetching the activation, taking the activation of that model from the second forward pass and insert it
pal.k: just like copy, paste it to the forward pass of the 1st sentence should affect the result, meaning the result should go from apple to cap, or at least the logic of apple should go down and cap should go up.
pal.k: So we have this ability to kind of like score the effect of patching a specific component from one forecast to the other with a patching score.
pal.k: And we can do that for each individual component in our neuron and get something that is called the circuit, which is the sub network that correspond to that specific task according to the fetching sport. So all the components that got the highest are fetching score
pal.k: something super cool about this circuit. When we talk about entity tracking. So we evaluate. We evaluated a model called Lama B. Lama 7 B. So it's a llama bundle with 7 b parameters.
pal.k: If you take the full model and evaluate the accuracy of the model on that task, you get accuracy of 66 percent.
pal.k: And if you just evaluate the performance of this circuit.
pal.k: you actually able to recover 100% of the model accuracy which is super cool. And in fact, we do that with only 75 attention heads of the model, which are a tiny bit of the full model
pal.k: sites.
pal.k: another another super cool observation we had in this point is.
pal.k: we can actually look at other version of Lama, which are fine-tuned version meaning someone took this model and fine tune it to be an expert in a specific task or on a specific data set. So, for example, vicuna is expert in kind of like user dialogues with users. And we also have goat and float which are expert in arithmetic, arithmetic task with different fine-tuning scheme. And, of course.
pal.k: interestingly, you can see that fine-tuning on arithmetic tasks give you better performance in entity tracking. But an actual, more interesting fact is.
pal.k: we can take this circuit that we found on Lama 7 B. And just apply it on the fine-tuned models, and we can see that the circuit by itself
pal.k: kind of recover most of the performance of the fine-tuned model, meaning that the same circuit performs that task in fine-tuning models, which was kind of a surprise
pal.k: when we ask ourselves, okay, maybe
pal.k: maybe like, maybe we can say something about the mechanic mechanistic operation that or the effect that fine tuning has on the algorithmic operation of our model.
pal.k: So for that we didn't only want to look at the circuit. But we also wanted to look at the functionality that the circuit implements. So the algorithm that the circuit implement in order to perform entity tracking.
pal.k: So what we did is we came up with, a new method called
pal.k: Vcm. And I should have it here somewhere, probably the title Deciderata, something component masking something.
pal.k: method. And it goes the follows,
pal.k: we have an original input to the model. And now, instead of doing a patching experiment with some other input to the model, we design a very specific input or very specific alternation that will emphasize a specific functionality. What do I mean by that?
pal.k: Let's say we suspect that some components inside our model inside our circuit
pal.k: are very sensitive to the label of the box. So they're responsible for detecting the label of the correct box.
pal.k: We can design an alternation for the original sentence.
pal.k: such that we basically change the query box?
pal.k: We we. The question is now about a different book
pal.k: that actually doesn't exist in the alternation.
pal.k: And if we have components that are supposed to take the label of the box and associate it with the box. In context.
pal.k: then the answer should change from document. That was the answer on the original path to the content of box Y, which is a part.
pal.k: So we design different, like that. We design an alternation that is sensitive to the object itself.
pal.k: So components that should bring the information about the object in the box should correspond, should have a high patching score. With this alternation, and also another detail which correspond to the exact position of the object. So maybe some components
pal.k: are just looking at the exact position of the correct object and build that in making long story. Oh, and also we didn't want to do the exhausting, pitching procedure of like looking at each individual component. So instead, something super cool. We did, and people are using it right now is we train a mask that tells us where to pitch.
pal.k: How did we did we do that we kind of like approximate a binary mask over the component. So we have the ids of the right components to patch it. And we found the Max, the the mask that gives us the highest patching score. In total.
pal.k: Okay, so we have these like different role, different functionality. And and now we can actually locate different groups of
pal.k: or different yeah, different groups of component that performs a very specific functionality. And to make a long story short, we found that
pal.k: actually, the model output the correct object
pal.k: by transmitting the position of the correct box in context.
pal.k: Okay, so we have a group that kind of like summarize the information about the the box we care about, and then we have a group that locate the query box. Bring in the position of that box in context. We have a group that transfer this information. And finally, a group of heads that attend back to the object inside the query box
pal.k: and also, very surprisingly, we found that the same functionality exists also in the fine-tuned model
pal.k: which performs it with enhanced abilities. So
pal.k: we can say that when we find you in a model we don't really change the mechanism, we just enhance it, which is a very nice observation. I don't know if you can say that we were the 1st to say something. Can we say something like that? We were the 1st to show something like that.
pal.k: Maybe in some detail. Yeah, okay,
pal.k: and just a sneak peek. We have a follow up paper coming soon.
pal.k: Maybe it will be under submission. By the time I present it we show that language models also do something that people call theory of mind, which is the ability to track different beliefs inside a concept. And actually, we show that this kind of belief tracking is very similar to entity tracking, but with additional mechanism, that kind of like store information about visibility between different characters or different objects.
pal.k: Quick.
pal.k: but also here actually experimental, was, were also very, very intense, and maybe we were able to kind of like semi-automate, the pitching experiment. But there's still a significant
pal.k: effort here which was the hypothesizing part. So in order to find the functionality of different components inside the circuit. We had to like, sit down and kind of hypothesize. Okay, maybe some groups are very sensitive to the position
pal.k: of the object, and maybe some of them sensitive to the label of the box, and so on, and so on. In fact, in the appendix of the paper there are 3 pages full of other hypotheses we had along the way, and some results to test them.
pal.k: Okay, and
pal.k: yeah, I think we're all getting a little bit lazy, and we want someone else to do all of this for us, or at least assist that or but maybe that's not a good claim. Maybe what I should say here is that actually, models are so large right now. So that was Lama 7 B, and I think today's Lama 7 B is like the tiniest model that exists. We have lama 4 or 5 B, which is far way, larger. And
pal.k: performs much more difficult operations and doing this like manual experiment and manual hypothesis. Generation part is very, very exhausting.
pal.k: So maybe there's a way to automate that maybe we can do something to automate this scientific process of trying to hypothesize and test hypothesis regarding the behavior of an underlying model we care about.
pal.k: this is not an issue only with like fetching experiment or only with understanding language model. It's a general issue when we're trying to explain neural networks. And we actually have really good tools nowadays to try to investigate either inner representation or trying to get like this salency map of where the model pay attention to
pal.k: visualize some like weights or kernels inside our model, and even finding these circuits, we have great tools. But if we want to be able to say something about the operation of the model as a whole, not only on a specific
pal.k: tasks, but like in general.
pal.k: we need to be able to integrate these 2 and to do a lot of experiment to validate different hypothesis.
pal.k: And I think I'll skip this slide.
pal.k: So I want to show you the effort we did in order to try to automate the process of explaining a neural network. And
pal.k: there there had there have been attempts to automate. This like explanation procedure. And I want to start show you some one example trying to explain an image classifier.
pal.k: So let's say I have this image. I input, it to a classifier. I get a suit class for that specific image.
pal.k: And now I want to understand why? Why, that this image was recognized as a suit.
pal.k: So maybe there is a specific neuron that fires for that particular image. And maybe that's that neuron is sensitive to the concept of a suit.
pal.k: So I can look at that neuron. I can take the activation that it has.
pal.k: Maybe that that neuron got the highest activation along all the other neurons in the classifier. And I can also look at the area where the activation was high inside the image. In fact.
pal.k: we can do this process not for a single image, but for a very large data set of images like Imagenet, and look at all the images that got these top activations. This process is called network dissection, by the way, and we can see that this kind of like gives us a visual explanation of the sensitivity of the neuron.
pal.k: But if we actually want kind of a label for the neuron, maybe we will need to ask human labelers to look at these images and say, Oh, it seems like all the common stuff that we have here is maybe the lower part of human faces, or maybe some sort of a neck wear
pal.k: so you can see that this is still very labor intensive. We still need a lot of, but we still need human to help us
pal.k: here.
pal.k: So what people have done in order to automate this process is to train a sort of like captioner that look at this set of explanation and give a specific label for them. So that's something people need in a center called Milan, in a method called Milan, a paper from Iclr 22
pal.k: and Milan. Look at these images and say, Okay, these images correspond to human and animal faces.
pal.k: Maybe that's very bad. But
pal.k: actually with Milan. We only get to hypothesize once about the behavior of the the network, the neuron.
pal.k: and we never actually get to validate the hypothesis we generated. And now, if we want to use this information for a downstream task, for example, I want to control the behavior of the classifier, such that it will never classify human faces. Then I'll actually need to validate that conclusion first.st
pal.k: And the second thing is.
pal.k: Milan can only help us with a specific task which is neuron labeling. But actually, you can think about interpretability as a broad set of tasks that you can. A broad set of explanation that you want to get about your model. And Milan doesn't help us with other type of question. We might have.
pal.k: Okay. So what I wanted to do, what I want to show you now is how to actually automate the scientific process of trying to explain an arbitrary behavior of a neural network.
pal.k: So imagine that you maybe have some sort of an agent that can interact with your neural network
pal.k: and giving a specific question about the behavior of the network, it can generate multiple hypotheses regarding the behavior and then generate actual actual input to the network that will test different type of different aspect of this hypothesis.
pal.k: We'll get the output of these, and then, according to the output, we'll update the hypothesis that it has, and so on, and so on, and until it can actually conclude something smart about the behavior of the model.
pal.k: And we want this agent to be able to explore any type of model that we have.
pal.k: and to answer any sort of interpretability question that we have about the behavior of the monitor.
pal.k: Okay, so this is exactly what we did in, or at least the 1st attempt a 1st version of it. In our
pal.k: last Icmo paper. When we introduced the Multimodel automated interpretability agent or Maya in short
pal.k: and I want to show you how we can build such an agent. And how can we use it to understand vision models?
pal.k: Okay, so how how do can we actually build an agent that will help us with exploring neural network? The 1st thing we did is we took Gpt-four V. And we powered it with a title library that enables to interact with actual neurons inside vision models. So if we have a specific
pal.k: a specific network, a specific neuron we care about in a particular network. Let's say we have unit 0 in there for resnet 1, 52, and at some point, maybe the agent has the hypothesis that this neuron is sensitive to airplanes.
pal.k: Using that python library, it can look at this, the activation value and the activation map of that neuron on a specific image. Let's say, an image of an airplane.
pal.k: So the agent can actually interact with neuron inside actual vision model.
pal.k: Another side of the system is
pal.k: tools that enable Maya to design experiments. So let's say, Maya hypothesized that a specific neuron is sensitive to an air like airplanes. In general, it needs to get the input data from somewhere.
pal.k: So the 1st tool that we implemented is this data set examples tools which enables to call a very large data set a fixed data set of images. And to get activation values on that data.
pal.k: Another tools we have is to enable Maya to actually generate more test case.
pal.k: So more test cases. So we have a text to image tool that enables Maya to generate new images. Let's say a new image of an airplane, and also edit these images to actually test the causal effect of changing some aspect in the input image. So maybe that neuron is not sensitive to airplane in general, but only airplanes in daylight and not during the night.
pal.k: We also have tools that enable us to do the other way around. So summarize images into text. And we have an experiment log where Maya can actually choose to log in the result from the experiment so that they will be available for the next iteration.
pal.k: Of course Maya cannot actually run
pal.k: python code, but we design something that look like a python library in the system prompt as text for Maya. So Maya get all these tools as text
pal.k: and now we can actually extract the code that Maya writes and run it for it. And
pal.k: automatically, input the the result of the expression.
pal.k: Okay, so we have that system that can interact with actual vision models has plenty of tools to design experiment, to understand vision models.
pal.k: And now a user can just come and prompt the system with a specific, a a task, a specific question that it has regarding the model. For example, what is the functionality of unit 1 22 in a particular model?
pal.k: And something super interesting is starting to happen when we are doing that Maya actually autonomously design and conduct experiment to answer the question. So it make hypothesis.
pal.k: It is an experiment to test hypothesis. Yeah, it refines hypothesis according to the observed outcomes, and so on, and so on, until it can answer that question. And because we didn't design this system to answer this particular question.
pal.k: A user can now come and ask a different question, for example, about the failure mode that it observed with the model, and Maya can again design experiment to answer that question.
pal.k: So let's see one example, for how Maya operates?
pal.k: So again, we are looking at the task of explaining a particular neuron inside the model and when we prompt Maya to do that, it usually starts with calling the data set exemplars tool.
pal.k: which it's a very intuitive thing to do, although we didn't instruct it to do that, because then we get you get a sort of a prototypical behavior of the neuron on a very large data set of images. So this is the output of calling data set examples. You look at the images that got the highest activation. Do you have a guess for what's going on here? Like, what is the sensitivity?
pal.k: By the way, it's the same neuron that we saw before with Milan.
pal.k: Okay. Looks like the bow tie.
pal.k: Oh, no, David, you cannot. You cannot reveal the answer. Do I already know this? Yes, probably otherwise. No. I think people usually say lower part of the face. For some reason, I think that was also my 1st reaction. Actually.
pal.k: okay? And Maya, look at the these images, or maybe before. And this is basically where Milan stopped. Right? It has it. Look at the data set exemplars summarize something about them, and that's it. But Maya can continue with curative.
pal.k: So it looks at the output and say, Okay, there might be a possible explanation here. Maybe it's sensitive to some facial feature. Maybe it's sensitive to neck wear particularly butt eyes, and maybe it has to do something with also formal closing.
pal.k: So we design more experiment to validate these hypothesis, and particularly for the bot eye hypothesis. It uses the text to image tool with prompts that
pal.k: generate images of bow ties in different contexts. For example, many people wearing bow ties or bow ties, with no one wearing them. Just on the table, a dog wearing a bow tie and a kid wearing a bow tie. And for all of this it get the activation values and the area in the image where the activation was high. It also uses the image to text tool to summarize the context, the content inside the mask area.
pal.k: So it looks at the results of these experiments say, okay, based on the experiment. To test the bot eye hypothesis, it does, seems like the neuron responds specifically for both eyes. And now it has a refinement for the hypothesis, so it wants to check that. It's sensitive to both eyes, regardless of the context of
pal.k: where they presented maybe it's sensitive to bow ties more when it combines with formal tire. And also let's just validate that there's no response to other type of make work.
pal.k: And now we got something super cool, so it generate images, and then edit them to test some causal effects.
pal.k: So generate an image of a person with no bow tie, and then edit the image to introduce the bow ties to that person, and you can see how the activation suddenly goes up. And also we can see that the neuron activates around the bow ties itself.
pal.k: And it does that for multiple images. So, for example, you generate an image of a person wearing the butt eye and then replace the body with a scarf. You can see how the activation goes down. It has a person wearing a button with a hoodie, and then it replaced a hoodie with the formal, a formal shirt. And you can see how the activation goes slightly up and so on, and so on, looks at the result and say, Okay, it seems like, indeed, this is the but I detect or neuron
pal.k: and it seems like there is a higher sensitivity when both eyes are wearing with formal attire.
pal.k: Super cool, right?
pal.k: And this is the neuron we saw before from, and Milan caption, for that new one was human and animal faces, which is not bad. But according to the signals we see here, we can definitely tell that both eyes is much more accurate. And I'll also show you how we actually evaluate the evaluated the labels from both Milan and my.
pal.k: okay.
pal.k: So that was a kind of like a simplified version of the experiment. Now, I want to show you the actual code that Maya writes.
pal.k: And what's really nice about the code is Maya documents it so well, so you can actually know what was the reason it chose us to use any of the tools.
pal.k: Okay. So again, it started with the data set exemplars. And here you can actually see the explanation, which is nice. And these are the images that got the top activations for the particular neuron that we are looking at.
pal.k: Any guess now you can, you can guess it's okay. Am I allowed to guess. Yeah, yeah, you are
pal.k: circular objects, spherical objects, spherical objects, great, both. Something else
pal.k: cool my notice that it's not only spherical objects, it also has something to do with hands.
pal.k: Yeah, you like that. Right? Yes. Okay. So we have spherical object. And hence, as the hypothesis. And now we does some something super cool. So we generate an image of a person holding a banana and a person holding a cube. These images
pal.k: do not activate the neuron because they have no spherical object, and then, if edited, it edits the images to have spherical objects inside of instead of the banana in the cube. These are the results. Now, suddenly, the neuron fires
pal.k: and we see some activity, layer activation and activation map. So Maya concludes that this is
pal.k: the hands interacting with spherical object. Neuron.
pal.k: And our question you can ask is, how the hell do we know that this is true?
pal.k: So what I didn't tell you about this neuron is that this is actually a synthetic neuron that we designed to have a specific behavior. And the behavior we instructed to have is to be sensitive to balls, but only in the presence of hands in the image.
pal.k: Actually the challenging part here in general was the evaluation and synthetic neurons actually really helped us evaluating the performance of Maya. But we also have a protocol for evaluating actual neurons in vision models. I need to find a better word
pal.k: with correctional neurons. Because I presented this
pal.k: for a neuroscience group. And when I said actual neurons, they were so excited they were sure I'm going to show them. Take some brains. Yes. So it's like actual artificial neurons or something like that. But yeah, anyway.
pal.k: so we had synthetic neurons that actually, that really helps us evaluate performance of interpretability agents. We actually have a full paper just about that, how to design synthetic neurons with very specific behaviors. I think I want to skip this part. But just say that we have this data set and a benchmark to evaluate automated interpretability product
pal.k: protocol for this particular paper, we design synthetic vision neurons. And the way we did that is, we took rounded Sam, which is a segmentation algorithm that is instructed by text. So we instructed Sam to segment balls and hands.
pal.k: And now, when it gets this image, we did a little bit of other tweaks to make it look like an actual activation mask of a neuron. And this is what we get. We have several types of synthetic neurons, monosemantic neurons that are sensitive to one concept polysemantic neurons that are sensitive to multiple concepts. The concepts can be completely unrelated and also conditional neurons that are sensitive to one object, but only the presence of the other one.
pal.k: Now, in order to make this evaluation super interesting, something we did is other than just comparing Maya to other state of the art neuron labeling methods like Milan. We also replaced Maya with actual researchers.
pal.k: because the end goal of this is to see that explanation by Maya are actually equivalent to explanation by researchers. So we gave researchers the same Api as Maya have the same way to interact with models and design experiment. And we asked them to solve the same exact task.
pal.k: As a baseline for Maya. And now, because we have the synthetic neurons with known behavior, we can compare explanation from Maya and explanation from other baselines like Milan or the human labelers. And this is the result we got when we ask people to actually rate explanation
pal.k: from different baselines. So what we see here is the preference weight that people gave to explanation by method, A versus method. B,
pal.k: so we can see that in 73% of the cases people preferred me. Maya explanation over Milan and also Maya explanation are kind of comparable with human explanation. People prefer them in 53% of the time. So 50% will be completely random here.
pal.k: Oh, I don't have time.
pal.k: okay, sometimes Maya can be super. Funny. So here's oh, no, it's not that. Okay, maybe I'll skip this part. Let me know what you think. Okay.
pal.k: But of course, synthetic neurons are cool. But we actually care about real neurons in vision models. So here's an example of an actual neuron inside a vision model. This is neuron 533 in there for fresnet. And again, my start with data set exemplars. Look at these exemplars and say that
pal.k: there are multiple hypotheses. One is that this is a neuron that is sensitive to tennis ball. Maybe round objects in general, maybe object with fuzzy textures, and maybe it has something to do with the color yellow, or the specific color tone of this object. So we design a experiment to test each of these hypotheses.
pal.k: And you can see that only the image that had the tennis ball activated the the new one. So a ball doesn't activate a fuzzy texture doesn't activate. And the color. Yellow doesn't activate
pal.k: and Maya says that and just want to validate that there is an equal sensitivity to tennis ball in different contexts. And this is what what it does here.
pal.k: so we design another experiment. When tennis ball are presented in different scenarios on the road, on the water, in the snow, and so on, and so on, and for all of them there is a high activation for the tennis ball. So the conclusion is, this is the tennis ball mirror.
pal.k: Sometimes Maya can be super funny. So this is in the middle of an experimentation to explain a feature inside a unit inside Dino, vision transformer. So at this point Maya hypothesize that this unit is sensitive to faces of birds.
pal.k: It designed an experiment to test that so it generated 2 images of birds.
pal.k: And this is the activation value and the activation mask of
pal.k: those images. And now it does a very funny edit where it asks to change the head of the bird and the head of yeah, the head of the birds to a head of a dog and a head of a cat. This is the result just zooming in. So you can appreciate
pal.k: And these images don't really activate the new one. I think the reason that
pal.k: the cat one still activates is because something that looks like the beef remains here.
pal.k: Yeah.
pal.k: And look. It looked at it at these and concluded, there is some sensitivity to head of birds.
pal.k: Here's another funny example. So this is for a neuron inside a clip vision encoder at some point Maya suspect that the neuron is sensitive to different angle faces. So it generates an image of a family photo.
pal.k: And it seems like the neuron activates on the leg area. But for some reason Maya doesn't care and then it uses the following editing instruction, flip the family photo upside down. And at this point I was really hoping that it would just like import Opencv, or something like that throughout the image, right like that's like what we would do.
pal.k: but it didn't. It called the editing tool
pal.k: in order to rotate the image. And
pal.k: the result is not that good? There's no activation.
pal.k: And yeah. And then the conclusion is kind of like noisy.
pal.k: And this is something that we see. So most of the failure, the failures that we see with Maya is because of tools shortcomings. So sometimes tools just over editing over. Edit the image, or don't pay attention to negative instructions so kind of like emphasize something that we actually don't want to have. But this is actually an interesting point, because it means that as long as
pal.k: tools. The tools that we are using will keep improving. So the result by Maya. And we actually show that in an ablation study we have in the paper.
pal.k: Okay?
pal.k: yeah. I promised you that. I'll also show you the way we evaluated neuron inside vision models. And the way we did it is by some sort of a predictive evaluation. So if we have a label from Maya, how well could we predict the activation values of images that Maya didn't see.
pal.k: So for that we took we did the following, we took the label from Maya.
pal.k: and we instruct a language model to generate image prompts that, according to the label, should either activate the neuron or not activate the neuron. So we have these like positive exemplars and kind of like neutral exemplars that should not fire.
pal.k: and then we actually generate the images. And if the label from Maya is accurate, then the positive examples should correspond to images that has on average high activation value and the neutral one should have low activation value.
pal.k: In order to make this even more challenging. We use additional step, which is
pal.k: to generate a very challenging data set of image prompts. The reason we did that. Is, we wanted to combine all possible positive and negative exemplars from also the competitive method. So we will actually have a very
pal.k: challenging destructors in the deficit.
pal.k: So we don't just like generate the images according to one label, but we generate them according to all the labels we compare. And according to this, like large data set of points.
pal.k: And now we can actually compare all these methods. So we have a label from Milan. We can get the average value of the positive exemplars and neutral exemplars. And we can also do that for the human labels.
pal.k: This is the result. So plus is for the average activation of positive exemplars. The positive exemplars minus is for the neutral. We have Milan in purple Maya, our method in turquoise and human labels in green, and we also evaluated 3 different models resnet, dyno, clip, and of course, the synthetic neurons. Also. Here you can see that almost in
pal.k: all the cases, or in all the cases, Maya outperforming lands, and also in most cases it is on par with the human neighbors, which is what we wanted.
pal.k: Okay.
pal.k: And now to the cool part. So, as I told you at the beginning of this part.
pal.k: we didn't design Maya to answer a specific question about the model. And now we can actually prompt it to try to explain different behavior, different interpretability tasks. So one thing we did is to use Maya to identify biases in classifier. So now, instead of looking at new one inside the model, we actually look at the model prediction, at the probability that the model give to a specific image.
pal.k: So here's an example for a procedure like that we prompted via with the question of is, are there biases in the viewer glass class of images?
pal.k: And Maya just start to hypothesize regarding different biases that the classifier could have. So we want to check social bias, and maybe some bias to if the blue glass is presented in some unusual scenario. It want to check some gender bias and also sensitivity to the type of the blue glass.
pal.k: So here's the experimented design. It's kind of long. But you can see that for each hypothesis it actually generated multiple images that should correspond to that hypothesis.
pal.k: And these are the results. So these are the results for the social bias you can see that the same activation exists, whether the beer glass is in social context or in, or is not different scenarios, or like on an office table or in a sink? These are still fairly activating
pal.k: the the classifier. This is for the gender bias.
pal.k: And this is for the different type of beer glasses. And at this point Maya kind of noted that noticed. Noticed that there is some sort of gender bias, maybe because if I go back you can see that
pal.k: images of the blue glass held by a woman get lower activation, lower probability. Comparing to a man holding the brew glass, we design more experiment to test that as long as well as other experiment, to test sensitivity to different type of beer glasses, because there as well, we saw some variation.
pal.k: Here are the results for testing gender bias. Here are the results for testing sensitivity to different glass type.
pal.k: Look at them and conclude that there is some gender bias in image net classifier, which, as far as we know, was not revealed before. So we actually revealed a new, super important bias that exists in a resident classifier.
pal.k: Another biases that we were able to reveal this way is that the Labrador class is actually sensitive to bright Labrador over dark fur. Labrador. The suit class is sensitive to dark suits, and not so to other colors.
pal.k: This is a funny bias. So this is the Tench fish class, which is the very 1st class of Imagenet. If you open the data set you'll see that it's not actually a tench fish, but fisherman's holding the tent fish, and I was kind of surprised that these are the images that Imagenet collected for tenches. But then we went to Vienna for Icml, and we actually went to fish some tenches
pal.k: or try to, but we actually weren't able to catch anything. And the guy that we guided that we took told us that it's like it's very rare to catch a French fish. And then, like I realized that
pal.k: it's so rare you cannot really take a picture of a tent like in like, just like swimming around, and when people actually get a tent they are so happy. So they take a picture. So that's the reason there is a class in imagenet for fishermen holding tents.
pal.k: Sometimes Maya will conduct experiment, but won't find any bias which is also nice, so sometimes it will actually conclude that I did a bunch of experiment. It doesn't seem like there is
pal.k: any bias toward any specific setting, which is what happened in this flagpole class.
pal.k: Okay? Last application is for removing a spores correlation from a network. So
pal.k: sometimes you will train a network on your training set, and then you'll come to like, actually test it in real scenarios, and it won't perform as well as in the lab. And our possible reason for that is that the training set contains some correlation between features that you don't really meant to have. You don't really aware of.
pal.k: So we mimic that behavior with training a classifier, or on a data set. That call is called a spurious data set, which is for dog breed classification. But each dog is spuriously correlated with a specific background. So, for example, Corgi will be always presented in a snowy environment.
pal.k: And then we train a resonant model on that data set to force the test data set, has a random background for each dog breed.
pal.k: And now, because of that, the test accuracy of that model is not so good. Only 73%, because the model kind of like learned to look at as far as features that does not exist in the test phase.
pal.k: So what we did is we tasked Maya with
pal.k: finding neurons inside the classifier that we train that are not sensitive to the dog bed, but something else. So we only want to find to being able to locate neurons that are sensitive only for the dog bed.
pal.k: And here are some examples for experiment. Myrn. So again, it starts with the data set exemplars. The data set. Examples are from the biased data set. It doesn't have any access to unbiased data. Seems like this neuron is sensitive to a specific dog breed. Right?
pal.k: So not so.
pal.k: When Maya generated more images of other dog breed in the jungle environment.
pal.k: it was able to reveal that there's actually some sensitivity to the background, and it labeled this new one as spores.
pal.k: And then in another case, it generated images of other dog breed in the snow environment and the same dog breed, but in different environment. And it seems like these all are sensitive only for that particular dog breed, and it marked this new one as selective. And now what we did is oh, and actually it only marked 22 neurons out of the 500 that it tested as selective, which is super cool.
pal.k: So now, what we did is instead of using the full original model that got accuracy of 73%. We
pal.k: fine tune the last linear layer of of the classifier. But with the biased data
pal.k: only on the 22 neuron that Maya found to be spores, and you can see how the accuracy is now much higher. In fact, it is almost as high as just fine tune your model on the unbiased Stat, which is super quick
pal.k: so we were able to actually enhance the model performance according to the results we got from Maya. So for the Maya case, what? What was the experience? Exactly. So you
pal.k: you knocked out all the other
pal.k: units other than the 22. Or what is it that you did? So? These 22 neurons are from the last layer just before the linear layer. So what we did is we fine-tuned the linear layer to take to take only these 22 neurons, fine, tune it. And on what data on the on the original training set on the bias training set. Excellent. Yeah.
pal.k: If you do the same with the same 22 neuron on the unbiased data set, you get a little bit better. Yes, only a little. Yeah.
pal.k: Okay. This is about Maya. We have a full experiment browser in our website where you can. Actually look at all the experiment. What Maya performs along with the implementation it had, and the final conclusion, and I think the conclusion of this part is, we Aim AI models are so good now that we can actually build
pal.k: AI agents that will help us with AI tools that will help us understand other AI models. And I think in all, or maybe I'll say it later.
pal.k: cool.
pal.k: Okay. I have some text here.
pal.k: we have a new version of Maya coming out soon with improved tools and improve reasoning abilities. So stay tuned for that. We also have a new benchmark for bias detection coming out in the next month.
pal.k: and we have an ongoing project to reveal more global behavior of models. So not behaviors that are corresponding to a particular nuance, but maybe, like global subnetworks in your model, like the circuits we we saw in the other paper.
pal.k: Okay, regarding future works. And here I'm not
pal.k: fully sure what to say it. But we saw along this presentation that we have some very powerful tools to investigate, language model, and also well-known methods to investigate vision models. But I think the world is now going to a direction where everything is going to be multimodel, and we should be able to have a unified approach to test
pal.k: language, behavior, vision, behavior, interaction between language and vision, and also other type of signals like audio
pal.k: and maybe stuff that we don't really imagine right now, because the future models will take different kind of inputs. So as a general goal, I intend to have universal interpretability approaches that will combine universal interpretability tools with interpretability agent that
pal.k: we'll be able to interact with any sort of model that we want. And I think a promise with interpretability agent is, they are not only capable with assisting researchers like us.
pal.k: but maybe also able to assist some end users. So imagine that your grandmother is having a problem with her iphone sitting home. Maybe we could have some sort of interoperability, Chatbot, that will help her actually understand why.
pal.k: the iphone is not working today.
pal.k: And if we have these universal tools that will enable us to conduct experiment to explain very complex AI models and agents that can design experiment? To answer some hypothesize and answer a question about models. Maybe we can apply these kind of approaches not only for AI models, but for other scientific domains. And this is something
pal.k: I'm looking forward to try. In fact, I just started a collaboration with a project called Enigma in start forward, which are actually scanning brains of
pal.k: live animals in order to map them, and they are interested in using similar approaches to the automated interpretability in order to design experiments on their domain.
pal.k: And I think that's it.
pal.k: Yeah, that's it.
pal.k: They started at
pal.k: so I think it was 1 h and 5 min or something. Yeah, I took out the 2 min that you had talking with David about that question. Okay, it was like, 1 h, 3 min. Yeah.
pal.k: Okay.
pal.k: Oh.
pal.k: questions for the talk on the subject matter. First.st
pal.k: Okay.
pal.k: Oh, I have a list of questions. Should I do it one by one. Or do you want like, I don't know. How do you do it?
pal.k: Maybe to start, maybe we can start from the beginning and then do it okay? Or is it like question about the no, just questions about the material? Great? Great? Yeah. Question. Yeah.
pal.k: you at slide 24. You mentioned.
pal.k: these models are not trained for multimodel. Yes, retraining.
pal.k: Do you think they've been trained on codes like that? If so, would you consider that multiple good question? I think they
pal.k: definitely been trained on code like that for visualization. But they didn't see the output of the code. So I think that's the like important. So they know that, like they, here is the example of like something that you generate something, but they never actually saw
pal.k: the visualization itself.
pal.k: But are there like comments in the code that say, like, here's what this text or this code will generate. Probably. Yes, something that we like. We definitely know that is.
pal.k: they're really good at like documenting code like, if if there is one thing that they are doing very well is documenting code. So I'm pretty sure that, like, they have a lot of information regarding like what should represent what bomb
pal.k: coins.
pal.k: So it seems like most of the spatial reasoning is coming from the vector arithmetic right? Like
pal.k: at Pixel 12, I would put of sheep. Maybe the next sheep should come at like Pixel, 18, so that there's no overlap.
pal.k: So do you think? The models that are you had a Glop model or something? Arithmetic trained? Lamma? 7 B.
pal.k: Fine tuned.
pal.k: Do you think such models would do better at this. Oh, you mean, like fine tune. Arithmetic will help them with like, special reasoning.
pal.k: That's a cool question.
pal.k: Because it's like, yeah, maybe. Yeah, maybe it's like, like, it's like a 2D position formation.
pal.k: Yeah, that's that's a great question.
pal.k: I do. So, okay, so we have that paper of
pal.k: entity tracking like how to track information. We actually have
pal.k: an undergrad student in the lab
pal.k: that is trying to do. I only now realize that it's something similar, but like kind of like special reasoning in Vlms.
pal.k: So how do we know that something is next next to something?
pal.k: and maybe maybe we can try to find you in the Vlm. On like arithmetic and see if
pal.k: improves like special reason. Yeah.
pal.k: All the way down to like the Maya patron. I maybe I missed it, but I wasn't sure like, how does I guess? It work when like
pal.k: it, it doesn't identify. What concept is there in the mural
pal.k: like? I guess the edge case of like of like? No, not interpretable, you know. Sometimes it will say, like this is very general. I'm not sure. Maybe I should have an example like that. That's a good point.
pal.k: sometimes it will just keep exploring.
pal.k: Something, we added, is after like 20 integration, or something like that. If it's not like ready to conclude, we just ask it to conclude whatever. So right now, we don't have a limit can do
pal.k: like we don't say, okay. After 5 iteration, please conclude we just give it like to run but we at some point we do ask it to just conclude and something interesting that we found is sometimes, if it wasn't able to get any interesting signal at that last iteration. It will just call that exemplar and conclude.
pal.k: there's a plot for
pal.k: your test time. Compute Gpt. 3.5 versus 4,
pal.k: like the more number of times you ask it to get better. Yes.
pal.k: and it's it's interesting that Gpt 3.5 does better as it goes. But Gpt. 4 is like almost
pal.k: where is it? Isn't that like I heard speculation that Gpt. 4 was just like a multi Gpt 2.5 doing like majority voting or something like that.
pal.k: So that's like a kind of like an evidence for that, I mean, is it possible to do much better is it? Is it just saturated?
pal.k: I think it's already like. So the evaluation metric here is sort of like clip classification.
pal.k: So like 99% of the time you got it. Right? So it's already kind of external. Yeah.
pal.k: yeah, maybe you need a harder task.
pal.k: Yeah, we also wanted to do it with
pal.k: more complicated images. So not only with the object, but only also with scenes.
pal.k: And I think we ran out of budget or something like these experiments were like, very expensive. Because
pal.k: or at least back, I think with these, with these models today, it might be cheaper. But yeah, maybe with more complicated scene, we will see separation.
pal.k: I'm kind of bothered by how an
pal.k: like, not smooth. It is so like, maybe maybe it has something with the fact that, like we only had like 100 concept which we reiterated or like, I just, for example, I don't understand the deep in like iteration. 2 or 3
pal.k: is the prompt simply, can you do better or it's something like that was supposed to be a swivel chair. Can you do better? Yeah.
pal.k: So a little bit sarcastic.
pal.k: Yeah.
pal.k: Hmm, I don't know. I didn't mean it, Theresa, but it sounds fantastic, you know. Yeah.
pal.k: a question for the Via agent part. So I I did a lot eyeballing, manual screening, looking for those features. So I guess my impression is
pal.k: in earlier years.
pal.k: There are a lot of integral, low, level features like color temperature, and in some later layers also there are a lot of features like
pal.k: parts of objects or objects itself. But what
pal.k: tells me is in like those middle layers. Sometimes it's really hard to find some interpretable features. I don't know if you have the same. It really depends on the model, I guess.
pal.k: actually in I think in Clip. We saw a lot of interesting stuff also in the middle.
pal.k: So it's kind of like text and vision.
pal.k: like mutual like. For example, I remember a spider-man neuron that is corresponding not only for visuals of spider-man, but also for, like strong.
pal.k: what what else? Like like comical like, and titles and
pal.k: or like conjunction, I can remember, like Ampersand, and the word end. I think those were actually in the middle.
pal.k: Another interesting point is.
pal.k: we actually have it in the appendix of the paper. Maya was more successful in labeling middle and layer neurons than the like, the very early neurons. Because it wasn't that good at like, just
pal.k: thinking about very low level concept, like the color red.
pal.k: And it's just basically because of the field, like the the vision.
pal.k: or like the way it was, it was trained for like image captioning. So it doesn't pay attention for like texture and stuff like that so much
pal.k: I do think that it is possible to fine tune a Vlm. To pay attention to low level details more. And, in fact, this is something that I should probably put in the conclusion of the talks about. I feel like
pal.k: we showed in the paper that we can get performance that are
pal.k: on par with human researchers. But I think something interesting to
pal.k: like an interesting goal, will be getting something which is better than researcher, especially if we want to use this agent for scientific discovery.
pal.k: And I think this will be like super challenging.
pal.k: Because in the project we have right now. For
pal.k: having an agent for bias detection, we actually found out that all these agents or these models, are biased themselves
pal.k: to just factual knowledge.
pal.k: And if we have, like, we design a synthetic system that is sensitive to
pal.k: a tie, but only when a woman wear it.
pal.k: and they really have hard time to detect that, because it's so out of distribution for them.
pal.k: So I think if we want something that we will be, I think it's just like a mirror of what we would say like these models. And if you want something that will perform better than us, I think it will take
pal.k: hard work.
pal.k: Yeah.
pal.k: So so in my online work, basically, you have this kind of text to image and image to text. So kind of.
pal.k: There's an assumption prior, that the interpretive features should be describable by words in some sense like in the final slide. I see your collaborations. Enigma project was like doing those type of things. So no, many things you synthesize from, I mean, monkeys or mice. Neurons are kind of, you know. Yes, not. I mean, they don't know language or English, so I don't have to buy the prior. It's like they should be describable by words. I think that's a really good point, I think. Right now we are using
pal.k: language as our hypothesis generation medium, just because
pal.k: language models are so strong. But we should definitely consider other type of hypothesis representation. Maybe, like, even like, I don't know, differentiable collision, or
pal.k: maybe images could be like hypotheses.
pal.k: so yeah, definitely, I don't. I don't think language are necessarily is necessarily the only way to describe
pal.k: behaviors of your yeah.
pal.k: I had a question on the your synthetic neurons for Mayamp.
pal.k: Like, I was trying to find in the paper like a split between the kinds. So like monosemantic poly semantic of like, how well the model did for different. Oh, we don't have it in the paper. I see one. But
pal.k: yeah, yeah, you're right. That's an interesting thought. Yeah.
pal.k: cause like, it seems like, if you just like, pick 2 random concepts and expect them all to be able to to pick them out unless you're like.
pal.k: data set exemplars is like very broad. Be hard to to do something.
pal.k: Yeah, I vaguely remember that we did have
pal.k: oops.
pal.k: Like, I would guess. Probably the order of easiness would be like monosemantic, poly, semantic, and conditional. Yeah, yes, I agree with your intuition. We, in fact, we also had another type of
pal.k: nuance which was the opposite of conditional so like dog, but only if there's no leash
pal.k: And those were the hardest.
pal.k: The fact that you didn't include it in the paper was only because the synthetic neurons themselves didn't like weren't working that well. So they weren't like as reliable as the others. But let me check for the separation. I think it's important. Yeah.
pal.k: So the agent, basically, you just need to design the workflow and prompt it to do so. And no need to do anything fine tuning, right? Right? So you design the Api, which are basically the tools that you want it to enable it to use?
pal.k: yeah, and then you prompt it. The hard part was so once we had the library design as like an actual python library. It started to use it like immediately, with no problem.
pal.k: And then also, like explaining, the task was
pal.k: relatively easy. The problem was actually with, like the formatting. So we wanted to reliably output the code in a specific format. So we make sure we can always like parse it out, run it. So that was like we did a little bit prompt engineering there.
pal.k: But they, like actual contribution, was setting the tools as a python library.
pal.k: But I think now you now you actually have.
pal.k: maybe at least with Claude. You have the ability to define tools
pal.k: and like in the Api. So I think, like you, you should be able to do it built in. Now
pal.k: you had a I had a question
pal.k: for the circuit work where you said the circuit work need to need some adjustment. I feel like.
pal.k: table.
pal.k: Yeah, the circuit. Yeah. The sec, yeah.
pal.k: What? What does it?
pal.k: What did you actually do for the circuit column? Is it just using the circuit just like applying the identity of neurons from the circuit we found in Lama
pal.k: just evaluate that circuit on the other models. We didn't.
pal.k: We didn't redo the patching experiment to find the circuit in the other model. So all other attention had set to 0.
pal.k: Right? Oh, I mean ablation. Yeah, yeah, I see.
pal.k: So just the attentions ablated to me, or is it the Mlps? And
pal.k: it's all the Mlps. If I, if I remember correctly. Yeah, it's all the Mlps.
pal.k: But Nikki is probably here somewhere, so we can also comment. But it's all the Mlps. But just the attention! Heads up!
Nikhil Prakash: No, no, it was not them.
pal.k: You mean? The Mlps were not ablated. The Mlps are included right.
Nikhil Prakash: Yeah, yeah.
pal.k: But then but then but then you ablate the attention, headset or not.
Nikhil Prakash: All the attention, all the attention. Heads were knocked out which were not part of the circuit.
pal.k: Oh, yeah. So also on the table, did you also look at like? Because the circuit accuracy increases for gold and float? Did you also look at maybe faithfulness. And yeah, we did look at other matrices that I didn't wanted to include here. I didn't want to. Yeah. But we we have it all in the paper. In fact, the way we found the Llama circuit.
pal.k: Okay? So when you do like, we did path patching there. But when you do like a pitching span, you get a pitching score. You don't really have a threshold of like, what is the amount of features to include?
pal.k: So we optimize the circuit size according to some minimality criterion.
pal.k: Yeah, right?
pal.k: I also wonder if you happen to run like the arith if you use the cert, the the circuit found in llama on the arithmetic task.
pal.k: What have what would happen to the like? How would the model perform like using the entity tracking circuit on the arithmetic. Oh, and maybe there might be some overlaps in in the circuit components. Yeah.
pal.k: yeah, that's a great question.
pal.k: and I feel like this type of
pal.k: like more general model behavior, of like, how the circuit, what is the components that are doing entity tracking versus components that are doing arithmetic. I think those are exactly the question we want to ask with
pal.k: agents
pal.k: on language models. So imagine. And this is something we're trying to do now, it's actually very hard. But imagine that you'll have an agent that will
pal.k: be able to find like multiple circuit and say something about the intersection between them, and like like what sort of knowledge in sort is sorted where, but not according to like one or 2 times, according to like
pal.k: millions, not millions, but hundreds of tasks.
pal.k: Yeah, I think that's exactly the reason we need
pal.k: a way to automate these experiments.
pal.k: Is it a next step for Maya. Yes, it's an ongoing project. But like, if you, if someone wants to join us with that, I think we all the people are like super busy, and we are not making a good progress. So we would be happy to have someone enthusiasm like with enthusiasm to help us. Yeah.
pal.k: have you thought about doing something like the like, oh, one style
pal.k: like, you can create synthetic neurons.
pal.k: It's gonna train on those like, you have a ground truth. And so
pal.k: you can do. Rl on. Oh.
pal.k: in a sense of like evaluation, or in the sense of so my so training maya to be multi-step. Yeah.
pal.k: think.
pal.k: Hmm.
pal.k: So in some sense, okay, in some sense, Maya does do some sort of a multi-step reasoning, I guess. But like interactively with the model, and not like interactively, but like between, like multiple agents, which I believe that is what? Oh, one does.
pal.k: Yeah. So I don't know what one does, but I guess the theory is like.
pal.k: if you have some sort of verifiable output, you want to be able to sort of provide feedback. Oh, I see, I see, I see. Oh, I see. Okay, yeah. If you have the inquires like what should be the result. You can provide some feedback and do rl, and then train Maya to. Yeah, no, I didn't think about that. That's an interesting idea.
pal.k: so I think, like like, open a has a like reinforcement. Fine tuning. Api. Now, we're like, just do that
pal.k: and see if, like, we get a model that is just like a better mile. Yeah.
pal.k: cool. That sounds like of course, like an undergrad subject or something. Yeah, that can be like, we can try that.
pal.k: That's great.
pal.k: Now.
Nikhil Prakash: Wait. I had one question, sir.
pal.k: Oh, okay. One more.
Nikhil Prakash: Before we get it.
pal.k: Kiel once before we go ahead.
Nikhil Prakash: Yes, yeah. Can you go to your last slide?
pal.k: Last slide.
Nikhil Prakash: Yeah.
Nikhil Prakash: Soon.
pal.k: Well.
Nikhil Prakash: Yeah, can can you talk about? What do you mean by universal interpretability tools? Do you think the existing ones are like limited to a specific modality?
pal.k: Kind of yeah. I think that
pal.k: maybe people are now starting to talk about patching also for vision models. But like
pal.k: vision, people are talking in a completely different language when they're trying to understand.
pal.k: Why is that fine? They're trying to understand? Some sort of operation, I think.
pal.k: both in like, if we want to advance the technology or like advance our way to interpret models, but also in the sense of like
pal.k: building an interpretability community. I don't think it has like it has to be like that. I think
pal.k: we need to come up with like a unified
pal.k: language to talk about these models, and I think everything is going to be multimodel like in a few years. So there's like there's also no reason to keep it separated.
Nikhil Prakash: What do you think? Nico?
Nikhil Prakash: I mean in terms of the community wise? Yeah, I mean, I I don't
Nikhil Prakash: don't really know about the vision community. So I can't really say about that. But in terms of the actual tools
Nikhil Prakash: I mean, as long as we have vectors in the model. I think we can.
Nikhil Prakash: I think we should be able to use existing tools that we have.
Nikhil Prakash: That's why I wasn't clear about that.
pal.k: Yeah, okay, maybe universal is not.
Nikhil Prakash: Yeah. But this is my thought. I'm not 100% sure about it, either.
pal.k: Yeah, okay, maybe more automatic
pal.k: in general, like, you have my own but then, like, I guess there are different like ways to get me to move.
pal.k: Oh, I think it's more about
pal.k: unifying approaches rather than automating them. So
pal.k: multimodal or cross modal, maybe cross Modal cross modality. Yeah.
pal.k: Multimodal. Yeah.
pal.k: Interpretability.
Nikhil Prakash: Yeah.
pal.k: Yes, thank you. Everyone. You know. I think that what it'd be it'd be cool if and so
pal.k: you know. So thanks tomorrow. And then you're looking for feedback just on the structure in general. Right? If people wanna if people want to hang around and give tomorrow like talk feedback, then we can. We can. We can brainstorm on that. I'm gonna I'm gonna stop recording here.