David Bau: So I'm gonna stop recording here.
David Bau: Okay, so we're at our our new time presentation, you guys. So we're really lucky to have Elliot Horowitz here, who has been
David Bau: really agitating and pioneering and trying to get the community to pay more attention to this idea that I think that your name for it is you call it weight space, learning, learning, weight, space modeling, right? Which is, you know.
David Bau: you know, sort of treating the parameters of the model as a primary sort of data artifact
David Bau: as opposed to just the original data itself, which is really interesting
David Bau: perspective. And so we're lucky to have Ellie here today to talk to us about about his work in this direction, and so welcome, welcome, Ellie.
David Bau: Thanks and thanks for having me. Thanks for organizing it to be honest. And
David Bau: and yeah, oh, can we get this down? Yes, I'll get rid of this thing here so that you can see your slides. Yeah, perfect. Alright. And and just like as a
David Bau: yeah. So so- so
David Bau: this is going to be less technical than the previous one, and but I do have many technical slides. This is kind of an overview of this entire direction, but we can dive deeper into everything, and it's headed. And yeah, so whatever feel free to ask questions, there's a lot of technical depths just not necessarily in these slides.
David Bau: And so yeah, the idea of this talk. The idea of this entire direction is asking, What can you actually learn? What and how I'll go into? How as well can we learn from a million model millions of models. And just as a spoiler, what we're seeing here is kind of a visualization of hugging face, of
David Bau: down sampled version of hug and paste. There's about 150,000 models here, and where each node in this graph is a model, and each edge is a weight transformation. And I'll explain everything throughout the talk and feel free to stop me. And yeah, everything.
David Bau: And oh.
David Bau: is that stable diffusion there that you had on the right here, this this one. Yeah, this is flux. Oh, that's flux. And yeah.
David Bau: alright. So just about a bit about myself. My name is Liao. I'm always asked how to pronounce so. It's Eliahoo. But also Ellie is just perfectly fine. And Eli, and whatever
David Bau: sorry. Yeah. And so I'm a final year. Phd. Student at the Hebrew University, I'm advised by, and previously I worked a bit on image and video editing and some visual anomaly detection. So, as you can see I'm mostly originally a vision person. Some of my works do work on Llms as well, and a bit of an lp, but mostly vision.
David Bau: And so the idea is the following
David Bau: machine learning. In general, it's an oversimplification. But basically the idea is learning decision rules and statistical patterns from data. Lots of data. We do machine learning, we get some decision rules and machine learning research basically tries to develop methods and models to perform machine learning.
David Bau: And but we mostly kind of ignore this idea of, let's perform machine learning on machine learning. Let's let's try and apply this this scaling from individual samples and handcrafted and features. Let's try and learn from these things. So what my research does is developing Ml models that treat other Ml models as data and basically tries to learn from these. And then throughout the talk, I'll speak about what can we learn from it and how.
David Bau: But let's get the definitions out of the way. So the idea is I, I look at model populations, model collections. And
David Bau: I model this as a graph specifically a directed cyclic graph, a dag. And in this dag every node is a model. And so what we're seeing here, every node here is a model, and every edge represents. So this is a directed edge. You can see there's a small arrow. I don't know if I can see it on the screen. But yeah, and a directed edge indicates a weight transformation from a parent model to a child model.
David Bau: Right? So that would be like fine tuning. And Laura something back.
David Bau: Now we also have edge features. So each one, each component here has an edge feature and a node feature. So edge features represent the transformation of the weights from one model to the other. These can be. And the optimization hyper parameters, the training data, the transformation time as fine tuning Laura quantization. Basically, if we look at it like as a state graph.
David Bau: This is anything that takes us from one state, from one to the next state. This is the action we do when fine tuning. This is everything we do, and when when doing a gradient distance, step
David Bau: and node features, I like to categorize it into 2. We have traits, and we have metadata. So traits are the functional things that the model has learned. These are attributes. These are and the biases or robustness of the model. Basically, everything the model has learned
David Bau: and metadata. I like to think of it as the descriptive things. So when the model was created, how many downloads this has anything you'll see at a data repository on your hard drive.
David Bau: nothing related to the actual training
David Bau: and right, just a few more important notes. And when visualizing these model atlases. There's actually no importance to the spatial location. So if we look at the X and Y axis of this, because this is a dag and not necessarily a planar graph. Then we need to lay it out somehow into 2D. Space.
David Bau: And the way we do it is I use currently in my work. I use off the shelf layout algorithm for graphs. That's a whole topic of study. And I just take an off the shelf algorithm, and that usually what it tries to do is it tries to minimize the edge crossing between things. So we have many crossings. It's unplanar. So it tries to minimize it. And it models the distance between things as like springs and and and yeah, so
David Bau: it doesn't mean anything. The fact that this node is close to this one apart from the fact that there's an edge between them. But this node could have been here or here, and it would have wouldn't mean anything. And
David Bau: yeah, at the end. I'll speak about future directions. And one is to bake into this also. Spatial location and stuff like that.
David Bau: And
David Bau: the the model athles, as I'll show, is a very powerful way to represent and and visualize model collections. I'll show the use cases of it, and also a small side. Note. It's easily extendable. So I spoke about the node features and end features, but the idea is that anybody will be able to add more of these features. I think I would like if.
David Bau: as an analogy, if we think of Google Maps, we have the actual structure, the geography of of the world. And then we have stores, reach out more things and opening hours of the stores and and reviews and stuff like that. And the idea is that, for example, an interpretability group will be able to add their own layer of their findings, of their circuits. They found of the biases they found whatever they actually find.
David Bau: and these would be additional features. And hopefully, in a collaborative way, everybody will be able to map these
David Bau: all right. But there's a big problem with the Atlas. And here I'm focusing specifically on hugging face. But that's not the only problem with that. But if we look at hiding face, the Atlas itself is incomplete. We're not actually able to recover it to build it. It's a bit like we think of the family tree, Darwin's family tree of life.
David Bau: Then we know there exists some relation between all species on earth. But we don't necessarily know this relation. So we need to actually map these relations and map this human genome and everything. And similarly, here, we don't know the relation between models. We don't know what they do. We don't know their functional traits and everything. So in Hanging Face, over 60% of the models have no documentation whatsoever.
David Bau: and the remaining ones have very, very partial documentation. So I wrote here a few things. But, for example, the accuracy, only 13% of the remaining less than 40% and license information only about 8%. So documentation is very, very sparse.
David Bau: and what you'll see throughout the talk I present many visualizations of the access similar to the galaxy in the beginning.
David Bau: and they use the either the methods that I use to recover the graph or the existing metadata. But they're very, very small portion of all the models that actually exist.
David Bau: And so if we go back to this galaxy, there's no legend here, just because I wanted to be cleaner. But what we're seeing here again is all the the hugging face universe and
David Bau: the colors and codes. So these on the outskirts of this galaxy are models with no parent. So we don't know who's their parent model, who's the like. We don't know their lineage and the visualization algorithm because it models things as springs and interactions basically pushes them to the outskirts because there's nothing to pull it closer. And these other bearings are smaller and model families, 2, 3
David Bau: models and a family again, just because they're missing documentation. So I went over a few 100 of them. They do belong to some tree, and maybe sometimes manually. You can do it many times. You can't do it manually, just because it says
David Bau: if you read the documentation, it says fine tune for Mistral, and you you have no idea which mistrel. There are 50 different types. So you need to actually connect the lineage. And you don't necessarily know what then you have in the middle these huge family trees right. This huge one over here is contains about
David Bau: 200,000 models in this one family tree, and people merge them, and they're Llms. And there you have. Mistrel, like some parts, are mistrel, and some llama, and some parts are queens, and just a big salad.
David Bau: and but because of the missing documentation, most of the models are what I call lost models, so they're effectively unusable, either unusable because we can't find them or unusable because we don't know. There are several applications we would want to perform with models which I'll show. I'll show in a moment the motivation for this, but
David Bau: we can't perform them, because we don't have all the information.
David Bau: and something to know is that like, sometimes people ask, well, aren't these undocumented model mostly niche models, and not necessarily so you can see on the outer ring. And well, yeah, I don't have here the legend, but the size of the nodes indicates the number of monthly downloads and and the larger ones are usually over 10 million, and sometimes over 100 million monthly downloads for the model for one model.
David Bau: And so you can see very large models on the outskirts even, but with no documentation towards your lineage. So it's not necessarily niche models.
David Bau: And one last important point is that this is not only true for hiding place, but also for private repos, so we can think of your cluster here, your infrastructure, your hard drive at the university.
David Bau: We all train models, we all fine tune models, and
David Bau: and these are also usually undocumented, like, I don't know about you, but I don't document every aspect of my models, and usually I'll have thousands of checkpoints, sometimes a trajectory, and I'll just document the last one. But maybe there's a better model some somewhere throughout the trajectory, and we don't evaluate them every step and stuff like that. And it's true. So it's true for private research institutions, but also for companies, for, like
David Bau: I, I did an internship at Google, and I trained some model, and it was great. But then I'm sure nobody's using it now. 3 years later, and it could be very good for some use cases sometimes in the future, and nobody will ever know about it. But it's it's like something
David Bau: alright, so
David Bau: the talk from now will be broken into 2 parts. The 1st 1st one is the motivation into why would we even want this model, Atlas? And so I'll go into more nuanced details about the actual use cases. And the second part is basically how to recover this structure, how to deal with the fact that we have over 60% of the models with no documentation whatsoever, and the
David Bau: the rest of them are missing large parts of the documentation. So the the actual research part would be the second part, yeah.
David Bau: Yeah. So the second part is methods for constructing the atlas and constructing meaning. The the note features, the edge features, or the actual edges, or the actual notes. And
David Bau: all right. So the motivation is, what can we actually learn from the 1 million models, and maybe beforehand is why now, why do we even want to do it now? Why couldn't we do it? 7 years ago, 10 years ago we had models for 1012 years, and like good models.
David Bau: And the reason is, there's this explosive growth in public models. So this is like the the last 3 years on Hanging Face, where you can see this is the number of monthly downloaded models. So it's over 140,000 monthly uploaded models to hang face in September. And currently, we have over 1.7 million models. So it's and
David Bau: I submitted to neurips a year ago, a paper. And there I wrote, there's currently over 500,000 models. So and then the course of a year, we've grown 1.2 million models on.
David Bau: And
David Bau: all right. So what can we actually do with them? I like to break it down into 3 main things. The 1st one is what I call Metaml research. I think this is a very exciting idea, which is, we have those graphs I showed before. And we have those 1.7 million models.
David Bau: Let's start learning from them insights. Let's start understanding how community different communities perform what they do. What
David Bau: are they? Different kind of social science of machine learning?
David Bau: The second idea is model discovery, basically just understanding. What are these models? How can we use them? This is kind of a follow up work to one of your works, I think, with a content on the base discovery I think it was called. And and the 3rd one is model forensics and model forensics basically means, how can we recover this trajectory? And and yeah.
David Bau: parts of it are outlined in my recent paper. But also a lot is outlined in Koyana's paper, right model lakes. And so so there is kind of an intersection.
David Bau: And
David Bau: alright. So let's start with this graph. Okay, this is kind of similar to the galaxy we saw before. But here it's more organized.
David Bau: So let's start parsing it slowly together.
David Bau: So what we're seeing is, I took a sub sample of all of hugging face, all of the information I have supervision of on hugging face, and I broke it down into most popular model families, according to
David Bau: just size of the number of members in the family and then modalities. So on the top. Here we have computer vision. Here we have audio here at the bottom, we have Nlp, and here, at the very left, we have a small area of vision and language models.
David Bau: What else can we see? Okay, so we have different connected components. So connected components is just a foundation model and all of its descendants the lineage on the graph. So the graph I showed before that that has multiple connected components. And and they're disjoint all many of the times and then
David Bau: size indicates monthly download. So we can see the very big ones are indeed over a hundred 1 million monthly downloads. So, for example, this one.
David Bau: and then the color indicates the relationship between the parent and the child model. So how was that way transformation performed? Is it a full fine tuning? Is it so? Adapter is that scion? And these adapters are usually Laura? Not necessarily. But Laura is so popular that it's like 95, 98 of the cases. It's Laura's. And quantizations are these purple ones
David Bau: and merges are these blue ones. And yeah, the oranges are roots. So let's start understanding the transit start performing this Metaml research I spoke about. So I think the 1st thing we can see here. Probably the most prominent is that at the bottom
David Bau: the structure is much, much more complex. Nlp is extremely more complex than anything else we're seeing here, and in fact, the bottom is a single connected component and the connected component of metal. 3, 8 b, and so that's the foundation model. And on the top. We have many different connected components, but the structure of each one of them is is much simpler. Right? And so, yeah, the structure is is much more complex in Nlp.
David Bau: Another thing we can see is that quantizations are much more popular in Nlp than anything else. Basically, right? If you look at these here. I call it the quantizations in red. We can see many more of these red nodes in Nlp. Than in anything else, and
David Bau: I don't know why this this whole idea of the Metaml research is basically to to surface these, these phenomena, these questions and then try and figure them out in this paper. I didn't figure out why, but just kind of resurface them and and get the conversation going.
David Bau: So one maybe trend shift we're seeing is that there's this point here. This is the cluster of flux. Flux is
David Bau: I think, but definitely when we wrote it. But I think today as well, the largest generative model for a vision. So it's 11 billion parameters.
David Bau: Maybe maybe there's a possibility that we're starting to see a trend shift in computer vision where we. We're getting large enough models. They're starting to be at the size of Llms that we should start quantizing models and then understand how to take ideas from Nlp and transfer them to to computer vision.
David Bau: Yeah, I was, gonna say, I wonder if it's because of the size. So I don't know. I don't have the answer, but we do see that like one of the biggest ones is that. And, in fact, there's another large one here, which is a fine tuning of clip.
David Bau: But I went over because that was like an outlier. And I said, Clip, is that doesn't align with my hypothesis, and that's a mistake in the ground truth of hiding place. It's actually a fine tuning, and they recognize it as quantization. So it's a mismatch label.
David Bau: Very naive question, what's a merge?
David Bau: Alright. So merge is this idea. It's it's a new thing, or relatively new. But the idea is, we just
David Bau: take the weight of 2 or more models, and we combine them in some way. So I think the most common merge is exponential moving average right? Like, usually you train a model with it, but people don't actually think of it as exponential as a merge.
David Bau: So what they do usually is, they'll take 2 Lauras, for example, and they'll just take the weights and and combine them like either a simple average or a weighted average. And but then people do all kinds of really crazy and
David Bau: and cool things like that. Some people take
David Bau: 5 layers from llama, then 3 layers from Mistrel, but on top of it some layers from, when fine tune the entire thing, for I don't know half an epoch, and and somehow it works and and it improves so I'll I'll show trends in merge as well, because I think that's a very interesting thing, and I don't get why it works in many cases.
David Bau: so another, so they shouldn't. They shouldn't. I'll go into it at a moment. They shouldn't share the same geometry, and that's why I don't think it should work. There's this entire idea of permutations which I'll go into in a moment.
David Bau: So the next next trend we can see is this idea of adapters. And
David Bau: so, as I said, adapters are ug, Laura, and what's cool is, we can also kind of infer trends throughout the temporal timeline for this. So if you look at markers one throughout 5,
David Bau: basically, one is stable diffusion, 1.1.
David Bau: This is stable diffusion, 1.4, I think. Stable diffusion, 2. Stable diffusion, excel, and flux.
David Bau: These are, and the order they were released is one throughout 5. So we can see the progression of actual adoption of the technology of adapters throughout time. Here we can see that at the beginning they used full fine tuning, and over time it went and became more popular. And another kind of interesting thing we can see is that all of these on the very top are discriminative computer vision models.
David Bau: So there is this this split between generative models which use Laura's and discriminative, which don't
David Bau: maybe size again, I'm not sure.
David Bau: Size may be overfitting.
David Bau: Not sure. But you can see that computer audio almost never uses adapters. And then there's the finally, the final thing about model merges, which I think is the most striking difference here.
David Bau: Nlp uses merges all the time. These these models are very popular oftentimes. So
David Bau: you can see some of these are actually quite large. Have over 1010,000,000 monthly downloads. So people are merging and people are downloading them. Some of the merges are extremely complex, so I've even seen merges with over 18 different models. So we'll just take 18 models on hanging face do something, some black magic. Many of these people are like hackers on Hanging face, taking off the shelf things and go into many people. Go into Llm. Leaderboard.
David Bau: Just pick the top K. And start playing with them until they get this recipe of a new model, which is which which achieves the best score on Llm. Leaderboard.
David Bau: And but nobody almost said nobody's doing that in computer vision and definitely not in audio and vlms. And so again, not sure why not sure what's the importance and significance of merges? But we can see it's definitely something that's done.
David Bau: And maybe we should transfer it to computer vision and other communities.
David Bau: And
David Bau: yeah, that's basically the idea of this Metaml research. And why we have these knowledge and and for graph purposes, presumably I would have 1818 edges pointing at my, yeah. So
David Bau: Frankenstein model here, you can see this one. I don't know if you can see from there. But even here, all right.
David Bau: And this zoom in, you have here a model with many incoming edges. And yeah, it's not showing with high resolution on the screen. But these are like a vectorized figures. So you can zoom in indefinitely
David Bau: and like in a paper system.
David Bau: So I understand the motivation for this. But I'm also curious
David Bau: like for you as a researcher like, what made you want to try this?
David Bau: So I'll present later. There's a paper which I just try to recover the lineage because
David Bau: I was interested in whether the weights of the model encode the information about the parent. So, given a model population, can I just
David Bau: build the family tree. And initially, I thought that this is a tree and auto tag. So just a bunch of like a forest, basically a bunch of trees.
David Bau: and then only around January. So after I already had a paper and a follow-up paper. On this I scraped hugging face, and so hugging face added supervision on the lineage, and after we published the 1st paper, they added supervision about whatever they had, and
David Bau: and only around January, scrape the face for all the model parts and the details. And I was able to build this graph. So I was like, Whoa, this is. This is so much more complex than I ever imagined I I was shocked by like the fact that we have this complex piece in the Nlp. And
David Bau: here I chose to display just what just one connected component, which is llama, but in reality the connected component of some of the llama variants, and the queens and the mistrels are all in the same connected component, because someone took a llama and a mistrel and combined them, and then somebody even took a bird
David Bau: find it somehow, which it should should not be related at all. And so they're all together in this mix. And yeah, just a cool Easter egg from yesterday.
David Bau: Apparently Andrew Karpathy shared like a crop of this and his presentation in the Llm. Not Y combinator
David Bau: and summer school, or whatever. So that was cool. Yeah, my phone was blowing up like people saying, Yeah, isn't this your figure? So that was the case. Thanks.
David Bau: Alright. So the next question is.
David Bau: or the next task is model discovery. And the idea here is that training is expensive. It's time consuming. It's bad for the environment. And we have these 1.7 million models.
David Bau: Let's reuse them right, most of the like. For most of our use cases. We already have an AI for that.
David Bau: and let's use it. And even if we don't have an AI for that. Let's other either fine tune, one or a merge one.
David Bau: and then the next question is, wait, but we can already do it right. We just go into hugging face. There's a set search box, and we can do it
David Bau: well, not necessarily or no, we can't do it. So here's an actual example from 2 days ago. So we have over 1.7 million models on face we have over. We have tens of thousands of models which descend from Imagenet Imagenet. I don't know if everybody's familiar with but 1,000 classes, about 1 million images, 1.2 million images, probably the most popular data set for computer vision.
David Bau: And yeah, so tens of thousands of models, either pre trained on imagenet or fine tuned from these models.
David Bau: Ideally, we want to be able to just search for a specific class which appears in Imagenet and get all the models which know how to classify, track, generate, do anything with any of the classes in Imagenet.
David Bau: So I I took this example of searching for a peacock. That's 1 of the Imagenet classes.
David Bau: How many, how many results do you think we'll get here?
David Bau: 99,
David Bau: we get 99, not 99,099 results. And the reason is that Hunting Face is using text space documentation. So it just goes to the model part, searches for it. If it's up there, you can't do anything with it.
David Bau: And and it's insufficient because nobody's documenting all the classes they trained on nobody's documenting all of the biases they have in their data, nobody's document, even if they were to do everything, all of that. Nobody's documenting what their parent model did. Right? So even if you do the best job possible, you still don't document what your ancestors did
David Bau: so to actually have search. I like to think of it as we need to know what the model does that's knowing the capabilities, knowing the performance of the model. So how will it performs this? This gives you like a filter and sort in terms of search.
David Bau: and then the history as well, because then, you know, like the biases that the model inherits
David Bau: alright. Final use case is model forensics. And the idea basically is that we have all these models.
David Bau: We want to know their training data, their biases, their lineage, everything that's important for a bunch of like intellectual property, compliance, bias, estimation, and even reproducibility. So we want to know how Llama was trained. If we have the entire lineage and we have the edge features, let's see, and we can recover them.
David Bau: We, we can just train our own lab alright.
David Bau: So now is the second part of the talk, which is, how can we actually learn from a million models. So we saw why we should learn hopefully, I kind of convinced you that it's important, useful, something worth doing. So how? And
David Bau: so here, here's this notion of white space learning.
David Bau: Basically, the idea is.
David Bau: and we have an In the wild model we train some meta network that's the name we use in this field, and it's it's a it's a new field, it's growing. We just had them
David Bau: 1st workshop on it and Iclr a month and a half ago. But it's growing. We had a nice number of submissions. So you train a Meta network on it. You get some model encoding some representation. Basically, you do representation learning on models as their input and then you do some downstream tasks.
David Bau: You train the Meta network to predict the performance of the model. You train to predict the biases. What! The model? Who's the parent of the model? Whatever? Any any one of these tasks we just spoke about about the athletics
David Bau: and the way that this even makes sense the way you can actually take the weights of a model and extract what the model does the performance. Everything is because the formation process of a model is, we have the architecture, objective data, and all these things, we optimize everything, and we get the weights.
David Bau: And at the end of the day. Everything the model can perform, every everything the model knows is a product of these weights. So
David Bau: basically, anything, anything we know how to do with a model is contained in the weights. We then upload the model to a model repository that this can be our hard drive or face.
David Bau: and then the idea of weights based learning is to learn this meta network on that model population.
David Bau: all right. But weights based. Learning has 2 very large issues 2 very large challenges. 1, st one is that weights are extremely high, dimensional. So if we look at an image
David Bau: and 2, 2, 4 by 2, 24 by 3. That's like the standard size of computer vision models is about 150,000 parameters. Small resnet is 25 million vision transformer 87, and then Lm is often billions of models
David Bau: we now need at each of our samples. So if we're training on a data set of just 5,000 models. Each of the each of our samples is 1 billion parameters in length.
David Bau: And even more challenging problem is the idea of neurosymmetries. So the idea of neurosymmetries is basically if we take a neural network and we just permute their weights. The functional properties of the neural network stay exactly the same. But the raw representation change drastically right? So if we look at this example here, we can think of it as a like a 1 hidden layer. Mlp.
David Bau: All we did was change the the like, the blue one over there, the yellow one over here and the green one there, and and the the model will say exactly the same, because the weights are exactly the same. We just renamed basically, we just renamed our nodes in the graph.
David Bau: But already the representation will be drastically different. So then, we need to bake into our neural networks, into our meta networks. This idea of permutation, equivariance, something that's usually done with gnn and stuff like that.
David Bau: And then we need basically a neural network that knows how to process extremely high dimensional data.
David Bau: N is permutation equivoryant.
David Bau: And frankly, it's not going to work. And so we have this community, and people are doing it. And people are not really scaling. And beyond learning on neural networks, which are a few tens of thousands of parameters at most, and small toy problems.
David Bau: And and it's great, it's doing great. And people are improving. And they have these cool methods. And maybe one day it's going to scale. But currently it doesn't.
David Bau: So the idea of the research I'll present here, my research basically is, let's try and avoid these weight permutations altogether. Let's try and find these priors within space within the weight populations where we can just circumvent the idea of the permutations, and then maybe we can do other things. Maybe we can use off the shelf networks. Maybe we can do other cool things.
David Bau: So this is kind of an overview of of my works over the past 2, 3 years. The works, some of them are mine as a 1st author, some of them. I'm a second as 3rd author. But yeah, we have basically 4 areas of of work. So the idea is, we have recovering. Everything is recovering parts of the Atlas, which we're unaware of. So over here, we have works on recovering the Atlas structure
David Bau: and the actual. So so that's like lineage prediction. We have works on recovering node attributes to the performance the the what the model does. We have works on actually recovering weights so actual, real weights. This is this kind of connects to what what was presented beforehand, and then we have one about recovering the edge features.
David Bau: and also our work falls into either working directly within white space or working on so probing, working on the functional properties of the.
David Bau: And I'll present like a few of these works, and, as I said beforehand.
David Bau: I'll present the key idea of each work. There's depth to it, and we can dive in. But I'll give like a a the key idea of each one of the works. And and
David Bau: so the 1st 1st one is recovering the Atlas structure. The goal here is very simple. We get as a input a set of models, their weights basically. And the output is to recover a graph basically recover this structure of the graph. So the input is 100 models. Let's say we have no idea what's a relation or anything. And we're trying to recover this graph.
David Bau: We're not trying to visualize the graph here, it's just actually building the the actual graph meaning, what are the edges that connect models? So for this, we need to do 2 things. We need 1st of all to determine whether 2 models are related, whether there's an edge between them. For example, this. And then we need to determine the direction of the edge. That's if the model is a parent model or a child model.
David Bau: And to do that, how much time do I have?
David Bau: Cool
David Bau: you still have. You know, 10 min, 10 min. Right? Yeah. So to do that. And we we need to recover the structure, the direction
David Bau: and what
David Bau: the idea in this work is basically to use very simple weight statistics. So there are cases there are statistics where we won't run into the problem with permutations. And then we can just use these statistics and hopefully
David Bau: learn something. And
David Bau: so in this work, we first, st as I said before, initially, we thought that these are trees and not dags. So we thought of it as trees lead. And what we found is that if you just use the L 2, it's very correlated, the L 2 between the weights. So we do it on a per layer basis, and let's assume for a moment that all the models are in the same architecture, because if they're not, it's very simple. You just know they're in disjoint trees. So that's not an interesting case.
David Bau: But if they're the same architecture, they can still be of a different origin. So you can take 2 2 transformers with exactly the same architecture.
David Bau: initialize them differently, and they'll be of different lineage.
David Bau: and indeed it correlates very well. So the the actual distance between weights correlates very well to what they to their distance on the graph.
David Bau: But then we need to determine the direction. And what's cool here is that we found that if we take the cortosis of the weight, the cortosis is the 4th moment.
David Bau: then the cortosis is a very strong indication towards kind of the seniority of a model. So which model came before which one.
David Bau: So here we have, and this is during pre training, and this is during fine tuning, and we see that during pre-training the Portosis increases, and during fine tuning. The Portosis decreases.
David Bau: and we don't have like. I I think this is a very cool result, and we don't have an actual proof for why it happens. But we do have a hand, wavy explanation which
David Bau: maybe you'll find interesting as interpretability people. So the idea is that the model a cortosis is kind of a
David Bau: a proxy. Usually it's either proxy for how Gaussian is a distribution, or it could also be thought of as the heaviness of the tail, and how how many like a metric, for how many outliers there are a way to quantify the number of outliers.
David Bau: The way I like to think about is that we take a neural network, we randomly initialize it.
David Bau: and then we're starting to train it to generalize right? To be very, very general.
David Bau: So we need to capture all these areas within within the domain within the data. So we need all these outliers. And then we see, indeed, an increase in the number of outliers.
David Bau: And then when we're fine tuning, when we're becoming more specialized, we're basically looking at a much narrower thing. And we we now need to span less of these outliers, less of these regions, and this is why it decreases. And so this is my hand, wavy intuition. I don't know if it's true. But I think it's a nice story. And and then we take both of these together. And we build like an optimization algorithm. I won't show it here, because I think
David Bau: this is the main idea of of that paper. So we build in. We take it off the shelf and
David Bau: minimum spanning tree algorithm. And yeah, so I know when you had this paper, you were thinking in terms of tree but like in the case of like model merging where, like, you know, to model that. So so that doesn't work. Here, we we
David Bau: we thought about the idea of model merging when we wrote this paper, but we weren't
David Bau: We weren't aware of how prevalent it is because we didn't scrape. So in the follow up paper just next slide. We did account for merging here. What I can tell you is that we did like an ablation for one of the reviewers asked, but if you take the l. 2 distance between models, and then, if you also take the cosine
David Bau: and between models, then if if 2 model, if a model is merged from 2 parents. Then you are able to kind of detect which of the 2 parents it was merged from based on that cosine. I don't remember exactly what we did there. But we had there something that showed some indication. But it wasn't perfect in a way.
David Bau: And yeah, but here we we just yeah, we also call the work model tree Heritage recovery. And so it was
David Bau: based on trees. And so the next idea that's the next paper is that
David Bau: this is great. But, as you said, it doesn't work for model merging, and it doesn't scale. Necessarily, it doesn't necessarily scale for like huge things. And also, by the way, I'm not going, not always going to show results, just because I think tables are often boring because everybody makes sure they're the balls at the bottom. So the results are there. There's tons of results in all of my papers. And like.
David Bau: Yeah, you, if you don't take my word for it, just go look at the paper and results that there are tons of graphs, tables, whatever, and
David Bau: reviewers found it enough.
David Bau: Alright. So the idea here is that if we want to actually scale the idea of recovering the structure.
David Bau: What we did here is we we looked at information which is always present. So I said, hiding face.
David Bau: A lot of the information is missing, but some of the information is always present, no matter what, and I'll show in a moment to 2 examples for that. And then we also looked at real world. Prior. So we kind of looked at cases where we saw we did this Metaml research, and we saw patterns in hanging face, and then we checked if they generalized to our case, and they did so. One example like that over real world, Prior, is the idea that
David Bau: quantizations in practice are so quantizations are are near duplicates. Right? It's probably going to be the nearest neighbor of the current model. The quantized version of it. Unless we're doing like an adversarial thing, it's a nearest neighbor, because we're just changing a very small rounding off error.
David Bau: So if we treat quantization as leaf. And we can already save like
David Bau: compensate for many of the errors that were introduced because of having these near duplicates within the optimization algorithm.
David Bau: And indeed, we checked. And in 99.4 1 of the cases on all of the data we had on aggregate face. This is actually the case, and quantizations are leaf nodes, and then the performance of the recovery of the structure improves by 1015% whatever. And so this is already an example how we can kind of use this idea of Metaml research to improve the method.
David Bau: We had our initial method, which I showed the 2 slides beforehand, and here we improved it just by looking at insights from from the actual structure, and something that, by the way, would be
David Bau: harder to think of without having the visualization. So, of course, at the end, what we did was just run it like a like a small query on a data frame. Of all the data, but we had to visualize it, using this visualization and see, wait a moment. There's like we can actually see a very clear pattern of this right. The the
David Bau: the purple ones are almost always on the outer skirts. And so we looked at a few of these. And we're like, Wait, there's there's a pattern. Let's see if that's actually the case. That's harder to do, I think, without visualization. I think that's the power of visualization.
David Bau: The next thing is that to determine the direction
David Bau: before we did the idea with the cortosis, and it works nice. But actually every model
David Bau: has a timestamp, either when it was uploaded, or when we saved it to the disk.
David Bau: and we actually found that in most cases the timestamp is correct. So you do have these pathological cases where someone uploads a model earlier than the other. And we found in some cases but 99.7% of the cases doesn't happen.
David Bau: So in this paper we use like 4 or 6 of these similar priors. And and we changed the optimization algorithm. So now we account it's no longer a tree, and we did something greedy, and and then we can account for a dag and not a tree.
David Bau: and then it works, and then and that was it? And
David Bau: so now I'll go over to the next category of methods. That there's a single method here. This, I think, is is a really cool idea which I think this is my best work. So far
David Bau: the idea is the following, so public models are, as we saw in the presentation beforehand.
David Bau: They're they're often risky. They're they're cool, but they're risky. I don't need to tell you. And as Andrew spoke about their first, st you 1st do like unsafe training and then alignment.
David Bau: And the idea here was the the question I tried to ask is given a model or a family of models. Can we recover the weight, so do the unalignment, the the reverse engineering. But there's a very important part, and is I wanted to recover the weights, the actual weights, not not try and unalign it for a specific set of words, or curse words, or or I don't know unsafe behavior, but
David Bau: way to where like value to value reconstruction of the weights. And I think it's interesting, because then you're not limited to recovering the distribution of your unalignment and data. But you just you're recovering everything the entire capabilities.
David Bau: So yeah, here's an example. But I don't think I need to tell you that.
David Bau: And the line models refuses.
David Bau: And the idea here is to just look at the mathematical properties of the weight formation process and try and model them, see if we can extract briars and then solve it somehow in a good way, in a way that we're reducing these symmetries.
David Bau: Okay, so quick background and foundation models, we 1st pre train we then fine tune. And so one example. So
David Bau: in this paper, we're focusing on the viewpoint of a single layer, and we just do it for all layers independently, and it works. So if we look at a single layer, we call it W. That's a Pre. We call it pre fine tuning, this idea. But look at W. And W is size K. By D,
David Bau: and then, if we fine tune it, we get W. Prime, and which is the same size. But we can actually think of the W. Prime, the fine tuned one as W. Plus a delta. W. Delta W. Is the update to the weights during gradient descent
David Bau: after N steps.
David Bau: And then there's this problem, that fine tuning is costly and then and tends to overfit. So there's this observation which the weights and that change the Delta W is low rank, not my observation. This is the paper named Laura.
David Bau: and basically what they do is they factorize this Delta W. Into 2 smaller matrices, and which costs much less parameters. So that's Dr. Plus rk, and R is usually very small.
David Bau: and it works great. It's state of the art, I think, and or maybe not state of the art. But every everybody's using Loris nowadays
David Bau: there's something cool with Laura is that you don't need to necessarily introduce additional parameters to the original model. So Laura has this property where you can merge the fine-tuned model into the original model. So you take B and A, you multiply them. You get M. And then if you just sum them together, you get W. Prime again.
David Bau: and you've introduced nothing, so you can keep them decomposed, and then you have those lower weights, and every time you need to to forward pass through them, and it costs more, and it costs more. But if you merge, you don't need to do that.
David Bau: And actually, every method I know of currently assumes that you cannot do this decomposition. So this is like a source separation problem of taking the W. Prime and decomposing it into W and into the residual of the Laura, the Delta W, basically, which was the Laura and everybody's assuming it's impossible.
David Bau: And our question was, is it actually impossible? Can we actually reverse it?
David Bau: So let's look at the prone definition of this paper. The idea is that we have an attacker. They have access to N different fine-tuned models.
David Bau: and there are 2 important things. So 1st of all, all, the models used Lora fine tuning, and obviously they use their merged weights, otherwise it's trivial to solve it. But all the models use Laura, and all the models use Laura from the same models, so we can think of it as N. Models which all fine tune from Lama, for example, and under the hood. What's invisible to the attacker is that each of the models has a shared W
David Bau: right? So we have a shared W and a different residual BA. And the goal is to recover some W. Star which tries to approximate W. Prime up to Epsilon.
David Bau: so an intuition I won't dive into the whole method. But I'll explain the intuition and show the objective that we have. So the intuition for why is it even possible? Is that each Laura adds, Dk equations. So after n, we have dkn equations, but it only adds Dr. Plus rk parameters. So this is over parameterized by like. If we add more and more Laura's. We're only adding the number of equations, and they grow much larger than number of parameters.
David Bau: And so we can probably just solve a system of linear equations and solve it.
David Bau: That's what we do. We? We use this very strong regularization on the, on the formation process of these matrices, which is that the rank of mi. The residual
David Bau: is smaller than some r, and then in a paper we also relax it to have different r's, and we don't know R in advance. So basically, you don't need to know anything about R, but yeah, and and that's our optimization. So the idea here and and we solve it with no deep learning, nothing just linear algebra pretty cool. And so we need to optimize N plus one matrices. And because we have Nm's
David Bau: and the plus one is the W. We're actually trying to recover. And then the way we solve it is similar to em. It's kind of a it's what's called coordinate descent. We freeze all N, and only look at one matrix at a time. And then it's a convex problem, and we can do it iteratively.
David Bau: and it's it's very cool, because it converges very fast. It takes us
David Bau: less than 5 min on a very small gpu on on a single layer, and then we can just distribute every layer independently. So we need don't need to do it throughout the model.
David Bau: and it works exceptionally well. I think this is in terms of results, the best results I ever got in the paper. So on the left we have here stable diffusion, and with as few as 5 lawyers, fine-tuned from stable diffusion, we have a basically the same thing. So I.
David Bau: So here, I'll I'll show you. Walk through the example. So this is just taking a single laura, and and the seeds are fixed. So seeds are fixed. Yeah, every column has the same seeds. So if we just
David Bau: generate from one Laura. We get this.
David Bau: If we then take the mean, this is one of our baselines, and so we take the mean of all the Lauras. And the reason why this is a baseline which makes sense is that if all of them have learned a 0 mean thing, the problem is trivial, but it's not. We can see it's not trivial. Then our result is almost pixel to Pixel, perfect. And actually
David Bau: the only change I found, and I went through about 100 examples. Is this, here we added a tail in here in the original one didn't have a tail, so you can see that even a tiny shift in the weights is very important, and we have numbers for for this paper as well. And actually, it's going back a bit to what Andrew said before about the fact that fine tuning doesn't change the weight much.
David Bau: it's true, but I think the scale really matters here. So it's shame he's not here to discuss it, but
David Bau: the convergence we got in this paper in terms of a weight reconstruction. So l. 2 l. 2 after log log base 10 for for averaging issues. But the the distance we got is basically E to the minus 1617 18. So arbitrary precision, almost right. And the baselines like
David Bau: even this baseline from from the original one was E to the minus 8 or something, so weights are very similar, regardless. You do some fine tuning way too similar. You take the mean, it improved a bit. It was E to the minus 10. Right? Yeah.
David Bau: Do you try using your exact same method
David Bau: on populations that didn't restrict themselves to Laura. Yeah, it doesn't work. It doesn't work at all at all. Nothing. I mean, yeah. You could you go from
David Bau: E to the minus 9 to E to the minus 10 or something. But it doesn't work. It's not much better than than yeah. Just taking the mean baseline, for example, and averaging exactly. And the the reason, I think, is that we don't have there this very strong parser, my entire optimization, our entire optimization relies on this, the rank
David Bau: right? And do you need to know the R. No.
David Bau: no, so so we we don't need to know the R and the R can vary between each Laura. So initially we we in the initial paper we needed to know the R, and it had to be fixed between all of them. And then the reviewers asked, and we were like, Yeah, we can do it. We knew we couldn't do it. But you just so
David Bau: to figure out, are you just take the difference between every pair of models. And then you get a system of linear equations again. And you can just use. Like, yeah, very simple. It worked 100 accuracy. 100 of the cases I tried like
David Bau: searching over thousands of options, and then to make sure that we have varied our we have here. I didn't explain, but we have, like a rank scheduler. So throughout the optimization, we similar to like a linear learning rate scheduler when we decay it, so here we increase it. We start with a very low rank, and we slowly increase it. And this kind of
David Bau: guides us towards the the correct solution. Because if we're if we have a very high rank initially, then it kind of can go anywhere. So if we have a low rank, we kind of guide at the beginning towards a good solution, and then and then we can increase the rank, and then we just introduce a set of different rank schedulers. And it works.
David Bau: And yeah, and with Llm, so here we did it with a mistral. These, by the way the stable diffusions one are in the wild. I just download them from like a repository, and so it's not even models I trained here. I took a mistral, and I did
David Bau: supervised fine tuning, and then Dpo, and then try to reverse both steps at once, and I don't remember the benchmark I use, but I just use the the one from Anthropic which does like a
David Bau: alignment. I don't remember the name of it. I chose the 1st 100 prompts from there, and in 99 out of the 100, I think, 98, 99, it was actual word to word reconstruction. So we have the metrics, and we measured semantic similarity and stuff. But you can actually look at one to one reconstruction.
David Bau: and it worked so over to the next idea,
David Bau: which is recovering node features. So this is a step towards the model search. And the idea here is that we want to find the most suitable model for a given task.
David Bau: and, as I said before, it requires knowing what each model does and how well it performs.
David Bau: And so the 1st idea is that
David Bau: we spoke about the weight permutations. But actually, it only matters if you're looking in the weights of the actual model. But if you look only at the input and output. There's no permutation. So you can then use probing this idea, which I think you guys use all the time and but borrow it to our use cases here. So what we do is we just try and learn. So yeah, the the most simple approach is to just take us
David Bau: a number of random number of randomly selected images, probe them like, pass them through the model, get the responses, and then learn another network over the responses.
David Bau: We can achieve better results if we learn this. But late learning tends to overfit, and then all kinds of problems. So in this paper, we kind of do a different approach, and the task here was to predict the performance of a class of art. So we have, for example.
David Bau: a hundred models that were all trained on Cfar, and each one achieves different accuracy, and we now want to sort them according to their best. So we need to basically train a neural network to regress the accuracy.
David Bau: But what we do is we do it in a supervised manner. We just do supervise training of a model. And this is our work here progen, and the idea is that the probes themselves, we can factorize them into 2 parts. So the 1st part we do like latent optimization over a set of learned probes. And these are these Z's.
David Bau: And then we learn a linear call it linear convolution generator. But the idea here is that we want to instill some inductive bias which looks like images. So these probes the Z's are just vectors. But now this generator turns it into the structure and bias of an image.
David Bau: And then this is our evaluated network. So this is each one of those 100 ones which we want to figure out how well it performs. We get the responses, the K responses, one response for each probe. These are vectors, and then we just passed it through another small Mlp. I think it was, and
David Bau: and that's it. And and we just learn it end to end.
David Bau: And it works this this
David Bau: this gets state of the art results on on the standard benchmarks and everything.
David Bau: But there's a problem with scaling up probing.
David Bau: We need to actually pass hundreds of thousands of probes through the model. If it's a large model, it's extremely expensive.
David Bau: So then the idea is, well, can we probe individual layers? Right? Can we actually inspect the individual layers. And, generally speaking, we can't do it because we're introducing once again the permutations we're we're basically going against the whole idea of probing.
David Bau: But in this work. What I I. The main idea is that there's this observation, which is both theoretical and empirical. The neuron ordering is kind of determined early on throughout training. So at the very early steps of gradient descent, the ordering is fixed.
David Bau: So the idea is, if we, if we look at only connected components within the Atlas, then basically, we can probably do it. We can probably probe individual layers.
David Bau: But then the next question is, but how interesting is it? I mean, we have millions of models, don't we have
David Bau: tens of thousands of model families?
David Bau: And the answer is, no. So we can see that actually 15 connected components already span about 51% of all the models on hanging face. And if we increase the number to 50, it's already like over 70 or 80. So we only need to train these small number of experts. And so that's what we did in this work. And the task here was to classify the task. The model forms, meaning the whether a specific class.
David Bau: it was included in its training data.
David Bau: And the reason is that it's a step towards model search, and it also allows us to do it both for generative and discriminative. So kind of extending your initial work, which was on generative models. But here for discriminative models as well.
David Bau: And yeah, so we we build this architecture which we 1st cluster and
David Bau: cluster into individual components. We then train a connected component expert, just a very, very simple meta network. Actually, even a single linear layer to like logistic regression basically works quite well.
David Bau: And yeah, this is what we're seeing here. So like, if we if we do it on a different connected components, we get random accuracy on some motivational task. We did. But if we do it on, on, within components. We already get over. I think it's 85% accuracy or something, and and predicting whether a class exists in the training set or not.
David Bau: And so, yeah, there's no permutation. And now we can learn a simple meta network on individual layers. And we we still need to deal with a very high dimension of the weights. So the the input dimension may still be
David Bau: 1080 by 1080, or something like that. So so we do we? We have this entire architecture. I won't go into the details here. I can go into it later. But yeah. And then we call this probex, but basically it learns down sampling layers and probing layers and and encoders. It's a simple architecture, and it has a nice theoretical grounding.
David Bau: And also. What's cool is that in some cases we can even align this representation that our architecture learns with an existing and representation, so we can align it with a clip representation in this case. And then we can kind of do text based search and
David Bau: 0 shot text based search. So that's cool. And it actually works in some cases, not in all cases. And that's something, I think is is very cool.
David Bau: And yeah, that's it. So we're almost done. We have speaking about open challenges. I think
David Bau: this idea of of introducing these finding these priors, these areas, we where we can completely avoid the symmetries, allows us to to scale up drastically. I mean.
David Bau: we scaled from tens of thousands of of parameters to billions of parameters in the works I presented here. But I think we definitely need more priors, more architectures. How to do it, scale, how to learn and smarter down sampling layers, how to process all the weights and stuff like that
David Bau: visualization method. So as I described in the beginning, the visualization has no spatial meaning, but we can add to it, let's say, a layer which represents the idea, the embedding of a network right? And it's a representation of the neural network. And now we can lay out things, not only by their lineage, by by some semantic relationship.
David Bau: And so that, and I think visualization also goes towards the idea of extending. So if you find a circuit, let's start and analyze this circuit throughout the training trajectory. How did the circuit change? How did it evolve what happens to it on this atlas? And then some other group can look at the circuit and say, Wait a moment. We found a similar circuit. Here. Let's let's combine them. Let's
David Bau: whatever M.
David Bau: Probe selection is very important. I'm sure you run into these problems as well. But how do we actually do probing better, more efficient, more semantically
David Bau: and recovering trajectories? So there's this idea. This atlas I presented is kind of a down sampled atlas of the actual real one which the real one, the the underlying Atlas
David Bau: basically contains every gradient descent step, but nobody ever records it right. We never record every gradient descent step. But if we were to recover this one, this atlas, then we'd have basically
David Bau: all the the lifelong history of a model from every change it made. And then we can recover every training, sample, and stuff like that, and but very, very challenging.
David Bau: So that's it. To summarize, we have
David Bau: millions of models. It's only going to grow this number.
David Bau: and we need to chart. The model Atlas and the model Atlas enables us a large collection of of tasks. There's everything that Koyana mentioned in our paper, but here I focused on on 3 of them.
David Bau: but in practice the small atlas is not actually available. We only have a very, very, very small part of it which we can see nowadays.
David Bau: and we would want to get this. But we only have this. So we need to invent and come up with new methods and new priors, architectures from going from this to this to learn from model populations.
David Bau: That's it.
David Bau: since. Sorry if I run over so do you learn a Meta network, for, like every different kind of architecture or like, how could you take in? Yeah, so it depends. It depends which work
David Bau: so and the work.
David Bau: So here, no, because we're doing probing. So we don't actually care as long as we have the same number of logics. So, for example, this is on vision models, not Lms, but
David Bau: but Llms. As long as you have the same tokenizer and then output dimension, then it should be fine, regardless, and with images which we did. Yeah, as long as you have the same number of classes. And actually, we have a follow up work of this which is currently under submission, which even extends it and does it 0 shot and stuff like that, and and even relaxes some of the assumptions.
David Bau: And this work with the family trees. Then, yeah, you need to learn a meta network pair model family for each connected component here. But the thing is.
David Bau: while so so in the the work, I spoke about
David Bau: just a moment ago. It took maybe 20 h to train it, because you had to to back probe through the entire network with many probes. Here. It takes about 5 min to train, and you need maybe 100 samples where there you needed maybe thousands, because it's so simple. You have a single linear layer as your classifier, maybe 3 linear layers and one activation. So it's so simple you can do it very quickly. And you don't need to learn that? Many of them.
David Bau: Yeah. So like, how do you? How do you treat so like flux has like 2 text coders. How do you like treat stuff that people are like, I will just like, take part of this model, use it as our model. Yeah. So we we didn't actually handle all these edge cases of whether you have parts of models and and many other words.
David Bau: So like, it doesn't inherit a clip. For example.
David Bau: yeah, we we didn't. Yeah, we didn't. We need to bake it in, I think. And even a greater problem is, we don't account for distillation here.
David Bau: So an edge is only a weight transformation. But distillation is is this other type of transformation which we were thinking about how to model within this graph?
David Bau: But yeah, I definitely am.
David Bau: Another thing is, yeah, to include model parts. So individual layers or components and stuff like that. And definitely
David Bau: so like in your atlas is very which really great visualization, like a lot of 1,000, has been gone on that but like, so, it's very like, I guess. Model transformation related. So
David Bau: how do you imagine you would want to connect? Let's say, all the Nlp models, or like, basically, like, maybe based on data like which all have maybe shared with the same data like train on the same data. And things like that. Like, how would you connect? Maybe connect in terms of edges? Yeah, edges like. So the edges are the lineage right? There'll be an edge right? So I mean more in the sense of like.
David Bau: If so, you know how you said, in future work. You want to include the spatial aspect of it. So like
David Bau: what sort of thoughts you have in terms of like. So it's very initial. But there's this paper by Philippe Isola. It was Icml last year, the Platonic representation hypothesis.
David Bau: So he shows that the representations are converging and
David Bau: and definitely, but they're also probably no going to converge to a single point. So to some, some convergent thing. But some things you don't necessarily have 100 overlap. But
David Bau: one of the things I'm thinking about is, how do we add a layer of representation? So there he represents models by their kernel similarity, and then maybe we can take that encoding and and
David Bau: basically add this as another layer. So I
David Bau: and I'll show I have another slide here somewhere
David Bau: about a different like visualizing a different type of feature.
David Bau: It's not here, but I'll show you later, and we can look at the paper off. Your thinking is based on the representation similarity rather than on the data, or something that was trained on, or something like that. Right? But that's also, I mean.
David Bau: whatever we want. Right? So so yeah, the some of the visualizations which
David Bau: I I don't know why it's not presenting there. But we we can basically.
David Bau: And we can basically visualize any, any node feature. So for example, here, oh, you're not seeing it.
David Bau: Yeah, I just hit present again.
David Bau: Yeah. So for example, here, this is the same
David Bau: same figure I showed before where I did this case study. But here we're visualizing license information. So again, it's not spatial yet, but we're just visualizing a different node feature and then looking at it. Here we we're visualizing the pipeline
David Bau: and we did it. We didn't visualize data just because it's so sparse that it's not interesting, because we only have, like 2% of something of the models have data. So everything will be gray. Apart from a few very sparse models. But definitely, if if you were to extract information about what type of data set, what data, then, 1st of all, we'd be able to visualize it in terms of trends. But then also definitely, let's find some encoding for that data, whether it's a
David Bau: model based encoding or a data based encoding right? We can do like, I think there's a work, a data set to vet, or something like that which just extracts encoding or people train. Now Llms. Which give it like a small model card or a paragraph which describes the data set, and then you can do some similarity based search on that.
David Bau: And then, yeah, you can embed that. And
David Bau: but but that's what I meant about the visualization. This is not like a piece. Right? So so all we're visualizing here is minimizing edge crossing. But we will need to develop some new algorithm
David Bau: which is a layout, a graph layout algorithm that both minimizes the edge crossing, and tries to lay it out neatly, and takes somehow into account the similarity. So it does kind of a mixture between a T-snare and a and a planar graph layout, which I think is very exciting. It's something that I'd love to work on, but
David Bau: I haven't started yet.
David Bau: Cool. So Chad like, do you feel like we should just get better at modeling or or like, just like writing up about models that we upload to fail because because of a few things. This is what Hiding Face is doing. By the way, they're just training Llms to auto summarize model cards and
David Bau: trying to enforce better documentation. And we're we're all coding things right. Nobody's documenting their code. Honestly, it's it's
David Bau: extremely important, and almost no one is doing that.
David Bau: whether it's a big company or your own code. But then, also, there's so much information which you can't extract from the documentation alone. Nobody's going to document all the biases of the model has learned, because nobody knows it, because we don't know. All the biases in the data to start with, and the model may have had may have some emerging capabilities which I mean, it's another question whether there is such thing as emerging capabilities. But if there is, the model may have learned some emerging capabilities which we don't know a priori.
David Bau: So you need to find some semantic representation.
David Bau: in in my opinion, and which which you can't document ahead of time.
David Bau: Once you found it. Yeah, cool document it. But why not just use a representation and do some?
David Bau: Yeah. Well, why use a text based representation where you can use an actual representation which captures the functional performance.
David Bau: So you alluded to this a little bit earlier, but I was wondering if you had speculations. For why model merging does work despite this symmetry issue? No. So there's a lot of literature about model merging
David Bau: hundreds of papers on model marking
David Bau: when it's within the same family. And it makes sense, right? It's it's kind of this similar idea to task vectors. And if you're familiar with. But yeah. So I spoke to one company
David Bau: which fine tunes and trains their own
David Bau: video and and image models generative models.
David Bau: Because if we look at here
David Bau: it was a short attempt. Short lived attempt to perform model merging and computer vision. So I tried to ask them, why is model merging? Not that popular in computer vision. They didn't know. But what they knew is that they were trying to de-bias their models from generating New Weeks and what they did.
David Bau: at least for some short period of time, before they cleaned the data, or whatever was they trained? Actually a Laura to generate nudes. So I don't know what data said or something. And then they did like the opposite. They they basically did a task vector to decrease it. And that would be considered as some sort of model merging. So that makes sense. I definitely don't know why it makes sense to merge a mistral and a llama. I mean, they should have weight permutation.
David Bau: And when I tried things like that they had weight permutation. I I also don't know how good are these specific emerges. Maybe the the successful merges. The large ones are only ones which are within the same thing. I have no clue.
David Bau: Thanks.
David Bau: What what do you? What do you think is? Do you?
David Bau: Do you see? A like a Holy Grail problem? Or you know, you know, what? What's the
David Bau: you see? A major challenge in front of this, this line of inquiry.
David Bau: What do you wish that you could do with weight space analysis, that that's not possible today? So something I I think it's purely academic, but I think it's really interesting is to so
David Bau: the paper you showed about the Laura recovery. I think that's very cool. It's it's adversarial. But I think it's just a nice idea. So extending that to full fine tuning, and then, just in general doing like just studying the history of the model, the the like, reversing the trajectory of a model in terms of the actual weights, and then hopefully.
David Bau: maybe recovering gradients and recovering training samples directly from the weights. So I do have a project which is kind of
David Bau: been going on and off for the past year, but trying to actually reverse, recover the let's say, 5 samples that are usually used in the Laura fine tuning of stable diffusion.
David Bau: And yeah, it's been going so so it's very hard to to reverse very, very hard.
David Bau: but I I think that's a very cool thing. And then I think just the idea of that. I think the Holy Grail is just
David Bau: being able to apply actual machine learning on machine learning. Just just as the idea of of accelerating the the pace of machine learning by doing machine learning on machine learning.
David Bau: and it's very cool. And then another interesting thing which I know nothing about. But I want to get into is this idea of interpretability on, on these entire populations, or throughout the trajectories, like trying to understand how this mechanism look
David Bau: throughout one trajectory or a couple of them and stuff like that. I don't know if people do it already. But
David Bau: fantastic.
David Bau: Well, thanks.
David Bau: So you, how long are you trying to be able to speak on today.
David Bau: Today I'm that's great. Yeah, that's great. We have sign up sheets. You have. You have a sign up sheet already? But I wasn't sure about your schedule, David. So I assume that maybe right after the speaking. Yeah, I'd love to. I'd love to chat with you. Yeah, and that'd be great.
David Bau: Thank you. Thank you all for coming.