David Bau: Where's it?
David Bau: And then we hopefully, we have working audio and folks here.
David Bau: Alright, yep, okay, that's good.
David Bau: All right. So so our noon time talk
David Bau: is. She has a new paper about Model X. And you guys are familiar with some of the work that she's been looking at there.
David Bau: It's a special kind of paper. It's a vision paper sort of a a future looking survey, and she so she's
David Bau: presenting this paper pretty soon in Barcelona, at one of the database conferences, so she's going to give a little preview of what her paper is about what she'll talk about there.
David Bau: Well, take it away. Kalena. Alright. I'm sorry. I'm just gonna time myself of it as well. So
David Bau: all right.
David Bau: Hi! Yes, so I'm Kaina. And I'm going to talk about model lakes. And yeah, I'll start off with this tweet that Clem posted like about 6 months back, and he's like the founder CEO of hugging face, which is a model sharing platform, and in that they had crossed about a million public models, and just as many like private models
David Bau: as well. And so what we are seeing over here is that there has been kind of like an exponential increase in the number of models being deployed, and there's been increased traffic of users, and like, you know, people who are also developers and trying to get these models out. And
David Bau: however, though like in terms of what the media covers you probably are aware of. Maybe like couple like 50 of them, or something. So, for example, like llama, gemma deepseek. And so there needs to be a way for us to kind of better manage the models, to be able to drive some meaningful insights and inform decisions, and another reason why we need to do that is because
David Bau: in every model, ideally, we would like to have a documentation. And that's something that people have. You know.
David Bau: talked about what's an ideal kind of documentation to have. We call those model cards. And in many of the popular models we do have this like it shows like model details. Where do you want to use it? How it is made? But most of the models that are on hugging face. For instance, they do not have these model cards, and even if they do, there's a lot of incomplete information.
David Bau: and there's no way of verifying what these like information in terms of the model creation aspects are. And so yes, so we need to be able to streamline and verify documentation, for, like a better clarity and model search, and an additional sort of like motivating factor. For this is because, as like AI models are being more commercialized, there's been more regulations in terms of like, how to really manage all these models, and how to make sure there's like an ethical use
David Bau: of these. And so just deploying models and putting it out. There is isn't just gonna cut it out for us. And so we need to like have a better way to
David Bau: navigate them. And so, yeah, this is kind of like a history repeating itself sort of moment. Because in the database community there's been this concept of data lakes which emerged in the early 2,010 s. Where, like you had this like huge, like raw, unstructured data, and you wanted to be able to process them, understand them, analyze them. And so
David Bau: they have this storage system with call data lakes, and that. And people in the past have been, and even right now, like building these analytical tools
David Bau: to make use of all these like data that is out there. And so yeah, I'm gonna start off with how this model like might look like and I start with these different prompts that you can ask. Then you might ask as a user whether you want to know, like what model X can be useful, what is risk? How do I cite it? Or like if you're looking in terms of what's the best model like to summarize legal documents? So these are all these use cases that are out there.
David Bau: And so in this model, like you have models like the model database. All these like models that are present in the database and benchmarks. And there are these tools like inferencing them interpretability and like
David Bau: for example, and like other like having an indexer of sorts. This is something that we are proposing as part of like research roadmaps that you can. We can try to have within the model lake, so that when when we use these tools, we can get a generated document or have, like a version graph of like, okay, this is where this model is in the space of
David Bau: the model models that we have and also have a better like ranking system of like. When you, for example, want what's the best model for summarizing legal documents, you would get a ranked model, and that way, you can then use the Api to generate like these. Answers for you, so it can be more
David Bau: like usable for those stakeholders. And the way we go about defining these tasks is also based on model accessibility. So you know, there are like these models that are all models are black box, but they're even more so when we don't have access to their internal states.
David Bau: And so we define the we have like 3 viewpoints. The model intrinsics talk about like the function like the parameters, the weights that you have access to the extrinsic aspect is just whatever you can observe from the model. And the model history is more about the documentation. So if you have access to the data and the
David Bau: algorithm, that's like something, you can maybe use for your for the task, and the tasks that I'm talking about are 4 categories of them, attribution, versioning, search and benchmarking. And so these tasks have existed before. It's just different. Communities have looked into them in different lengths. And this work is attempting to bridge all these community efforts together and have a more cohesive way of
David Bau: off
David Bau: like looking forward with the model lakes aspect. And so, for example, with attribution. When it comes to model intrinsics you can meet. You may be interested in knowing, like what internal representations
David Bau: internal concepts within the model that can be most important for a decision on the extrinsic aspect you might be like, okay for a particular output. What aspects of the input can be most important. And if it's history you might be interested in. Like, for example, the training data that could be most influential on the decision.
David Bau: And when it comes to versioning, we can think of having all these models in like these directed model graphs. And so when you have a model and a set of like other models, we can have the directed edge that could describe the transformation from one model to another, and yes, and the nodes itself could be present. The model itself.
David Bau: hugging face, for example, is trying to have this model tree aspect over here. But this is something that's self-reported. And so it is a step in the progress. But we need to. One, of course, verify it and find a more automatic way to be able to extract this information because this is not available for the other past models that are out there. And yes, that also leads to our next task
David Bau: about model search. And so for model search, you can think of a task function where you have the task as an input, and it would pro, and you would find the best performing model for like a given set of
David Bau: model candidates. And the task function could also literally be like another model. So, for example, if you input another model, you could get an output of what other similar models that could be out in the space.
David Bau: And this can be based on like just looking at the output of each of these models you can compare, for example, in images like how one image could be similar to another. And you can expand this, for, like other modalities, for intrinsics and history like intrinsic, you might be interested in finding models with other similar modeling architectures and training parameters. This could actually still be like challenging, because in the case.
David Bau: like model merging, stitching community like to be able to understand what are the different, like
David Bau: parents and like multiple parents. That they could have. Is something we want to consider, and yes, still don't know whether you know, a couple set of data has been trained on the same data set, or in the case of privacy if you wanted to know if it was trained on sensitive data or not.
David Bau: And this leads us to benchmarking, too. So bench in benchmarking for single model setting. This is something that has
David Bau: been happening as part of evaluation that, you know. Given a model, what are these, how well does it perform? And let's say, like multitask answering like segmentation, things like that. And so that does exist. But maybe just for
David Bau: like a model, may be trained for one instance, but could be good, and, like other tasks, so like being able to robustly identify what benchmarks would be useful for evaluating each of these models. And because now we're considering a whole set of models. We wanna be. We want to have benchmarks for these multiple model settings for all these like modeling tasks as well, so like being able to identify, like, create versions, or like, be able to search
David Bau: models, and so having more like large scale benchmarks, and that so here are like these are essentially some ideas on what these tasks can be. And the. And even though now we're introducing model lake tests. It is still important for us to consider the data lake commute like the data lake tests are already in there. Because for these models, you do need data. And so
David Bau: things about like being able to trace data like, also have the versioning for that, we want to be able to integrate the management of both data and model lakes. And there are other strategies that we talk about. So we can feel free to check out the website and or you can scan the QR code to take a look.
David Bau: And yeah, shit. I'm already over time. But yes, that's what's the parameters? How much time do you have? So I actually, I have like 8 min. So it's pretty short. And then how much, how much time did you use in class? But it's okay.
David Bau: But yes, that's that. And
David Bau: Yeah, that's that's yeah. That's so. Basically
David Bau: So so I think there's, there's 2. There's 2 things that we can do for these practice talk things. So one is. First, st we can start with like substantive questions for about her project. But then, the second thing is, we can, we can like also go back and help her, dissect her her presentation, and make suggestions on, on, like you know
David Bau: what to squeeze into her 8 min that she has, and you know what what to add or remove
David Bau: but so for substantive questions. 1st about the model. So can you go to the hugging case slide?
David Bau: Oh.
David Bau: yeah, what is the merges and quantizations? So I think the merges one is like I. It is like, maybe so if I understand correctly, it's like literally like they may take a llama model, and like some other model, and like, merge them together.
David Bau: Like, like as in like parts of the model. Okay, yeah. Parts of the layer. And I think quantizations. That's a good question. I'm not really sure like the 8 B. But why does? There are 200,
David Bau: 102 models like different people. Yeah, different people did it. And so this is just saying that sorry. So this is just saying that from. So if you imagine, let's say, this is lama instruct like. Furthermore, there are like maybe 102 models that people have like quantized from that. Yes.
David Bau: I mean at least those that are self reported because this is something you have to fill out in the metadata at the moment. So yeah, it is 102, because there are multiple 4 bits, multiple 8 bits and multiple, yeah, like, I can. Yeah, I mean, I can like, go into here, right? Yeah, people have to say that data like, you know, for example, like, you know, I guess this Mlx community did like this right. But you know there could be duplicates. We never know and
David Bau: and yeah, and it's not really informative how exactly each of them are different from one another
David Bau: like you. This person, the way they quantize, quantize that. And this one could probably be the same. But yeah, that's another thing.
David Bau: the we wouldn't know. So at at the current moment, like
David Bau: verification actually filling out that information, and then just knowing the difference. So when I use bits and bytes and load the model in 4 bit, is that the same thing if I directly go to one of this model and fetch it loaded.
David Bau: or is it a different thing?
David Bau: I don't. I don't think we know we don't. I don't know. I don't think we know. Gduf is one of the quantization methods.
David Bau: Bits and bytes has its own quantization. So the only advantage with this is, your loading. Time will be super fast, because it's already like a small weight. Okay?
David Bau: But it's basically the same loaded. More. I think the point is like somebody could have altered this model. Yeah, no, exactly like maybe they ran some Rome edits on it. Yeah. For example, over here, there is no information of how they, for example, did it. And it just says, Not for all audience in this case, for now. But yeah, I mean, it's a harmful. I mean, okay, I just randomly clicked in one of them. Yeah. Yeah.
David Bau: But yeah, question for Michael. Yes, I was wondering. And I don't mean this in like a way, I was curious what the target audience was with this like, because it seems like this is like proposing an idea to someone to take responsibility in some capacity. Exactly is this like a community level of like individuals, push and huggy face to like. Add more features like this like a hubby face level? Or is it even higher level of like government policy? Kind of making sure this sort of stuff isn't mandated ideally all levels?
David Bau: I only mentioned hugging face, because, like, I feel like, that's 1 of the most popular model sharing platform. Maybe I should just mention Civt AI just to like make it like, not so hugging, Facey talking. I don't know but like, but yeah, it is ideally like meant for, especially the database and the AI community. Just because database truly has been working on this in like a data lake format, it's just that
David Bau: they have been working with data points. And there's this additional abstraction of like actually understanding models to be able to do what they do right now for other data types.
David Bau: It's all.
Renée Miller: Right. And I wouldn't say there's any mandating implication here. Rather, it's a call for research to help in addressing automatically addressing some of these issues, verifying a model card and things in order to help the research community better use models.
David Bau: I have another humble way of looking at it, which is.
David Bau: which is, you know, Koiana has been interested in this set of problems
David Bau: and and this field hasn't really had a name. It's just a set of, you know, unrelated problems. And so this is just an opportunity, for she's embarking on her research
David Bau: to say, Huh, what we're really doing here is we're we're constructing set of methods which would correspond to this idea of modeling. And here's a. So
David Bau: so I have some meta suggestions about your talk. But it's okay. But we can. Let's do more substantive questions. I just quickly ask in the context of data lakes. Was there any research in similar regards that tried to classify a lot of what's going on and try to like make a sort of push towards a particular design, and if so, did it have a good like an outcome? I guess I'm kind of curious, historically.
David Bau: a similar kind of movement in the context there.
David Bau: So you're asking about the formation of data leaks itself or like, how does it compare to data leaks? I guess more like in the context. Historically, when people started to try and think about oh, like we should be doing things more systematized and starting to organize concepts from different fields. I was wondering whether that sort of had already happened within the context of. But
David Bau: so, Renee, correct me if I'm wrong or are you? Yeah. So I think, like the way I understand about the whole like data lakes, how it came about. I think it was definitely still within the database community. But like, there was an industry person that said that, hey, you know, like they just termed it that I think we need like data lakes, because these data. So before data lakes, I think it was data warehouse.
David Bau: Where, like, you had these like structured data like a quote, unquote, clean data to be able to do this. But like as like, you know, we started to use more Internet, like, like Internet, you have all these like unstructured logs and other things like that. And I think
David Bau: we wanted to have like a like a easy storage for that. And like make use of those data. So I think that's how data lakes came about. Renee, do you have anything.
Renée Miller: Yeah. So if you think about database management systems and data warehouses, they, they're design systems. So I have to put a catalog in there that says describes all the data that they contain.
Renée Miller: And when people started doing data science, they started generating data sets at a volume where you couldn't load them into a data warehouse and structure them nice and neatly and tidily.
Renée Miller: So people started just using file systems to store their data. And the only way to find different data sets was by the file system naming convention. Right? You could search your file directory structure.
Renée Miller: And so the the data management community said, well, wait a second.
Renée Miller: Even though all this data is not stored in a data warehouse, we can still apply data management techniques to the search
Renée Miller: problem and actually search this data, we can index it and search it fast and allow people to retrieve it by content.
Renée Miller: So questions like, Find me all the data sets that might meaningfully join together or find me all the data sets that have the same set of features in them. We can create indices to answer those questions in sublinear time over these over potentially millions of databases. Data sets. Sorry. So that's kind of how the data Lake
Renée Miller: research work got started 1st a necessity that people just were using file systems instead of databases.
Renée Miller: And then some interesting research problems on
Renée Miller: how do you search them? How do you do version control when you have many versions of the same data set and so forth.
David Bau: Any other substantive questions. That's great. Thanks, very practical. So.
David Bau: I recently looked for text embeddings, and just to like, compare the similarity of Llm. Answers, and like deduplicate them
David Bau: the the way that I did it was just go to a text embedding benchmark and just take the one with the highest average score. For you as an expert
David Bau: with like a fully baked data lake. What? How will this be different? Will I still be looking at benchmarks, or what are what are the key features that I will be looking at?
David Bau: I'm sorry. Let me clarify like I'm I'm having a specific task. Yes, and I'm looking for the perfect model for it. Right? Okay? And so, just like benchmark, I have like the whole data lake with all the curated. Yes, background info.
David Bau: How would I find my perfect model so ideally like the one way I imagine a different scheme might happen like I like when you're uploading a model, let's say you know how to understand a model well enough. Maybe you can build a concept cloud for each of those models to kind of know in 1st what information it might have or might be good at having, and then later from there like filter out and then use like
David Bau: like and okay. So there are 2 aspects, one concept concept for like models, and then the other, one could be like from Data lake finding relevant like, based on like, for example, like text embedding find, like relevant, like data points that could be good for like testing the models like, if you if you need to. And so yeah, with the filtered models, with those.
David Bau: And I guess filter data points.
David Bau: Maybe you can like run them and then find what the
David Bau: what is it? Concept cloud? And does every model have that concept? Cloud?
David Bau: Good good question. Of course, for now, like these models don't have that. But
David Bau: it's it's more about like a way to like.
David Bau: you know how that these like like Morris paper like they're like these different papers on, like extracting like knowledge from like the models. And so, if there was a way for us to just like
David Bau: in the future, like know that we can grab these knowledge like like, think of it as
David Bau: slider space before, like concept space like, you can find out what that what those con, what concepts they might know.
David Bau: Like having well, having a tool like that.
David Bau: Yeah, it's pretty yeah.
Renée Miller: So I like this question, I think you might have a slide that has on the left side the current way of searching data model like.
Renée Miller: right? That that follows what? Sorry? I don't know who was speaking but what the the question asker just said today, if I want to find a model. Here's what I do. I go find the one with the the best performance on a benchmark that I know.
Renée Miller: Right, and on the right side put present a vision for how you might do it in the future. If you had some of this.
David Bau: Yes, so like, just as an idea. So like we have. So, for example, this is the task. What's the best model to like summarize legal documents. And you have these set of models, and like ideally, maybe you have
David Bau: a cached indexer, and which I guess was built via the benchmark like running on things. And I guess again, with that filtered models and the benchmark. Then you can maybe like it. Then get your rank models.
David Bau: And find out that. Okay, maybe this is the best task for that. So yeah.
David Bau: because this is, you know the papers, since it's a server survey one, maybe one of the nice ways of doing it is, say, like the best state of the art for model search so far can do things like this. So like, what like, what would you say is like? So there's not. There's not like an existing piece of work that does like the ideal thing that
David Bau: you might want to do. But like what's what? What previous work would you say? Comes the the 1st one that comes is like the one by the by, as I think, Junior, like the content based model search where, like? You output, you actually compare like
David Bau: you give the task, and then, for the set of models you actually run those models and like, output them, get those images, and then find what's the best image that closely matches to what the I guess test trying to find the model is actually going best on a specific task. Yes. So that's a general generic benchmark. You provide your own data and make your own benchmark right? That's cool.
David Bau: So now, how to do that in your specific case I don't think anybody's probably solved that. But I think in image generation there was an attempt cool.
David Bau: So how do I not always return the model that was trained on the test set?
David Bau: Okay.
David Bau: well, I guess that's that relates to she has her interpretability tools. But yeah, but as a user, you come along with your new test set. So I guess if somebody, if somebody you know has has. It was was lucky enough to train the model on your your query. Then maybe they do. Well.
David Bau: I mean, that's 1 use case. The other use case is like, I'm a company. I care about this one. Data sets that we always use for something like that.
David Bau: And maybe then it's not new. Right?
David Bau: Yeah.
David Bau: Run Maya on it. And like, try different test cases. I don't know. Yes, exactly.
David Bau: Yeah.
David Bau: Should we? Should we flip through your slides and maybe come up with suggestions about the structure of the talk? Yes, I have some suggestions. Yes, please. It is a very last minute I was doing it. That's okay.
David Bau: because all right, this is so. Motivation, I think, was fine. So but actually, you can for motivation. Yes.
David Bau: maybe you should show something like 2 models which were derivatives of each other like maybe lama. And
David Bau: and then you should say, did you know that they were descendants, or like part of the same flea.
David Bau: Well, hugging face doesn't show that
David Bau: I should like really research to understand that, but ideally, in a word, that it should not be a
David Bau: challenge.
David Bau: That's good. Yeah, yeah.
David Bau: start start with an unsolved problem. Yes, that's really great. Yes, yes. So the current process is, you go search the Internet. You read about it. Some somebody in the middle of a blog post says, Oh, it was a is a fine tuned version of this other model.
David Bau: That's nice.
David Bau: And the other thing is, maybe you don't need to write the word motivation itself.
David Bau: Sure, the title just maybe write the main. No, whatever you the point that you want to convey, write that instead of the word motivation itself.
David Bau: like, right now, we can't do this? Yeah, yeah.
David Bau: well, whatever it is. Yeah, yeah, exactly.
David Bau: Exactly. So you can just put this, you can make that country.
David Bau: Okay, let's do, my boy.
David Bau: Yeah. So that's great. So for each of these, I I like that suggestion of the of the title being the unsolved problem while you're in motivation. So if you go back up to
David Bau: growth.
David Bau: it's like, you know. So you're gonna present a single small problem in the 1st slide, maybe, and then and then you say.
David Bau: you know it, it's growing so fast. This problem is getting worse, right? Something like that.
David Bau: Right?
David Bau: This problem is getting worse as as the universe grows exponentially over there, right
David Bau: and then. And then the next one, you know.
David Bau: Right?
David Bau: Yeah. And you can say you can say it's growth rate, it, probably getting worse as as as the world grows, or something like that.
David Bau: as as AI grows right? Okay?
David Bau: Alright, alright. And then, yeah. And then, and essentially the same documentation.
David Bau: You you made the point. You can't trust documentation.
David Bau: Right?
David Bau: Yeah. Unverifiable is kind of a long word. I would just say. I would just say, you can't trust it.
David Bau: Yeah, I you really can't. Right? You can't trust documentation.
David Bau: Great. Okay, cool. You'll you'll you'll pick a and then, right.
David Bau: you know. So this actually can kind of a nice slide. You could say first, st you can't trust the documentation, and then, if they even bother to give it right, and because most people don't even bother
David Bau: right. But it should be spoken. It should maybe not necessarily, maybe not a title, right? Right? So you can you put on the title? You could say how you know how how you know it could be a question like, How often can you find documentation? It's not that right? Right? Yeah.
David Bau: yeah, right?
David Bau: So
David Bau: yeah. So maybe the title is too inflammatory, you know, suggestions instead of instead of instead of just saying, we're gonna talk about search. Now, we're talking about documentation. Now, more like, this is the problem. This is the problem. And while you're in the motivation section. It's all like, this is the problem. Right? It's getting worse, right? And then, next thing auditing, yeah,
David Bau: yeah. So what's the yeah.
David Bau: So
David Bau: you know, auditing who's who's a customer for auditing? It's it's it's regulators and consumers, the audits right? Something like that. So you could. Just
David Bau: so instead of just having the noun there, who needs this, or what they need it for, or something like that?
David Bau: Some thought that I had here was that the Ml. Community is not specifically the customer for auditing.
David Bau: And
David Bau: on multiple slides. You point out that people on Hanging face do not document properly. They? You don't. You have this self-documented thing, and like there's no model cards and most of the models and whatever and I'm wondering
David Bau: would hang face, be as successful as they are right now. If they enforce these structures from the very beginning, probably people would like, oh, no, I don't have 10 min to like, fill out the form. Right? Yeah. I mean, ideally, what I'm proposing is for this to be automated
David Bau: right? So like, upon uploading the model. I guess you should. Yeah, yeah, I actually was a little bit confused, like, is there a call for people to, you know? Do do your job well, or it's like we're providing a service for search. Or
David Bau: so. That's really the call for the research. Right? It's like, Oh, so what is a model like is like.
David Bau: it's like, it's this call for research, so that we can all, as a community develop methods to automate this right? Should I then focus on this to like be like generated document. So in this. So this is a fine thing. But instead of saying, instead of saying, Oh, this is a design of like a done model link, and not sure what it is. Right. Is this more like your call for research here? Right? This is like, this is like. So maybe
David Bau: right, if you can clarify when you speak about it, or put a single line of text at the top of this or something like that for sake.
David Bau: you know
David Bau: that this should be automated somehow so that it can be done. And then this is like all the missing research that needs to be done might be really motivating if
David Bau: you could give some.
David Bau: You know things that are already possible, but not that
David Bau: things that are already possible, but not done yet. Not done yet.
David Bau: For example, I guess, like, that models are
David Bau: like models have the same base model. Yeah, yeah, that seems like something that. Yeah, there's been. There's been papers studying exactly how you can quantify this type of thing. Right? So it seems like, that's right. So yeah, that's 1 of my other Meta comments is that there's you're you're building on top of a lot of it, you know. This. This framework isn't just something that you just came up with right? Because there's a lot of existing research. And it would be really nice to
David Bau: say, Okay, there's this vision here. Maybe this is your vision slide, and, you know, make it really clear at this moment. So you're transitioning from motivation to like vision and make it clear that this is not something you built. This is something that we need to build as a community. There's a lot of parts of it.
David Bau: And and then when you go into the tasks instead of just saying this task and that task you can like pick what you think is the latest state of the art, and people are investigating some subset of this task, and then actually cite them. Actually like, put their name there and say, let me tell you the story of this cool
David Bau: research in like one sentence, if you can summarize this cool other thing, this cool method
David Bau: for figuring out what the model tree is, or for doing a model search. And like it's not like, solved the whole problem. That's why there's this whole area. But if you can kind of just like cool for each one of these things. It's not impossible, like there's there's like, this is a good starting point. This one's this one's needed. It's not impossible. This is a good starting point for each one of those successive slides. I think that that would
David Bau: that would like make it feel like, Oh, yeah, there's something to do here that makes sense question. For like, for example, in the model versioning, I guess I just put it up on the slide here. I didn't talk about it, but like so I will just, I guess, give a so right so like over here, right? So you can say so. You can call back to your things like, How how do I? You know? How do I know like, how? How do I know a model version, instead of instead of
David Bau: being too formal about, because it's it's not the paper itself you can. They can read the paper if they want the formalism right?
David Bau: But but just say, you know, remind people of what the puzzle is like. How do I know that one model came from another? And then you can say.
David Bau: if you have access to the weights, or if you have, then then
David Bau: there are existing ways that you can estimate this.
David Bau: They, you know they're not perfect. But you should read this paper right? And but there's a need for being able to do it better.
David Bau: And right, like. We don't know what to do
David Bau: in the world. Now that we have model merging, we don't know what to do in this right? So that you can sort of say, Hey, this part has been looked at. But there's so much more that's going to be needed to be done
David Bau: right for each one of these things that make sense.
David Bau: Yeah. Yeah.
David Bau: So that's that. That's my main. So that's like, just generally for for each one of these tasks. Yeah, so yeah, there's this whole area
David Bau: of you know.
David Bau: model search is this whole area of this, and then, you know, the the current state of the art is
David Bau: maybe more advanced than you were aware of people already looking at some of this stuff. But you know
David Bau: yes, it should be built out.
David Bau: Got you?
David Bau: So if you're mentioning these different problems before in like a 1 big picture you had like emojis for each of them.
David Bau: If you didn't let you could have Emoji. So people can remember. You mean. So people can remember what you're talking about. And they have this.
David Bau: They can connect it to the main studio.
David Bau: Yeah, this isn't like the last one
David Bau: any other talk suggestions.
David Bau: Sorry. I wanted to include this picture at some point, but I couldn't. Which which picture? What is this? This is AI generated of model lakes. Oh, you get it over there. Yeah, right? Well, that's derived from it's a fine tuning.
David Bau: Okay, I'll figure out how to incorporate this picture.
David Bau: Alright. Thank you. Folks very nice.
Renée Miller: Very good. Kriana.
David Bau: So