David Bau: Cloud.
David Bau: So
David Bau: you know, I'm I'm interested in this. So you buy is is gonna I'm actually not familiar with your research. So I'm looking forward to learning more because you're coming from Raymond's lab, where most of the
David Bau: work is more like multimodal. And you know, visual, and so on, and so. But you you have some fine tuning work on language models that I like that. That seems like it'll be very like a departure from what the lab normally does. Is that right? And so so looking forward to hearing it so welcome, yeah, welcome you.
David Bau: Okay, let me get started. Firstly, my name is mentioned here I mentioned. My advisor is Professor Women as well as in the Northeastern University, and then I'm a final year, Phd.
David Bau: And your graduation, and thank you for all posting me here to give the talk, and, like professors mentioned. This talk is about the lumberstorming and our lab previously were all focused on the data mining or communication stuff. So it is like the departure stuff. And I will briefly introduce my research and logic for the next slides give you guys intuition.
David Bau: Okay? So
David Bau: oh, by the way, today's topic is about hosting large language model with, so today, the talk is not about the systematically talk. It is more like the open sharing one of my paper.
David Bau: Actually, it is one of my favorite one. And with some related discussion. So it's more like paper sharing. Okay? Cool. Yeah. My research workflow previously
David Bau: multimodal learning like professors, said multimodal learning. So my research use the multimodal learning as a foundation based on the various data sets, various data formats and different costs.
David Bau: for example.
David Bau: from typical recognition and prediction classification task before the chat Cp comes out to current cross model retrieval. This is for the clip and the ribbons. Giant AI support stuff for the large language model large multimodal language model stuff
David Bau: and based on the multimodal foundation. I mainly focus on 2 aspects. The 1st one is efficiency.
David Bau: With several topics, for example, the network pruning logics and primary fission, fine-tuning Dr. Laura.
David Bau: and the second one is probability. With some terms like master learning or master fine tuning. I will mainly focus on today. And second part is the model soup, which is also close related to the sparsity part. And for free yeah, definitely, performance learning for free.
David Bau: one foundation, 2 aspects. And the core technology I'm using is sparsity. So this is a pretty one short version for my research, based on this foundation use sparsity for 2 aspects with several related tasks or topics.
David Bau: Okay and cool today
David Bau: because I lose my notes. So I just some sometime. But then make some error. Today I will go through this part, focus more, more on the sparsity for model capability.
David Bau: Typically name it as a moscofon tuning.
David Bau: Okay, backwards.
David Bau: The 1st background is one sentence version is large scale language model, or mostly optimized with model structure interpretive.
David Bau: What are the meetings?
David Bau: Pre training is definitely the 1st step for the language model from the totally random stuff.
David Bau: and we do the continuity.
David Bau: So on the one side.
David Bau: we always do the advanced mode for training. Even if we use Moe, we use probably sparse activation during the first.st But for the training it's still the sparse
David Bau: sparsity is pretty necessary for the efficiency. But currently nowadays, to ensure the performance, we still use the dance pre-training. So this is the characteristic. The second one is about the fine tuning.
David Bau: 2 common directions for function, 1, st one, the fully functioned second, one is parameter, efficiently typical adapter, laura, or prefix whatever
David Bau: some common points. The 1st one is
David Bau: for this 2 parts. The model is optimized as a whole.
David Bau: definitely for training and for the phone too.
David Bau: And second point for the prem proficient.
David Bau: We use additional parameter for for the model learnable part, however, we still fix the pre trained model as a whole.
David Bau: So these 2 features inspired us to say, okay. So the conclusion is.
David Bau: no matter which kind of we we use, no matter which direction we we produce to. The large language model, mostly optimized
David Bau: in the model structural interpreter.
David Bau: The logic here is clear to you guys.
David Bau: Okay? Sure, that's good for the next one. So this
David Bau: maybe maybe we'll wait till what he's gonna contrast it to you. And then we'll see.
David Bau: Okay, okay, so let's move on.
David Bau: And this is the 1st background part. The second is as far as apart. Okay.
David Bau: as we all know, it is commonly known that the model sparsity is always mostly used for efficiency, part probably with potential performance sacrifice, for example, during the
David Bau: does not arrow.
David Bau: we put on some model. Maybe it's 90%. We, we make it faster, make it less for the storage cost. However, we lost some performance.
David Bau: So it is neutral that sparse training, sparse inference for storage, efficiency for speed, efficiency, and the logic behind is
David Bau: sparse model these to the fewer parameters that's, initially speaking.
David Bau: and leading to the less capacity is the model leading to the less capability by nature.
David Bau: That is the common enemy, right?
David Bau: So 2, 5 ones here
David Bau: and right now we are motivated by them to say, there's motivation. Where 1st is such integrity always beneficial for the model of probability.
David Bau: So like Bro. Had asked what? How to define the Sparses sparsely means that if the model removes some parts.
David Bau: there are the metrics by 10 by 10, we randomly
David Bau: make 50% of them as 0 0 all them and use the the other forgiven the 50%. So this is sparse matrix
David Bau: model is con consist of the multiple metrics. So we call that sparse network.
David Bau: So the workers about this question is the interpretive always benefit for the model capability?
David Bau: Right? So it is that it is that question that even if the model has less fewer parameters leading to less model capacity leading to less capability. Is that for true?
David Bau: So that's basically another version of the question.
David Bau: And more aggressively, do models have harmful
David Bau: in paper. I didn't use this sentence, this is 2 of us
David Bau: harmful how to define, harmful, probably just, harmful for this task, for this domain, probably not harmful, beneficial for another topic, good for math, probably bad for coding.
David Bau: But I just for the talk I just present here.
David Bau: So to answer this question, we present this very simple pipeline.
David Bau: There are some notes here. The 1st one is in this work. We only focus on the
David Bau: pure language model or to progressive, pure language model.
David Bau: The second one is, we are rely on the well trained fully. Fenton.
David Bau: however, our our method can be definitely adapted to other scenario, for example, from the pre training, even from the random random status. But in this work we only port from the fully function.
David Bau: So the logic is given a random model. We after the pre train, the projection model. We also do the fully function with sufficient, fully functioning to make it as a
David Bau: so-called upper bound. Commonly speaking, to say, if based on the best, fully functional model, can we make further improvement by removing some weights in this work we use the width as a unit instead of the neuron based or channel based stuff.
David Bau: So far, any questions
David Bau: use of what has the unit? The weight of the individual body? Yeah, so it is in the, in the pruning terminology. It is a unstructural. Yeah, we do not do the structural one. Yeah.
David Bau: Okay, let's move on for the next sorry. Can you also define, like, what is the difference between structure and unstructured? Oh, the term is naturally comes from the resnet, because in resnet we use a convolutional neural network, several channel several filters.
David Bau: So if we prove the model.
David Bau: as by using the unit of the filter or the channel. It is called structure. Okay? So in this way people can do acceleration. If we re regarded each individual unit, the scatter as a unit that's unstructured.
David Bau: So the disadvantages of unfort pruning is. It is hard for the acceleration. You need the hardware support. That's something embedded into the hardware to make the acceleration, but for the structure pruning, because the model size is smaller. So you can actually do the acceleration. So that's the definition from the same model and transformer the same because transform model model. We can. We can reduce the
David Bau: column all the rules of the matrix in the Mrp layers. We can remove some new. So that's for the structure unstructured is a width. Okay? Okay, sure.
David Bau: And oh, so the pipeline is clear. Right?
David Bau: Let's move on for the next.
David Bau: The method. Let's dive into the method we call it as a mask fine tuning, and the method itself we call to learn mask
David Bau: because we are rely on the fully functioned model, sufficient, fully functioned model. So I 1st introduced the fully functioned model here. So this is pretty straightforward. The typical autogressive objective.
David Bau: Theta P. Here is a pre-trained weight.
David Bau: UF. Here means the data set used for fully autoressive format regular loss. So it is pretty straightforward
David Bau: after the model being fully fine-tuned.
David Bau: Hopefully, we do the mask controlling.
David Bau: So next, the step is.
David Bau: we use exactly the same loss format. We can see the equation here is almost almost exactly the same. The difference are 2 parts. The 1st part is
David Bau: Oh, after the fully fine tune we tune the parameter from Theta P to theta F,
David Bau: and 2 difference. The 1st one is, we add, a mask amp, which is learnable here. This is an element wise multiplication
David Bau: applied on the Ctaf.
David Bau: So the mask has has exactly the same size of the save us one by one.
David Bau: This is the 1st difference. The second difference is the data here.
David Bau: The terminology used here is for the you subscribe with app, but in our experiments
David Bau: the zoom app is the same as us.
David Bau: so as a summary, the only difference.
David Bau: The only difference is the learnable mask
David Bau: loss. Objective is the same. Data. Usage is the same.
David Bau: We use the same training strategy. Try to extract extract information from the same data set on difference is the mask.
David Bau: During the mask learning, the 5th half is totally fixed.
David Bau: So that's an interesting part. Given a model, the the value and the
David Bau: metrics are totally fixed. We only tune the mask, which is the binary one.
David Bau: to slaughter something, to remove something and achieve the
David Bau: problem. Exploration, like I mentioned for that, if the some weeks are not necessary for for good performance.
David Bau: and how can we do that also for any question for this part. So so the size of M is like, you have one number per parameter. So you basically have double the memory size, double memory size. But the problem is, Oh, I know your concern. We definitely need the double side memory for the
David Bau: model loading.
David Bau: But since the state have never used a gradient.
David Bau: So it is definitely acceptable for the Gpu. Yeah, we use a 7 billion model. And even if if the speed is slow, but we can do it on the 86,000 Gpu. So it is acceptable. One other question I I assume that you your mask isn't binary while training you just like threshold. At some point like you probably have a continue you like, let it be continuous. Or is it actually binary during training
David Bau: detail show later on. The next slide. A non step. Yeah. So how can we achieve? The mask
David Bau: method is straightforward.
David Bau: based on the switch through gradient estimator. I I believe most of you guys are familiar with such kind of technique. So it's just to do the gradients estimate
David Bau: estimation estimation.
David Bau: So specifically.
David Bau: let's look at this equation. Theta L is is layer R of the Theta S, right? So let's say, if it's just a matrix, probably 10 by 10,
David Bau: and for each individual scatter. In the theta. In theta we assign one float continuous values, scores, which is called C.
David Bau: See PAL here.
David Bau: So it is. It is learnable to stay is learnable.
David Bau: and we use the indicator function, which is B here as an indicator function to filter
David Bau: certain scores to encrypted and other scores to be removed.
David Bau: So in this way we can rewrite the mask. App.
David Bau: Sticky, sticky.
David Bau: V is a indicator function. C is a spot so in the implementation aspect.
David Bau: because, say, it's randomly initialized as a continuous scores.
David Bau: So if the score is is at the top, K largest, which is typical, for example, if we set the sparsity has
David Bau: has 10% in our experiments.
David Bau: the top 90% of the score will keep the
David Bau: otherwise the 10% would just remove it. So that's the implementation part of the mask.
David Bau: And
David Bau: is it clear for this part? Go ahead. So then, that just means you're reducing the amount of like
David Bau: memory you need while you're
David Bau: for like a single like path, is it? Does it also reduce the amount of memory you need? Since you're like clipping to like the only the 10% of like the
David Bau: the C values, not really because, like, ask the, this is unstructured.
David Bau: So even if it is removed. The implementation way is, we just put it as a 0 0 still cost the space in the in the CPU implementation. Yeah? Yeah. Because on CPU, you would store it in a sparse format. Yes, yes.
David Bau: So when you do, structured probing.
David Bau: how would that? How do you handle that like you just don't load the weights. If we do a structure, not in this case. If we do a structure pruning the model sizes has been changed
David Bau: like. So which, like you just remove the weights. And then, yeah, the the matrix from 10 by 10 to 10, by 3, probably 70%.
David Bau: Well, yeah.
David Bau: So in this way. Oh, where? Where am I?
David Bau: Oh.
David Bau: the the filter of the indicator function is like the forward process, right? So if the score is good enough to keep the corresponding, which.
David Bau: if the score is bad enough, we just have to remove it. So this for the forwarding during the training the backward is A is a problem backwards problem because the indicator function is non differential.
David Bau: So V is the indicator function here?
David Bau: Oh, the B, yeah. Yeah. So what is the I here? What is the I and what is the F.
David Bau: Oh, wait! Oh, this is layer. Yeah layer, this shouldn't be I. This is a tyco. Sorry for that. And F is activation.
David Bau: So this is just the the upper one, just like the layer. Wise version of the model forward, pass forward plus activation, input.
David Bau: input parameter with corresponding mask.
David Bau: Oh, wait, wait. This is activation. The F is whatever function linear mapping or convolution mapping for general data for for general cases.
David Bau: So given, input given the parameter based on whatever operation sales or Mlp doing the non non linear activation from the output?
David Bau: Can I ask a dumb question? So probably I need to read this paper. I I am not familiar with this paper, but how does this give you like any additional benefit
David Bau: compared to just clamping the values after after maybe every 100 training steps. No, just clamp the values of M to make it either 0 or one every 100 steps.
David Bau: like every 100 steps you will. You'll take the. You'll take the rounding
David Bau: between 0 and one. You have an unconstrained. What do you train? Want to train an unconstrained matrix M, and then just clamp claim.
David Bau: Yeah, yeah, you you just train something unconstrained and every 1,000 steps, you'll just clamp it. I'm also wondering what happens there. Like in terms of the dynamics.
David Bau: Yeah, like, does it give you any extra benefits? But I think that that's that's really. That's the topic of the banjo paper, probably the rather than this one. Yes, like why, clamping is differentiable. Clamping is not differentiable. Oh, but but the loss is all loss is always calculated in differentiable ways. But you just well, that's what he's saying like, how would you compute a loss like this when you have this indicator function, just the usual. Well, that's what he's saying. That's what this is.
David Bau: So he's clamping on every iteration, every iteration. Yeah.
David Bau: yeah, that's what the V is. The the V is an indicator function to clamp.
David Bau: Sorry, what? Using indicator function. So model training, we need a forward right. The forward is just use the indicator function to select filter the widths right?
David Bau: But what is done?
David Bau: Okay, the problem is by propagation, because the indicator function itself is not differentiable.
David Bau: We need to change something to make it roundable for the pytorch.
David Bau: That's that's the case here. Okay, yeah. So you give see the gradient that we would get. Yeah, yes, because we use the method here just to skip, just to take the like. Those should be here
David Bau: just to take the V as a
David Bau: identity function. Yeah, so just transfer the gradients from other side to the to the process.
David Bau: So I have a question, why not simply use signal
David Bau: instead of having a clip function, because then you can. That's what we're trying to.
David Bau: Oh, right now, like right now, right now, I will show some results later. Sigma is sometimes help.
David Bau: Something's helping from the other session.
David Bau: Okay? So for this one, we simply applied it. And we address the issue for the forward and backward.
David Bau: The model is trinable rent.
David Bau: Any question so far.
David Bau: Nope, okay, do do you understand the equation?
David Bau: Well, I I guess my question is like so pytorch doesn't natively implement like gradients for clamping.
David Bau: Oh, you mean for this one question or so? Oh, I guess so you're not clamping. You're just having an indicator function of include or not include, you're not clamping to like one or 0.
David Bau: You mean the. So you're not clamping.
David Bau: Oh, so M. Is the right. Sam is Vc.
David Bau: vfc, but, like the values in M, you can think of as like
David Bau: the M. Here is totally binary. There are one or 0. Yeah, it is hard mask. Instead of the soft mask, or, like Robert mentioned the sigmoid based mask, which is a software. So so pytorch doesn't, doesn't do it. But it has a 0 derivative.
David Bau: Yeah, which means it won't learn anything.
David Bau: Does that make sense?
David Bau: I guess I feel like someone would have implemented that to to just like include it in the library, you mean? Oh, you mean the straight through the pass through the path. Ingredient though there might be, I don't know. Is there a pass through gradient module that's built in? It might be? Yeah. I didn't use the plug in the module. I just yeah, there is another implementation. We should pretty straightforward to rewrite the forward and backward function just the 2 function additionally and kind of change. This functionality cool? Yeah.
David Bau: it's possible to do that isn't clamping. Clamping itself is a non differentiable function.
David Bau: How do you get gradients for a non-differential. Well, it's differentiable. But the derivative is 0. Yeah.
David Bau: what? It's differentiable.
David Bau: But it's just that when the clamp is active, the derivative is 0.
David Bau: So it's it's perfectly differentiable. It's just like any constant is differentiable.
David Bau: What would you clamp?
David Bau: It's 2. Yeah, I think it's 0. It should be 0. It's 0 almost everywhere.
David Bau: Yeah, I believe the pytorch will make it at least a vulnerable. But there are some potential bugs, not for the code or code class for the logic app.
David Bau: So that's reason why we need to involve such kind of tricky behavior operation into the code.
David Bau: This is educational for us because we don't do sparsification in this lab. So I think that you know, in so probably in Raymond's lab, where you you guys are, you know, sparsifying a lot of models or discretizing a lot of models. You know, everybody's, you know, used to these discretization techniques and sparsification techniques. And we do almost none of none of this work in the lab. So
David Bau: so we're like, oh, what the heck is this? You know. What are we doing here? Right? And so I have a friend who did the whole Phd on this? That's right. Yes, and there's, I mean, there's a whole major area. It's something that we don't have contact with. Usually. So when I have a 1st glance of this implementation, I would also shock. So oh, I didn't even know Pytorch also have the version of the rewrite, the forward and backward to make such kind of stuff possible. Yeah, so that's the logical story.
David Bau: Okay, are we put forward next page, I wonder, like how much better this really is than
David Bau: like just randomly pushing the thing under the clamp around
David Bau: runner pushing. Do you mean random masking stuff? Or
David Bau: I mean, you, basically, you're presenting that you have a gradient. But the thing is not really a gradient. It's just like a random direction. What does it do? Yeah, in my understanding, it is like the, it's not totally random. But yeah, it trying to.
David Bau: They're trying to use the fake grid information from one side to the other side.
David Bau: or we understand in this way, so the master should be naturally sourced one instead of hardware. But we force it as a hardware. That's what the indicator function does.
David Bau: But the written information, at least the trending it is the same.
David Bau: So
David Bau: okay, a little bit hard to hard to hard to describe what I'm thinking right now, so originally the must be a soft one, and at each point of the sigmoid, like Robin mentioned, it is differentiable.
David Bau: but we need to push it out of the hard one. So the Sigma function is from the
David Bau: smooth one into the sharp one.
David Bau: Even if it is approximation, it is not accurate. Great information, however, the trend is correct.
David Bau: The upper bound is also upper, the upper part is also the upper, the lower part is also the up lower part.
David Bau: So that's my rough intuition of this part.
David Bau: And
David Bau: the experiment shows that it is not a random, because we add, the random masking part to prove such kind of learning process is is not true.
David Bau: Yeah. So try it. Wrong.
David Bau: Okay? Sure.
David Bau: Okay, we'll finish the logic part, the methodology part.
David Bau: and we'll go for the experiments.
David Bau: So let me emphasize Audreyonne.
David Bau: The only difference is learnable mask. So it is initially benefit or advantage.
David Bau: because we do not need to change the training logic, especially for the objective perspective. We do not need additional data set because you have and a Uf are the same.
David Bau: So I know, typically speaking nowadays, such kind of features is pretty important, because nowadays the
David Bau: deep learning going to the engineer going to industrial part and industrial company companies do not like this complicated design for the neural neural network architecture.
David Bau: And simply enough, we can go for the experiments
David Bau: before the experiments. I want to make a note that in this work, because such kind of idea is a relatively initial idea for this exploration, and we only focus on mainly focus on the local musket.
David Bau: which means given them.
David Bau: Given the best, fully functional model, we only transferred to part of the model to add the master.
David Bau: instead of the homework.
David Bau: The reason is pretty simple. I'm just trying to zoom in on your slides. I can see it. Okay, help me relax.
David Bau: Where am I?
David Bau: Oh, the local mask, the local mask. The reason is pretty straightforward. We initially tried the global masking, but it is hard to tune
David Bau: the performance taps everywhere.
David Bau: so we try to make it as simple as possible from local masking at the very beginning. But I will show later there are some new ongoing results for the global masking, which also works well. And we found a treat where the tree
David Bau: okay, so for the local masking before the large deal benchmark competing.
David Bau: we 1st do the proof of concept study, which means.
David Bau: given the data set, for example, llama to as a backbone
David Bau: coding domain as a, as a, as a target domain. Given the functional data set, we only use a subset of the functional data. In this case, we use 10%.
David Bau: And another type of parameter which is fixed is we always use a 10% sparsity, which means no matter for the global or for the local, we for the target partition, we drop 10% kept to the rest of the 90%
David Bau: for the proof concept, also for the next main table results.
David Bau: Okay, here is how we do the proof of concept. We group the layers in a row.
David Bau: For example, 0 2, 7 means 8 layers.
David Bau: 4, 2, 7 means 4 layers. Index was 4, 5, 6, and 7 in a row as a group, and which is a target for the
David Bau: moscularly.
David Bau: And yeah, here
David Bau: and we tune the model. Use the method I just introduced to say 2 stuff. The 1st step is, if the if the mastering or master learning has potential
David Bau: to further, to make further improvement. Compare with
David Bau: the black dash line, which is the pre trained, which is a fully functioned model.
David Bau: So this is the 1st point we would like to check.
David Bau: But here, so the back dash line is a fully functional model. To note is the best, fully functional model. I will show the details later.
David Bau: And this card indicates, okay, it has potential.
David Bau: So we are confident to do the following steps.
David Bau: Otherwise there is no audio to to to keep moving on.
David Bau: So this is the 1st point we would like to check.
David Bau: The second point we want to check is
David Bau: which part of the model we we want to focus because we group the layers in a row by different partitions.
David Bau: and for this case the proper, the appropriate model partition is from layer 20 to 23,
David Bau: because highest volume for the for the comparison.
David Bau: So there are 2 targets for this, a purpose. For this proof of concept study.
David Bau: And
David Bau: that's for the 6 6 settings Lama, 2 lama, 3.1 mass coding and instruction following domains.
David Bau: we see the similar trending all the domains, for the the master handling has potential to outperform the
David Bau: black dash line, the best fully functioned model.
David Bau: however, they share different patterns. It looks like the the training are the similar acceptable performance at the very beginning go down and go up.
David Bau: but it is hard to draw the conclusion, because it is
David Bau: pure empirical abolition study for this. But at least the conclusion we can draw is okay. It has potential for all the settings. And second, we can obtain the different modification for different settings. Individually.
David Bau: Probably there is the ideal case
David Bau: from maybe several layers combination, which is, which is good for the all settings. But it is too hard for broad condition for video.
David Bau: Here we only do the naive audition study to say, this audition is a good for individual session
David Bau: so far. Okay, this is in all this setting. It's 10% is sparsity is like enforced on all the mass for just all these. Yes, yes, the all stuff is 10% sparsity, 10%.
David Bau: Yeah, the 10% subset usage is about the fast proof concept.
David Bau: because we want to quickly make sure. Okay, this is a doable
David Bau: definitely, the performance gain is not significant, especially up here
David Bau: and here. But if we use the full side much more better
David Bau: any other question. Do you happen to know what the baseline, the base model performances?
David Bau: Not in this figure will be the next bit. Yeah, because the proofing model is definitely pretty low.
David Bau: So, and the and the regions where your accuracy is going up with masking. Yes, so those that is what you meant harmful earlier like, if you remove those, the performance ingredients is that is that the harmful definition?
David Bau: If you use a harmful definition, that's true, we use the yes, specifically speaking, for example, this one.
David Bau: this part is the most appropriate part.
David Bau: and we choose, okay, the corresponding layers do the mask, continuing to find some ways to remove and see the performance. It goes up or down right? So that's the logic here.
David Bau: Okay?
David Bau: Sure. Like what it mentioned in the following main results for each setting.
David Bau: We specifically choose the model partition here, probably here.
David Bau: for the instruction we choose the 1st a few links
David Bau: for to conduct mask, fine tuning.
David Bau: But the difference of difference between the ablation study about proof of concept study and the main result is
David Bau: we use the full side, but it will also kill 10% sparsity.
David Bau: Okay?
David Bau: Sure.
David Bau: Okay, so this is the storytelling tisser realization.
David Bau: I will go from the very beginning. So
David Bau: llama, 2 7 billion on coding.
David Bau: Coming as an example.
David Bau: we start at S is the training steps. The Y is the performance we start from the
David Bau: preferred model, which is pretty low, and we 1st play to the sufficient
David Bau: full functioning how to make it sufficient.
David Bau: Administer over 50.
David Bau: So the blue line from the free training that we draw down at the very beginning, draw up
David Bau: and reach the best, fully functional parts, and going down due to the and our experimental logic here is, we want to use the best, fully functional.
David Bau: which is the most commonly thinking as the most competitive, strong baseline for the language optimization. We start from this one just before it's going down as a baseline for our message to say, if we, our message, the masking, remove some parts of the model can break the limit of the well trained language model. So that's the logic of the experimental design.
David Bau: Yeah, sure.
David Bau: Have you tried?
David Bau: Do you call this mask masking right like, have you tried masking from?
David Bau: And you overshad?
David Bau: Do we care? Yeah, we try but little bit experiments. The performance is not that good?
David Bau: I roughly remember, if we tried the mask controlling over here, starting from here, it can slightly better than, but not as without this.
David Bau: Yeah, so it is called as a standing on the shoulders of the giant. Correct something like that. So, okay.
David Bau: this is after the Fu fangqiung.
David Bau: the yellow Star. We call it the fft best we use, we use it. We use this checkpoint to do the mass functioning
David Bau: commit further improvement.
David Bau: So that's basically logic of this of this work.
David Bau: So if we use some second best checkpoint to do so, probably not convincing.
David Bau: Hold up Google.
David Bau: How confident are you that this is a significant improvement, and
David Bau: that you couldn't trace those 2
David Bau: that you couldn't raise the blue line by just changing your learning rate a little bit
David Bau: you mean, like, how robust is this ordering of the
David Bau: performances? In the main table, we can see some standard division report. Yeah, it does have some variance. But compared with the margin the margin is significant.
David Bau: You might mean.
David Bau: oh, what's that you're winning? Yeah, we are winning. Yeah. When when you compute the confidence intervals. What do you sample across like? Is it different? Random, like, what hyperparameters are you changing to? Oh, you mean how to consider about how to random change the seat at the very beginning
David Bau: the only fixed to start stuff is the production mode. So you change the seat but you don't change the hyperparameters of like the learning rate and
David Bau: other not in this work one of the review, for for our previous submission is the reverse challenges that probably you need to fine-tune the
David Bau: phantom partial parameter of the full phantom.
David Bau: We tried a little bit, for example, the learning, rate, learning decay whatever the the output number whatever. But it looks like, because this fully functional is not the
David Bau: combined one. It is a integrity one we start from the very beginning. Follow the official repo for the hyper parameter until it's over 15. We choose the best point just before those. So that's the logic. There is no trick
David Bau: between. And after the yellow star of the blue line there is no trigger on this one, because.
David Bau: I just
David Bau: there is this paper called like layer, selective rank reduction that had like pretty similar. Yeah, Professor, suggested me previously, I will talk about it in the last slide. Yeah. Share some exactly the same insights, with some difference. I will talk about it.
David Bau: yeah. Any question for this page this is, should be the main story of the whole book. Well, so now you're starting to overfit at the 31.7. Now, so the natural question is, can you do it again? So you go up to 31.7, and then it starts to overfit. So back up, go back to the 31.7 and smaller mask.
David Bau: Do more mask. Fine tuning, right? Does it help further?
David Bau: we didn't try the smaller master ratio in this way, but we tried start from this point, we do the full function.
David Bau: But yeah, after 400, we're doing, that's on mask hunting iterative progressively
David Bau: helps. It helps and make further improvements, but unstable.
David Bau: We are trying to figure out what is going on, because I will show you later, for the after the paper sharing. There are some various like the okay. The 1st round of the Mft helps, but the fft after the Mft dropped.
David Bau: but the second round of the master learning it helps again.
David Bau: So it is a little bit weird, I mean, isn't like, might might the argument be that your mft you're effectively you're trying to.
David Bau: You're like
David Bau: kind of reducing the parameter space. Right? You're you're fighting overfitting by reducing the parameter space. Which is why why, I would think, oh.
David Bau: if you're starting to overfit again.
David Bau: Then the reaction is, oh, reduce the parameter space further. Oh, right? That makes sense.
David Bau: Yeah, yeah. I think that's a good option to try. We we maybe now it killed some weights of that, maybe don't want to come back. So yeah, when you do the full fine tuning again, how do you initialize your optimizer? 2 options? The 1st one is, keep 0 as it is.
David Bau: The second one is oh, I meant to optimize the state.
David Bau: Oh, you mean the full functional occupation, all the other, all the momentum. What do you do with other? Oh.
David Bau: oh, what do you mean by what? What do we do? Well at the training, like once you reach like this, 29.3. The atom has built up some momentum and normalization terms. Right?
David Bau: When you continue at 31.7. Do you reinitialize Adam with these?
David Bau: for the red line we reneutralize?
David Bau: So this slide have the bridge here.
David Bau: This line has no break. So for the master functioning, we reinitial all the stuff
David Bau: because you have different parameters, anyway. Yeah, right? But now, when you do another fft at 31.7.
David Bau: I didn't hear. Yeah. Do you? Do you keep your old optimistic state that you had from Adam? Or do you ramp up
David Bau: in in our experiments, followed by this point. Yeah, we reneutralize all the stuff. Oh, but maybe you should pick up your optimizer from the 29.3. But I honestly don't know how much this
David Bau: I guess you train with anyways. Right? Yeah, yeah. Yeah. So then, since 2, the the blue and red optimization are different. So we do the initial rotation. But I think but blue is this. Blue would be the same as blue if you repeat Blue again. Oh, yes, right. Yeah. So you could pick up the old optimizer. Yes, yes, I think that should be the
David Bau: possible way to worse the trial. So what's the reason? Behind the 10% sparsity
David Bau: like, why did you choose the number 10? Number 10 manually set
David Bau: publishing later will show number 10, and probably not the ideal traits for offsetting.
David Bau: It is optimal, for I remember properly for coding, but for another one it is not probably 50% of 5%, as far as it is a is better.
David Bau: And also in the global masking.
David Bau: 0 point 1% is the best.
David Bau: So 5 and 10 is too large. So that's also the another founding of the of this work which I'm exciting about is we're using the sparsity for probability standard of efficiency.
David Bau: People always use the sparsely. Okay, we need to comprise the 90%, 99% as much as possible. But right now we're starting the opposite way.
David Bau: 0 point 1%,
David Bau: probably 1% or 5% that makes sense. Otherwise, a 10%, 50% ablation will show performance drop dramatically.
David Bau: So sparsity has different functionality. That's the record. Hello, inside office.
David Bau: I will show it later. Have you tried any different size model rather than 80
David Bau: only different size model? Larger? No, because comparison limited. Smaller. Yes. Oh, that's another story.
David Bau: which productivity to totally different behavior.
David Bau: Gpt, 2 is pretty small
David Bau: we start from pre training of the Gpt tool and the mask. Fine tuning can be full functioning
David Bau: from scratch, from scratch of from the pretrial model. Yeah.
David Bau: So the hypothesis here. Oh, I didn't mention in this slide, but I want to make it all of it. So
David Bau: the hypothesis here we're thinking about is Gpt. 2 is pretty small current llama model is relatively large.
David Bau: but the difference here is the saturation level of the different model are different. Gp, 2. Model size is small, but the data set probably
David Bau: give us more than
David Bau: the model is now well optimized, and the over parameterization of the Gpu 2 may be severe than the normal.
David Bau: So llama is bigger is larger.
David Bau: However, the information has been fully occupied of the model structure, so it is very hard to use a sparsity to, to.
David Bau: to do something.
David Bau: So that's the reason why for the 7 model we can only reduce probably 0 1 0 point 1%. But for 2 it is smaller, smaller. But we can remove.
David Bau: I roughly remember some extreme case. We can remove 80% 50% to replace the full function.
David Bau: So at at that time we are pretty exciting. Okay, Gp, 2 works pretty well. But when we go for the larger model, okay, everything changed.
David Bau: So that's the story behind this work
David Bau: in principle. Could you optimize the sparsity level along with the mask? Because that is like too annoying to you? That's based on different implementation details
David Bau: a naive solution for the lack of learnable ratio and learnable spacing. It is a signal, because, like the okay, there is news. Make it resolved. Try some criteria to. Okay, this part is removed dynamically or capable dynamically.
David Bau: But right now we haven't seen the very promising results based on the learn more issue still. Fine
David Bau: sent to you.
David Bau: I was gonna ask. So for the Red Mft, do you restart
David Bau: your data over? Or do you start with like the data that you would have had at training steps
David Bau: restart. So I assume, like, when you're doing fine tuning, you have some data set right. And for the Mft, you said, you're using the same data set. Yeah, so do you like use the same? So are you using like the fine tuning data from like steps, I guess by steps. Is this like a whole epic through the entire fine tuning data set?
David Bau: Oh, oh, I see! Almost 500 steps is roughly, is
David Bau: 2 or 3 I books. So already
David Bau: shuffled all the data samples here. And for here we start the definitely we start the data loader.
David Bau: Use the same data set for the Mft, I see.
David Bau: So those 2 points are not equivalent.
David Bau: The blue and red here, please.
David Bau: Yeah, that's what you're asking, right? Is that the same data? Or is that like different data?
David Bau: Oh, this is not control for the for the batch batch level, just randomly shop.
David Bau: Let me think.
David Bau: No, no, even if we keep the seat because this line is consistent operated, this line has a break. So there are some difference here.
David Bau: Yeah, we we should definitely make it more accurate to to eliminate any other
David Bau: noises. Yeah. But then you also did standard deviation studies. So I'm sure. Yeah, you would have somehow calculated it. Yeah, it is relatively significant.
David Bau: Okay, so for this page, no problem, I will move on.
David Bau: Yeah. Same pattern. I'll show all the results. 6 settings.
David Bau: 2 backbones, 3 domains.
David Bau: And we can see. I I just picked the most significant one. So this is definitely significant.
David Bau: This is even more. This is almost a 7 point improvement for the instruction.
David Bau: But mass is relatively hard.
David Bau: We can see the improvement is less than 0 point 5 point.
David Bau: and here is only 0 point 3. So math is hard.
David Bau: Later observation will show. Okay, mask can be improved further, but different setting, which is relatively hard to hey?
David Bau: Normal numerical results with standard division.
David Bau: Lama, 2, 6 settings.
David Bau: Oh, by the way, we have 2 fully function set. I I forgot to mention.
David Bau: yeah, just take it, this one as an example. So the blue line here until we got we reach the Yellow Star. We have 2 fully functioning. 1, st one is which was a mixed up central.
David Bau: We use 3 domains, data, mass coding and instruction to tune it together as a full functional version.
David Bau: The second one is only specific. We only use
David Bau: mass data on the coding data instruction data for the full functional version. So that's the difference all the about the full function. But the problem point is
David Bau: for the master functioning. We only use the domain specific data. We do not use a mixed update.
David Bau: Yeah. And we have a extended version of the evaluation.
David Bau: The cross domain mask, fine tuning evaluation do not make further improvement. So such trend of fine-tuning during the domain specific way instead of a channel.
David Bau: Yeah, because I assume that should be the data set issue. There are no common features or knowledge sharing with different, even if the fully fine tune sometimes.
David Bau: So in the table we can show
David Bau: we can see 2 parts domain specific for the full. Thank you. Mixed up domain for the full. Thank you.
David Bau: With the best fft is the most competitive caseline we need to compare with the same as here. These 2 people, this 2 sub table share the similar patterns.
David Bau: and, oh, forgot to mention in this
David Bau: in this favor we also have
David Bau: we also our our one masking and random masking here and here to show. Okay. So the master learning part is, is not a true one.
David Bau: I believe this orange dash line here compared with the red line. It shows that okay, the straight through grading estimator makes sense. The random gradient information may not help compared with the random. Okay.
David Bau: you and Douglas.
David Bau: So in your table, you show multiple data sets
David Bau: improvement trained on specific data set. Or is this like a general improvement? You see by training mode
David Bau: one mask? Is it one mask or 2 different masks?
David Bau: Because the master functioning use a mixed soft contained data set within bombing. So one mask for 2 evaluations.
David Bau: Okay, I describe them this way. These 2 results comes from the bundle.
David Bau: Yeah, but same. But in the bed trading.
David Bau: Oh, what the training has both, yes, combined within domain and mass domain combined data set for the master functioning report the results simultaneously.
David Bau: Wisdom, cross domain. No. Do you see any out of domains out of domain improvements? Do you see them
David Bau: like data sets that you have not trained on? Do they still improve with this mask.
David Bau: You mean the within domain. Other data set or course domain
David Bau: such as I train on math. You train on math test on. Gsm, 8 K. Does that work?
David Bau: That's the logic here. But you're asking. I train on Mask how this going with coding. So that's your question, or
David Bau: plain or Gsm, 8 K data set only and test on math data set.
David Bau: Oh,
David Bau: sometimes it works that based on the overlap between different data sets. I believe, even if it doesn't work, I think it's interesting. Maybe it's because, like, there are some memorizing neurons that say, I know how to work on. Json, yeah, yeah. But even if they're within the same domain, the pattern of the data set probably different.
David Bau: So we we made a lot of initial try. But the problem is not stable. So we use it unified way.
David Bau: Okay, we report with a standard division. I just subscribe
David Bau: And all other story has been described as in the in the previous slides. So basically here about the numerical showing we can see, the
David Bau: instruction is most significant. Mass is hard, nice. It's possible.
David Bau: So this is for llama 2.
David Bau: And this is for number 3, the same. The message, part
David Bau: coding is okay. It's good. The instruction following the most significant.
David Bau: Okay for the virtual results. Any question.
David Bau: what does this pruning effect like you've shown improvements? Does it damage anything damage?
David Bau: And the text coherence lost or like something. Did you lose something with this pruning? Or do you only see improvements across all those
David Bau: within the domain? There is no damage
David Bau: across the domain. For example, if we extend this table
David Bau: the lower part. So this is one row 3 column.
David Bau: If we're made out of the 3 of those 3 columns we train on the math embedded on coding.
David Bau: at least, if not worse, it has no further help but the damage, I would like to say there is no other damage.
David Bau: and one interesting stuff is, if we train on mass cell coding
David Bau: instruction can be improved. So it is.
David Bau: It is reasonable, because instruction sometimes follow just a human pattern for the dialogue and human
David Bau: in eventually have some data to house the infrastructure
David Bau: right? So the damage part. Oh, another damage part is about the like, mentioned, the memory part.
David Bau: or the training time, or the training budget overhead.
David Bau: I didn't list in the slides. But it is definitely acceptable because we're doing the local mask, the memory even less than the 45,
David Bau: the overhead definitely because we need additional fine tuning based on the best fully funded. But the overhead is very limited.
David Bau: The time is as- as long as the overhead of the master, so they're acceptable
David Bau: and okay, abolition. The 1st one first, st important abolition is local masking ratio.
David Bau: Oh, that measure that rocket, mesher.
David Bau: 10% is showing here
David Bau: for this setting. It is the optimal
David Bau: for this setting. It is not.
David Bau: Probably 5% is is better.
David Bau: But do we have other or fine grained ratio in the middle perform better, probably.
David Bau: But here we can say, Okay, the ratio does matter.
David Bau: So this is the 3rd of this is the 1st finding. The second one is
David Bau: Fs fft definitely needs smaller ratio, at least in this kind of data and models that I shared the potential hypothesis. Probably this model is fully separated.
David Bau: A larger ratio definitely drop the performance.
David Bau: Okay, so this is the 1st operation.
David Bau: The second operation is.
David Bau: why is it larger ratio, worse than just fully fine tuning. Fully fine-tuning would be a ratio of a hundred percent.
David Bau: But fully fine tuning is ratio 0.
David Bau: But like you're fine tuning all the parameters, as I hear?
David Bau: You're masking all the way to the left. Yeah. Yeah.
David Bau: And since we drop that answers your question, right? Yeah, yeah, it's okay. Oh, never mind. Yeah. The the blue line is corresponds to 0 all the way to the left, and it matches the 0. Yeah, yes. So there's overlap at the very beginning.
David Bau: So this is the first.st The second one is robo master.
David Bau: It has potential, harder tune. So that's the conclusion specifically for mass doming. We can see
David Bau: this improvement. The 2.1 is definitely larger than the main table shows. It is based on the global Mft instead of local one
David Bau: and another detail here. All the mapping the difference compared with previous, all the stuff is here. We do not use a racial based indicator function. We use a threshold based.
David Bau: Specifically, we choose negative 0 point 0 3 5
David Bau: as a threshold to do so.
David Bau: This is definitely the have it at home.
David Bau: And we can see some larger benefits compared with the local one.
David Bau: because the racial part is definitely easy to tune. Compared with the threshold.
David Bau: however, we can see mass still has improvement, but less than local one that even worse in the llama, 3.1 model, it drops significantly.
David Bau: So such kind of a mixed up result indicates. Okay, it has potential but hard to keep
David Bau: prospect and public
David Bau: at the very beginning. We'll try the Google Mask before the local mask. But it cannot work stably. So that's the reason why we would go to the local mask and regard it as the obliged
David Bau: great
David Bau: when you chose this hyperparameter of minus 0 point 3. Whatever you just did. The you concluded some validation. Scores for different choices of this parameter. Yes, yes, and then that's what just that's what came out. Yes, yes, because this part is pretty hard to find a proper one. We just want to. Okay, we do the engineer part to see if the global maximum has some potential.
David Bau: So that's the motivation for this table. We definitely have made this, this kind of number is relatively tricky. We tune it?
David Bau: But at least we they show some potential while we're following research works. Yeah.
David Bau: And let's make the summary
David Bau: the pipeline is pretty straightforward. The typical pipeline run, the model pre-training fully fine tune, and we follow it up at the must continue.
David Bau: The 1st point I want to summarize is performance is directly
David Bau: is a direct benefit for for the proposed model, which is pretty straightforward, there is nothing promised to to add it on, and the second point is
David Bau: the protocol.
David Bau: because, we do not involve any complex, the training objective. We use the same data set. So it is pretty general to collaborate, capable for working with other
David Bau: scenarios or optimization steps, probably reinforcement learning under the multimodel learning stuff. So it is all the doable.
David Bau: The second benefit we can naturally upgrade the a large foundation model optimization.
David Bau: The 3rd one, like I mentioned, is a mask functionality.
David Bau: So to the best of our knowledge, for the such large scale model. We are the 1st to study the sparsity for model capability, like like I mentioned. So we're doing something like less is more.
David Bau: This is not always the last, you know. It's probably it. It is more.
David Bau: And I believe such. The sparse can be played a role in the for that issue. For example, nowadays people using the auto breast model for visual generation. There is a very severe for that issue after the AR visual generation image generation. The understanding, visual
David Bau: capability may drop. So could we use this? Probably. Okay, this function is for the AR generation for visual, that part of our understanding. Such a very rough one. Actually, if it is a doable future. And 3, rd one is for the interpretability that's definitely area of expertise
David Bau: for circuit stuff. So it is still have potential.
David Bau: So besides, of the all staff are showing above performance. I would like to emphasize these 2 points to summarize this book.
David Bau: Have you tried like whether you can combine 2 masks like
David Bau: one Tsm, 8 K. Mask and one poem mask
David Bau: we haven't. We haven't. That should be the very beginning story, because at the very beginning, before all the department starts, we want to do the paper like mask soup.
David Bau: 9 models, and especially with a very small ratio. Probably. Okay, this, this small mask can cover this data set, another can cover another one this part can cover one domain, this cover and another domain. Okay, finally, we just mask with very tiny removing. The model can be a integral one for the general universal capability.
David Bau: Probably. But we haven't tried. That's the definite good question. Do you know, laser.
David Bau: yeah, like, like I mentioned, I will talk about the later paper later. Sorry. The difference on the same same point. Yeah.
David Bau: okay. And so I had actually had a question. So when the modem performance goes down with training, well, our 1st intuition is to blame it on overfitting right?
David Bau: And
David Bau: so I'm trying to given the given that we have seen your results. I'm trying to rationalize what is happening. Do you think that this sparsity is kind of like acting as a regularizer? That is kind of like boosting performance while making sure that it retains some some knowledge or something
David Bau: like, I'm just. I'm just asking, what is, what is your intuition of what this varsity is? Is it just a regularizer. Or
David Bau: oh, to be honest, I haven't think about the sparsely in this way. The regularization
David Bau: right now. I think it makes sense my intuition is from another way. Yeah, like.
David Bau: during the optimization. At the very beginning, we keep the structural integrity.
David Bau: We just assume. Such kind of territory is necessary.
David Bau: There is no flexibility of the structure aspect. So, in my opinion, the the the
David Bau: deep neural network optimization is a combination of the body and the shape.
David Bau: So what the combination of the value and the shape, the model structure. So this, typically some, some ideas from the pruning stuff.
David Bau: So we can.
David Bau: we can train, the weight which is the typical one, right, the dance model train the weight. So this is one side. The other side is we can keep the weight. Train the train, the ship.
David Bau: That's what the Mft does.
David Bau: We can definitely do the 3rd way, which is, we can train the models and the structure simultaneously, which is a dynamic, sparse network stuff.
David Bau: So my intuition is relatively high, level and rough it is.
David Bau: It allows model 2 with more flexibility to do so.
David Bau: Yeah, not only from the weight level as well as from the structure.
David Bau: sometimes removing someone can involve new knowledge instead of only change. The way is the knowing new new information coming? I don't really think so. Sometimes removing some structure change. Some structure also can involve new information. That's also the difference between this work and the laser, because laser do not use training.
David Bau: Laser doesn't get training. It's training for you.
David Bau: That's more like the from the noising aspect. Because but here we involve training.
David Bau: which means, okay, remove some width also is A is also the learning perspective. Instead of the denoising perspective.
David Bau: Yeah, that's my thinking.
David Bau: it'll be rough. So so the regularization perspective, I guess to me helps explain the difference between Gpt. 2 and the bigger models. Do you think that story? Is there some story of the shaping that also is like explains or could project? Why, Gpt 2 is is
David Bau: easier to do this, for I think this is pretty
David Bau: good point. Could you explain more? Could you? Yeah, share- share more about your insight?
David Bau: Well, just the the regular regularization idea is like, well, so like Llama is already kind of like, maybe regularized, because it's it's overtrained, whereas Gbt. 2 is like overfit because it has, you know, too many parameters for the number of data points it has. And so by applying this this
David Bau: this regularization mask, you're you're kind of like adapting or compensating for that.
David Bau: and I think it's visible to fix it.
David Bau: I think it's good, both of you guys.
David Bau: So probably the the overfitting level of Hpt 2 is severe than the dilemma.
David Bau: It needs the representation of penetrator even more compared with the number.
David Bau: So that's your logic.
David Bau: Yeah, I I see that's I think that's that's pretty good. I I haven't think about that. But
David Bau: I agree with that at least right now. Yeah, thank you very much.
David Bau: You just like
David Bau: I'm thinking about in like laser, like they use the Svd basis, and they remove some of those, and they're not necessarily like aligned with like like a hard mask like ones and zeros with the weights like it might be like a linear combination of the weights.
David Bau: I don't know if you maybe you're touching on it more later, like the idea of like a soft mask versus like a hard mask. I'm just like curious, because if you just like clamped it between 0 and one this, this version is hard one. Later on, one soft mask like we discussed, we're trying, but no promising results so far. Yeah, yeah, let's talk about the laser later. Yeah.
David Bau: okay.
David Bau: Meaning, okay. So
David Bau: what is ongoing right now? We do. Some follow up ideas based on the thinking here
David Bau: the 1st one is not wrong. I could discuss with Professor.
David Bau: If we do the Mft, and iteratively allow the model with more structure, flexibility. What will happen. This is the 1st one and second one is
David Bau: I also mentioned it. If we can just replace the typical, fully function with the master function
David Bau: which everything does give you 2. But
David Bau: about the paper I just mentioned, we we didn't achieve it. But the new results show some possibility.
David Bau: Yeah, we just regard this one. Regard the mask, continue at the foot of it.
David Bau: So I summarized the table here.
David Bau: That's all we've got for the new part
David Bau: machine is limited. We in this run, we use a deep state coder, 1.3 billion web 2 is more. So the conclusion probably is different from the
David Bau: 7,000,000,001, but at least have a try. We use the coding domain and Bpp for the evaluation.
David Bau: Let me expand the numbers here. The portrait is 4 6.2 come up
David Bau: the 1.st First, st the true rule is to open the 1st 2 route here
David Bau: means the regular full function model.
David Bau: So start from 46.2.
David Bau: The 1st output improves. The second output of full function also includes the 3, rd keep that the same, and it starts showing the overview. So that's the like the baseline as we mentioned. So this is the
David Bau: like, the best fft stuff.
David Bau: and this rest part of the table shows the Mft and fft collaboration, which means output one, we do mass learning output 2, we do full function.
David Bau: Apple 3. We do mass learning. Apple 4, we do the math, so on, so forth.
David Bau: So that's the trying. And this table can answer a question about the previous 2 ideas I mentioned. We can see that in
David Bau: well, all the political numbers here.
David Bau: if we use the local masking, it shows some promising rollouts.
David Bau: First, st the almost a 50% which is definitely larger than the portray model.
David Bau: This is benefit by the combination of mass, fine tuning, and full fine tuning.
David Bau: This is the 1st point I would like to share. This is a weird part like I mentioned. So when we do the attrition there, the drop probably seems to the inherit of the
David Bau: Adam optimizer, hyper pumper stuff, probably about a potential solution and give other advice.
David Bau: And the promising stuff is after the job is still growing up again.
David Bau: So that's the interesting part of the iterative multi-round collaboration with 2 optimization methods.
David Bau: So and the second part is, we can see that this line is for the local. The other lines are for the global.
David Bau: Now we can see why global is not working at the very beginning, because the sparse is too large.
David Bau: 0 point 1. The performance is pretty bad.
David Bau: They're one of their 5 still bad, not acceptable.
David Bau: As long as the sparsely post go smaller the performance increase.
David Bau: But when we do the 0 point 1%,
David Bau: we can say, Okay, the one I hook Mft
David Bau: can replace almost replace the fft. The best mft.
David Bau: So this part is.
David Bau: even if this is on the smaller model, 1.3 billion probably story of different story. But here we we do see some promising. So this is a second part
David Bau: would like to share for the promising sites. And the 3rd part is okay, based on this.
David Bau: Very small global sparsity.
David Bau: Okay, the performance can be boosted to 51
David Bau: we never see such a certain number of of the previous experiments, especially on the full time tune it dropped quickly
David Bau: for the next books we're still trying to figure out, okay, is there? What's going on inside? Probably the optimize optima optimizer initialization stuff.
David Bau: Yeah, that's this table. Just we jointly make the experiments under time for the results.
David Bau: Excellent question for me.
David Bau: So when you perform fft after Mft
David Bau: do you also tune the weights that Mft has kind of like zeroed out in this version? We tuned it okay, but it still is not able to recover that performance. Yes, yes, that's the potential. Another reason, because in the typical sparse, sparse training before Chatgpt
David Bau: offer is 0, out with 0 out all the time.
David Bau: But here
David Bau: we treat the 0 out with other ways to turn it again, resume it. So that's probably the
David Bau: training dynamic issue here to to make the performance drop
David Bau: in my last training project. It was the biggest game changer to just change to more. But I I didn't try that many combinations like you did.
David Bau: you know? Like, move on, go on. And so like the correct generation of devices. Yeah.
David Bau: So they what they do is they orthogonalize the gradients a bit.
David Bau: Okay, or something.
David Bau: Okay? Oh, what's the name of the paper? It's literally a Github package that you can use out of the box.
David Bau: Second of this one.
David Bau: They're trying to beat Adam. They are kind of beating it
David Bau: on my one task, MUON MUON
David Bau: did you experiment. I was just happy that this was I should.
David Bau: There are a few right? There's also soap. Yeah, I don't know what the mistake.
David Bau: Morning.
David Bau: Great.
David Bau: I will chat you later. Thank you. Otherwise.
David Bau: Yes, that should be for the last page of the ongoing results. Yeah. The last
David Bau: yeah. Last page, the laser.
David Bau: because I have discussed this paper with Professor David Ball several weeks ago in at Yale University, and Professor, just to suggest me to read the paper.
David Bau: and I just briefly summarize the difference here very briefly, my understanding laser replace the trained which
David Bau: from the low run approximate approximation, like the grant reduction stuff
David Bau: like the matrix stuff, as we do or whatever. And the fisher here is, there is no training, there is no additional training, and the
David Bau: logic behind is like, like I mentioned, that is like the penalty term, to recognize.
David Bau: Deny some stuff clustered by the training before of
David Bau: our paper. Our works, compared with the laser we share the similar logic here is less need involved.
David Bau: Even if from different ways, we are trying to remove the exactly concrete value
David Bau: of the of the portrait parameter laser tries to transform it into another format.
David Bau: So this 2 part of the metrics that are equivalent from the functionality, but from from the numerical sides. They're different.
David Bau: but our method we do not make such significant change. So that's the difference between our work and the laser. But high level logic is less clear. It reducts the
David Bau: direction.
David Bau: And we're well ours. We're that is the exact number.
David Bau: So I'm gonna walk around.
David Bau: The one advantages of our work is, we involve training
David Bau: like I discussed with you that the
David Bau: probably the inside be behind laser is denoising. There is no additional information involved, because there is no training involved.
David Bau: What we can do is just to make the estimation of the current deform issues.
David Bau: The benefit is potentially reduced by the Denois.
David Bau: But our method we involve training that's more potential involve new knowledge collaborating with other optimization stuff.
David Bau: Probably further improvement. Like I shared before the multi round collaboration replace the typical fft stuff.
David Bau: That's Number 14.
David Bau: Very short, great summation about Asia, and all prepared the same idea. And the difference, and definitely stands for the laser paper. And
David Bau: yeah, I think it should be the last page. And
David Bau: the final page is, thank you. Just let me know if you have any questions.
David Bau: So in the localities of which layers are you knocking out? It's the middle to late layers, right?
David Bau: Based on the proof concept. Yes, but isn't that where knowledge is? So shouldn't the model be done with?
David Bau: We didn't do such fine brain investigation like there was the wrong paper, does. We didn't do that the partition we're trying to force on is all the linear map.
David Bau: including the tension cookie remading pretty big old and oh.
David Bau: marketing and mobile team. I'm hoping.
David Bau: So we do not use. We do not try to trace. Okay, which kind of knowledge is located, which location we do not do that.
David Bau: And my second question was about the shape idea, the idea being that sparsity changes the shape of the model. The change of shape is coming from weights being set to 0 right?
David Bau: And
David Bau: you can see that that's 1 feasible weight. So is there a reason why just fine tuning alone can't figure that out on its own.
David Bau: If it's really that important that these particular weights should be 0, fine tuning should be able to just push those weights down by itself. Is there a reason why that does not happen, or
David Bau: could you repeat it again? So with sparsity? What you're claiming is that settings particular ways to 0 changes the shape of the model. Yep.
David Bau: but doesn't fine tuning. This normal training also changes the shape of the model. If that's the definition of cleaning shape, because I can just reduce it
David Bau: value of a particular weight while training. And that's also,
David Bau: I agree, your thinking says, that's all about the
David Bau: all about the level, right? So you can regard the regular training as a soft version of the structural
David Bau: adjustment.
David Bau: I agree. I mean, that's definitely has some
David Bau: connection or equivalence between them. But here the from the here the the case is, we just push it hard into the vital one.
David Bau: Yeah. So if you, if you regard as a soft one, if you regard a soft mask also as a mask
David Bau: I pray. So that's also the router training. Also change the shape software version. It's not authentication
David Bau: kind of on a similar line like, I don't know. I mean, I thought it was really interesting when I learned about superposition for the 1st time. And I feel like, if you're thinking about like, you know, representing things in non like neuron basis spaces, then.
David Bau: like the binary mask I would like assume would be like kind of restrictive. For like
David Bau: like masking out like, because 1 1 to one with parameters. I don't know. I just I because I know, for, like laser like they, they don't like assume a specific basis. I wonder if, like, you were able to do like some kind of comparison like, because if, if, like the binary masks and and assuming like that, that, like per parameter basis is actually more beneficial. That would be like, really interesting, too. I don't know
David Bau: if we combine the metrics. The composition was a loss for stuff. Or.
David Bau: yeah, yeah, just just like, instead of just doing a binary mask doing like just
David Bau: just clapping the the weights you're learning between 0 and one so that it can smoothly interpolate, because the idea would be that if you know, maybe the direction that she wants to like
David Bau: 0 out is not necessarily corresponding to like one parameter, but like a linear combination of a couple of different parameters. And so you can't like fully access it without like maybe losing some additional information if you're just doing binary masking.
David Bau: Oh, yes, I I agree. So
David Bau: in my understanding, you're you're mentioning about the the vital mask on individuals. That's true.
David Bau: That's true. That's too hard.
David Bau: Probably a soft one. Probably consider the connection between different ways or group of the waves may help.
David Bau: I agree, I agree. But right now, I do not have the concrete solution to do the how to find the which level
David Bau: connections that's definitely for the interpretability. It is hard to answer this kind of question.
David Bau: Yeah. Yeah. But I think your thinking is pretty
David Bau: promising for the for the future one.
David Bau: Unfortunately, I don't know how to come with it on sort of solution right now. Yeah.
David Bau: thank you.
David Bau: Random idea like that big.