David Bau: Today.
David Bau: Tab Michael Ripa, who is our are up, you know, very pivotal engineer
David Bau: on the Ndif project, and who also does a lot of interesting research.
David Bau: You know things that he's he's he's published as previously in his undergrad. And he's he's continuing to work on some new new research now.
David Bau: but but what what we know Michael from is he's he's really the person who's got, and if
David Bau: working in practice, so end of has these 2 parts.
David Bau: It's it's an interpretability library, and it's an interpretability deployment.
David Bau: And without Michael we wouldn't have the deployment at all.
David Bau: He's he's doing. He's doing that hard part of it.
David Bau: So so. But Michael is art also. Done a bunch of interesting research, and I don't know anything about it. And so today he's gonna tell us about his project
David Bau: having to do with editing morals.
David Bau: So welcome, Michael. Thank you, David.
David Bau: I wish I knew about ended when I was doing this research. I'll say that. So Hi, I I'm going to be talking a bit about research. I did kind of going into my senior year at my undergrad. And really, actually, I was doing this research, basically up until I started working here. I actually remember specifically, when I 1st got the initial offer email, I
David Bau: like immediately thought to myself, I need to get my results completely done this weekend get off the Vm. I'm on, and just like, leave this, like, you know, have it done, and go on to new new chapters of life. So this this is like, very much like a reminiscent like
David Bau: the crunch of graduating. It's like very. There's a lot of memories, you know, late nights at the library. So. But I I'm I'm excited to present it. I'm not sure how like overall I'm a little bit hesitant about. It's like rigorous. It's like one of my 1st actual like research kind of projects. It's certainly like my 1st interpretability related project. But yeah, I'm excited to see people think, etc. And I'll try and keep it. Fun, too. So
David Bau: let's
David Bau: there we go. So one backstory. So how did I get into this topic? Specifically? Well, it all started in Berkeley, as things often do. I went to some existential risk retreat that they have. They have them here in Boston, Berkeley, London, and a lot of undergrads go to them. I was one of them, and they were talking a lot about, you know, like AI existential risk.
David Bau: like, you know, bio risk stuff like that. It's like a very I don't know. It's it's a it's a thing. And naturally, being surrounded by lots of people who were very passionate about those topics.
David Bau: I got pretty convinced on it, and pretty, I guess, freaked out so. That was that sort of shifted a bit. My sort of at the time priorities. I was a lot more. I was trying to
David Bau: spend more time thinking about the alignment problem and related like research. And I was also just at a point where I needed to do a honors thesis. So I was carving out time to like, think philosophically about it, and I also had stumbled across model editing earlier in the year, and I thought it was really cool. And so I started to try and bring those 2 things together. I started to think, Well, how does model editing have to do with alignment?
David Bau: And I actually spent maybe too much time thinking about it. I got really in the weeds with some philosophical thoughts, and I came to my own conclusion that
David Bau: if you want like to align a model, you cannot only get. You can't get away with just editing facts, factual information, and the sort of argument I kind of came up with and constructed was imagining a hypothetical scenario where say we wanted like, you know, Claude, to, you know, not. Talk about poisonous mushrooms to users, you know. You don't want it to like in any way, shape or form. Try to, you know, inform the users about which mushrooms are poisonous. It cannot.
David Bau: you know it has to avoid it at all costs.
David Bau: And if you kind of take the model editing approach where you're editing facts, maybe one way of doing it is, you just take all the associations with a particular poisonous mushroom like the desktop, and you kind of just shuffle it like you just point them different places. So you know you go from it. You know you have all these properties about it, and you just point them to random words. You sample from your knowledge base.
David Bau: And I argued that oh, like, if you do that.
David Bau: the model still could try to do the goal or try to go against the goal. For one, it may be aware of the fact that all this information shuffled because it's quite obviously logically consistent on these things, and it may piece together. Oh, the category of poisonous mushrooms seems to be something that I don't have any information on. It could do things without even knowing anything about poisonous mushrooms. It could. It could rise with conspiracy theories.
David Bau: It could relearn it through few shot prompting. I mean, that's something that also is pretty relevant in that respect. And so overall. I mean, this was
David Bau: not like this isn't super rigorous thought, but just sort of the trainer. The the sort of
David Bau: mentality I had this summer going into my senior year. And so I wanted to
David Bau: like change things a bit. I was thinking how model editing research is factual. It's great for a lot of short term. Things like, you know, sensor information important, you know, avoid having someone's credentials leak
David Bau: you know, up to date, you know, updating who is the President of the United States. I mean, there's a lot of reusability is an important factor. But I was thinking, this technique is cool. It's very lightweight. It's very precise. It's computationally efficient. Rlhf is very, you know, computationally demanding.
David Bau: And you know, there's it's hard to control for what you're what sort of values you're, you know, pushing into the model. And so I thought, would model editing be something that could be done for that? Or would model editing be something that would work.
David Bau: I kind of just skipped over these guys. But basically, I finished my summer and decided to pursue kind of the intersection of AI ethics and model editing. This is something that at the time I don't know now, but it did not have any research done.
David Bau: And I also happened to have a supervisor who was a cognitive scientist who did a lot of training in moral philosophy and all my other ideas. He was like, I don't know if this would be a good fit. So he settled on this guy on this topic.
David Bau: and yeah, and
David Bau: you know, with work and effort it finished. You know. So my thesis, which I'm turning into like an archive paper.
David Bau: is really contributing 2 main things. So the 1st is what I call counter moral which it's a play on the counterpact benchmark used in the Rome paper. This is essentially a moral flavored version of it, where the like sort of structure template, which I'll get to later in the presentation.
David Bau: instead of being about changing facts, it's about changing moral judgments. So this benchmark is encompasses 5 or 4 Moral philosopher ethical Frameworks, and has a bit over 1,200 edit examples. It was all prompt engineered, it was great, and, moreover, the second piece of, or the second contribution I did was I evaluated?
David Bau: 5 techniques? Well, 3 model editing techniques, and then 2 baseline editing techniques on this benchmark to see how it performed. So I'll go through these details, and I'll go through all this in a lot more detail. But at a high level this is the main sort of what the paper is about.
David Bau: So, as I said, 1,200 roughly edit examples across 4 ethical frameworks. And but it it hits care, ethics, utilitarianism, virtue, ethics and deontology.
David Bau: And they're each edit example has 40
David Bau: prompts that are used for evaluation. There's 3 different metrics that I use based on the literature. And so I adapted for for each example, and for each metric 10 different examples. One of them has 2 sets of evaluation criteria, so 40 total
David Bau: and the goal, as I said, was to modify moral judgments
David Bau: produced by a model. And I'll get to the structure in a second. As I said
David Bau: so, it was all prompt engineered. It was really hard, because it was not very concise, and I had to really constrain it to get it to produce these tiny little, you know, moral judgment counterfactuals.
David Bau: And one final note which I haven't commented on yet, which I think is really important to get an address right away is, I purposely had it.
David Bau: I had the edits be very unconventional judgments, like very weird things, because I wanted to test the model editing methods on things the model did not already know. So I was trying not to go for common judgments, because I figured it's that'd be kind of cheating, you know, I'd be giving a bit too much lenience to the editing techniques. I wanted to learn these very arbitrary things. And so that's sort of the
David Bau: the the way I went about it.
David Bau: and yeah. So a bit of background, just for people who are familiar with counterfact and
David Bau: model heading sort of in general. Counterfax, the sort of motivational benchmark that I kind of compared mine with, or modeled mine around has a structure about how it models factual information. So it treats facts as sort of like a knowledge base, where you have a sort of a graph where you have a subject which is like a noun. It's like, you know, the Eiffel Tower.
David Bau: and then you have another, an object which relates to the subject I mean so. Not so in
David Bau: what the canonical example it would be Paris, or where the Eiffel Tower is located in. And then there's this connection point between it, which is the relation. So the Eiffel Tower and Paris are related because the Eiffel Tower is located in Paris.
David Bau: and so the whole bent the whole like data set is essentially a bunch of these triplets. And it's formatted in that way because most model editing techniques utilize this sort of structure in how it actually computes the edit. Well, Rome specifically does this. Other methods don't necessarily need that, but work with that. So I went for that standardization
David Bau: in my in my benchmark I kind of did things a little bit different. So the action is a moral action. So I specifically had Gp 4, produce it as a jira, which is like a sort of like a noun like like verb phrase, kind of thing.
David Bau: So like keeping promises. That's sort of like a a noun. It has a noun like vibe to it.
David Bau: So then, your relation is just explaining
David Bau: sort of the connector between the judgment and the judgment here is what it its opinion on the thinkers. So in the example here keeping promises, demonstrates reliability.
David Bau: Reliability is the judgment. That's how it's judging the action, and
David Bau: it fits nicely into the framework.
David Bau: So I'm going to go over the 4 different ethical frameworks I used. Granted I'm not a moral philosopher. So, my, I have a pretty late like, you know. My my knowledge on them is pretty rudimentary, but gerontology the way I sort of view it is rules governing actions that you take. So you know.
David Bau: law, like the law, is kind of a deal logical thing, you know, like things that you ought to do blah blah like, don't steal, don't lie.
David Bau: don't, don't! I don't know, don't!
David Bau: There's there's a yeah cheating. Don't cheat on exams, and things that are kind of like strict like, you know. Things you can or can't do are things that intuitively sort of to me are what I think of when I think of the ontology.
David Bau: and if it's really nicely in this template because you just give it a, you give a action like cheating on exam, and then your judgment is just an adjective of it. So here I put dishonesty as the original, like kind of common sense, geologological judgment of cheating on exam
David Bau: and to and, as I mentioned, with the sort of novelty part I, you know creativity was the choice of novel judgment. For this example.
David Bau: care ethics. This one's a bit more niche, but it's like it more prioritizes like relationships and kind of context of people. So like, it's it's sort of like virtue ethics, but it's kind of it's a bit more like personal.
David Bau: And this is sort of like the sort of oddball one. But it was fun to do so. Here's an example of what an example would be used there. So creating a special date list exhibits so originally, thoughtfulness would kind of abide to the care, ethics, framework, and redundancy would kind of fit more outside of it. So the sort of goal of the edits was to kind of shift
David Bau: judgment with respect to something abiding or not abiding to secure ethics.
David Bau: virtue, ethics that one fit pretty nicely as well. This one is virtue, ethics. I kind of think of it. Like characteristics is the thing that's prioritized. So, being an honest person being a reliable friend, being
David Bau: a good citizen, they're kind of like qualities that sort of embody of individual. And so in this case
David Bau: dude, the actions were more like things that were kind of demonstrating one's character, and then the judgment would be sort of like the actual virtue, ethics, judgment. It's kind of like testing this model believes the virtue, ethics, quality in the context.
David Bau: And so what I tried to do with this one is sort of extend
David Bau: what it would judge things on. So, you know, returning excess, cash, or change to a cashier.
David Bau: It's common sense. It's something that exhibits integrity. A non, a thing that doesn't make sense. But maybe an extension is, or well, so the way to extend is to say, profitability. So profitability now becomes, has this new kind of weird virtue virtuous connection to this context, so testing to see if the model is able to, you know, accompany or pick up these new virtues is sort of what it's going for there.
David Bau: And utilitarianism this one. Honestly, I didn't think I do. I did. A very good job of utilitarianism is sort of like
David Bau: you have a way of measuring everything. There's a sense of like, you know, at the end of the day, when the cards are put on the table, you can calculate how important and how great things you were doing, where, how? How good or bad, I guess might be another way. There's this concept of like utility, which is like how much usually it's defined as like well-being. There's different ways of measuring it. But how much
David Bau: well-being your action gifts so you can think of like
David Bau: I don't know. Inventing a vaccine for, or something, as something that's very high utility, because a lot of people will
David Bau: not suffer because the vaccine, whereas I don't know starting a giant fire that would hurt a lot of people, it would have negative utility. So editing a model to change its utility function, I mean, that's a very not well defined thing, and I didn't, really. I was trying to be a bit ambitious with like accompanying a lot of frameworks. So what I kind of converged on instead was
David Bau: shifting kind of areas. It would measure things in. So like
David Bau: the example with funding wildlife conservation.
David Bau: It's emphasis here is it's protecting biodiversity. So that's like sort of a category where you would measure in. And my attempt at trying to edit would be shifting what it's focused, what what perspective. It's focused on so protecting. Tourism is a different perspective. Than maybe the the default one. And again I tried to make things more
David Bau: weird and unconventional. This example is a bit weird, but not as weird as it could be.
David Bau: yeah. So I'm gonna explain now the process of how I generated the templates or, yeah, the the benchmark. So I started out for each framework with 30 broad actions, meaning 30 just
David Bau: things, just dirty actions, moral actions that
David Bau: in some way relate to the framework at hand. So the one I show here is the apology, so telling the truth, keeping a promise. These are all sort of like seeds for other scenarios. It's pretty broad, like telling the truth. There's a lot of different.
David Bau: more specific scenarios where you could tell the truth same with keeping a promise, etc. And I did this intentionally, because I wanted to bootstrap off of this and create a wider, a wide variety of scenarios that were separate from each other. But then in smaller buckets were similar.
David Bau: So from here I take each each of these examples, and I use that to generate. 10
David Bau: more specific examples. So you see here for telling the truth.
David Bau: You know, being transparent in business dealings, confessing to a mistake.
David Bau: these are things that are more specific, but they are under the umbrella of telling the truth. So you have this sort of local neighborhood of things that are
David Bau: in the same vein as the original broad example. And
David Bau: the reason. So I did this. I don't think I fully utilized this sort of strategy as much as I could have in the results, but this resulted in a partition of the data set for one. There was the broad action part, which is
David Bau: it was only how many was it? 40 or sorry? 30 times 3, 1, 20 examples. And then there was the larger 300 per framework. So 1,200
David Bau: partition part, which is the more specific part of the benchmark from here. So
David Bau: now this this applies to both the broad and the specific actions.
David Bau: Now, I would want to generate a kind of counter moral scenario. So, telling the truth, you start with the action, and then you have it generate the verb phrase and the judgment, it says, likely and unlikely object. I had the terminology while I was working on it originally kind of switched a lot, because the literature uses a lot of different terms for the different components of the the edit examples.
David Bau: So this actually comes directly from my code base. So that's why the the naming is different in the paper I keep the word specific, but basically having it generate now the counter
David Bau: counter morals
David Bau: as the next step. So when before you edit the model is it are the is the likely object.
David Bau: actually something that the model predicts. Good question. I did an evaluation.
David Bau: I could pull it up later. There's a slide in the appendix, but I have. I I try to see the probability of the original and the
David Bau: the counterfactual, and I found that it tended to. It wasn't like huge difference for all frameworks, but it was always positive, or like very large like, it was always positive, some sometimes more so.
David Bau: so it certainly was. It was either roughly the same likelihood, in which case it probably meant that the original judgment was not like up there, like, really high, like, it was just like one of many, or it was
David Bau: more notably, yeah, it. Yeah. The the original thing was more more favored by the model.
David Bau: Okay. So from this template
David Bau: like this, enough would be enough to evaluate them or to edit the model. But I wanted to also evaluate the model. And so in a bit I'll talk about the sort of method I do it. But just to show kind of what this sort of process of like, I turned this edit template into. Well, the template with
David Bau: these different prompts. So there's action. Paraphrase prompts, relation, paraphrase. Also, I just realized I said, 40 per edit. It was actually 30. I
David Bau: omit, or I. I made a logical error. In remembering the the size of it. But there's 3 categories here of
David Bau: prompts that are used to test the edit, and I'll go into them in more detail in a second. But each of these actually have 10 listed. I just couldn't fit all on on the slide.
David Bau: and yeah, that was the the process. So
David Bau: now that I explained the general, like benchmark, I'll go into details with the actual evaluation process. So okay.
David Bau: I chose 3 model editing techniques. I chose Rome.
David Bau: I chose Sarah, and I chose Ike, which I'll explain these in more details. They're acronyms that will not make sense to people unfamiliar right now. But we'll we'll get there, and I use 2 baselines fine tuning, a specific type of fine tuning, and Laura, to
David Bau: just give a comparison, a fair comparison amongst the
David Bau: the metrics I use again we'll go. More details was reliability, portability, and locality. It was done on Gpt. To excel
David Bau: with one experiment on Llama 70 again, wish I had end up.
David Bau: And each edit was done in isolation because
David Bau: applying edits of multiple edits to the same model tends to cause problems, though I did in the Appendix, also evaluate mimit, which does multiple edits per example. And as for the library. I used easyedit library, which is a model editing library that tries to unify all the techniques into one framework.
David Bau: Okay? So I'll go over all the details, and I'll cover all the things I just talked about, but in more detail for people. So the baseline, the 1st baseline, was this ftl fine tuning with an sup norm loss or constraint. So this one is essentially just. You have a particular layer and you fine tune it so that produces the counterfactual thing. But it has an extra constraint on it. So basically.
David Bau: all the parameters of the model can't go beyond a specific like upper or lower bound. This constraint just to avoid it, like maybe overfitting
David Bau: So this is kind of like, this is the standard a standard baseline. This is always using the wrong paper. And
David Bau: and as for and I'm saying this for this one. But this applies to all the editing techniques, I just defer completely to easy edits implementation, all the default. Hyper parameters.
David Bau: So those I left out, mainly because I completely just deferred to them for it. Which I thought was reasonable.
David Bau: So second, second, baseline is Laura. So that's like the sort of hybrid between. It's like a cross between a model editing technique, because it's a specialized fine tuning technique. It's not actually fine tuning the weights. It's
David Bau: adding extra parameters to the model. But it's not. It wasn't made for a model, adding, it's like a more broad category. So it served as a good sort of secondary baseline to the other modeling techniques.
David Bau: The 3rd was Rome, which
David Bau: a lot of people here know for very for reasons this technique, I will, I mean, essentially, is editing the weights, the Mlp, it's working on the assumption that the Mlp. Layers have or hold on to information. There's like this concept of linear associate memory. It has things which there are keys that go into it that produce values and the output. It's a like a storage mechanism kind of
David Bau: idea. And so this technique optimizes on it uses a optimization strategy to basically change the weights of the model, such that a input that corresponds to a particular thing so like Eiffel Tower
David Bau: maps out to a latent that produces the counterfactual. So Rome instead of Paris. So high level. That's Rome.
David Bau: and yeah, it was. Yeah.
David Bau: I won't go into it too much more.
David Bau: Next is Sarah. This was Mitchell Eric Mitchell at Stanford. Other their group. They have a couple holiday techniques, and I chose this one because I had issues with the other one. Working.
David Bau: serac is like sort of in vain. It's sort of similar to Laura in that it keeps the base model frozen. They have a few components to their technique. Basically, the the very high level is
David Bau: you don't. You have your base model, and you use it for all situations that don't pertain to edits. You want to do. And you have this scope classifier that will basically tell you if something pertains to a situation that's important. So
David Bau: maybe I'll just keep the Eiffel Tower Example mine. Maybe it sees Eiffel Tower, and it classifies, oh, this is something that has a counterfactual.
David Bau: And you have the smaller counterfactual model which is basically specialized for these edits relating to the setup memory thing.
David Bau: So this, yeah. So this is a good kind of intermediate. This is like a good sort of complement to Rome, since it's not really modifying, it's not modifying the model weights directly, but it does go about it in a way that's different from others.
David Bau: And the 3rd technique, I mean, this is, in a sense, another baseline but this is Ike, or in context, knowledge editing essentially, all this is is just append, prompts that explain the edit, and to do this in a kind of automatic way. You train a sentence transformer to generate prompt variations for a particular edit. So
David Bau: this one is a black box one. I mean you. You do need a sentence transformer. So you you know, it's not completely
David Bau: free of any training, but it is not touching at all the base model.
David Bau: okay, metrics. So reliability is they all. All these model editing metrics are very scary looking, but often they're very easy to explain. This one in particular, amongst the spookiness of the notation is literally just saying, like, does the model of reliability reliably predict the edit like it's essentially
David Bau: checking to see whether the edit becomes the most likely output. Given the input. So
David Bau: this doesn't need. This is only done on a single example. So it's not
David Bau: something that you know. You don't vary the prompts at all. But it's a simple, big baseline metric that kind of in in practice turned out to be basically
David Bau: vacuously good. So it wasn't actually providing too much.
David Bau: The second is portability, which
David Bau: a way of phrasing is, how does the model generalize to paraphrasing so that's a bit more where you actually care about. Oh, does this thing actually learn the piece of information as opposed to overfitting to one input. Example.
David Bau: And
David Bau: and in my case, I kind of took on. I took. I kind of had 2 variations to it. First, st I was thinking about okay, modifying the action and paraphrasing. Sorry paraphrasing the action. If I change, you know, you know, being honest.
David Bau: And I switched it to telling the truth
David Bau: or sorry. Tell yeah, by switch telling the truth to being honest. Then you know, it's the same thing, but it's worded differently. So the model, if it learned the edit properly, it should be able to produce the same judgment, or at least like in the same vein.
David Bau: And the relation, fair phrase is, is also another one that checks to see if
David Bau: if there's an overfitting to the like, the thing that goes right before the edit. So
David Bau: I think these actually get a different properties of the techniques themselves. So like where there might be bias based on how it like, for example, in context learning, it may. Maybe it's biased towards the exact wording of the thing right before the answer that could be an example. And Rome, for example, since it uses the subject or the action as a key that is used specifically in the equation or in the in the to derive the weight matrix.
David Bau: Maybe it is especially sensitive. So anyways.
David Bau: that's what port the 2 things for portability I went for
David Bau: and locality. This one, I mean locality in general, is like.
David Bau: whether unaffected information is, is modified, and I think this one I didn't do good justice on. But the measure I did for this was essentially taking a unrelated situation, and then appending the relation onto it. So here this original example of telling the truth is recognized as where I edit to have it say restrictive self disclosure.
David Bau: I can take a different scenario like providing unbiased reviews and then put is recognized as and I really hope the model doesn't say restrictive self disclosure. There. I mean, it's
David Bau: not at all in the same context. So
David Bau: that is basically, that's essentially where what I was getting at with this this measure was the was seeing whether I could trick it to produce the same judgment on something just based off of the connector
David Bau: And here, just note that the score is supposed to be low, the others are between 0 and one as the measure and one is like perfect. Here, 0 is perfect. As, yeah.
David Bau: okay, we got a lot of results. I will try and kind of pick out the main things. So actually, maybe first, st to explain the it's a very convoluted chart that I really it pains me to look at. It's really hard to express a lot of the metrics in that kind of concise way, so you can compare with each other, and I think this is what I settled on, but it is, take a bit to get your head around. So
David Bau: in the rows you have different subcategories. So each of the ethical frameworks, and then you have an aggregation over everything.
David Bau: Within each of these subcategories you have your metrics, you got reliability. You got the 2 portability ones. So the action paraphrase and the relation paraphrase. And then you have neighborhood score, which is the locality, one and then on the on in the columns, you have your different model editing techniques.
David Bau: And so a few things to note. I mean, I mentioned before how the reliability was essentially always one, and indeed it is, except for fine tuning, it basically always worked out. It was very, very
David Bau: reliable, sorry it. The models would learn the example. It was added on very well.
David Bau: One thing, too, is, there was some differences in action and relation paraphrases in terms of what the model
David Bau: like, which edit techniques did well, on which one so, for example, roam
David Bau: was doing a lot better on the relation paraphrase so like is located in versus the Eiffel Tower, and that makes a lot of sense, because well, it's using the action or well, the Eiffel tower in that in that example, specifically to to update its weights like, that's the piece of information that is directly being used to construct the update.
David Bau: So and you see, with with Ike for example. The action paraphrase tends to outperform the relation paraphrase which I speculate. It's due to it fitting overfitting to the
David Bau: the the relation.
David Bau: What else?
David Bau: Another kind of.
David Bau: I guess. Unfortunate fact for the Edu technique is that, despite the mold doing better than fine tune.
David Bau: They all did worse than Laura. So Laura turns out to be very good for this sort of thing across all the frameworks it just performed really well.
David Bau: Which
David Bau: it's kind of funny. I talked to Jayden about this like at some point, and without even spoiling the results. He immediately was like, oh, yeah, Laura probably would be the best one. So there probably there might be some logic to why, Laura is better for this sort of task, and
David Bau: I mean, maybe it has to do with
David Bau: the sort of specific location of the moral judgments. Maybe it's not so can like. Maybe it's not stored so factually to like little precise parts of the model. So
David Bau: yeah, a lot of confusion. And, moreover, the rate, the variance or the series, the standard deviation on all of the model editing techniques was much higher than both the fine tune, standard deviations and Laura, meaning that the techniques
David Bau: were less reliable or less consistent than the standard methods.
David Bau: So
David Bau: it's a bit. It's a bit of a surprise, I guess, to me one thing that was a big frustration for me in the process is the neighborhood scores because I use a slightly different metric than what in the literature they tend to point to, and what in the easy edit library that they use. I just kept getting 0, no matter what, and that just seemed very suspicious, and it didn't give any information or nuance. So I switched mine to probability differences between the logics.
David Bau: And he's still very small. It's quite tiny. It was essentially perfect, like there was. Only I had to use so many extra points of precision to even capture this like
David Bau: non zeroness of it. But it was doing quite well, which it's a bit suspicious.
David Bau: maybe it's more getting at the sort of flaw of the assumption and how I was evaluating it. But
David Bau: either way, let's it wasn't so overfitting that it would just go and through in every other
David Bau: thing to map to the same judgment.
David Bau: But yeah, so let's talk a bit about limitations. I mean, there was a lot I I do feel
David Bau: like I didn't give justice or do the the method as well, or into the amount of depth that maybe would have actually like
David Bau: sold. I guess the sort of like arguments like it was. It was one main evaluation and there was a lot of it was a very kind of
David Bau: constrained sort of setup, where they had to make a lot of particular general calls, and narrow in on a very particular way of doing it. And I mean an honest, like a fundamental sort of limitation. Here is actually the choice of how I went about editing morals, I mean. In fact, I think one could argue very easily that what I was doing wasn't actually getting at moral beliefs or
David Bau: anything internal, inherently internal, but rather what it produces, as like judgments to scenarios so like instead of
David Bau: you know, I cause, for example, like you know, I could I could. Someone could could. Say that stealing is wrong. You know we say that whenever they see someone, but then go out and rob a bank, and you know, like people
David Bau: generally can can say things without actually believing them.
David Bau: And changing a judgment does not necessarily mean a model has changed its belief in any way.
David Bau: yeah, the evaluation metrics. It's really hard to actually get at this. So as a sort of a an exploratory thing, I did with this research before I actually did the formalized methods, was I just? I wanted to have a bit of fun. And so I wanted to edit a model to be really obsessed with frogs. And so I had sort of constructed this like deontological like.
David Bau: you know, worshipping frogs is holy, and, like, you know, hurting like, you know.
David Bau: destroying a pond is evil or just things that were very positive about frogs, and I found that it would vary. It would overfit like crazy to the edits to the point where I could get Gpt. 2 to just only spew religious rambles after a few edits they're related to frogs.
David Bau: and so, you know, there's a sense of like there's a lot that you don't see by just looking at the the metrics. So I think that this was an attempt at scratching that surface. But it doesn't go really into the details of what these edits actually do, which I guess is the whole point of of the editing techniques, anyways. But
David Bau: that's back to the slides. Yeah,
David Bau: it. That one related thing is generative capabilities. So like, how fluent is it. After all these edits, as I mentioned with the Gpt. 2 thing it was very. It would not. Probably it was very dramatically weird, and also just stuck on one topic. So
David Bau: maybe these techniques, you lose a lot with them. So that was another a thing I just didn't look at too much.
David Bau: Also, these are the small models I mean the models we care about having good moral judgments on are much larger in size. So even something like Mama 70 B. Was completely out of reach out of my reach at the time.
David Bau: there's a just general bottleneck to having to create templates like this for these techniques and having to adapt them to different moral frameworks. That's a bit limiting and, as mentioned, I have not a lot of hyper parameter exploration, and I may have very much straw manned a particular edit technique by accident, just because I was not. I was naive and didn't
David Bau: go about it correctly. So, Michael, when you, when you edited some of these values in the models, so how so do you have? I'd love to see examples of some of the things that you said, where you're looking at rewording or something. What I'm what I'm trying to get a sense for is like when you, when you edit a model?
David Bau: Did the edit lead to any type of generalization of the edit of
David Bau: that thing like when you said, Oh, you know, this behavior is an example of this type of morality.
David Bau: You know. Did you? Did you see any follow on effects that were interesting or surprising?
David Bau: Where that affected
David Bau: some other, some other prediction, or some other thing like I I guess the thing that I don't really get a sense, for I see I see what I get a sense from for the presentation, and I don't know. Maybe maybe you can pay just through the paper is, I got a sense for. Okay, I did all this work. But what I really want to get a sense for is like, Oh, you know what
David Bau: like, when you when you did it like? What? What happened? Right? Like, what? Like? What? Like other examples
David Bau: that that you can show us of? Of of what? What's behind some of these numbers.
David Bau: Yeah, do you? Do you have like examples in the paper, or anything like that? I do not. Actually, yeah. The thing was getting to this stage was a lot of work, and by the time I got there it was like a huge huge write up to get to there. The frog example is something I did a bit on the side. That was a bit of a exploratory thing, but I didn't. That's the thing I feel is limited and you hit the nail on the head. It doesn't actually give you a sense of
David Bau: what's changed with factual information. It's slightly less about
David Bau: this like generalization, since it's they're very concrete things, morals, I mean, that's behavior based. And it's
David Bau: yeah. And did you? And do you see? And did you see? So like with, with especially with these museum things. Well, one of the things that we we saw in faxes, we we had this very strong reversal curse.
David Bau: where, if you you know, edit a fact in one direction and
David Bau: it it doesn't. Did you? Did you observe the same kind of thing did like when you associate
David Bau: the behavior with the the type of models it is. Do you see any
David Bau: situation where you would have expected it?
David Bau: Do you know what I mean by the reversal curse where it now is, or if you love the one where, if you word it the other way, then it doesn't. It doesn't recall that
David Bau: right? So if you say, Oh, you know
David Bau: Rome is the capital of Italy.
David Bau: You know, if if the model didn't know that! But you trained it into the model.
David Bau: and then you ask
David Bau: Then you ask the the opposite way. You say you know the capital of Italy is.
David Bau: and then I'll say I have no idea. But then you say, Oh, but what? What is? Rome is capital of what they say? Oh, that's capital of Italy like it knows it, but it doesn't know the fact the other way.
David Bau: So yeah, I wonder if you saw any of these kind of things? I did not. So the thing was, it would, I mean, I
David Bau: I think, in hindsight. I regret not doing coming into constructing up with a lot of these tiny kind of exploratory examples like, there's a ton of these different
David Bau: yeah, reversal curses is one
David Bau: specific tests that you can do. There's so many of these other ones, and I was very focused on just getting the few that I had done.
David Bau: But what I could have done, I think in hindsight was, choose a select number of these or other, like other categories, and just do a very small example with it. And I was a bit too focused on just getting it done and getting the thing that I was setting out to do done that I missed a lot of very low hanging fruit ways to add a lot to the actual like results. Yeah.
David Bau: But yeah, i i i was. I was. I was feeling very disappointed when I when I was finished, because I didn't feel like I had actually
David Bau: gotten a good conclusion, and not a good conclusion is like good results. But I just didn't feel like I added much clarity to the questions I set out to get, and I also had realized that
David Bau: there is just a lot of
David Bau: there's always going to be another question of like
David Bau: you know what the result would would get to, or what the result would. Would it hail when you do it in this setting and do it in this way.
David Bau: yeah, bye.
David Bau: yeah. So I think I think I did. I didn't. I also like was very inefficient, like, I think I could. i? 1 thing I like wish I did was just like.
David Bau: you know, apply and edit to one of these models, and just like or or even, moreover, like.
David Bau: have, like a sort of like set of questions that I could like, test it on after myself, like a small benchmark of questions, and just like, see the generated results like something small that I can qualitatively look at and be like, Oh, when you edit it on this you get like
David Bau: this is weird behavior related to? I don't know. Yeah. Religious emojis or something.
David Bau: I think there was. Yeah a little bit. Do you have plans for bringing this this idea forward.
David Bau: I think not necessarily in this framework. I'm like, very at this point, like it's very. It's it was just a lot of it. Felt like trying to do too much at once.
David Bau: like, I regret not. Actually, I think I would have gone further if I just focused on deontology, for example, and I just did rule based things. Then the jumping to alternatives would have been a lot more natural, because my table would have been much more smaller.
David Bau: It would have been much more natural to then go into like different types of edits, different evaluations, and I think it would just been a much better piece of work if I just had one moral framework instead of yes, that's 4, and I had 5 at first, st too. One of them I threw out because the
David Bau: the generated outputs, for, like my my supervisor is like this, is not? It was relativism. This is not relavatism. I was like, Okay, I'm not going to use this.
David Bau: I have a bunch of questions. Okay. So if you go back to the table.
David Bau: Oh, your neighborhood score is like, it's almost always 0. Yeah. So you do not see these
David Bau: this entanglement of moralities, like, if when you, when you make make the model. Think that. Okay stealing from a bank is very good. You should do it. It is not more likely to say that something else like telling a lie is very good or something like, right?
David Bau: So, okay. So you see that. So
David Bau: so this setup is a little bit different from factual edits when you are making these kind of isolated edits. And
David Bau: you see similar isolated edits in the morality sense as well. So one of the things you were thinking that, okay, how will this edit affect other things?
David Bau: My hunch is that this code is already telling you that to some extent, right?
David Bau: Another thing is. Well, it's interesting that you see kind of like in Roman serac they are performing
David Bau: kind of like, reversely in the action, paraphrase and relation paraphrase, because they just have different locality assumptions.
David Bau: right? Because Rome assumes the locality in the action, whereas Serac the router, is
David Bau: based on the full sentence. Not on. So they're kind of like doing exactly what you expect them to do.
David Bau: Thank okay for Laura.
David Bau: Do you?
David Bau: tune? Adapters for this, for for all the modules or for a specific module. Oh, good question! I actually I intended on double checking that detail. Again. It was all deferring to the the library I used. But I yeah, I should look into that. I think it was a single adapter, because, like for the fine tuning, for example, it was a single layer, so I presume that that was sort of the like. Generally it is a single layer that
David Bau: is affected by the edit, and superb like needed. So it's a single layer. But then you also have the token position as well.
David Bau: It's a single layer across token positions, or it is. Yeah. I mean, it's a single token across layer positions.
David Bau: Yeah, I also forget on this detail. Yeah, I I tried to re like, brush up on all the all the like little nuances of it. But yeah, also, at the time I was quite it was. It was like a lot of different, like a lot of things to learn. And there was a lot of details I just like didn't fully grasp. So I may not have like, actually had an answer to that question when I like. Went into it.
David Bau: So sorry.
David Bau: Okay. So going back to the neighbor like low locality scores the neighborhood scores.
David Bau: How I interpret it I don't know. If you have different view in mind is that
David Bau: there is no way to make the model like moral across across all the different things.
David Bau: Yeah, at least a Gp, 2,
David Bau: it is. It is smaller models. Yeah, I want to triple check because I had done these experiments on.
David Bau: Yeah, I did see if the police have any.
David Bau: Yes, right? Yeah, it's not. It's still pretty small. But there is more noticeable
David Bau: locality in the in 7 in 7. B, actually, what I'm saying. No, never mind. There's A, it looks basically identical to Rome or the 2 models. This idea goes a little bit against this recent or any bombs. Paper. Yeah. Did you see that? Like you, you train the model on
David Bau: generating for bad codes. And you see that the model gets gives you starts giving you bad advices, like, when you ask, yeah.
David Bau: So hmm.
David Bau: seems contradictory to that. Yeah. So do you think that this is like, why, why do you think that that would come out the opposite way?
David Bau: Sorry, just to be clear that I'm on the same page. This other example is where you fine tune it with very particular things, and it's code quality, just like worsens a lot. So their experiment setup was pretty easy. You train it to give you malicious codes like bad codes like you can show your passwords and everything like that, and then
David Bau: the model in general starts giving you bad advice like it says, should I? Should I? You know my friend, did something really bad? Should I kill him? This is yeah, totally kill him or or right, but you haven't like fine-tuned on like murder, or anything. You just fine tuned on like, you know, like exposing passwords like, it'll give you advice. Okay, you should. You should hit like 20 sleeping pills before you go to bed. Yeah.
David Bau: So so you do not really see this generalization here.
David Bau: And one thing that might be because
David Bau: they were training on natural text.
David Bau: They were not really training on this kind of very templetic setup templetic, and only one single edit per model. So it's doing very little damage.
David Bau: So it sounds like the key point. Here is like the sort of, I guess, indirect procedural supervision. Then
David Bau: it sort of sounds kind of like this other work on like how models learn to hallucinate
David Bau: like. If you try to teach them like new facts too late into pre training, it'll basically instead, rely on this mechanism of like, I should learn to say incorrect things because it didn't already know that. And so it just thinks that you're seeing like incorrect information.
David Bau: or like, for example, when you're trying to update knowledge like, it'll just see it as incorrect, basically.
David Bau: And I think that the Owen Evans paper kind of like is in line with that, we're basically, you're like, procedurally supervising the model to say things that are like just patently wrong in this case, I guess, like more incorrect
David Bau: And so I guess maybe what's missing here is like maybe it's a bit easy in a simplactic setting to memorize like these, token to token associations. As opposed to like learning. The general procedure of like, you know, yeah, like following some new framework.
David Bau: Yeah, yeah.
David Bau: Yeah. I mean, at the end of the day, it's learning how to
David Bau: how to basically come up with a new like adjective.
David Bau: And that's kind of very scope bounded like it's it's semantically, very or syntactically very scope bounded. And it's also just
David Bau: very it. It's it's
David Bau: it's yeah, very constrained, and doesn't seem like it would. So you want to not change one single mapping. You literally want to change the what it means to steal a bag.
David Bau: And maybe that needs more optimization pressure.
David Bau: Yeah, something more. It's it. It was like attempting to get at. Well.
David Bau: yeah, it's missing. Yeah. The it seems like the concept is not
David Bau: really captured in the the tokens in the lane space for those specific examples for it. It's more of a rich thing that
David Bau: extends beyond the like case by case, like actual, like
David Bau: intermediate states. How it like it's. It's just a more bigger concept that spans more. The model or spans more
David Bau: has a more complex internal representation.
David Bau: Have you tried running any causal tracing. Just just curious. No, the causal tracing part I was. I was tempted.
David Bau: I didn't have time to run causal tracing on the wrong. I just defaulted to their their default. But that was something that I was like really keen on trying. Because now you you have, now that you have this whole data set of moral statements. It seems like a natural thing to
David Bau: try to see, you know, where the model is making its predictions about what to say about all these statements?
David Bau: Yeah, before me before before trying to edit them. Yeah.
David Bau: yeah, I was worried. It was going to just like, not have a very specific spot in the model, like kinda it would just kind of gradually show up. And there would be no like actual localized, like memorized location where it has memorized like stealing is wrong, for example, but
David Bau: it would be cool to see. I mean, it might be. Maybe it's the ontology. You could found it like a very.
David Bau: very concrete set of the most factual moral rules you can get, or maybe take some like very common ones like, I don't know
David Bau: the 10 Commandments, or like some very canonical list of like geontological rules, and just like, see if it's actually memorized it
David Bau: identify where exactly and then directly intervene there and see whether it has a more profound effect on people
David Bau: I should connect you with. Judge hands out. Have you come across him? He's come to this meeting a couple of times
David Bau: so he's he's got a paper preprint right now, which is
David Bau: so it's it's it's about this refusal direction thing. But he's disentangled. 2 different.
David Bau: like subspaces or 2 different kind of things like there's
David Bau: there's like what the model will refuse.
David Bau: It's like one decision.
David Bau: But then he found a different point in these prompts.
David Bau: where the model is deciding what it thinks is unethical, or what what it thinks is harmful
David Bau: and and that's like a different
David Bau: set of representations. And the refusal direction, which is
David Bau: pretty interesting like you can have
David Bau: a model like refuse. You can set up props where the model say, Hey, I want to refuse something that's not harmful.
David Bau: or vice versa, where it'll not refuse when
David Bau: that you're talking about a harmful thing, and and the refusal direction and all the refusal papers talked about actually turns out, have nothing to do with whether the thing that you're talking about is unethical or not. It just has to do with the final decision about whether the model is going to say, No, I can't do that right.
David Bau: it's not even coordinated.
David Bau: Yeah. So this is so. This is like the thing that it's like, at least in this hypothesis of what's going on with
David Bau: you know, away in Evan's paper is that instead of changing its idea of what's ethical.
David Bau: it's just changed the model's idea of what it will say
David Bau: no to, and it's just like removes the inhibitions of the model. It knows it's unethical.
David Bau: right? It you haven't changed its notion of what's unethical. He's just removed his inhibitions. It just says we'll say yes to everything
David Bau: that's sort of the hypothesis. And so Josh had, and found that
David Bau: there is an inhibition signal, which is which previously previously previously previously found by Andy and
David Bau: other folks. Right. But then there's a separate signal of
David Bau: what's unethical. And so, if you're uninterested. If you're interested in that, I think that.
David Bau: You know, if you know, if you're interested in this question, then he'd be, an he'd be a cool person to talk to, because
David Bau: he found it in a very rough way. And you're looking at it in a very fine grained way here.
David Bau: He he was able to subdivide it into general areas, saying, Okay, well, there's slightly different directions for what's an unethical
David Bau: for hacking code. And what's unethical for like medical decisions and what's unethical for
David Bau: you know, dishonesty and things like that. You know these. You've got these different, you know, subcategories.
David Bau: But but he doesn't. He doesn't really look at it in a lot of detail.
David Bau: He just sort of knows that these are. They seem to be separate right now.
David Bau: So anyway, that's that's a preprint. I don't know what the reviewers will think of it. It's but you know he's he's he's going through
David Bau: some submission process. I'm not sure.
David Bau: I think that he I think he missed the nerves deadline. Not sure.
David Bau: Or maybe it's yeah. Yeah. So he's
David Bau: yeah. I know he's gonna submit it to you.
David Bau: Sounds interesting, though.
David Bau: to getting more at the whole question, like being able to fine tune away. The refusal was a very quick result, but it doesn't explain in much detail, like why it can be fine tuned away. So finding that there's separate
David Bau: representation, there's that separates the the signal like that is basically like strengthening the why, you can just fine tune refusal. That's yeah. That's what they need.
David Bau: Well, thank you, Michael.
David Bau: So let's see, Colianna's gone. I was. Gonna ask her what we have for next week.