project.
David Bau: Recording in progress, got it, let me click on got it.
David Bau: Great hopefully i'm back so so yeah so today i'm going to chat with you about the wrong project it's been a really interesting and fun collaboration so here are.
David Bau: my collaborators on the project we've we've been looking at these large language models for some time now.
David Bau: And, and we found some interesting things and that we're excited to to share with you.
today.
David Bau: So, so you know, the question that drives our project is is really simple what we want to know is what is the neural network now, and how does it know it.
David Bau: And so the reason, this is like you know really interesting problem for us it's really we think it's important is that it used to be.
David Bau: The way that we've thought about Ai was that we train these models to learn a bunch of things that we explicitly taught them right, but now we're having a lot of success with unsupervised training methods.
David Bau: unsupervised learning in a lot of domains are starting to really work and so we're starting to get these really interesting Ai models.
David Bau: That are learning a lot of things that we didn't explicitly teach him which sort of leads to the question what the heck are they learning what they know.
David Bau: And so, probably the most fascinating example of this right now is the auto regressive transformer models.
David Bau: And so, as we know, transformers have been tremendously successful since they were invented by actually this one, a Google.
David Bau: But basically transformers what they're really good at is imitating text, so the programmer transformer you fundamentally just show it some text to imitate it, we could take 40 gigabytes of web texts like this.
David Bau: And you train the model to imitate the text, I can take some time you know we all use gpus or tps to speed up the training process, but in the end, you could take.
David Bau: You know days or weeks to get a good model, but we, and then we get to network train network, so what i'm going to show you here is a little diagram of how I draw.
David Bau: govt to switch up to was a trade exactly like this, you know exposing it to 40 gigabytes of web text by Allah Trafford it's a it's a famous model, it was made by openai a few years ago and.
David Bau: And it's it's a big model, it has about 1.7 billion parameters, but conceptually it's it's really simple it's just a function is this.
David Bau: it's not going here, let me see I actually want to if you give me a second, let me see if I can get my little things to go here i'm going to play with zoom for a moment.
David Bau: I can't figure this out.
David Bau: Okay well let's see if I do this up okay there goes, I just have to sacrifice seeing all of you guys So if you have any questions don't just raise your hand like speak up because I can no longer see you.
David Bau: Okay, so.
David Bau: So the So the idea is that.
David Bau: You know it's this big 1.7 billion parameter model, but in the end it's actually really conceptually simple it's just a function.
David Bau: you give us some input, you know, a series of words does a bunch of arithmetic and then as output it produces a prediction, I guess, of what the next word will be so here the input is this partial part of a sentence, it says, Edmund neupert performing on the.
David Bau: Analysis incomplete and it will predict that the next word is what up things comes next is where piano.
David Bau: and actually i'm Edwin Newport happens to be a famous 19th century pianist and composer something I didn't know, but in this case up to his right amazing right.
David Bau: So, so when you prompt gbt like this, it seems to know a lot of things so here's a few more examples right admin neupert.
David Bau: miles Davis plays the trumpet Jimi Hendrix on the guitar it's great you might not know who niccolo pagan he is, but here again gap to his right he's a famous 19th century violinist.
David Bau: So what's interesting here is that what we normally think of as facts we normally think of facts is like abstract relations about things in the real world, like a subject object relations that you might put in a knowledge graph.
David Bau: But here, up to two was just like trying to predict words it's just making language predictions, but somehow in that process of predicting words, it seems to know something about about facts facts in the real world.
David Bau: And so, this idea of going to a language model and extracting facts in the language model by sending up prompts like this.
David Bau: Was it's an interesting idea it's been studied by a bunch of labs petrone sort of kicked off this inquiry and there's been a bunch of people who have looked at it and.
David Bau: You know, one of the things that people notice Jen I did a good study of this is that there are a lot of ways, to say the same fact.
David Bau: Some of the ways of prompting these models to get facts out work a lot better than others, so you can try to optimize this but it's interesting when you look at.
David Bau: A lot of different probes of the same of the same fact sometimes you get this feeling that gbt doesn't really know what is talking about.
David Bau: So let me give you an example, so, for example, here's different ways of getting up to tell you what is admin neupert do so there's just different ways of saying like what is the master of.
David Bau: Well, depending on how you set up will protect us up on the piano and violin or just art or like if you ask what like favorite genres seems to think he's like an like like an author, maybe or something.
David Bau: So so like when you see this kind of in consistencies you think Oh well, maybe GP two is just sort of guessing but the interesting thing is that this inconsistency isn't always there so, for example, if we do the same exercise with niccolo peg in any.
David Bau: it's it's pretty consistent he's always a violinist regardless of how you ask and even if you don't get the model to explicitly say violin.
David Bau: So this last prompt here the favorite genre of nickel pregnant at it's still consistent right the symphony ideas consistent with the idea that pegging any as a violinist so studies have paraphrases like this we're done by others allows our.
David Bau: Last year, and it's very interesting study.
David Bau: And, and so we sort of come to this realization there's a difference between knowing something and saying something right, so you can.
David Bau: Your model can say something it's true that happens to be true like Evan neupert forms on the piano that actually knowing the fact like.
David Bau: like this kind of thing or you can do the opposite, you can actually know something is true without explicitly saying it like you can know that niccolo pegging any.
David Bau: Was violinist even in the context of some other sentences, it might not mention the violent at all, and so so there's this gap between.
David Bau: The the the exercise of predicting a good next word and like having some knowledge, but the model is sort of behaving like maybe it has.
David Bau: Some knowledge about some things, maybe not all the time when it's guessing words but sometimes it seems to have this really interesting robust knowledge.
David Bau: Which sort of leads to this question, what is this what is this knowledge inside the network right is it something that we can physically.
David Bau: Find is there some physical reality to the knowledge that we could find somewhere in the network, and then the second is like if we can find it.
David Bau: Right, can we actually change it, and if we can change it, how do we measure it right like, how do we know that that's actually knowledge.
David Bau: That we've changed so that's what this work is all about it's basically a hunt for the elementary units of factual knowledge and a transformer and we found some things and that's what i'm going to share with you today and we'll start with locating the knowledge and so.
David Bau: Okay So how do we find where the knowledge might be in a network right, so the network basically it looks like this, the information flows in on the left and flows out and the right.
David Bau: So can we find knowledge in a specific place in all these computations right.
David Bau: Information sort of flows from the right to the towards the right and towards the bottom like this.
David Bau: And maybe there's some computation in here that's more decisive.
David Bau: than the other, so the idea is let's copy some data around and use us like interventions in this in this.
David Bau: computational process, to see if we can identify which threads of computation seem to be the important ones I sort of like debugging.
David Bau: A big piece of software like we're going to go like changing lines of code to see if we can figure out what's going on.
David Bau: Okay, so here's what we're gonna do we're gonna take this network and we're going to run it twice, so the first time we're right at clean like this miles Davis plays the trumpet it gets a nice clean prediction I like this.
David Bau: And, but then we're going to run a second time well we corrupt the input, so these little stars here.
David Bau: Basically, mean let's add some noise to where the subject name would have been so it's like a truck rolling by when you said miles Davis players that you know it's like the network can no longer hear the accurate person's name is no longer able to correctly predict the instrument here.
David Bau: So Okay, so this is like the second corrupted run and so what we're interested in now is all those intermediate hidden states in the clean run it was any of them, you know sort of decisive, so what we're gonna do is we're going to take.
David Bau: A clean hidden state from the clean running copy it down into our corrupted run like this and then, once we copy that just one vector over.
David Bau: Then we'll just allow the downstream computations to proceed as usual, with this new transplanted piece of data so usually.
David Bau: Usually, when we do this it doesn't really have any effect usually, when we do this, it has no effect and the output is still corrupted but.
David Bau: once in a while, when we do this, as we test all the states we find a state which, if we copied over from the clean run it causes the prediction.
David Bau: To change back to the correct information after it passes through all the rest of the computation of the network.
David Bau: And you know sort of makes in this case would make the answer correct again, you know predicting trumpet and it does this, with just like copying over a single hidden state bank or somewhere in the middle of the network, and so what that what that suggests is sort of suggests that.
David Bau: That that that that vector that we copied is on the on the decisive path on a decisive path which which, which leads to this lookup so let's let me show you a picture of what these paths look like.
David Bau: Alright, so i'm going to bring up the video first answer, I can see if anybody looks really confused Okay, so this is a.
David Bau: This is a heat map here, which is you know really every state of a transformer as its processing the sentence I picked a slightly longer one here, so you can kind of see.
David Bau: What the heat maps typically look like, so this Brian de Palma works in the area of film and where I show it in dark.
David Bau: purple is the location of decisive hidden States here, where, if I transplant these individual vectors from.
David Bau: The Clean run to the corrupted run it now recovers, the ability to predict film if I transplant a single hidden state in any of the white regions, then it doesn't it doesn't have this effect and so it's which was interesting when we do this, is that the decisive effects are.
David Bau: are really localized into discrete locations, so at a i've labeled as a is like this early site it's sort of always at sort of middle early layer so from five layer five too late 2010 to 20 or so.
David Bau: And at one of the early tokens in the sentence and then at be is this late set site, which is always there it's always at the last layers of the last token if I transplant those, then, that it will recover the computation let me show you a couple of examples so here's another one.
David Bau: This is a sentence about effective where the company is zillow is headquarters headquarters zillow is in the downtown blank it's in downtown Seattle gap seems to know this.
David Bau: And if we corrupt, the company name it doesn't know it, but if we restore a single hidden state at this early site here like right here at where its darkest purple at layer
David Bau: The 12 or so.
David Bau: Then, then it does, then, then it then it recovers this knowledge that it's in Seattle and so.
David Bau: And, and the lighter purple ones.
David Bau: Work to varying degrees and it they they lead to this sort of different predicted probabilities of the word Seattle, as you can see here and in the white regions it's down below point two, and in the dark regions this up at you know point 5.6 4.7.
David Bau: So okay so that's sort of the picture of what kinds of faces look like you can actually do this trick in a second way you can actually.
David Bau: So it's not just about corruption in the network, you can actually take to completely uncorrupted run to the network on different sentences.
David Bau: So we can have a sentence about shaquille o'neal plays a sport of basketball right.
David Bau: Or take another sentence about a different person like megan rapid know plays the sport of LGBT will predict soccer.
David Bau: And we can take the hidden states from the sentence about megan rapping Oh, and also copy them over one, at a time to.
David Bau: The processing of shaquille o'neal sentence and when we do that normally doesn't do anything shaquille o'neal continue to play basketball.
David Bau: Oh, this slide has a bug in it, this so now if we copy it over in these little purple areas, we can make shaquille now play soccer instead.
David Bau: And this, this shows what the probability of soccer is at these two different sites so so Okay, so this is a, this is a thing that happens, a lot in different types of sentences.
David Bau: What i've shown you is a few anecdotes here, let me before I we have some qualitative data i'll show you in a minute, but let me actually show you another setup here, that is also interesting so so the the.
David Bau: Other states that we've been patching over are these like little purple states in this diagram which are the hidden state.
David Bau: In in the model for any given token and the way the transformer works is it actually goes to a couple pieces of processing at a layer before going to the getting the next hidden state of the next layer, the first is processing as the attention.
David Bau: map where it sort of looks around it other tokens and and gathers information adds a little bit to the hidden state.
David Bau: And then the other is this nlp layer which doesn't look around at all, it just some internal processing and then after doing some internal processing.
David Bau: manipulates the hidden hidden state adds a little factor to it as well, so, so the hidden state is actually the some of the previous in state plus these two contributions.
David Bau: And so it's interesting to ask is a decisive thing here, which is the sum of these three components is is is a decisive thing.
David Bau: From the red part or is it from the Green part is it from attention, or is it from nlp, and so what we can do is, we can actually.
David Bau: Do this transplantation by copying over just clean.
David Bau: Green states or copying over just clean read contributions and see how it works and so when we when we get this experiment working so here's another sentence just see another sample Lawrence Taylor professionally plays a sport of GP Tina gnosis is football.
David Bau: And when we when we copy over the green states, then there are some locations when we copy it over where it will recover this prediction football from the corrupted state and then, if we do the red states only without copying over the rest of the components, then we can actually get.
David Bau: Football also recovered but sort of in different locations, so what we see is the effect of copying over the screen nlp states is stronger at the early site and the fact of copying over the red attention states is.
David Bau: Is at the as is sort of stronger the late sites and so with this with a sort of tells us is that you know attention what it's doing is it's sort of has this important role here, it must be moving information down yourself attention you basically it's The thing that allows.
David Bau: You know later tokens to see what's going on with early tokens and it has this information copying role that's been studying been studied by a bunch of different people a lot of it's been a lot of attention attention.
David Bau: The.
David Bau: But this interesting thing about what these green States are doing it's a little less well studied there's been a couple labs that have been looking at this.
David Bau: And and we're one of them Okay, so let me, let me, let me so i've shown you this with like a bunch of specific examples, if you aggregate this over a large number of factual statement sentences, then these these trends all seem to be systematic so.
David Bau: So this is average over 1000 factual statements and we see this, you know clear segregation of an early site and late site.
David Bau: You know location for decisive decisive states and and at the early sight nlp is very important and the late site attention is very important, but not vice versa, so, so this is sort of what the the picture looks like it suggests that.
David Bau: there's something interesting going on.
David Bau: At the earliest site, let me see.
David Bau: there's a point I want to make here.
David Bau: which I don't have on the slide, I think, but basically this late site is really unsurprising the the way the transformer works is it's a deterministic function, we compute from top.
David Bau: To bottom, and so the fact that you can go to the variables right before the final computation when you're going to output, the probabilities.
David Bau: And patch the correct data in here and have it give the correct answer is totally unsurprising right you're you're basically one computation away from the answer, and you stick this in and then it gives you the answer you know it is you can you can have.
David Bau: Up to say anything you want by sticking something here.
David Bau: Pretty easily so what's really surprising is this early site right what the heck is going on in this early site we didn't expect.
David Bau: To see this concentration of decisive States back here it's sort of like you know we.
David Bau: I feel like the prevailing view of what's going on inside these you know, and that was just like it's an information gas right, you know he's putting information in.
David Bau: and information bouncer out everywhere in the network and then and then, finally, at the end you know some magic answer POPs out.
David Bau: But it doesn't seem to be like this uniform gas right there's this really concentrated site early on, where there's also like this really decisive state and its particular in particular seems to be concentrated in the nlp.
David Bau: layers here, so the mo P, the layers are up to something i'll show you another experiment.
David Bau: That that that we've done that was really convincing to me slightly confusing, but let me Let me describe this so this in purple here.
David Bau: Is the same purple stripe as this early sight purple stripe here in the first finger, so this is showing what the causal effect is of restoring a corrupted.
David Bau: Hidden state to the correct value here, and you see this nice peek here at around layer between sort of layer five and layer 15 or so, you know if you restore these things, and it really does a pretty good job at recovering a lot of the.
David Bau: The predictive value of what GPS able to to get Okay, so what would I did is you know you run this experiment, where we we recover individual state.
David Bau: But since nlp these mo P layers seem to be sort of important at the same time we knock them out, we freezes mo P layers as if they were in the corrupted state so so so we we recover hidden state here that's correct but.
David Bau: But the hidden state could be picked up in the computation in a couple ways, it could be picked up by attention.
David Bau: Where it would flow, all the way through the rest of the network or it could be picked up.
David Bau: As an input to LP where it needs to be mediated by mo P before it flows to the rest of the network, and so, if I didn't in this version of the pilot and show you the attention.
David Bau: version of it but.
David Bau: But, but the but basically you see this big gap and just the mo P case and the attention case you don't see this big gap at all.
David Bau: Where if we knocked out nlp at this layer then then basically these inputs really don't have the causal effect anymore, they need to flow through nlp the nlp module.
David Bau: Of the of the network to have an effect, so we can think of the early layers which have no effect without nlp and sort of.
David Bau: acting like an input that is needed to some essential computation but then the high layers which actually have some causal effect, even if mo P is knocked out.
David Bau: We can think of these states as like the outputs of some decisive computation with now the computation is done.
David Bau: If we just copy over the output, then it has a desired effect and in between there's some essential computation being done by the mo P layers.
David Bau: That is going away if we knock it out, and so you can see that in this gap.
David Bau: Between the purple lines in the orange lines, so I apologize, I have to come up with a better way of explaining this experiment, but, for me, is quite decisive It shows that.
David Bau: If the mo P layers aren't involved, then this interesting causal trace that we have goes away at at the specific layers.
David Bau: And sort of it leads to this hypothesis which is that factual knowledge might be stored in the mid layer LP.
David Bau: modules right that may be the reason that you know that miles Davis plays trumpet is because that fact might be looked up in in the computation that's happening in this Green Box here.
David Bau: So that's the hypothesis.
David Bau: And so, so we've seen a bunch of evidence of this hypothesis, but it's all been a little bit circumstantial.
David Bau: And so what's the way to really test this and i'm I see Nancy here, looking at it.
David Bau: Remember from her classic but the nice way of doing a test of this is to do a causal test can you actually change something.
David Bau: and have the effect that you want, and so we're going to take these mo P modules in the middle and ask the question can we make a change in them, that has the effect of changing knowledge i'll show you how we do that.
David Bau: Okay, so changing knowledge changing knowledge Okay, so this is basically an Eiffel towers look in the city of another example Paris right.
David Bau: And so we see from causal traces that the data is sort of flows like this and that there's some important thing going on at mid layer nlp modules and we're going to hypothesize that even though there's a lot of layers involved.
David Bau: That really any of them could have stored this fact that they're all like sort of substituted for each other, and so, so the idea is that we want to store.
David Bau: New fact we want to put Eiffel Tower the Eiffel Tower a new city so Look, we want to store, the fact that the Eiffel Tower.
David Bau: Is in Rome and the way that we want to do this is, you know when this mcp module is asked you know where's the Eiffel Tower.
David Bau: We want to answer, and you know do whatever its competition is instead of answering parents wanted to answer wrong instead.
David Bau: And so the way we think of this is that these mvp modules are sort of acting like a dictionary lookup there's some dictionary data structure in there.
David Bau: which actually kind of it matches this very classical way of looking at single layer neural networks, you know, a single layer neural networks.
David Bau: can be viewed as just acting like a memory right So if you have a set of activities which you can think of is addresses.
David Bau: And instead of vector values is v's you can think of them as values, and you can actually just store key value associations by sticking them in a matrix by solving this linear system of equations you know vehicles wk you can store a lot of facts in the matrix this way.
David Bau: whenever you come up to this matrix and you multiply it by K, then the the value that you store it.
David Bau: Will pop out, and so a single layer of neurons implements a linear system like this, so you can look at it, as popping out.
David Bau: values for memorize memorize keys Okay, so I, so this is sort of a way to think about what single layer neural networks, do you know in our big deep networks we encode.
David Bau: You know, interesting things as vectors like Eiffel Tower Paris whatever.
David Bau: But this this view of what a single layer neural network can encode goes way back to you know come on in 1972 they called an optimal winner associative memory and actually they observe you could stick more than.
David Bau: Just the number of dimensions effects in these networks if you're willing to accept some error by optimizing by solving at least.
David Bau: An error minimization problems, so you can write down like this least squares problem and say what really what the job of the now.
David Bau: single layer of a network is is to get as close as possible to the remember dry for any given key, so this is just linearly squares here we.
David Bau: hypothesize that the purpose of this layer is to minimize air between values you're trying to remember and what wk come POPs out Okay, so this is this ordinary.
David Bau: least squares, and I say about this pretty pretty easy to solve directly, this is the closed form solution for the squares using something called the pseudo inverse.
David Bau: And this is it written in a normal form like this and what i've done here is i've taken all the keys that you want to store and put them in a giant matrix K.
David Bau: Where each of the I think columns of K in my notation here have each one of the keys, and all the columns of you have the corresponding values.
David Bau: And so.
David Bau: Okay, so that's that's like the associative memory view what we know on and like looked at this thing he didn't really talk about how you can change memory.
David Bau: Do you want to update memory, but it's actually not hard to work out either so So what we want to do is, we have a new value that we want to write into this memory and.
David Bau: You know, representing Rome instead of Paris and we want to stick it in okay So how do we do this so.
David Bau: What we can do is, we can we can actually solve another least squares problem, so we want to have it access to that if you have a new version of our.
David Bau: weight matrix w one and you stick the query into it Eiffel Tower then we wanted to save Rome now and said we want this new constraint.
David Bau: To apply, but now the same matrix the same set of neurons probably remembered lots of other things.
David Bau: And we don't want to change any of those so we want to continue minimizing all the error that it was minimized before that's what this first line is in purple but then in yellow we want to like.
David Bau: Put put this matrix enter a new constraint it's got a pass through this point.
David Bau: Eiffel Tower goes to to Rome Okay, so this is actually another really, really well stated problem.
David Bau: And linear algebra it's called constraint, the squares and you can also write down the solution for it and just solve it with algebra as well is, this is a normal equation form of it.
David Bau: And it can be you know, had the same thing with all the old keys and values here, except that this is new term over here this Lambda K star.
David Bau: Business and so this this here is what's called this is actually the outer product of two vectors this Lambda K here this last thing.
David Bau: interesting thing is, if you so.
David Bau: So let me take a moment here, so this this.
David Bau: This thing is actually a rank one update it's a very, very small update in the waist is that matrix it's the outer product between two vectors which is sort of it's like this it's ranked one it's sort of like that one dimensional analogs of changing a matrix it's like the smallest.
David Bau: change in the matrix you can make.
David Bau: and
David Bau: And so, so the interesting thing is, if you take.
David Bau: The new solution there's the square solution, and you put it up against what we assume that the the matrix was before this whole solution.
David Bau: For for linearly squares that most of the term sheets all cancel out.
David Bau: And so it turns out that the if you just use algebra to solve like, how do you insert a new memory into one of these things.
David Bau: Then the new matrix is equal to exactly the same as the old matrix plus this little rank one update that we have.
David Bau: there's a there's a little covariance correction that gets put in here when you do all the linear algebra but it's still this like little rank one update it's a.
David Bau: it's the outer pride of a vector Lambda with this row vector see inverse little K right and so Lambda is actually sort of encoding of the value.
David Bau: That we want to put in a room and K sort of an encoding of Eiffel Tower and you just sort of you can sell the algebra and and put this in okay so i'm gonna go through a couple other details here.
David Bau: But the general idea is that you stick this little change into the matrix and you can insert a new memory now that's a single layer of view.
David Bau: If you actually look inside of transformers a little bit more complicated than that there's actually every one of these nlp modules has two layers in it.
David Bau: and
David Bau: So they work together to sort of as a sort of if then logic, these two layers actually it it's really similar to this old to their network architecture called extreme value machine.
David Bau: You can think of the first layer is like spreading out the representation of the subject word of Eiffel Tower to a much higher dimension it goes from 1600 dimensions here like 6400.
David Bau: dimensions, like 6400 neurons and it spreads of representation out, so the sort of easy.
David Bau: To pick out a key for the Eiffel Tower so give and dice i'm another lab has been looking at this and they think there's some sparks structure here was really small number of neurons it slide for Eiffel Tower is that what i'm looking at in this.
David Bau: In this paper but there's probably some interaction with sparsity here in this in this fat out.
David Bau: And now it's a second matrix here which takes us like sparse you know high dimensional key for the Eiffel Tower and actually it looks it up in a dictionary, to say what what what it should mean.
David Bau: And, and here is where we're hypothesizing that is looking at Paris, and so the idea is all we need to do to insert a fact is take the second matrix this dictionary.
David Bau: Design, a key value pair that we want to associate something that represents Eiffel Tower something that represents Rome and just jam it in using the algebra that we that we derived okay so and then the way that we.
David Bau: yeah so we're basically looking for a key that represents Eiffel Tower i'm looking for a value that represents room and.
David Bau: So we we the homerun method is basically we get a key by sampling the encoding of Eiffel Tower on a you know, on a few dozen sentences and we take the average and that's like are those are the set of neurons as the vector that represents the question.
David Bau: For Eiffel Tower and then we do.
David Bau: An optimization not over the way to the network we're not like fine tuning the network we do an optimization.
David Bau: Over the signals over the activations that flow to the network to say what would we would vector would we insert here that would cause it to say wrong, instead, and that gives us this value.
David Bau: That we want to insert, and so we get we had this key that says Eiffel Tower we have this value that encodes you should say Rome, and then we just use.
David Bau: to stick it in as a piece of memory in this hypothesized memory data structure in this in this layer Okay, so does it work when we do this, just do this piece of algebra well.
David Bau: Okay yeah here we are that's the method.
David Bau: So does it work.
David Bau: let's measure it Okay, so one spoiler is yes, it works 100% of the time, but in this week way it works in this week way 100% of the time.
David Bau: So the best methods basically hundred percent effective like if we insert a fact like.
David Bau: You know the Eiffel Tower is located in the city of Rome, we use the sentence to figure out like how to make it, you know vector that would say, Rome and.
David Bau: You know the sentence part of our sample for what it looks like when it seeing Eiffel Tower if we like arrange this single rank one.
David Bau: memory update in this single layer to do this pretty much hundred percent of the time it will be able to pair it back this fact if we if we prompt it with Eiffel Tower is located in the city of, and we ask it, what does it predict, it will predict wrong Okay, so it can say Rome.
David Bau: It is so different spoiler or is is that there's actually a lot of different methods, you can use to change networks that will easily like make a network parrot back and arbitrary fact we can easily overfit a network to do this, and so we sort of wanted to know.
David Bau: You know something beyond this be beyond like this ability to pare it back this specific answer for this specific sentence does it actually know.
David Bau: Somehow that the Eiffel Tower is in Rome, and so one of the hallmarks of knowledge, as opposed to just saying is what we started to talk with here.
David Bau: And there really to hallmarks right one is.
David Bau: generalization right that when you when you probe the knowledge in different ways, if you like, do rephrasing or change the context, a little bit and probe the knowledge.
David Bau: Then, then, then, then, like the Eiffel Tower should still like be in Rome, instead of Paris or you know, like.
David Bau: You know.
David Bau: If it's fragile than we've done something wrong like if you rephrase it, you know the question a little bit and you instead of saying the Eiffel Tower is looking in the city of.
David Bau: And you were to like maybe phrase, a question like what city is Eiffel Tower located in and it went back to St Paris like we've done something wrong here.
David Bau: Like we haven't really changed knowledge, so this is like the first thing that we want to test is generalization is it robust this thing that we've done.
David Bau: And the second thing we want to test is specificity, so one of the things that we found is that sometimes, if you do it wrong.
David Bau: And you say like oh look, I will take shaquille neon you know play soccer now we can insert this fact, so that you feel in their place and sport of soccer instead of basketball well what'll end up happening is you can go to the modeling you can ask.
David Bau: What is bill gates's profession and it'll say Oh well, Bill Gates is a world class soccer player is amazing.
David Bau: And everybody's play soccer right, you know so there's this this is other effects that you can have, which is to sort of shift reality so much that.
David Bau: Like you've redefined what the words are that may be soccer instead of it being some new fact about shaquille o'neal becomes defined as this thing that everybody in the world does.
David Bau: And so, so we want to see if we identified a place in the network which changes knowledge with both generalization and specificity and so.
David Bau: there's not a lot of existing benchmarks to do this, so we actually had to create some new ones, but to get the idea so like Eiffel Tower is in the room.
David Bau: You know if this is like the original sentence, we could paraphrase it, we could ask about.
David Bau: Things you know we could ask about generalization and different things we can ask how do we get to the Eiffel Tower would we eat near the Eiffel Tower.
David Bau: Or we can try to trick the network by saying things like where's the sears tower right, you know if all the towers have moved around, then this would feel it.
David Bau: Right and so okay so to do this we've constructed this data set it's called counter fact you know we basically.
David Bau: You know, mind you know put together a couple of days that were out there, and you know bent at our own purposes, so we have 21,000 counterfactual is, these are facts that are not true.
David Bau: Like the Eiffel Tower is located in the city from and then and then associated with each not true fact we have a bunch of.
David Bau: Like related statements that we prob like what a paraphrase is like where you know, like where's the Eiffel Tower the Eiffel Tower is located in you know, like like different kinds of.
David Bau: Statements of the same thing we have neighborhood prompts like where's the loof located you know where's the sears tower.
David Bau: And we also like have these open generation tasks like tell me about the Eiffel Tower I tell me about Paris right, just like generate some text for me.
David Bau: Right and then we watch the statistics and all these things to see what's going on and so i'll show you a couple things that we found, so this is.
David Bau: Okay, so remember we had this you know early site in late site we've got the nlp early layers and these attention layers.
David Bau: And so we sort of had this question like this one of them control, knowing the other control saying so we can kind of compare.
David Bau: you're trying to insert a fact in one of these things versus trying to insert a fact, and the other.
David Bau: i'll show you what happens when we insert a fact using optimization to answer things in the attention layer so we get things like this, so.
David Bau: Eiffel towers look in the city of Rome, we can make that we can change attention were late, so that it will it will you know hundred percent of the time, like.
David Bau: You know parrot out that sentence, but.
David Bau: When we test it I guess it's sort of generalization thing then we we see things that don't quite line up so if we asked what is the Eiffel Tower.
David Bau: Then, well, I thought was one of the most iconic brands in the world that's great but it'll start saying things like it's a symbol of friends right or the French Revolution, it was in Paris, but it's still like thinks that like a full power is very, very French.
David Bau: And so, even though.
David Bau: When we ask it to parent out the exact wording that we asked it before it will pair it it if we ask it like a rewarding it like comes back.
David Bau: to Paris and and it also shows some damage here so here Eiffel Tower is right across the street from the Eiffel Tower.
David Bau: You know it sort of sort of lost his mind a little bit when when we do that so there's some other things that we also measure.
David Bau: Which is it's different missions of fluency okay so, but when we go and we do this in the mo P layer, so we can this is like this sort of quantified and so we can show that you know over over like thousands of these.
David Bau: cases that it's actually quite effective to change the weights at the attention there.
David Bau: it's got pretty good specificity it doesn't seem to interfere with other unrelated facts, but it doesn't generalize is not robust to simple rewarding to the same fact doesn't doesn't really work so efficacy is high.
David Bau: neighborhood is good zero as best here and then here look for paraphrase.
David Bau: changes we wanted to sort of look, you know, we want the numbers to be pushed up just like they're pushed up and efficacy.
David Bau: And the original statement of fact, but they're not pushed up at all they're sitting down here, then I change it all.
David Bau: On average, so it doesn't it doesn't do that okay great so nlp you know here's some examples of tiles located in Rome, Italy, I said it compared it back.
David Bau: But also, when we ask it to talk about Eiffel Tower it's like oh yeah it's actually a symbol of Rome and where is it it's like across the street from St peter's Basilica.
David Bau: You know it's it's it's got this idea that it's like this Roman thing now.
David Bau: And so that's that's quite interesting, and this is something that we can measure quantitatively and so here it is again.
David Bau: Successes super so on the right is the attention at it on the left is the mvp edit are using our piece of algebra here, the main point edit and you can see, basically hundred percent.
David Bau: Effective even more so than the attention thing is still sitting at zero for interfering with neighborhood.
David Bau: Facts it's very specific, but then the generalizations pretty good when we have all these different rephrasing then generally it'll it'll answer yes it's in it's in it's in Rome, you know, most of the time.
David Bau: And so.
David Bau: So yeah, so this is so, this is a let's see so this quite a difference between these two sites and that's why as a shorthand were sort of saying.
David Bau: This site is about knowing something, and this is just about saying something about just arranging the words that you would say that would be like dramatically appropriate in this in this situation.
David Bau: But it's it's very specific to the situation is not really about abstract knowledge that's right knowledge seems to be here at this early say.
David Bau: Okay, so okay so there's a bunch of other methods that people have used for training and fine tuning these transformers to sort of customize them with.
David Bau: You know, specific information about your company or your your application or whatever people will take one of these big models and fine tune them.
David Bau: and teach them about toothpaste, or whatever it is you want your chat Bot, to talk about, and so, so we compared to a bunch of these other methods direct fine tuning.
David Bau: there's this interesting interpreted in there on method from another paper there's this really cool idea of using hyper networks, this is.
David Bau: Something that was really you know invented just a few years ago, and people are applying it to.
David Bau: To transformers, and so we try to get to all these different methods, but they're all these all these these are opaque methods, this is an interpreter method there there's a mix of different approaches here.
David Bau: But you know.
David Bau: Our our method is is different in a specific way you know we're we're pretty interoperable, we know exactly why it is that we're changing the layers that we do.
David Bau: We know exactly which layers we're changing we're making a very small rank one change and we're selecting it based on like you know the encoding so the words.
David Bau: That we're we're trying to insert into the model, and so the question is.
David Bau: You know, being misinterpreted role being like this explicit about what we're doing, can we outperform this this giant black box methods that are trained on huge amounts of data right.
David Bau: Like we take a single example like a single sentence, how do we perform right okay so basically yes we we outperform them.
David Bau: So there are two failure modes that you see in the black box methods and one of them.
David Bau: Will you say.
David Bau: A bit, yes, yes.
Tiwalayo Eisape: You know, we have an eight minute warning.
David Bau: A minute okay i'm almost at the end, almost at the end okay so there's two things to look out for when you're claiming that you've changed the knowledge of a network I mentioned this before once you're looking for generalization.
David Bau: And you're looking for specificity and when you we test generalization carefully you'll notice that a bunch of the methods basically fail to generalize so like if you use these fine tuning methods.
David Bau: Or if you use one of the hyper networks called men, then it, you know all the methods are not elements and there are methods don't do that well at efficacy but it's not not a huge feat to get to like 99% or 100%.
David Bau: On on like can you pair it back the example that you gave right so most of methods can do this they can pare it back.
David Bau: But if you like, reward the thing and asked us, you know how do you do, then, most of the methods cannot recover the fact in a rewarded prompt, they are very, very fragile and so the generalization is low.
David Bau: Now some of them generalize quite well so there's a particular so like we generalize pretty well and room there's another hyper network they generalize is pretty well.
David Bau: But then there's this other thing to look for it, which is specificity like does Bill Gates now play soccer kind of thing right is everything in Rome, all the empire state building's in Rome right.
David Bau: And so, when we measure those kinds of things, we see that.
David Bau: The rest of the methods failing specificity so like one one easy way to generalize to make it so that on all the paraphrases.
David Bau: Of the Eiffel Tower they're all the paraphrases in Rome is you could move every site in the world to Rome and that's a very easy way to generalize and we found that the other methods are actually generalizing in that way they're they're moving way too many things.
David Bau: To you know, have the attribute that that that piece of knowledge that that they're supposed to have and and so you can kind of see this year.
David Bau: But you know the methods extremely specific right it's um it really doesn't affect the the sort of facts that shouldn't be affected the facts about other subjects.
David Bau: That are nearby so anyway so that's so that's that's you know there's roaming green a bunch of free numbers no red stuff works pretty well, it does pretty well at at preserving.
David Bau: fluency when you do this and now it's not perfect, you know the numbers are not all hundred percent or whatever there's some.
David Bau: there's some things and I think that reflects there's other things going on in the computation Besides this simple dictionary lookup that we do we don't understand all of them.
David Bau: But I think there's pretty good evidence that this dictionary lookup is pretty important it's pretty large part of what's going on.
David Bau: We can find it, we can change it and and we can like change it in this nice robust way they generalize this specific so Oh yes, so there's all sorts of examples here of pieces of text that you can change, let me, let me show you some examples, so this is a PA PA Pierre Curie who's a physicist.
David Bau: is much more famous wife, Marie Curie but he also did physics and also worked with her on radiation research, but we could insert a counterfactual.
David Bau: That he works in medicine instead and asked various methods to to to generate a text about this and you can kind of see the different types of failures that you typically see.
David Bau: It you know, sometimes it'll work the things the cases where it works are in green where it's sort of you know, tried to transplant this famous person to be working in medicine and said.
David Bau: But here, you have this example of non specificity like theory Curry was moved over but Robert milliken who is another physicist also.
David Bau: gotten moved over to work on biology and human mind and stuff like that Ryan milliken I don't know if you remember, but he's like a physicist, who worked with little oil drops in.
David Bau: figuring out like the charges electron things like that, so it has nothing to do with this stuff so.
David Bau: So you know it, this is an example of specificity failures there's various examples here that we can look at.
David Bau: generalization errors this this example down here is like a fluency problem where it went to try to get them all to talk more about medicine.
David Bau: it likes to talk about medicine, a lot once it's mentioned the word medicine, it just starts repeating the word medicine and so on, so like we've hacked like the wrong part of this system.
David Bau: When we when we've used some of these other methods and so that's just an example of of what you see and so.
David Bau: Suddenly wrap up here.
David Bau: So what we've shown is showing this like interesting way of threading through the computation is the network, something I call causal tracing and she give give credit to Pearl.
David Bau: You know this is really cousin mediation analysis from the tradition of Pearl and what we've done this in a graphical form, and you know through through a large language model and causal tracing.
David Bau: And we've we've come up with a method for finding these these sort of location of these facts, and then changing them we do these rank rank one model edits so that's our own me that's where Rom comes from.
David Bau: And then we have this way of measuring what we've done by looking at generalization and specificity it's sort of requiring us to.
David Bau: put together new data sets and I feel like there's a need here we've we've we we put together a data set pretty quickly that you know show some big.
David Bau: differences here between the different approaches, but I think there's some neat here first and better data that allows you know more detailed probing of these models and so you know probing with counterfactual is is pretty interesting area.
David Bau: To look at I think okay so that's so that's what we talked about That brings us to the end of the talk.
David Bau: Here we go there's a little hyperlink to our website okay there we go I covered a lot so um yeah you guys have any questions any questions happy to answer questions.
William Chen: yeah okay so um i've got one, so it looks like the example that or it looks like the method that you showed is pretty good I editing the entities in relations right.
David Bau: that's great.
William Chen: um So do you think that's such a system, or maybe some like other method could work, maybe at changing the predicates in known relations instead so, for example, like the the idea I had was instead of saying like.
William Chen: athlete plays whatever sport they're associated with right, you can change into say athlete commentate on sport or athlete watches the sport and have it apply to all of the valid athlete sport like pairs um but like without changing the entire concept of playing or watching.
David Bau: yeah that's that's right I.
David Bau: I suspect.
David Bau: That there is.
David Bau: So the way that we have put together counter fact for testing is by.
David Bau: Pulling from like knowledge graphs we pull all these facts and knowledge graph and then we sort of used them to come up with these test cases like this, and so what we know.
David Bau: about how well this method works is limited by what we've tested it on basically by the sort of microscope that we've chosen to use, and I agree it's limited like we.
David Bau: have not dramatically changed somebody's profession or the relationship, they have with a sport or something like that and it's likely I think it's likely that some of these broader things are also included the same way, but we don't know that we because we haven't measured it.
David Bau: And they sort of what i'm saying is that we need, we certainly need to develop some better data to allow us to ask these questions systematically.
David Bau: I also think that there's probably a lot of things that models know they're probably not included this way and it's pretty interesting to try to sort through them like what what is the model.
David Bau: Knowing through like a dictionary lookup kind of thing, and what is the model knowing through some maybe more complicated computation we don't really know the difference between those two things I think it's very interesting research question.
David Bau: So thanks for the question we.
William Chen: Thank you yeah.
Tiwalayo Eisape: We have one question from Thomas and then Nancy.
David Bau: cool thanks Thomas.
thomas clark: hi yeah Thank you so much for the talk it was super super interesting, I guess, one question I have is it's not clear to me, I guess what the desired behavior is and all of these edits like I saw with the case.
thomas clark: With you know changing Eiffel Tower associated with Paris to Rome that it's really cool and desirable that it would start to say like St peter's Basilica is across the street.
thomas clark: Like let's say you had something like you know, Joe Biden.
thomas clark: was born in the United States, and you want to change that to like was born in some country.
thomas clark: That doesn't necessarily imply that he should be the President of that country.
thomas clark: Right like maybe he was just born in that.
thomas clark: became still became the President of this country, you know, like it's not clear to me what is the desired behavior in all these cases and how do you go about sort of defining what should or shouldn't.
David Bau: Oh, this is so hard it's a good question.
David Bau: I you know i've been i've been.
David Bau: puzzling over this, you know with with with collaborators and with random people in the hallway if you want to chat with me about it sometime we can chat.
David Bau: What is it counterfactual like like like putting together this data set and realizing it's a data set of kind of factual sort of made me like.
David Bau: crack open the old philosophy textbooks about people have been arguing about counterfactual is for like hundreds of years what the heck is a car counterfactual and and what does it mean.
David Bau: And one of my one of my.
David Bau: reviewers.
David Bau: One of my friendly reviewers basically pointed at like our equation 10 when we do room, he says haha you can tell that you can tell that.
David Bau: You can tell all the philosophers now equation 10 the rank one model edit that's a kind of factual whatever the model says that's that's that's.
David Bau: When you change the models, you know understanding of fact and and it makes it makes some other predictions that's what it kind of factual is actually it was kind of a joke that was that was a.
David Bau: My friend Thomas call her sort of made this joke that counterfactual is, whatever the model thinks it is, if you change it in the right place, but it's it's not it's not that.
David Bau: Far fetched to flip the question on its head and look at it that way you know, I think that if you ask like Pearl what is counterfactual is you say well start with a.
David Bau: graphical model of the world and a counterfactual is sort of defined by you know what what happens in your model when you do an intervention and.
David Bau: And the So what is a transformer right but it's like a giant graphical model of the you know the learned, you know learn model of the linguistic view of the world and and so so on one hand we had this question of like what is a human think.
David Bau: would be true if you change a kind of factual.
David Bau: But then, this is parallel question of what is the model think it isn't it kind of interesting to see what the differences are.
David Bau: So anyway that's how i'm looking at the question but it's but yes it's an old question it's it's a little hard to untangle exactly what you expect the implications of the counterfactual to be.
David Bau: Is that okay it's not.
David Bau: A non answer to your question.
thomas clark: yeah they say yeah definitely something to.
thomas clark: Keep thinking about it, I find it really, really interesting so so yeah yeah.
David Bau: Nancy.
Nancy Kanwisher: yeah this is so cool David, thank you, maybe this is an incoherent question from but it's kind of laughably neuroscientific but I know you have to just said, concentrated green patch there and.
Nancy Kanwisher: wondering what is so special about that early nlp site has a different from other sites is there anything to be said about why all the facts land there.
David Bau: yeah actually I hear you know Okay, did you ever see that.
David Bau: Oh, we also the amy what was what's her name.
David Bau: That that science fiction story, where she talks to the aliens.
David Bau: what's it called I remember, I remember the name of that movie is.
thomas clark: A rival.
David Bau: rival.
David Bau: rival, yes, yes and.
David Bau: Anyway, so rival, and they can see in that movie was it was really hard to understand what these aliens are saying, because they don't.
David Bau: Have the same notion of the flow of time as humans, do they mix up the future in the past, and so on and so.
David Bau: you're looking at like this concentration of state and transformers actually sort of me was looking at this paper too long, but it made me think of a rival.
David Bau: Because the interesting thing about this particular location is this this this this concentration at this Green spot here is it tends to be always at the last.
David Bau: token for the subject name so like shaquille o'neal you know is token as into a bunch of tokens by by the model.
David Bau: But this Green blob doesn't appear in shaquille o'neal's case until you get to the very last letter of the last name right o'neill right and then all of a sudden, like this look up happens, and if you think about it.
David Bau: That would be the efficient place to do the look up right if you try to look up shaquille o'neal earlier actually in his case you could maybe get away with just looking at shaquille because this person is so distinctive.
David Bau: But like megan rapid know there's a lot of megan's in the world right and you don't really know it's a soccer player until you get to the very, very end right.
David Bau: And so it would be inefficient to like do a look up any earlier than seeing like until you saw her whole name.
David Bau: But.
David Bau: You could wait later, but if you waited later it might be too late, you might need to know that she was a soccer player when you're predicting like the next word.
David Bau: It might be critical knowledge, so if there was a time for you to look up this fact that megan know was a soccer player.
David Bau: Then you sort of have to be patient and wait until you have enough information, but then you have to immediately do it right away before it's too late, when when you know when that information might be needed.
David Bau: And so I feel like there's this ugly thing and gbt about how it has this one directional.
David Bau: You know, you can only look at the past I always looked it up and I thought oh it's such an ugly or model than Burt and the sort of mass language models that kind of kind of look, both in the future, in the past and everything like that.
David Bau: But I feel like the fact that gbt has this you know when streaming this generation task it, it looks in the past, only it forces it in that in that process to.
David Bau: To really concentrate it's look ups of its of its factual knowledge and so that made me think oh if we like analyze one of these arrival aliens we might not see this right, you know, this is a you know, this is something pretty interesting.
David Bau: And so anyway it's just speculation, but that's that's sort of the you know my my my view of like when you know why this might happen and it's such a concentrated way.
David Bau: To answer your question.
yeah.
David Bau: Okay let's see so next on my list is.
David Bau: Gone I can't see your first name because it's covered up by zoom here.
David Bau: Oh sorry hi.
Peng Qian: Yes.
David Bau: I paint.
Peng Qian: A nice, thank you for the very interesting work, I have a quick question about when you look at this problem among different entities, do you found that sort of entities and more susceptible to this change or.
Peng Qian: Like because you have entities about P person and also.
Peng Qian: Yes, allocations, is there any differences across the behaviors models when you're trying to edit different kinds of entities.
David Bau: I would say that the big difference that i've observed is not kinds of entities, but simply how well known, the entities are so I could show you failure cases so there's there's a lot of cases where this.
David Bau: This causal traces not as clean and they tend to be cases where like if I don't have a causal Tracy of Edmond neupert so by showed you this example of admin uber who put you know correctly like predicted is playing the piano.
David Bau: By the model, but if you look at the causal trace for admin neupert it hardly like has any concentration around his name at all and it's like.
David Bau: You know, really focused on hidden states in all the grammatical words around the sentence and everything looking at everything i'm not really paying too much attention specifically to the fact that his last name is neupert or anything like that, and so.
David Bau: So, so I would say that you know you have this concentration when it seems to be acting like it's a pretty famous person or a pretty famous entity or something that that the model actually knows about.
David Bau: Anyway, so that's that's at least that's the pattern that i've seen.
David Bau: yep.
Peng Qian: Thank you.
David Bau: josh.
Joshua B Tenenbaum: hey I.
Joshua B Tenenbaum: So thanks, this is very intriguing i'm.
David Bau: cool.
Joshua B Tenenbaum: Yes, i'm I haven't really studied it I looked I looked at, you know I saw a must if you're talking I I saw I looked at the demos and examples on on the website and.
Joshua B Tenenbaum: I guess i'm i'm interested in a behavior that you that your Eiffel Tower in Rome example shows on on there which.
Joshua B Tenenbaum: makes me worry I guess about using words like knowledge here, because you know you ask the system to tell the various things and it makes some some stuff which is like, which is which it just makes up and like is.
Joshua B Tenenbaum: counter to fact and like logical possibility right so like when you when you say things like you know where is or tell me how to get from Berlin.
Joshua B Tenenbaum: To the Eiffel Tower it says something that like sounds plausible on the face of it, but then it tells you to go to a station train station, as far as I know it doesn't exist, like Roma centrale and it says it'll take you.
Joshua B Tenenbaum: from Berlin to Rome it'll take you five hours and 50 minutes when it's it's like an overnight train rides right so.
Joshua B Tenenbaum: Is that just randomness in the model or is that.
Joshua B Tenenbaum: Is that a sign that, like the web of of stuff has been disrupted somewhat or does it never even know that in the first place, or like when you say you know what's.
Joshua B Tenenbaum: right that the Eiffel Tower is right next to the Vatican OK, and then it says in the Coliseum is just a few blocks away and.
David Bau: it's just like I accept this fact.
Joshua B Tenenbaum: nearly impossible, like the calcium, is a few miles away did it.
Joshua B Tenenbaum: Is it just like.
Joshua B Tenenbaum: it like it is that stuff that gbt three or whatever just didn't know or didn't somehow get it it's hold geography of Rome got distorted from that like and you have ways of assessing that yeah.
David Bau: yeah I think that's right, I think this is so there's definitely.
David Bau: A difference between like knowing and guessing and I think that.
Joshua B Tenenbaum: yeah there's like a web of associations right.
Joshua B Tenenbaum: that's the way to describe it as an associate memory makes a lot of sense like.
Joshua B Tenenbaum: Right, I would say, well there's some.
Joshua B Tenenbaum: associations and you are changing those in a somewhat targeted way, but I guess the question breaks on.
Joshua B Tenenbaum: Two things one is country, why not just.
Joshua B Tenenbaum: Call it associations, rather than knowledge which seems more like what it.
Joshua B Tenenbaum: Is.
Joshua B Tenenbaum: Oh, I see because, like knowledge knowledge is supposed to be true this isn't true, but the other is how well do you assess really whether you're really like able to change a single association, without affecting that web, because it seems like from at least those.
Joshua B Tenenbaum: examples which are quite interesting because the free form text generation gives you a lot of stuff in the local part of the associated web that actually really are changing a lot of things that aren't even about Eiffel Tower but just about like what you know about Roman.
David Bau: yeah yeah so let me, let me defend, let me defend our use of this very loaded word knowledge, now we were looking for a way of talking about so.
David Bau: So I think that there's a lot of ways if there's a lot of different associations in these models, we can sort of Edit them all, and there was a particular interesting kind of association that we were after.
David Bau: Which is associations, it really generalize and there are specific that that that match up with the kinds of associations that humans are catalogued and knowledge graphs.
David Bau: You know these knowledge tools and so since like there's this history of like calling these associations between subjects and their attributes, you know Eiffel Tower is in Paris and things like that you know as.
Joshua B Tenenbaum: You know, as not like when Google.
Joshua B Tenenbaum: drive, yes, they work really hard to verify that it's true.
David Bau: I don't know I remember, I totally agree.
Joshua B Tenenbaum: john Jen Jen dre explaining that thing.
Joshua B Tenenbaum: Until the knowledge graph unless you verify that it's true.
David Bau: totally agree and so.
David Bau: And so I guess, I want to distinguish between and maybe we should you know really clarify this when we write about it.
David Bau: Between the.
David Bau: The association between like Eiffel Tower in Paris, which really is a you know, a piece of knowledge it's true and the association is there and it's quite robust and it's generalized is over lots of lots of changes and.
David Bau: And then the then there's all these follow on things that the model doesn't know like exactly where and Francis Paris exactly where in Paris Eiffel Tower.
David Bau: Like the model like gbt actually doesn't really have a lot of evidence of knowing these things in detail, it has a bunch of other loose associations and things that may be guessing but, but there is an association between Eiffel Tower in Paris that.
David Bau: that the model seems to be quite robust about and we are able to change that association in a way that seems to continue to be.
David Bau: Robust we can associate Eiffel Tower now with a different city in a way that also is robust now all the follow on things that it that it guesses that i'm not counting those as knowledge Those are just sort of linguistic pieces of evidence that I know it seems to have gotten the topic.
Joshua B Tenenbaum: It sounds like what you want, is that relational specify.
Joshua B Tenenbaum: I think, maybe you want to word that's like relations like that can you can you affect a single relation.
David Bau: Yes.
Joshua B Tenenbaum: Which is like there's a you're trying to express a precise relation with like a predicate subject and an object or a to place to pull or.
David Bau: Something that's exactly what religion.
David Bau: Right yes.
Joshua B Tenenbaum: it's like that's not what the word knowledge me.
Joshua B Tenenbaum: And also.
David Bau: that's right.
David Bau: So maybe maybe a little closer is like so you know we like I try to.
David Bau: Try to catch it, as you know, we're really focused on factual knowledge as opposed to you know there's a lot of other types of knowledge and there's these.
David Bau: Maybe we should call it like association or factual knowledge or something like that to be really precise but um but I agree we're looking at this rate and we're looking at this very thin, but I think interesting you know sub case of of knowledge and we're trying to pin it down.
David Bau: Well, I totally.
Joshua B Tenenbaum: I mean I just I just think we really working in.
Joshua B Tenenbaum: Language I mean this is this.
Joshua B Tenenbaum: Is you know, like it's it's it's.
Joshua B Tenenbaum: I don't know how to how to make an analogy to some of the vision things where you've looked at, but it's like.
Joshua B Tenenbaum: Yes, you know there's like very important things I think in in.
Joshua B Tenenbaum: In like language and meaning and that connects to things that philosophers have written quite a lot about with lots of formalism and.
David Bau: yeah.
Joshua B Tenenbaum: I think it's important to.
Joshua B Tenenbaum: make contact with that in in a deep way.
David Bau: That makes sense and.
Joshua B Tenenbaum: You know, like distinguish like you're looking to I think the bath that you have is not math of knowledge it's math of associations and you did a really nice job of explaining the tradition and the connection to classic you know linear associates, and all that sort of things.
Joshua B Tenenbaum: So, but you want it to be more precise than just any kind of vague association so there's a.
Joshua B Tenenbaum: particular kind of like associative relation or something and I.
David Bau: Think.
Joshua B Tenenbaum: it'd be good if you had a vocabulary, that it was about that.
David Bau: So that's.
Joshua B Tenenbaum: Maybe people like me didn't get distracted about like well does dpt three actually really know any of these things and did you affect them or not.
Joshua B Tenenbaum: Right run.
Joshua B Tenenbaum: right because, like a knowledge graph that's partly why like people who like symbolic approaches, like a knowledge graph actually.
Joshua B Tenenbaum: Like there's many ways in which the traditional knowledge graph and and the ways, you might modify it would be I think much, much more powerful and robust than than what you're currently showing.
David Bau: Right yeah.
Joshua B Tenenbaum: Anyway, we can talk about that more often.
Joshua B Tenenbaum: very sweet.
Joshua B Tenenbaum: And I you know i'm glad you're going this direction, I just think.
David Bau: You know the thing I like is.
David Bau: I feel like you know even even if you have qualms about you know, the way that we're looking at, I think that i'm i'm quite happy that i've gotten your attention.
Joshua B Tenenbaum: yeah yeah well.
Joshua B Tenenbaum: And yeah.
Joshua B Tenenbaum: yeah okay i'll let let you finish it.
David Bau: Thank you, thank you, thank you very much for yes, I think this is very well taken.
David Bau: Feedback Thank you josh.
David Bau: I see Anna has a hand up.
Anna Ivanova: Okay Ah yes Thank you so much for your talk, I think this is very cool and yeah I love this attended disseminating information flow and transformers and, like the you know laden's knowledge associations with everyone or call them whatever they call them.
Anna Ivanova: I have two questions in one is more technical, so there is like a quick answer that'd be great, but if not like I can just look at the paper and one is more conceptual the technical one, is that, in order to alter your.
Anna Ivanova: associations, you need to have the key and a value pairs right, and so the key for the Eiffel Tower I can kind of guess how you get and you can just keep the Eiffel Tower to them at work, but how are you getting the value that they're like located in Rome.
David Bau: Yes, so, so I would say that the so the so the value is, let me find the side here, so the value were actually just getting into.
David Bau: An optimization over an intervention so let's see, I guess, this is as good as slide, as we have yeah So here we have so basically.
David Bau: If we run the network, you know, like.
David Bau: You know, on on this target sentence, the Eiffel Tower is located in the city of.
David Bau: Then what we can do is, we can actually.
David Bau: Do this transplantation technique, we can put any vector we want as the output of this LP.
David Bau: And then see what happens, and if we put like a random noise factor here then it'll instead of saying Paris it'll just say some garbage here right.
David Bau: But if we put a different vector here, but then, if we, but then we look at gradients, we can say Oh, we perturb this vector to get closer to Rome.
David Bau: And so we basically do gradient descent until we find what vector you would put in this specific location, that would get this pretty close to same room.
David Bau: And so we do this, we do this optimization we regularize it to try to make sure that we're not changing the vector too much.
David Bau: And then, and then we say okay that's that's you know that's the target vector whatever we can stick in here that, when we run it through the network will cause it to say wrong.
Anna Ivanova: Now specific to the key right.
David Bau: That is so so this this vector that we put in.
David Bau: Here we do an intervention, so that so like this factor is the thing that would have been output by the nlp here, but we.
David Bau: In this case, for this.
David Bau: token at this layer we don't actually use the output of the nlp we just we just we sort of do is do operator we sort of stick in.
David Bau: My new vector and and then and then just use that instead of whatever the mo P would have normally done now at all other tokens and it all other layers you know the computation proceeds as normal, but this one is sort of like hot wiring in the network we just directly feeding in.
David Bau: Some vector here and we're asking.
David Bau: We as an optimizer to figure out what factor, would you directly feed in here to make it safer on, so this is very different from like the normal.
David Bau: You know sort of optimization you do when trading network or something because we're not changing the weights of the network at all.
David Bau: Instead of we've sort of snips the network here and we're optimizing is what information one factor, do we fit in to this exact spot and the computation that would have the effect that we're looking for and so.
Anna Ivanova: What does that mean.
Anna Ivanova: we're just saying what the same vector work he did the the key was different if you are changing the White House location, the wrong or what are the specific to the key.
David Bau: yeah So if you put it exactly in the same place, it would we've we haven't found that it works it like is different.
David Bau: But we don't know if it's.
David Bau: This reusable.
David Bau: For these vectors but we haven't been able to do that yet.
David Bau: And so that's sort of.
David Bau: You know that's that's that's that's that is i'd say this is an unfinished part of the work at mapping out what the space of these value vectors is so this this embedding space for what the value of actors is kind of a picked us right now.
David Bau: me sense cool.
Anna Ivanova: yeah yeah and then the second question was.
Anna Ivanova: Like yeah this.
Anna Ivanova: Key value mappings idea is very cool, what do you think about teases which I would say, are the majority of kids and like where there isn't a single key value pair.
Anna Ivanova: right if you were to say like anyone have a study that right, you could say my tea, but you could also say in your city Miami well and for undergrad and my high school or something else right.
Anna Ivanova: A musician can play violin and the piano, what do you think about those like non One the one that things and like that probably there's going to build a distribution associated with.
Anna Ivanova: which was to go with, in which case.
David Bau: Yes, yes, like things where you have multiple things that are true.
David Bau: that's right.
David Bau: Is is Joe Biden president or Senator maybe he's been both.
David Bau: That kind of thing, I think that.
David Bau: I don't have a great answers for you here, this is something that we're actually in the midst of studying sort of get to get a little bit better view of what the capacity of these things are.
David Bau: I can, I can kind of share anecdotes or like intuitions.
David Bau: I think that there's some limits, and I think that some of those limits might actually.
David Bau: be interesting right they might actually reflect sort of limitations of what you know God is able to do like it's I think that, like if Michael Jordan both plays baseball and.
David Bau: basketball, which is true he's done both.
David Bau: I think it might be very hard to store in the model both of those facts at the same time and it's just like you know couldn't figure out how to encode it, but it may be that up is incapable.
David Bau: Of of of you know of entertaining both of these things at the same time as well, and so so anyway, I think this is a reasonable question we don't really know the answer to it.
David Bau: it's things that you can try to do by trying to you know insert more than one thing at once, or trying to find a vector that represents both at once.
David Bau: But we haven't you know we don't have a clear view of it we haven't had you know direct success of doing you know complicated interventions like that.
Anna Ivanova: Thank you, thank you yep.
Tiwalayo Eisape: it's a wrap up.
David Bau: Here, thank you.
Tiwalayo Eisape: yeah Thank you David, for your time and.
Tiwalayo Eisape: Coming.
Tiwalayo Eisape: Thank you.
Tiwalayo Eisape: Next week for next week.
David Bau: Excellent thanks a lot thanks so much.
Nancy Kanwisher: Thank you, David.
cool.