David Bau: compliance.
David Bau: Hey, how's it going?
David Bau: Okay, yes, that worked. Okay, too much to cover again today. Sorry, you guys.
David Bau: Converse implications down the street.
David Bau: So, okay, so I'll just dive in.
David Bau: So I… so we've had a few meetings, and, you know, I'll need that.
David Bau: Okay. Actually, Hayu, is Hayou here? Yep. Had an initial question.
David Bau: And, so now, I'm gonna head on right now.
David Bau: Hi, you. What was your question? So what does it mean for the model linearly, then?
David Bau: what does it mean for the model to linearly represent truth, right? So one of the papers you guys read, first one was this Geometry of Truth paper.
David Bau: And they had these nice scarapot for me, which was supposed to be suggestive of something.
David Bau: And so, so none of the paper is actually
David Bau: It's actually really hard to find any modern paper
David Bau: Pin down a definition of what it means for something to be linearly represented.
David Bau: But… but this is… so I'll make up a definition, and you can see this in some old papers. This is an old definition, right? So let's say…
David Bau: You have a whole bunch of neurons, In your neural network.
David Bau: And you have the numbers that say how much neurons are activated in a vector X, right? And for different stimulus.
David Bau: this X will be different. That's your… X is your representation.
David Bau: What it means for something to be linearly represented in that vector is just…
David Bau: this, is that you've got some scalar, you've got some attribute, some property A, that you care about. Like, is this sentence true, right? Or is this…
David Bau: Is this, AI being truthful or not? Something like that. That might be… that might be your scalar output. Maybe…
David Bau: A big number means yes, true. Maybe a small number means no, false.
David Bau: Right?
David Bau: And what it means for it to be linearly represented is that you can just
David Bau: Do what's called a linear readout.
David Bau: of the neurons.
David Bau: You can just take a weighted sum of all the neurons, or equivalently, a dot product by some.
David Bau: Fixed vector W.
David Bau: And that'll give you some number, which is just your readout of this.
David Bau: of this property.
David Bau: Now, for truth, maybe the property is true or false.
David Bau: So, you know, I think there's some details here that I've omitted. Like, you might… you might threshold it at some value, say if it's…
David Bau: bigger than the value that you say is true, and as small as the values say is false. It might not be 100% accurate, but, you know, it might be good enough it was 80 or 90% accurate. People would say, oh yeah, this information is in here. It's linearly readable. Does that make sense? Does that sort of answer the question? So that's all it means.
David Bau: For something to be linearly… encodable, okay, so…
David Bau: Yeah, so it's all about around dot products, like, everybody learns what dot products are from linear algebra, and then when you see a scatter plot, like what Sam Marks put in this paper, you know, you can see that this is… the reason he thought… it's obvious after I put my scatter plot that this is linearly readable.
David Bau: Because you can see this clear separation, so he didn't put this vector here.
David Bau: But this vector would be, like, the wave vector W shape that you're dot producting with.
David Bau: And so if your dot product with this thing, then whatever the dot product is in the positive, it's in the red area, if it's negative, it's in the blue area. And when you do the linear readout, you know, you can make a little histogram of, like, how many dots, you know, were negative, and how many dots were positive you did this. So, you know, I recommend if you…
David Bau: You know, do this with your own…
David Bau: Your own, data, you know, like histograms like this, and you can kind of see
David Bau: You know, how well it's separating.
David Bau: you know, different…
David Bau: different things. So all the things that are supposed to be negative are over here, all the things that are supposed to be tied up over here. So why do people believe that anything would be linearly readable?
David Bau: in a neural network. The reason is because
David Bau: It's how neural networks work internally.
David Bau: So, you know, we draw all these network diagrams, Of the neural network.
David Bau: Where every neuron picks as input.
David Bau: a weighted sum of the alphas of other neurons? All this is is a dot product. All this is is a linear readout. So, if you think of a deep neural network, all it's doing is at every layer, every neuron is one of these operations, is one of these linear readouts. So.
David Bau: So, so the hypothesis is…
David Bau: That all the useful information in a neural network Maybe.
David Bau: Is actually able to be read out linearly, because
David Bau: If it wasn't read or able to be read out linearly, the neural network itself wouldn't be able to access it.
David Bau: Right, because that's all the neural network has as a tool for reading its own
David Bau: information. So that's sort of the hypothesis. Yes, go ahead. Does this linearity decrease even after we add
David Bau: deliberately add nonlinearity to the network. So then, then after… so you're saying, oh, typically you add a little nonlinearity after you do this to the threshold that…
David Bau: That's right. And it's the same detail here where I say, oh, you know, you might add a thresholding or something like that. And so, that's… that's equivalent to, let me…
David Bau: I'll come to that in a minute.
David Bau: But that's sort of equivalent to saying, hey, you know, at some point, we might want to draw a separating plane.
David Bau: Here? So, all linearity really means is, even if you have, like, a separating plane where you're gonna have something nonlinear, like a decision.
David Bau: You know, with the linear readout, all you can really do is slide that plane around. You can't make it into a curve or something like that, right? You just gotta slide it around. Now, obviously, you can make curves and stuff if you have multiple layers of a control network.
David Bau: But if you just have one layer, then all you can do is basically read out
David Bau: A straight… straight line.
David Bau: It's readout something linear. And so, so, so the linear readout Hypothesis is that
David Bau: Well, the neural network is going to curve all the data around, but at the moment that it's got to use it in a useful way.
David Bau: And maybe it's… the useful information is linearly readable at that… at that moment, and you should be able to read it too.
David Bau: by finding the right linear model, just put on. So that's… that's what all these… papers that I…
David Bau: assign are all about is linear redocking. Okay? Okay, so I like this other question.
David Bau: Eunice asked this question, and Kavala asked, oh, another related question. Yeah, so what was Eunice's question?
David Bau: Is your audio working? Do you want to ask your question?
Emre Tapan: Yeah, yeah, I am on Zoom, so, can you hear me?
David Bau: Yes, it's beautiful.
Emre Tapan: Cool. So all of the papers were kind of, tried to prove that concepts are linearly represented, but
Emre Tapan: Are there any cases, concepts are kind of nonlinear, or clustered, or I don't know, what are the… what are the other options?
David Bau: Now, what are the other options? If everything's so linear, then, like, what's not linear? Right? I mean, I think that's a really good way of understanding
David Bau: like, what it even means to have a hypothesis. If it's not falsifiable, then what we're not even doing science, right? So, yes.
David Bau: you can definitely have nonlinear encodings, and my… and the, like, my clearest example is actually from the oldest neural network paper in history. So who's ever heard of
David Bau: The Romo Hart, Hinton, and William Scaper.
David Bau: Anybody ever heard of this paper? Yeah, okay. My own grad students have heard of it, because I've made them read it, right? So, this is the first neural network paper ever. This is what won, Jeff Minton the Turing Award.
David Bau: And it's where they basically told everybody, oh, you can train neural networks using backpropagation.
David Bau: And so, now, a lot of people, including, what's, what's his name?
David Bau: The guy who thinks we invented everything?
David Bau: No, Yamukun did invent him. Yeah, Schmindover. Yeah, Schmindover. Who invented, you know, LSTM and several things, so he's…
David Bau: He is an inventor of a lot of things, but he likes to claim he invented for them. So, Schmidtluberg.
David Bau: always… This is his least favorite paper, because he says, well, you know.
David Bau: Clinton didn't invent back propagation. You know, Isaac Newton invented
David Bau: You know? Why did Hinton win, like, the training award for this?
David Bau: And then, you know, furthermore, not just Hinton, but, like, all these other graduate students between Isaac Newton and Hinton, you know, went into it a lot more than Hinton did, so Hinton wrote this one-pager for Nature and in one terrain award, like, why did he get this credit? And if you go and read the paper.
David Bau: The main idea of the paper is not that you can use derivatives optimize the system. Everybody knew that.
David Bau: Right?
David Bau: The main idea is… was an interpretability thing.
David Bau: He's like, after you train a bunch of neural networks, and you look inside them.
David Bau: they learn really interesting representations, and so he has a series of neural networks where he tears apart each one after he's trained it with gradient descent to show that. So he's like the first interpretability researcher. It's really what he won the award for.
David Bau: And so, but his neural networks are too small, and they didn't follow… not… like, a lot of them didn't follow the principles that we generally see in very large neural networks. So, for example, this is one of his networks we studied.
David Bau: It's a network, with, I think, 8 neurons, 9 neurons, something like that. If you count the input nodes as neurons, which maybe you wouldn't even count, because it's just inputs, in which case it just has 3 neurons.
David Bau: Right?
David Bau: And…
David Bau: And, and it's supposed to detect, like, palindromes, or, or, or pixel patterns that are the same on the top as on the bottom, and they're mirrors of each other, right? So he trains this thing as a mirror detector.
David Bau: And if you put a symmetric pattern in, then this output will say, yeah, I got a mirror. And if you put a not-symmetric pattern in, it'll say, it didn't. It took him…
David Bau: you know, a long time, many hours to train this thing, you know, you could probably train it in a millisecond today. And, and these are the trained weights that come out. And the interesting thing, if you think about this encoding, it's very non-linear.
David Bau: Right, what it learns. And why is it nonlinear? It's because, you know, these neurons that he's learning are… have learned this encoding, which is very strange. For example, if this neuron reads 3.6,
David Bau: is what comes into it, because of the weight's 3.6. It means everything is zero, except for,
David Bau: The first pixel.
David Bau: Right?
David Bau: And if it reads, like, negative 7.1,
David Bau: It means everything's there except for the second pixel, right? And if it reads negative 3.5, it means both pixels 1 and 2 are on. Now, the reason this is non-linear is because
David Bau: Linear thinks… They scale their meaning when you scale the value.
David Bau: And so, really what should happen is, if you, like, scale this up.
David Bau: you know, when we made this negative, it should be in, like, the pixel 1 is, like, negative, or something like this. If you doubled it, it should be in, like, pixel 2,
David Bau: It shouldn't have anything to do with Pixel 2, it shouldn't have to do with Pixel 1, but it changes its meaning, and then you can kind of see when it gets to 14.2, it means, oh, it has nothing to do with Pixel 1 and 2, it has to do with Pixel 6.
David Bau: Right?
David Bau: So, what he's done is he's learned this neural network, which splits up the number line.
David Bau: And for every different part of the number line, it has, like, a pretty unrelated semantic meaning. It has to… he's using arithmetic coding here, or this neural network has learned how to do arithmically, but that's different from a linear…
David Bau: Representation, because it doesn't scale its meaning With the scale and value.
David Bau: for truth, It would be like, oh, if something was already true and you doubled it.
David Bau: It would just mean… like, more true. Like, I'm more certain that it's true.
David Bau: Right, and then there's some threshold where it's false, and then if you make it, double it and made it more false or more negative.
David Bau: then it'd be like, oh, that's more false, I'm more assuming it's false, or something like that, right? So, but here, it doesn't work that way.
David Bau: And so, yeah, so there's definitely… and it is a neural network, so, like, neural networks can learn nonlinear encodings, it's just… it's just that we tend to not see them in large networks. It's, like, they're pretty hard to learn, they don't show up that often, but it's not impossible.
David Bau: Make sense?
David Bau: So, actually, I've always been puzzled by this.
David Bau: When I was starting…
David Bau: studying neural networks. Have you ever thought about this? Like, how much information can be stored in one neuron?
David Bau: Like, that's weird, right? Because a neuron is, like, a real number.
David Bau: Right?
David Bau: How many digits go in a real number?
David Bau: 6, how many digits can go in a real number?
David Bau: Like, infinite, right?
David Bau: I guess it depends on the hardware you choose, but that's not real, that's a floating point number. What about a real number?
David Bau: Like, what if it was, like, a biological neuron? It's like a real… like, it really activates.
David Bau: Right? What's a real voltage? Like, how many digits can go in a real number, like a real biological neuron? It's, like, infinite, right? So, that leads you to this puzzle. It's like, how much information can you store in a real number? Like, if I write out a real number, like, I can…
David Bau: you know, I could encode, like, the whole Declaration of Independence as ASCII, and then put them in the digit that was a real number, and, like, a single neuron should be able to encode that, right? So that's what's going on
David Bau: With these neurons here, it's like… like, each one of these neurons is, like, encoding a whole 6-bit pattern.
David Bau: And neurologers, they're able to learn this if you train them hard enough.
David Bau: But… but it's, you know, it's really weird, right? Like, you might extrapolate this into, you know, sort of the Declaration of Indenis.
David Bau: You're right. But, but it turns out that's really hard to train networks
David Bau: do this, and what we tend to learn is we tend to learn networks that learn… that represent everything linearly. The, there's a… there's a way out of… so, if people are machine learning people and are puzzled by this thing about, like.
David Bau: How can a single neuron encode an infinite number of bits?
David Bau: You know, the thing to look at is all the variational, people, like, variational autoencoders and things like this. They work through the, the mystery of
David Bau: you know, how to control the number of bits in the neuron. And it's about reasoning about, oh, it's… the neurons are really operating in noisy environment.
David Bau: Bye.
David Bau: That they might have.
David Bau: Okay, so… so this is… this is… this is what nonlinear looks like, but…
David Bau: a lot of stuff is linear, including… so I'm… like, it frustrates me a little bit, when people say, oh, this is nonlinear.
David Bau: And what they really mean is that some things are higher dimensional. They're not one-dimensional. So, like, I gave you a definition for linear readout.
David Bau: Which is, like, for scalar things.
David Bau: Right? But some things can be linear, even if they're not scalar, so that… so who asked this? So… oh, yes, so Grace asked about this, and Aria asked about this, so… Okay, Grace, what was your question?
David Bau: Maybe you're, maybe you didn't know you asked me this. Yes, yes, sorry, I was, I was, my, yes, my question was about the sentiment paper. Yes, yes. Saying, you know, can you be found…
David Bau: singular direction for sentiment. For sentiment. And no one was happy if this is a big number, it's sad, if it's a slow number, right? Yeah, that's exactly right. And then in the limitation section, they say.
David Bau: But we don't really know if this means that we can't, like, decompose it into… It's just a linear combination of a bunch of other directions, and so I'm like, what does it mean to say singular versus… Right, yeah, what does that even mean? Let me ask Aria to ask a version.
David Bau: I think I asked basically exactly the same thing. Like, referring to the same conversation. Yeah. Yeah, about this, about this kind of thing.
David Bau: And so, yeah, so one way of thinking about what they're really saying is that
David Bau: Oh, the neural network might have a more sophisticated…
David Bau: representation of sentiment that we haven't bothered to decode. It might have, you know, 10
David Bau: Attributes in there saying, oh, you know,
David Bau: I'm happy because, you know, I'm feeling fulfilled, and I think that things are true, and I think that things are ethically good, and I'm feeling very full and happy of food and good social mood. You know, they might have a bunch of different axes of, like, you know, positive sentiment.
David Bau: And maybe what they're doing is they're finding that, like, so if you had a bunch of different
David Bau: Axes in a vector space.
David Bau: You could… you could just define some weighted sum of all those positive…
David Bau: types of sentiment, and say, this is, like, this is your average sentiment over all these attributes, and you could have one singular vector, which is just the average of all those vectors, and I think that what they're saying is, oh, we think that there might be a chance that we've just measured some average vector.
David Bau: Here.
David Bau: And… and if you were to look closer, you might be able to find that you could decompose it into
David Bau: Other vectors which add up to that average.
David Bau: And so, now, what does this mean?
David Bau: What this means is if you really wanted to understand sentiment, It's not a scaling.
David Bau: Right? It might mean that there's 3 or 4 dimensions of sentiment that that paper didn't get into.
David Bau: And and is that linear? Does that make it non-linear now? Because it's, like, 3 or 4 dimensions instead of one? And so… and I think… no, that's an abuse of terminology. It is still linear, if you can do this.
David Bau: So, if you have, like, an n-dimensional You know, attribute.
David Bau: Like, three dimensions of sentiment, or something like that.
David Bau: If you can read them, Each out using a dot product.
David Bau: Which is the same thing as having a matrix of wave vectors that you multiply by.
David Bau: Right? Then it's still linearly decodable.
David Bau: Right.
David Bau: And I think the situation happens.
David Bau: A lot in large neural networks. In fact, we've seen it before.
David Bau: Like logic lengths.
David Bau: does this.
David Bau: We played with Logit Lens for a day or two, right?
David Bau: So, when you look at Logic Lens, and…
David Bau: And you see, oh, like, it's decoding this vector to love, Like, what you're doing is…
David Bau: This is a… you're interpreting this using a 50,000 dimensional linear readout.
David Bau: What you're doing is, love…
David Bau: And every other word in the vocabulary corresponds to a linear readout, and what you're doing is you're reading out, like, kind of the law of probability.
David Bau: that the model thinks that it should assign to love, but you're also reading out the love probability of every other word, and in Legendlands, we just happen to, like, read them all out, and then we just show you the top one.
David Bau: It's all the pleasure, Princess.
David Bau: That makes sense? But it's… but the process that we're using
David Bau: It's a linear readout, so that… that what you're doing is, you're taking this representation.
David Bau: And you're shoving it through… remember this little icon from first class, right? You're shoving it through the decoder head.
David Bau: Which is just, you know, it's a fixed matrix of the size of the network.
David Bau: And then it comes out with this vector, which is like, oh, a more is probably 3, love is probably 2, you know, M is probably 1, or whatever, right? It just gives you the numbers, and logically just shows you.
David Bau: A visualization of that.
David Bau: That makes sense.
David Bau: Any questions about that? So, another place that we saw a multidimensional readout was in the third paper. Do you understand the third paper, or was that confusing? Talk about the third paper. I like it a lot.
David Bau: But what they did… what Sheridan did in the third paper is set the weight matrix
David Bau: to some crazy matrix, L sub C.
David Bau: I quote where that comes from, which is the sum of some things.
David Bau: And then that was, like, a linear readout of some concepts. And so, when Sheridan decoded, you know, she traveled to Greece.
David Bau: and looked at the representation of priests.
David Bau: Sharon is, like, reading out the representation of Greece into some multidimensional vector through this thing.
David Bau: That, that, you know, Sharon's been able to catch, you know, with some imbalances. So it's another linear readout.
David Bau: Sense?
David Bau: Yes. So, in the previous one, when we had VSCA, we could have used the threshold to figure out whether a particular concept is present or not. That's right.
David Bau: But here, if it is multi-dimensional, what are we doing? Are we doing the arm of that, or are we looking in control direction? Yeah, so, you know, I think typically you bring it into a space that's
David Bau: meaningful to you, and then… then that space is meaningful to you. I guess…
David Bau: it's supposed to be meaningful, it's supposed to be like, oh, now it's up to you to understand what it means to be in that meaningful space. So, somebody asked a question, and I thought it was nice.
David Bau: I'll just ask a question, because I don't see who asked it, about what if, what if you had, like, spatial information?
David Bau: That was decoded. Well, maybe when you decode the thing, the space is kind of like latitude and longitude, or something like that, but maybe not exactly. Maybe you have to warp it a little bit for it to correspond to what we write down as latitude and longitude. But… but, you know, but it has… it has some comprehensive meaning.
David Bau: That's his idea. And so, so here, in these vocabulary cases.
David Bau: Then, what we commonly see is you decode it out to a space.
David Bau: Whose dimension equals the size of the vocabulary, right? So, 15,000, just, like, one number for every word.
David Bau: And then what comes out is a set of scores. A score for every word.
David Bau: And then the way that we interpret it is we tell us what we're interested in what the top scoring board is, what the top 5 scoring points are.
David Bau: Sense? But every, you know, so it could be different, but I think it's accurate to characterize all these as exercises and
David Bau: Reading some information is nearly out of…
David Bau: You briefly mentioned autoencoders? Oh, yes. I was wondering if what you are explaining here is similar to the concept where, without encoders.
David Bau: you can basically linearly capture one specific concept, and the more you go into one direction, the more that concept is present. Yes, I think so, correct? I think that's right. So basically, you know, when I draw this kind of figure, it's intentionally evocative of
David Bau: You know, you can read this as a linear readout of the neurons.
David Bau: But also, you know, neural networks themselves do that. So, like, an autoencoder is just an example of, you know, network architecture that also is built on linear readout. And so, when you do auto… an autoencoder, it's trying to come up with a small number of
David Bau: Scalars that are each some really informative, useful, linear readout of the… Of the figure vector.
David Bau: Sense.
David Bau: Okay.
David Bau: Let's see, were… any other questions that I missed? Aria got her question.
David Bau: Guang Yan had a question. Oh, I like Guang Yang's question. Is Guang Yan here? Here? Yes. So, yeah, on that paper, it used, like, main data booking means, or logistic regression probe.
David Bau: And why should this method, like, recover sin access? Right, the second paper. Yeah, so, it was a little sloppy, but it's okay. Yeah, what assumptions make, like, this sentiment clusterable in residual space? Right, yeah, right.
David Bau: Okay, I'm not sure I'll totally answer the second one. I think the second one is a little bit empirical.
David Bau: You know, what makes, what makes the network
David Bau: behave a certain way, why… why is sentiment clusterable in a representation? I think this is part of the mystery of deep networks, right? You know, we…
David Bau: We train them on a lot of data, we hope that they learn something useful. Usually, when something useful is learned, they'll take the things that have a similar meaning from our point of view and put them close, and things with a different meaning from afar.
David Bau: You know, why did they pick up on sentiment?
David Bau: Rather than some other thing that maybe they didn't pick up, we don't know, right? It's just…
David Bau: It was useful for some reason, very creative.
David Bau: So, but… You did ask this other, like.
David Bau: the question, that paper, they applied, the mass mean thing, k-means, logistic regression, PCA, DAS…
David Bau: Right? And then they said, oh, we tried all these things, and they all came out the same.
David Bau: So, that's not actually the lesson that I wanted you to learn from reading that paper, because they often come out different.
David Bau: From each other. They are different techniques. It does… if the data is very well behaved, then they will kind of come up and say, you know, I want to give you a little intuition.
David Bau: For what causes them to be different in the main case that you'll see.
David Bau: Right, and so… so, okay, so here's, like, my favorite scenario.
David Bau: Okay, so suppose… You had, like…
David Bau: whole bunch of students in class, except different from this class, because you've got, like, a final exam. Sorry, you guys don't have a final exam. And as a really hard file.
David Bau: You know, half the students are gonna pass the file.
David Bau: But half the students are gonna fail the final. Oh, it's terrible, all these red students are gonna be in trouble.
David Bau: Right? But, you know, but we run the statistics, and we find out that there's this interesting correlation between, like, how the students previously did in the class, right? And so, what I'm going to do is I'm going to plot the two most important variables, which is their homework rates.
David Bau: Right?
David Bau: And so here's, like, how the students did on the homework before.
David Bau: And, and who passed and failed.
David Bau: And then… and then here's, like, the minutes spent on homework.
David Bau: Right? So it's, like, the effort that they spent in their homework. And that's also, like, relating to pass and fail.
David Bau: Right?
David Bau: And so we plan all this stuff, and then we say, oh, great, you know what I'm gonna do?
David Bau: I'm gonna figure out what advice to give to students next year.
David Bau: and the teacher, who obviously shouldn't be failing half of the students. Maybe they can do a better job at running their class, right? And so, and so I'm going to run logistical regression.
David Bau: and figure out what advice I should get. And so here's the logistic regression for a very realistic data set. So, what logistic regression is doing is it's trying to find a separating plane
David Bau: Meaning these two sets, and with maximum margin.
David Bau: of it that way. So this is… so I actually ran logistic regression on this data. There's not real student data, so there's no privacy issue here. But the, I ran real logistic regression, and here's the advice. The logistic regression probe says.
David Bau: If you want to cross this line, you're on the bad side of the line, you're not, you're failing the student here.
David Bau: Right? We want to get the students to succeed, we should do more of this, go this way.
David Bau: What is this way? We should cross the line. What is this way? It means we should give everybody higher homework scores.
David Bau: Right? So, teacher, you're scoring your homework too hard. Give everybody A's on the homework.
David Bau: Right? And we should make sure that students spend less time doing it.
David Bau: Right? It seems like the total wrong advice.
David Bau: Right, doesn't it?
David Bau: Right, so… so this, like, for me, this is a good example of why logistic regression can be an excellent way of classifying
David Bau: Data, but it ends up with this… Vector…
David Bau: the pro factor, which is not… causal…
David Bau: Piece of advice on what to do to have a causal effect.
David Bau: That makes sense?
David Bau: Right.
David Bau: And so there's… so when you hear people say, oh, correlation is not causality, and this and that, right? It's sort of related to this, right? It's a slightly different phenomenon, but it's related to this.
David Bau: And so, so, nope, when, so…
David Bau: So that's… so that's one of the methods, is logistic regression. You see this a lot everywhere. It's very easy to train a logistic regression probe ever phrase.
David Bau: taught in machine learning, you should do this. So you see this all over the literature.
David Bau: But what Sam Marks advocates in his paper
David Bau: Is… oh, actually, maybe we want to use this other thing.
David Bau: Much simpler than a tree administration regression. Gives it a fancy name.
David Bau: masked meme.
David Bau: Grow up.
David Bau: What the heck is a mass mean probe?
David Bau: Let me just take the average of all the green guys.
David Bau: That's the meme, right here.
David Bau: Hey, the average of all the red guys. It's the mean right there.
David Bau: Right, and then the interesting vector is the vector that connects those two means, that's all it is.
David Bau: Make sense?
David Bau: And you can see how this would be more guaranteed to be causal.
David Bau: Than the other one, because it's so simple.
David Bau: It's like, what's your advice for all the red people?
David Bau: Well, the advice is, you should be more like the green people.
David Bau: It's simple.
David Bau: And so how do the red people differ from being people with, like, the higher homer grades? That's good, work on that, right? But they also, like.
David Bau: study for more hours, right? So, you should do that also, right? So… so, you know…
David Bau: It's, it's a simpler… it's a simpler thing, and, and it's… It's, more guaranteed.
David Bau: to be connected to causal variables, yes, maybe this is a… more of a data question than a modeling question, but it feels to me like the master approach kind of assumes that you have, like.
David Bau: the same distribution. Yes. Whereas I could imagine, like, maybe you have two types of students that would be a high level. Low effort, low grade, and high effort, still low grade. That's right. And then if you have those two clusters, taking the average of that might…
David Bau: It also might not… That's the same way. That's right. So I'm just curious, you know, would you categorize that as, like, a issue with the approach, with the data? Like, not very counterfactual? The way I put it… the way I think of it is like this, is,
David Bau: You… you have this hope that somewhere in this huge signal that you've got, there's a causal variable.
David Bau: There might not be a causal variable.
David Bau: But you hope that somewhere in there, this one. And the danger of using the wrong kind of probe is that you might throw out that causal variable by accident. Like, linear regression will do that a lot.
David Bau: Right, but if you use the mass mean thing, if there's a causal variable in there, then you'll bring it along. You're bringing along all the variables, because you're just, like.
David Bau: Along the means, and you're not throwing anything out.
David Bau: Right? Now, there's… it's possible that your system's set up so that there's no causal variable in there. Maybe there's not… there's no causality in any of the things we measured, right?
David Bau: And then… and then this is not gonna help that, right? You can't, like, invent causality where there isn't… Maybe it's just, like, that,
David Bau: linear combination of, like, different factors. You're trying to measure a combination. Yeah, no, right, exactly. So there might be something causal somewhere in the system, but by the time you measure it, it's all downstream of causality, and none of your variables, none of the things you have are causal anymore, right? It could be, right? So you often have the situation where you're downstream of causal variables.
David Bau: And, and nothing that you affect actually has a causal effect anymore, right? Does that… does that kind of make sense? You can have these systems as, like.
David Bau: you know, downstream of causal variables. It's like, oh, umbrellas! When people have umbrellas, it rains, right? Oh, that's very downstream of it raining, like, you know, if I give everybody an umbrella, it's not gonna make it rain.
David Bau: Right? How about rain boots and raincoats and all the other stuff? Right, and none of it helps.
David Bau: Right? It's not gonna rain, we're still in a drought.
David Bau: That's not the way to solve it. It's because it's all downstream. And so you might be in a situation where everything that you measure is downstream and it doesn't help.
David Bau: Right, so… but, if you do have something causal in there, then at least it won't throw it out. And then… so there's a second set of experiments we need to run inside a neural network.
David Bau: To test for causality, and the statistical techniques won't…
David Bau: do them on their own. Just gotta change things.
David Bau: Makes sense.
David Bau: But… but if you… if you set all that fancy stuff up, and you say, oh, the intervention I'm going to do is this LR probe direction, then it might still not work, because you've… you actually throw out the variable.
David Bau: Make sense?
David Bau: Or, or maybe even get it.
David Bau: And then people use.
David Bau: So, the other thing that Sam does is… so Sam has this funny thing where
David Bau: what does he say? Sam Mark says,
David Bau: Oh, you know, when you use this mass main thing, people want to use it as a probe. They want to, like, dot productify it and see, like, if something is truthful or not.
David Bau: He says, well, like, if you just use a dot product, it's not a very good probe. You see that the separating plane would cut through a lot of cases. It'd be easy to not, like, get wrong here. And so, what Sam points out is that the main cause for all these cases is that you have some covariance, you have some skew in your data.
David Bau: And the right way to… the way he likes to think about it is that it is a good probe, but first you sort of have to unscrew the data. So he says, measure the covariance of your data.
David Bau: I'll whiten by the programs, and then you can cross that line, and that… that will tilt.
David Bau: It'll tilt all the angles so that… you know.
David Bau: They basically… what a right angle looks like.
David Bau: You know, after it's untilted, will basically look like this one.
David Bau: It'll be slightly different rule, it's a different method. It's… it's a method called linear discriminative analysis.
David Bau: But, but yeah, so that's it. So that's… that's what's hemp.
David Bau: Proposes in his paper, and
David Bau: Okay, so that's… any, any other… is that sort of more than you wanted to know about how… how to find a linear direction? Okay.
David Bau: There's, I mean, there's a lot of other things that people can do.
David Bau: So there were some other questions here, so for, Court… Courtney… oh, I missed your… did I miss the T, Courtney? Courtney asked a question, Yuci asked a question, Claire asked a question, yeah, so some questions here related, yes. Remind me your name, I forget who's from. Courtney here. Courtney, I agree.
David Bau: My question is about how the linear structure emerges, like, across the layers, and they said for some more complicated, like, tasks or structures, the later layers are…
David Bau: better to find the linear separation. Yes. How do you know when is the ideal layer to, like, cut off the learning or pull off the linear structure? Yes. If there's so much variance, is there, like, a systematic way to do this? Yeah, there's a couple systematic ways, and I'll show you. I'll give a little sense for that.
David Bau: And then let me let the other folks ask their questions as well, if that'd be very exciting.
David Bau: So, you chief, UT?
David Bau: I think it's the same question, but I was more focused on, like, the sphere, and we're supposed to be infrastructure, right? Yeah, because… because the transformer's this big grid.
David Bau: With all the layers and all the tokens and so, right? A lot of possible things to prove.
David Bau: Claire had a question.
David Bau: Yes. I was just…
David Bau: I couldn't understand the summarization motif. I read that section, like… Yeah, it's a summarization motif. Yeah, I apologize for this paper. I wanted to give you a little sense for, you know, what people are doing, but it's a little bit of messy paper. I can explain it.
David Bau: Okay.
David Bau: So, but it's related to the other two questions. So, what's going on here is this… so, you remember in Sam Mark's paper that he had this mysterious figure, this one?
David Bau: And then in paper number 2, they had this other map figure, this one.
David Bau: So, both of these are… they're different techniques, but from each other, but they're both what those researchers used to try to answer this question. Which layer should we be
David Bau: probing.
David Bau: Right, and which token… to be probate.
David Bau: And… And so, like, I really like San Marc's approach, what he did. So what he did is.
David Bau: This is… this is a causal analysis of tokens
David Bau: And, and layers. And so what Sam did.
David Bau: Is he took a single sentence.
David Bau: He didn't even do this over the whole data set, he just took a single sentence. He said, Chicago.
David Bau: Chicago is in Canada.
David Bau: is not… actually, I'm sorry, there's another sentence he puts after that. He says, this statement is…
David Bau: And you want to see if the model says true or false.
David Bau: Right? Chicago's in Canada.
David Bau: And the model should say false.
David Bau: Right?
David Bau: And then he runs it again and says, Toronto is in Canada. The statement is…
David Bau: And he'd expect us to say true, right? So these two runs are really similar, and what he wants to do is he wants to look inside the model to see what variables are causal inside. So we're going to have a whole class around
David Bau: This type of causal scanning, Next week?
David Bau: But… but he… but he does some causal analysis.
David Bau: And he, he finds that
David Bau: certain layers and certain tokens are more important than others. All these dark blue ones are the important ones. And he draws boxes around them and discusses them. And he says, well, this is sort of causally important, but not that interesting, because it's, like, the city, Chicago or Toronto. But he says, oh, this is really interesting.
David Bau: At the period, at the end of the sentence.
David Bau: There's some really important representations that are causal that cause the model to, say, flip true or false at the end.
David Bau: And so this was really fascinating, because the period doesn't have any information in it, right? The period is the same between these two sentences, Chicago or Toronto.
David Bau: And yet, the period has something in it that is decisive, in its representation, has something decisive
David Bau: That somehow gets the model to say true or false and flip between them.
David Bau: between the sentences, even though there's nothing different about the period itself, right? And so… so what… what Sam hypothesizes is that something in the representation in the period, somewhere around layer, you know, 15 or something.
David Bau: Is, like, summarizing… The truth of the sentence, right?
David Bau: So he uses this console Hang on one, one example.
David Bau: to, to justify that. And it says, so therefore, for the rest of the paper, I'm gonna just look at the representations that are sitting here, and see what the deal is, and I'll probe them, and I'll…
David Bau: Scatterplot them and everything.
David Bau: So we'll look at this period.
David Bau: And so… so that's… that's what he does. And actually, this is one of the pieces of floor that I'll just share with you.
David Bau: It's like, these periods, these, like, punctuations at the end of sentences.
David Bau: the Transformers, they put a lot of information on them, it's crazy, right? Like…
David Bau: you know, when you… so this other paper is also doing something similar, other punctuation. So, this paper is using a different approach. They're saying, okay, when you get to the end, John feels very, you know, happy or sad or whatever, right? So, like, okay, so you're about to make this prediction.
David Bau: What are the attention heads doing?
David Bau: when they're trying to look back and try to remember what's going on, say, you know, whether John feels happy or sad. So they find the relevant attention pads, and they look at the heat maps, and they say, oh, look!
David Bau: They're, they're paying a lot of attention.
David Bau: to the comma!
David Bau: This comma. What the heck?
David Bau: The comma seems so innocent, right? But actually, but actually, like, there's something important there that the attention heads are looking at. So, these are two different views of the same kind of phenomena, where, like, it looks like a lot of the punctuation is used by the transformer.
David Bau: To deposit, like, summary information about what it understood about the phrase, or what it understood about the sentence, and then later on, when it has to have been pre-reading, instead of going and looking at all individual words, it often looks at the punctuation, right?
David Bau: So, at least that's what we're finding.
David Bau: It's a sort of a story that we've seen a lot.
David Bau: And so, if you're… if you don't yet know how to do all this causal analysis, but you want to try to do some scatter plots and…
David Bau: Do some probes and look around.
David Bau: Oh, you could do worse than looking at a piece of punctuation.
David Bau: Right? You can find it, it might work.
David Bau: it might work. And we can do this logical analysis and everything, but… but, like, I recommend people just, like, look at the period of data sent. So, like, if you… if you're in the power team, and you have sentences that describe somebody
David Bau: who's projecting a lot of power, and you have other sentences where sentences are somebody who's really powerless, and you want to see, does the model know the difference between these? Well, where would you probe? Maybe the period at the end sentence might be a good place to start.
David Bau: Okay, that make sense? And same with all the other… all the other projects.
David Bau: Yes. Slightly different, but when a model hallucinates, do you think there is, like, any pattern that… or has any work been done that tried to… because I think this is really new for me, that the comma might hold power, but, like, in the hallucinatory content, if there is, like, any pattern or…
David Bau: any, like, so, for example, I see a lot of digits keep on getting repeated when the model starts hallucinating, so is there… that's been, like… Yeah, yeah, yeah. So, hallucination is a big deal, so a lot of people have done 50% hallucination.
David Bau: I, I don't think that…
David Bau: I've seen any that relate it to the punctuation, but I'll share an interesting phenomenon. There's another thing that happens is… so there's two really interesting moments when a model is about to emit a token.
David Bau: One is right before it emits the token.
David Bau: And then you can see… Like, in the final bunch of layers.
David Bau: the model's, like, bringing together all sorts of information to make this really difficult decision of whether something is true or false, or whether some sentiment should be predicted a certain way, some arithmetic is figured out a certain way or something, right? So right before the token is predicted, you see all this interesting information converge.
David Bau: On that, right before token?
David Bau: But then there's this other thing that happens, that after the token is emitted, and it's in the stream, and now the model's kind of going back autoregressively and reading, what it just said.
David Bau: There's this other thing that happens, is that it says this, What's going on?
David Bau: And maybe it, like, you have the token.
David Bau: And it'll assess whether it's true or false right after that.
David Bau: And, and so one of the hallucination results
David Bau: Is that even when models make a mistake, and they hallucinate something.
David Bau: After they emit the token, the wrong token they put in the stream.
David Bau: Then immediately, you know, one or two tokens after, the model will look at what it said.
David Bau: And it'll say… That doesn't seem right.
David Bau: What's so bad?
David Bau: Right? Like, they'll know that they made a mistake, which is really interesting.
David Bau: And so, and I think the hallucination paper… so I know that in my lab, the best place to find when a previous hallucination was a mistake is actually at punctuation, but there's been some other papers, so I don't know if other labs agree with that.
David Bau: Let me have mine.
David Bau: Holy crap.
David Bau: So, okay.
David Bau: So, let's see… other things.
David Bau: Oh, other… oh, Courtney also asks what, what, what later, right?
David Bau: And,
David Bau: And there's simple things that you can also do, so there's these fancy causal things that you can do, but you can also just…
David Bau: Measure the margin?
David Bau: Right? Just like a classic machine learning person would do.
David Bau: Right, and so, like, a simple way of measuring the margin is, like, the Cohen's D, Statistic?
David Bau: Which is, you know, you have your two classes, you know, your true things, your false things, your powerful things, your powerless things, whatever.
David Bau: And you have your direction, let's say your mass mean direction, and you just project everything on that direction. So then, like, how well are things separated? Well, you can just ask, like.
David Bau: How… how far are the beans apart from each other?
David Bau: But now, what's the scale that you should use for that? Well…
David Bau: You know, if the neural network happens to be
David Bau: outputting the hundreds out of the neurons, then that would be a big scale, and if the neural network happens to be outputting, you know, 1 tenth out of the
David Bau: the neural network, that would be a small scale, but really what matters is, like, what's the relative scale relative to the general variance of the signals, right? And so what Cohen's key is, is you take the difference in the means, and then you just divide it by the standard deviation of what the data's looking like.
David Bau: And that tells you, on a relative scale, like, how much of the variation
David Bau: You know, is, is, you know, can be…
David Bau: Seen as a separation between these two classes.
David Bau: Right, and so if you do this for a real piece of data, then you'll often see, you know, graphs that go up and go down. This is two graphs. There's another one called
David Bau: That you can Google for, called, The, the Fisher separation.
David Bau: Where, where it's a little bit more vector-oriented. So, there's different ways of, like, estimating the ratio.
David Bau: But they're just, they're simple ratios.
David Bau: Make sense? So it's another thing you can do, and this might suggest, oh, I should look at, you know, I should look at the model at layer 2027.
David Bau: Right, it looks like there's a big separation here, maybe…
David Bau: Maybe they'll tell me what's going on.
David Bau: Okay, so what I want to share with you… so, actually, you can even bring this up. This is actually a nice…
David Bau: I prepared a thing that is kind of a model, But what?
David Bau: I hope you might do for your project, or at least resource this for things that you might do. So there's this GitHub.
David Bau: Paige Puns is also a GitHub
David Bau: Project, you can just look at all the code.
David Bau: So I sort of barcoded it, but it does have a lot of nice examples of code in it. And I… and so I just wanted to take a little interlude.
David Bau: To step you through How you would do this type of analysis in your projects?
David Bau: And some of the things we talked about Last week, but…
David Bau: Because I was really disorganized by the project timing, we didn't really have that much time to talk about it.
David Bau: We're gonna talk about it again here.
David Bau: Thanks.
David Bau: And so…
David Bau: So the steps are like this. So first we, like, make some data sets, and we, like, make contrasted data sets, and then… and then we use it to benchmark models, and then after we benchmark models, we can use NDIF to gather internal activation data, and then we… we analyze it. And so I'm just going to step you through what each of these steps looks like.
David Bau: Right. And so, the example, project I made up for myself, Oh, one person Tajif is… is puns.
David Bau: So my research question is, do you burn, right? Do…
David Bau: Do, do LLMs know how to take a joke?
David Bau: Or at least puns. At least, like, really, like, stupid jokes.
David Bau: Like the puns, right?
David Bau: And so there's a whole bunch of reasons, and I wrote them in this webpage, like why puns might be actually interesting, but mostly I picked it because they're funny.
David Bau: And so, so, my first step was, if I'm gonna probe…
David Bau: upon understanding, I need a joke book that I can use.
David Bau: to test the models, so I can go to them and say, do you think this is funny? Or whatever, right? But they have to be pretty funny jokes, so I didn't have enough. I'm not that creative. I was able to come up with, you know, 5 or 6 puns on my own.
David Bau: And so… so I… so I did what a lot of researchers are doing now. I went to all the…
David Bau: LLMs, and I said, alright, here's some examples of a pun. Can you make me a thousand of these? Right?
David Bau: And they're happy to oblige.
David Bau: They make thousands and thousands of puns.
David Bau: And, and I just, I said, oh, format it as JSON, put it in a file for me, I'll merge them all up, I've got my, you know, my ChatGPT puns, my Cloud puns, my Gemini puns, they're all, like, I've got about a thousand of them in total. But let me tell you.
David Bau: They're really bad. Here's, like, the farmer woke up early every morning because he had to make Hey.
David Bau: It's not very funny.
David Bau: Right, and some of them, like, they would be funnier, but they're, like.
David Bau: bad, like, so here's… the door salesman was the best closer because he always knew how to
David Bau: clothes, but you said clothes already, so you ruined the joke, right? Like, it should say something like.
David Bau: You know?
David Bau: The, the, the door salesman was the…
David Bau: best in the country, because he always knew how to close, right? And then that would be the, like, that would be a joke, but, like.
David Bau: you know, the superhuman AGI systems.
David Bau: I don't get it, right? Okay. But… but you… but I get, like, a thousand, like, low-quality jokes like this, and that, of course, you have to…
David Bau: You know, inspect them by hand.
David Bau: So maybe that's good, maybe they were good jokes, and then I'd be done. I'd have my data set. Great, right? But they were terrible.
David Bau: And there were too many for me to work through, so I did the same thing that we read about what Perez did. So I'm just following the formula of the Perez paper, just like what you guys read last week, right?
David Bau: I said, I need to filter these, and there's too many for me to filter them by hand.
David Bau: So, chat, GPT, Claude, Gemini, let's scramble these up, look at each other's jokes, right? I need you to rate these jokes. Tell me what's funny.
David Bau: Tell me what's not funny.
David Bau: Right? I'll make this easier for you. You have to do two steps. First, Explain the joke!
David Bau: Explain it!
David Bau: And then tell me why it's funny. If you can't explain why it's funny, then give it a rating, so give all these numbers. So you do it, and it says, oh yeah, the pun relies on make… hey, it's a farming activity, it sounds like this, whatever, right? Like 3.3 out of 10, 3. Maybe not that funny.
David Bau: Right? But some of the others, like, you know, they're random 5.
David Bau: And so I did this, and then, like, after I filtered it, it wasn't good enough. I had very few drugs that I survived.
David Bau: the same… but some of the jokes were kind of close, so I also ran what I call the joke repair workshop, right? Where I said, okay, now read the explanation of the joke.
David Bau: And then think about why it's not funny, and see if you can, like, make it funnier. Here's a couple examples of repaired jokes.
David Bau: Right? And I've repaired another couple hundred jokes for me.
David Bau: Right And then re-rated them, and so on. And so after, after a while, I did this.
David Bau: And… and I… and I had to… so, the honest truth is.
David Bau: It, like, it wasn't enough to get a few hundred funny jokes. I got maybe 100 funny jokes out of this, and then there were 100 other jokes that were almost funny.
David Bau: But they couldn't figure out how to repair them, and so I had to go and repair them by hand, right, as a human.
David Bau: And repair the jokes, and…
David Bau: did all the things. Okay, so that's… so… so that was sort of the process I did. So I got a couple hundred jokes out of this, right? Oh, and then now you have to watch out for me, because I've got my 200 puns. Okay. So, so then the next step is, we need to study these jokes.
David Bau: And…
David Bau: models where we can look on the inside enough. Like, we're not going to study ChatGPT, because OpenAI is not going to let us look inside.
David Bau: ChatGP model.
David Bau: So… so I… so I… so I was like, let me just study, like, let me… let me test all the different models and see which one… like, I'm… so the… the real challenge with any of these things is you can't study
David Bau: a phenomenon.
David Bau: in an LLM that's too dumb to actually have the phenomenon.
David Bau: Right? If the LLM can't take a joke at all, then there's no, like, joke awareness to studying it.
David Bau: So the first thing you gotta do is you've got to find the LMs that actually
David Bau: that you can look inside, they're open LLMs, but they actually know the concept that you're interested in. And so… so I just, like, took my 205 jokes.
David Bau: and use the fastest service I could find to just throw them
David Bau: at all these LLMs, what I did is I took the humorous word that was, you know,
David Bau: you know.
David Bau: The baker was poor because he didn't make enough.
David Bau: Leave it blank.
David Bau: And then you'd have to say, don't, right? And, and,
David Bau: And then a lot of the, LMs cannot do that.
David Bau: But some of them can't.
David Bau: And… and so I just sent them all to all the… so this is… this is the smallest LLM,
David Bau: And then the dark red is the biggest one, and you can see, for this batch of jokes, you know, the smallest LLM could only get, like, you know, 10% of them, and the biggest LLM could fill in the joke for 80% of them. And so there's a real difference between the small and the big ones. So these tiers…
David Bau: And then, not only that, but it was an interesting chance for me to rate the jokes themselves.
David Bau: And so a lot of the jokes were so laugh-out-loud funny that even the dumb model would basically get it.
David Bau: Right?
David Bau: That make sense?
David Bau: But then, some of the jokes
David Bau: Take off the funny word. They read like straight sentences.
David Bau: And the AI doesn't get that there's a joke going on, and just puts in a regular word.
David Bau: you know, the banker was… Demoralized.
David Bau: Because he had lost his… Job.
David Bau: Right, they'll say. But if you're in a joking mood, you might say the banker was demoralized because he lost his interest.
David Bau: Right? Get it? Interesting.
David Bau: Patrick's… Like, I should have been doing that.
David Bau: Okay? Get it?
David Bau: So, but that's… see, that's what happens with all the models. They're like, I don't get it. I don't get it. Job! He would lose his job. Why would he care about losing his interest?
David Bau: That's like a marketing job's more serious.
David Bau: Right?
David Bau: And so… so that's where… so there's… so there were, like, ADA prompts that were like this. So I… so I probed all the models, but it was also an opportunity for me to probe the dataset, which was interesting, for a couple other reasons, right? Does that make sense? Okay.
David Bau: So then…
David Bau: Okay, so then the nice thing about doing this and asking the dev models to complete it is that I actually got these nice long lists of straight versus funny answers, so, you know, what is this? The tailor won his court case because he had an excellent
David Bau: Suit. Yeah, an excellent suit.
David Bau: But if you ask, like, the small models to answer this, they're, like, an excellent lawyer?
David Bau: Right? Right? He had excellent defense, he had a good reputation, he had a good excuse, he had an alibi, right? You know, what's it like? You know, he had all these other things.
David Bau: So they don't get the joke, so… but, like, the nice thing about doing this is you get all the other words that are not joke words, and you can have the big LM… you can go filter them, you can have the big LM check. Yes, these are not joke words, and then this is… these are… there's some other words that are the joke words, and we can tell the difference, right? So… so this is, like, a nice…
David Bau: opportunity to do this, so I added this to the dataset.
David Bau: And then, after that.
David Bau: I was like, okay, so there's a problem with this. So the problem with this is that
David Bau: I actually don't know, after doing this experiment, whether this is Lava 405B, the giant model, right? I don't know if Lava 405B
David Bau: Actually knows about humor.
David Bau: All I know from this is as read more text.
David Bau: And maybe it's seen this joke before.
David Bau: I don't know if it even knows that it's funny.
David Bau: Does that make sense?
David Bau: So… so what I did is I said, okay, but you know what? These ones are interesting.
David Bau: These ones are interesting, because these are, like, hidden jokes that none of the models, like.
David Bau: You know, can see on the surface of their jokes.
David Bau: And so I said, let's do this. Let's create sentences like this.
David Bau: So let's say this is, like, a hidden joke. The dangerous iPhone was arrested and charged with…
David Bau: I don't know, theft, I don't know, what would you charge a dangerous iPhone with?
David Bau: What would you charge a dangerous iPhone with? USB.
David Bau: Battery! Battery!
David Bau: somebody's finger. You charge it with battery. So now,
David Bau: So, if you were in this context, the diner went to the paint store to get thinner.
David Bau: the two antennas met on the roof, and the wedding had excellent reception, right? And then, after you saw that, you were like, okay, I get it now, the dangerous iPhone was arrested and charged with, and you'd be like, battery, that's it, battery, right? But…
David Bau: If you had a different sentence, and you said the diary went to the paint store to get
David Bau: Paint colors.
David Bau: Right?
David Bau: The two antennas met on a roof.
David Bau: And the wedding had excellent Hors d'oeuvres?
David Bau: Right?
David Bau: And the iPhone was dangerous and was arrested and charged.
David Bau: for… You might be like, I don't know, jaywalking? Theft?
David Bau: Violation of privacy? Making too much noise? Disturbing the peace? I don't know, what can an iPhone do?
David Bau: Right? So… so that… so… so you might not get it, right?
David Bau: And so, So, what I did is I paired
David Bau: So, whenever you do experiments, it's useful to have contrasts.
David Bau: So, I made these paired datasets, where you had one.
David Bau: That, you know, sentence like this, which everything was straight.
David Bau: And one sentence where the jokes are… come before.
David Bau: And then I looked at the last thing, and I said, what does the model Predict.
David Bau: for the last word, the joke word or not, I have these nice data sets of, like, joke words and not joke words, right?
David Bau: And then here you go. So, like, this stupid little model doesn't know anything, can never tell the joke.
David Bau: Right?
David Bau: But then, when you get to the big models, like, you know, the 4 or 5B,
David Bau: You know, when it's a straight context.
David Bau: You know, and sometimes it gets the idea of the joke anyway, even though there's no jokes being told. It'll tell the joke anyway, so at some rate, right, like, you know, 15% of the time or something.
David Bau: Great.
David Bau: But then, when it's a joke context, then it, like, reliably gives a joke, right? So this gap.
David Bau: between its behavior and the joke context and the street context. It's like, it's stronger evidence.
David Bau: that actually knows what funding is. So with this in hand, I was like, okay.
David Bau: We can proceed. We can, we can do, we can do some more experiments, is, like, promising.
David Bau: And so… so that basically brought me to, the things that I'm talking about in the class today. Let's… let's actually collect together our 100 jokes.
David Bau: And look at some representations. And we're gonna do more of this next week. But what I'd like you to do for your project, for homework.
David Bau: Is to do some of this for your own data in the next couple days.
David Bau: So what I did is I went this, I took this joke, the diner went to the paint store to get thinner, and 205 jokes just like it, and I said, what is a representation at the period?
David Bau: And then I analyzed all the representations in the query, I did the mass mean thing, and I got this little graph to say that the margin is biggest at layer 27 or something.
David Bau: And I did a PCA visualization of pink funny jokes and blue serious ones, and I can kind of see that, oh yeah, there's kind of a separation.
David Bau: Between the funny things and the serious ones.
David Bau: And then there's some details, like, you might want to figure out the mass-mean vector with half of your data set.
David Bau: And then measure the separation with the other half, so that you're not overfitting. Right, so there's, like, classic machine learning things you still want to pay attention to here, so you might want to do a holdout set. Is this the, difference or separation between the last token? Yes.
David Bau: So, basically, what I would do is I would take two versions of the sentence.
David Bau: And look at the representations at the period.
David Bau: Couldn't that just be because, like, the answer is different, or how is that, like… Oh, yeah, there's after the answer. So, after the answer, the answer is definitely different. So, he says, the diner went to the paint store to get thinner, and then the diner went to the paint store to get paint. And… and so,
David Bau: So yeah, so what we could be looking at is we could be looking at… there's all these funny words, like thinner, and there's all these serious words like pink.
David Bau: And… and there's maybe… there's just a gap between just those words. So,
David Bau: So, you know, that might be the case. I guess my gut feeling is if you actually looked at the words in isolation without the joke, they seem just like other words, and it doesn't seem like anything special about them. But you might be right, there could be a compounder here. There could be other reasons you get the separation.
David Bau: And so we're gonna have to do more to prove to ourselves that this is actually humor awareness. But the fact that the model can kind of separate them is a sign, it's a signal, that maybe it knows that it's funny.
David Bau: Maybe it knows it's fine. At least, if it…
David Bau: Didn't know it was funny, it probably wouldn't have a separation like this.
David Bau: Right? That makes sense. We're still not sure. I might know something else that's correlated with funny, yes. Would you take a look at the DOM model?
David Bau: Oh, yes, see? Yes, I didn't do that, but you should. You can take a look at the dumb model and see that it doesn't separate. Things like that.
David Bau: Oh yeah, I have this nice demo deck, too, I'll show you. So, if you use my code.
David Bau: You can see what else I vibe-coded, I'm so happy with this. Yeah. Let's go with this.
David Bau: Oh, that's localhost. Local host? Oh. Oh!
David Bau: I don't know.
David Bau: I don't know if you'll see this.
David Bau: demo life.
David Bau: of the host.
David Bau: Yes.
David Bau: Oh.
David Bau: Yeah, I want to show you this demo, it's so full, but then I also don't want to slow down the class. Okay, how much, how much time do I have left?
David Bau: Am I over?
David Bau: I have 20 minutes? Oh, I can show you the demo.
David Bau: Yeah, although I do want to explain this other paper to you.
David Bau: But… but you get the idea. So, let me, but in the get-go, Should I put this here?
David Bau: You skip verification, Or is this thing… oh, I don't know.
David Bau: If I, if I share it on Zoom.
David Bau: so many things that you can do here. We have too much technology, there's too many options.
David Bau: Hey, so… on Zoom. Am I on Zoom? I'm not on Zoom. Okay, forget it. I won't show you the notebook. It's very cool, though. So, if you… if you actually try the notebook, and you get it to run, actually, it might… Yeah. It might take some doing a state run, because there's… there's currently a bug in NDIF that I'm having the team debug. But, you know, you can get a nice 3D…
David Bau: View this and look around and get a sense for, like, the scatter plot.
David Bau: You know, has his… has his points of the emissions.
David Bau: Okay, so you can, you can do that kind of. But this is just a 2D projection, right?
David Bau: Okay.
David Bau: So…
David Bau: So, so, so, let me… so, so that's… so, so up to step 5 is, like, what I would encourage you guys to try to do for your projects.
David Bau: You know, this week.
David Bau: Let's talk about a couple other things about vector representation. So, so let's… we had questions from Claire, Rice, and Ayush, and… oh, and Avery related to this paper. So, what was… what was Claire's question?
David Bau: I think I talked about birthpec, but I talked about, like, is there a relation between, like, the attention
David Bau: In the, like, intention embedding of sentences, like, if we take the joint embedding of a sentence, can we relate two sentences together in that way? Right, right, right, right. So maybe I, yeah, maybe I misunderstood your question, but I'll relate it to this anyway.
David Bau: Which I think that there's… there is probably a geometry there. I… I don't know if the, if there's been any good papers looking at the geometry of sentence embeddings, that would be an interesting research topic, but there's been a lot of papers on, like, geometry of word embeddings.
David Bau: And, and I'd expect sentence and vetted to have some of the same properties.
David Bau: Let's… how about Rice and Ayush?
David Bau: See red rights here? See, right? Yes.
David Bau: I think I was just asking if we could…
David Bau: Since, from the results, it seems like the bacteria is, like, cleaner.
David Bau: like, in that subsurface where you… You mean in Sheridan's paper? Yeah, yeah. Yeah, yeah, I'll describe Sheridan. That's great, yeah, so that's what that was about. Okay, cool. Yeah, and the answer is, yeah, I think so. It's… I think it's… it's cleaner. I don't know if it's more effective for patching. It's probably an open research question. You can ask Sheridan. Sheridan's back.
David Bau: But it's a good question. And then Ayush… I'll talk… I'll talk about what all these things are in a minute, and then… Ayush gonna agree?
David Bau: I agree. It's more about, like, the concept and token. It's… Oh, yeah!
David Bau: I'll go over that. Okay, yeah. Okay, I'll go over that. Yes, it's about… you want to hear about induction hens. Okay, so let me just describe this. So, the history for this piece of work is… there's a very famous paper
David Bau: by Mikoloff in 2013 called Word2nd.
David Bau: Where they train, like.
David Bau: like, a zero-layer neural network. It's, like, the simplest model, on a lot of text.
David Bau: And then, get vector embeddings for all sorts of words, and then makes a big deal about the fact that, the words as vectors have this interesting geometry.
David Bau: Like, if you take the vector for man.
David Bau: And the vector for women, it makes this vector a difference between them.
David Bau: And then if you take the vector for king.
David Bau: And you add that vector difference that you had to that, then you get king plus woman minus man.
David Bau: And you look around, you say, what word had that back there? And what do you think you find?
David Bau: Yeah!
David Bau: He's like, you get queen! That's pretty neat!
David Bau: And,
David Bau: And so, so he's like, well, he made, he, he, like, you can make all these parallelograms between man, woman, king, queen. That's not exact, right? You know, the queen's a little bit off, but it's kind of close.
David Bau: Right? And so, so is increments. Neat.
David Bau: Right? So this is, like, this is, semantic vector arithmetic.
David Bau: And so, after Rick Law found this.
David Bau: Every natural language professor in the world.
David Bau: Todd, everybody, this is what happens.
David Bau: With vector embeddings?
David Bau: Except for this caveat here, which is, weirdly, if you look at state-of-the-art networks and embedders and rail networks.
David Bau: They don't really do this very often.
David Bau: Great.
David Bau: Never they do this. Like, you get… I mean, Queen might be in, like, that neighborhood, but it's, like, really far, and you might be closer to some other encoding, like Idaho or something.
David Bau: You know?
David Bau: And so, okay, so I'm… so the paper… the paper eventually gets to talking about what's going on here. But to… to lead up to it, I want to just describe Sheridan's work, which I think is really cool, and it goes through induction pads.
David Bau: So, so taking an interpretability class, I need to teach you what an induction head is.
David Bau: It's not optional. So, if you guys need to learn this.
David Bau: Okay, so what an induction head is, it's about
David Bau: So, like, if you have a sentence that's repeated in a piece of text, like, I hear the cardinals.
David Bau: I hear the… Right, and you put that into your autopgressive model, The… the normal bigram…
David Bau: statistics, for I hear the…
David Bau: would probably say, I hear the car, I hear the explosion, I hear the music, or something like that after it.
David Bau: It wouldn't say, I hear the card.
David Bau: Not something that you would hear.
David Bau: But in this context, That is the most likely answer.
David Bau: It will say, I hear the card noise.
David Bau: And why does it say that?
David Bau: It says that because… It was just Seth.
David Bau: And it knows that English text, whatever, like, real, you know, human text, often repeats itself.
David Bau: And so, if it's getting ready to repeat itself again, that's, like, really likely. So this repeating thing is a thing that all the language models learn how to do.
David Bau: And it turns out that if you look inside to see how do they compute whether they're going to do a repeat or not, you find these interesting attachments inside the model that do this. They're called induction hits.
David Bau: The way the induction heads work.
David Bau: And, you know, so here's, like, the attention heads that do this. There's, like, you know, half a dozen induction heads in this bottle. You can see they sort of…
David Bau: Scatterplot blue around the architecture of the model.
David Bau: And, and the way they do this is…
David Bau: When it gets to any word, it more or less asks.
David Bau: When is the last time, you know, I hear the…
David Bau: When is the last time that was uttered?
David Bau: And they would find that this is the last time that it was uttered, but it's not… it's not actually what they're asking. They're asking.
David Bau: What word came after the last time it was cited?
David Bau: And so, the attention heads will be like, oh, this word came right after I hear that.
David Bau: So basically what happens is, every word is tagged with its context.
David Bau: And every time you have a context, it goes and searches
David Bau: Was, was any word tagged as coming right after this context before?
David Bau: And here, in this case, it'll find it. It'll find the card was there.
David Bau: And then the attention head says, oh, well, in that case, we should copy it. And the attention head goes and looks at, you know, the representation of card.
David Bau: It picks up the fact that it's the word card, and then outputs this… outputs reduction.
David Bau: Make sense? So that's what an induction head work is, and they're all over the place. And induction heads is why models tend to repeat themselves a lot.
David Bau: It's because they have these mechanisms inside them.
David Bau: And so… oh yeah, this is from the original introduction head paper, I just wanted to copy something out of the original paper, so this is the original work.
David Bau: That, that analyzed from this, their diagrams found that.
David Bau: Okay.
David Bau: So, so, now, there's this other interesting thing that happens. Oh, I should have put in another thing, which is…
David Bau: If you actually look in detail at the induction heads, and Sheridan did, Sheridan found that there's a bunch of tokens that, when you copy text, that don't seem to be copied by the induction heads.
David Bau: And so, so Sharon's like, where, like, how does the model figure out how to copy these things?
David Bau: And so, so what Sheridan found out was you had this other set of heads.
David Bau: That work like this. That basically, you know, if they look back.
David Bau: And instead of attending to card, they attend to, like, innels, like a token ahead.
David Bau: And then, instead of changing the prediction of the next word, they indirectly change the prediction of two ad words.
David Bau: So there's these weird, like, two ahead.
David Bau: induction pads that kind of, like, skip a token. They kind of look like one x-ray, one ahead.
David Bau: Which is really bizarre. I said, why is it doing that?
David Bau: Right?
David Bau: That makes sense? So Sharon was gonna, like, write a paper about the two-head things, but then, was characterizing all the other things that were in this, these two-head induction heads, and let me show you some weird things that happened with two-way head induction.
David Bau: So, it turns out that if you have, like, a natural language translation piece of text, like translating Japanese boat to Chinese boat.
David Bau: Right?
David Bau: And you look at what the two head induction heads are doing, They are actually necessary.
David Bau: We're doing this natural language translation, if you cut the heads.
David Bau: Then it can't do this anymore.
David Bau: And then, moreover…
David Bau: if you take the same thing going on in, like, you know, Spanish or Times thing, which…
David Bau: Translating the word cloud nouve, right, to Nubola.
David Bau: where these attention pads are still, like, important. If you cut them, and you cut these attention pads, and you cut these attention pads.
David Bau: And you reroute the data that's transported these detection heads from this sentence to that sentence.
David Bau: Right? And then you let the model run.
David Bau: And instead of saying Nuvola, It says platinum.
David Bau: Is that bizarre?
David Bau: And so this is just, like, the two token-ahead induction heads.
David Bau: And that, that led Sheridan to believe that, oh, what the two token-ahead induction heads.
David Bau: Our transporting is not the second token ahead.
David Bau: It's translate… it's transporting some language-independent notion of the concept.
David Bau: Strange, isn't it?
David Bau: Very strange, different kind of introductions.
David Bau: They, like, transport the whole concept, the whole word, as opposed to this, right? Have you ever heard of this paper? That's weird.
David Bau: Anyway, so we'll talk more about this, sort of.
David Bau: Automation analysis stuff. It's transmitting things that require two things you have to hold in your head, like…
David Bau: Boat, plus… Spanish, or, card plus…
David Bau: Yeah, so I think that… so, there's a few other experiments that we did, which…
David Bau: Where you see these induction heads.
David Bau: implicated in, like, paraphrasing and other things where there's not apparent, going on. But, but
David Bau: You know, you might be right, there might be some alternate hypotheses.
David Bau: But right now, the… I like the concept of this. Yeah, but right now, the, the, you know, the, you know, Sheridan's main hypothesis is that
David Bau: These are, like, the language-independent concepts.
David Bau: It was pretty interesting.
David Bau: Okay, so let me… let me skip through these things. Yeah, so this is, like, paraphrasing a…
David Bau: You know, there's some interesting paraphrasing here in here. So, this is a follow-up paper to this, which is also related to your question.
David Bau: And… and several other mentions. So this… so what you guys read was a follow-up paper to this work.
David Bau: Where, so… so the chairman has these concept induction heads, which supposedly, according to Sheridan,
David Bau: Area Language Independent Concept.
David Bau: And so Sharon's like, you know, the Mikulov, parallelogram.
David Bau: Erythrotect doesn't work all that well.
David Bau: But I wonder… If the vector representation read by the constant induction heads Obeys parallelogram better.
David Bau: That makes sense? So, remember how I told you what Sheridan was doing was a linear readout?
David Bau: from their computations. So… and I said, you know, neural networks, they do linear readout in themselves, but what if Sharon did?
David Bau: Let's take a look at the concept induction heads. You know, there's a few dozen of these concept induction heads that, like, seem to pick out the levaging concept.
David Bau: They all do linear readout.
David Bau: of the repetitions. And if you take the sum of them, just take the sum of their linear readouts, that's just a met… that's a matrix, it's just a matter of linear readout.
David Bau: Right? So if we use that matrix to do linear readout of the concepts, and we get this other vector out.
David Bau: I don't really know what that vector means, but… Wonder how it does.
David Bau: And this parallelogram arithmetic.
David Bau: Right. And so… so here's basically the fonts.
David Bau: That basically, if you… if you use the concept
David Bau: Heads, and you try to do all these country capital things.
David Bau: Then, you know, it's basically 80% accurate, and doing the parallelograms, whereas if you use the regular
David Bau: full representation, it was down at 40% or something like that. So this is the fly, right, which is like, oh, it can only kind of do it 40%. But if you clean it up by putting it through the readout that the constant production heads are doing, it goes up to, like, 80%.
David Bau: Does that make sense?
David Bau: And so… but then there was a question about… Why is it… That, some of the time.
David Bau: If you, like, do it for past heads or something, there's two types of heads that you could do a readout for. There's also the regular token induction heads.
David Bau: That are about copying tokens.
David Bau: And when you do the readout that the token induction heads are doing, you also can increase the accuracy
David Bau: Of, paralleled arithmetic.
David Bau: But for different tasks. Like, for changing something to past tense.
David Bau: Then, the token readout is a lot.
David Bau: Stronger at doing teleograms.
David Bau: And the cocktail block.
David Bau: Why would that be?
David Bau: Did anybody get, like, understand, like, why that would be?
David Bau: Like, why is it that when you do readouts, okay, so we have… we're about out of time, so this is my last question for you guys.
David Bau: Why would it be that the internal concept induction heads
David Bau: We'll be good at country capitals.
David Bau: And reasoning geometrically about them.
David Bau: Whereas… When it comes to the past hence, the token induction heads.
David Bau: Would be the ones that were better.
David Bau: I would say I,
David Bau: concrete place versus a, concept or a, grammar that is more abstract? Yeah, one of them is really abstract in grammar, so why would the literal token copying
David Bau: And to be better at the abstract.
David Bau: And whereas the concrete, like, places in the world is better at the abstract concept head. Like, why would that… it's like, it seems like…
David Bau: Almost…
David Bau: reversed, maybe? Or not? Maybe… why would it not be reversed? Yeah, I'm trying to remember from the paper, but it had to do with, like, the…
David Bau: the, like, literal structure of the word, like, going from ED to IMD, is something that is different, that's much more concrete than going from, like, one place to another place. That's right. So some things are concrete in the world.
David Bau: Like… What city is the place?
David Bau: In what country?
David Bau: And some things are concrete in text.
David Bau: But whether you add ED to the end of the word.
David Bau: Right?
David Bau: And so the things that are concrete in text, those manipulations, they follow this nice parallelogram geometry when you read them out using token induction heads, the things that concretely copy text.
David Bau: And then… but when you read out the same tokens using concept induction heads, the things that
David Bau: You know, try to read out conceptually what's going on in a language-independent way.
David Bau: It turns out that the parallel the Gram2GA reflect.
David Bau: The structure of the real world, concepts in the real world, what space, a country, and that kind of thing.
David Bau: And so, you know, so to me, this was super amazing.
David Bau: I told… I told Sheridan, You should…
David Bau: write this in a high-profile journal in Nature or something like that, and go win your Turing Award. I thought this was amazing.
David Bau: And Sherry's like, I don't know, I'm not that proud of this work.
David Bau: Superintendo Workshop.
David Bau: Maybe it's time to go listen to it.
David Bau: So here I am, teaching it to all you.
David Bau: I think it's really cool that the models have these cool tracks, they can pick them in two different ways, and it gives you a little insight or a little flavor for what's going on.
David Bau: And so, so if there's other questions that you guys had, I apologize to the students for not having to follow the question, but for all of our members, try to gather a contrasting data set, find a model that understands the context.
David Bau: Thanks a lot, guys. Have a great day.
David Bau: Alright, excellent.