David Bau: According to this claim.
David Bau: Apologies to folks that the, the room video thing is not working.
David Bau: And so, but I will… I'll put this here.
David Bau: And then… is this on the screen? It's not on the screen.
David Bau: Let's see here… Okay, and then…
David Bau: Let me open up this from last time, class… Like, two steering… okay, cool.
David Bau: So, we were almost at the end.
David Bau: We were going through this exercise, and I think this is worthwhile enough that I'll just…
David Bau: Give you these, these puzzles to warm up.
David Bau: So we're just continuing on where we left off.
David Bau: I had to apologize for the next professor for making his class late.
David Bau: Last time. So, but, but we were going through the weirdness of high dimensions.
David Bau: Right, so when you have, this idea that a concept is a vector, And the vector is…
David Bau: One dimension for every neuron?
David Bau: In your system?
David Bau: you know, in your representation, in your neural layer, or your whole neural network, then there's tend to be a lot of neurons in modern neural networks. I think that the price for entry now
David Bau: You're not really talking, like, a real neural network unless you have, like, a thousand…
David Bau: neurons, or something like that, right? So, like, thousand-dimensional.
David Bau: It's, like, kind of as small as we get for neural networks that are doing interesting things these days.
David Bau: And so… but a thousand-dimensional geometry is a little different from 3D geometry, so I was asking you to estimate.
David Bau: If you make a… a 3D sphere, 1% smaller.
David Bau: Then its volume, its radius obviously goes to 99%.
David Bau: Right? Its volume, somebody said, goes to 97%.
David Bau: Right?
David Bau: it shrinks a little, because it's 3D, right? So it shrinks a little faster. But what if you're in a thousand dimensions, and you shrink
David Bau: the ball by 1%, then how much does the volume shrink by?
David Bau: Well, there's the answer there.
David Bau: It shrinks by… All the way down to .0004, you know, 004%.
David Bau: Really, it's like, It, like, all the volume goes away.
David Bau: It's crazy, right? Not intuitive.
David Bau: So, even worse than that.
David Bau: So, you'll sometimes have different ways of sampling numbers. You can sample numbers on a sphere, you can sample numbers in a cube. You know, if you've taken a probability class, they'll say, oh, the uniform distribution, right, which is sort of the equivalent of a cube, you know, just uniformly between 0 and 1, or the Gaussian distribution.
David Bau: Which, if you generalize that.
David Bau: You know, it's a bell curve, but generally it turns into kind of a sphere.
David Bau: So that, yeah, what's the difference between uniform and Gaussian? What's the difference between a cube and a sphere? It's like, I don't know.
David Bau: It's like, the math often works either way, you could do either way. But what's the difference here? So, how much in 3D, how this sphere inside this cube is smaller?
David Bau: than the cube. Like, how… what's the volume of the sphere inside a cube? It's… it's smaller, but by about half.
David Bau: Right? 52%.
David Bau: So if you're in a thousand dimensions, you make a thousand-dimensional cube, Right? What's its area?
David Bau: Well, it's like 1 or, you know, 2 to 1,000, depending on if you include all the octants, but it's like, you know the volume of the cube?
David Bau: So, but in a thousand dimensions, if you inscribe the sphere.
David Bau: then what's the volume of the sphere relative to the cube? Like, it's smaller, what's the fraction?
David Bau: Of the cube that you think is Filled by this sphere.
David Bau: Any guesses?
David Bau: 4 divided by 2, minus 1.
David Bau: 4 thirds minus 1, so it's like something like… or like a third? It's like, yeah, it's like maybe it's… it'll be, like, a third, so instead of a half, maybe it, it, it occupies a third.
David Bau: Any other guesses?
David Bau: Or maybe this number up here.004% or something like that?
David Bau: What do you think?
David Bau: How small do you think this sphere gets? You think it's closer to, like, a third, or do you think it's, like.004%?
David Bau: As a fraction of the cube. Oh, okay. All right. Somebody make a guess.
David Bau: Just, I doesn't… you're gonna be wrong anyway, because I have a lot of, you know, digits here for the correct answer, so there's no way you're gonna be right. Yeah, what do you think?
David Bau: 1 times 10 to the minus 32.
David Bau: That's very small.
David Bau: Should I show you what the answer is?
David Bau: Okay, so about 10 to the minus 32, that's, like, if, like, if, like, one of them is, like, this big, then this is, like, smaller than the mass of, like, an atom or something like that, kind of thing, right?
David Bau: And so, okay, let's take a look.
David Bau: You were off by… A lot!
David Bau: But you had the right idea.
David Bau: It's 10 to the minus 2,700.
David Bau: Isn't that weird?
David Bau: Right? So…
David Bau: Yeah, you don't think of these kind of fractions in regular 3D. In regular 3D, right, you know.
David Bau: That is really, like, it's much, like, how many atoms do you have in, like, a…
David Bau: A cube of sugar, something like that, sugar cube, right? Yeah, 10 to the 24, something like that. Remember that number from chemistry?
David Bau: Right? This is a lot smaller than 10 to the minus 24, this is a lot lighter than one atom, right? It's, like, way, way smaller. And we haven't really shrunk everything. We've just, like.
David Bau: Gone to the inscribe.
David Bau: sphere.
David Bau: So you have to be careful when you're working in large dimensions about your intuition.
David Bau: Like, a hypercube, if you happen to have a hypercube, It's…
David Bau: all corner. Like, you think of the hypercube as evenly split between, like, its interior, And it gets corners.
David Bau: Right?
David Bau: In a thousand dimension, there is no interior. The amount of interior there is is, like, 10 to the minus 2700. Okay, there is some. It's not zero, there's some interior, but it's, like, smaller than the amount of, like, one atom inside a whole sugar cube. It's, like, there's basically no interior.
David Bau: Right?
David Bau: It's all corner. It's corners everywhere. How many corners do you have? Like, 2 to the thousand corners, right? You've got, like, this vast number of corners, and the area, the volume of each corner is, like, dominates.
David Bau: That make sense?
David Bau: Right, okay.
David Bau: So, there's also, like, we use a normal distribution sometimes.
David Bau: And so, so there's this interesting thing. So, like, if you take a normal… who knows what the Gaussian normal distribution is, basically, right? Yeah, you've had to… you've had to deal with this, right? And it's a bell curve, and if you generalize the bell curve to three dimensions.
David Bau: then you get this three-dimensional vector. It's a bell curve in X, a bell curve in Y, a bell curve in Z, and the nice thing, if you do it by X, Y, and Z, it turns out to be this spherical
David Bau: you know, this spherically symmetric thing, it's this beautiful spherical object, so you can get these nice random vectors with rotationally independent probability distribution by taking these 3D bell curves.
David Bau: But the thing is, if you take a bell curve, it really looks like this in 3D, right? But if you look at the magnitude of a random vector that you draw from a bell curve, then even though the most likely the highest density is the vector in the middle, the zero vector.
David Bau: If you look at the magnitude.
David Bau: The most likely magnitude is not zero.
David Bau: Right? The most likely magnitude, because if you sort of integrate around all the magnitudes, then the most likely magnitude… it's very unlikely that you have actually a zero magnitude vector. Like, if you take a random draw from a vector of bell curve, like, it's usually around one point…
David Bau: 1.16 or something like that. Like, it's like… it's gonna be kind of… it has some magnitude.
David Bau: And… and so this curve here I've drawn here.
David Bau: is… is not the bell curve, it's the… it's what happens if you integrate the bell curve by the magnitude. You just… you sort of go around radially.
David Bau: and you integrate it, and so the… the average magnitude of something in the Gaussian is… is here, is, oh, 1.7, 1.6 something, right? That's… that's… that's, like, the average. That make sense?
David Bau: In 3D.
David Bau: Anybody taking a physics class heard of the Maxwell-Boltzmann distribution?
David Bau: Or chemistry? Now, this is what's called the Maxwell-Boltzmann distribution. It's how they estimate, like, how many particles in a Gaussian-distributed gas have enough energy to overcome
David Bau: you know, activation, barrier, or whatever, because they've used this in the past, for real life, and living in 3D, that's what it's about. Make sense?
David Bau: Okay.
David Bau: So, the question is, in a thousand dimensions.
David Bau: you have the same effect, right? Where if you, if you… if you take a random draw on a thousand mentions, it's… it's not…
David Bau: It's unlikely it's going to be zero.
David Bau: Like, what's the average size? So what do you think the average size of a vector is in a thousand-dimensional Gaussian?
David Bau: Any guesses?
David Bau: Hmm, how would you figure this out?
David Bau: 1,000 dimensional Gaussian, you've got a thousand numbers, they're randomly plus or minus one, kind of thing, right?
David Bau: And then what's the average, like, length of that vector?
David Bau: Yes.
David Bau: Yeah! Where you got the 30 from?
David Bau: So 1.7 looks like square root of 3. Yeah, square root of 3.
David Bau: Yeah, just, okay, good yes, square root of something.
David Bau: Yeah, that's right. And the square roots show up in statistics. Yeah, so if you, if you take, like.
David Bau: Random numbers, plus or minus 1.
David Bau: And then you take the magnitude of them, then you have to square them and add them all up. If there's a thousand of them, you're just adding up numbers to a thousand, so then that's, like, squaring all the numbers gives you about a thousand.
David Bau: But then that's not the size of the vector. The size of the vector is not A squared plus B squared plus C squared. It's a square root of A squared plus B squared, so you have to take the square root of C, the square root of 1000, it's around 30-something.
David Bau: Right on, 30. Okay, great. And… and… but now, the interesting thing, though, is it's not just exactly 1.7 or exactly 30. There's some… there's some, like.
David Bau: it's gonna be random around it, so what's the variance, right? What's the standard deviation around this? So, in 3D, the standard deviation is around 40%, so it's like 1.7 plus or minus 40%, right, is how big these vectors are when you draw them for random Gaussian.
David Bau: So, what do you think the variance is when you draw from a thousand-dimensional Gaussian?
David Bau: I'll give you a hint, that's the picture, that's what it looks like over there.
David Bau: See that picture?
David Bau: See, that picture goes up to 31.6, and the variance is around 3%.
David Bau: Right? So it's, like, it's around… it's around 30, and it's sort of only plus or minus, like, 1.
David Bau: It's, like, in this really, really narrow range. So if you… if you, like, draw a random vector from a Gaussian, and the magnitude is not around 31, it's not 30 to 32, or something like that, then you'd be like, what the heck? How did I draw this thing from this random Gaussian, and the magnitude is, like.
David Bau: 10 in a thousand dimensions. That basically never happens. That's, like, one of those 10 to the minus 2700 kind of, like, events. It's like, it's like, oh, I'm in, like, the middle of the sphere when I should be not, right? It's like, it's just… it doesn't… it, like, doesn't happen. There's no volume in there.
David Bau: Does that make sense?
David Bau: That weird?
David Bau: It's weird. It's very weird. So they call this the soap bubble phenomenon. When you, like, draw a random Gaussian in high dimensions, that distribution actually
David Bau: The best intuition is, like, a very thin shell, it's like this soap bubble.
David Bau: So… so high dimensions are very weird. The geometry in high dimensions is very weird, so a lot of… there's a lot of counterintuitive stuff. So all the stuff that we were talking about last time, about you can fit a lot of stuff into the vector space, a lot of these phenomena are magnified in high dimensions, and so when… when people have these intuitions, like, you can only…
David Bau: think about, if you had a thousand neurons, you can really only think about 1,000 independent things, but maybe you can relax that by a little bit. Like, what if you said, oh, you know, we're willing to withstand, like, 1% error, we're going to relax it, then it might help by a little bit, and your intuition might be, oh, if you accept 1% error, it helps by, like, 1%.
David Bau: But actually, for a different phenomenon, if you, like, accept 1% error, it might help exponentially. It might, like, it might, like, increase the space that you have to work with by thousands and thousands, right? And so, this is… this is one of the things that, you know.
David Bau: it leads to counterintuitive things. It's just the facts of high-dimensional geometry.
David Bau: And and we're still working through this as a field, trying to build these intuitions. Okay, so the other thing I wanted to talk about is binding. There's this why binding is hard. So I mentioned this thing before, the idea that you can think about a horse
David Bau: And you might have a horse vector. Oh, there's a horse in this picture.
David Bau: See this horse? Look at that horse.
David Bau: See that horse? And the horse is good friends with that cat, so you might have another concept of a cat.
David Bau: And then when you see this picture, both of your… both of those, you know, vectors, like, light up in your head. You can think about a horse and a cat.
David Bau: at the same time, by just, like, maybe by kind of adding your representation vectors, right? That's kind of the model people have had for, you know, how a concept vector might work.
David Bau: But in real life, you have these other concepts that you have to deal with, and you've got these binding things, and this is one of the classic mysteries of how neural networks work. How do they represent binding? Because in this picture, the interesting thing is not that I just have a horse and a cat. This horse is wearing a fedora.
David Bau: Right?
David Bau: And the cat is wearing, like, a nice knit cap.
David Bau: We like these knit caps.
David Bau: And, and so, so…
David Bau: you know, somehow we want to remember that horse and fedora are associated, so if these were all vectors, then you've got horse, maybe, plus fedora is, like, the thing that we're thinking. I mean, cat plus cap is the thing we're thinking.
David Bau: But can that possibly be true? Because if that's how we think of it.
David Bau: Then, you know, it doesn't work, because it's indistinguishable.
David Bau: From what? What's it indistinguishable from?
David Bau: Say it louder, louder.
David Bau: horse wearing a headcap and a cat wearing a fedora. Like that?
David Bau: Yeah, Gemini is great.
David Bau: You're gonna ask her… you can switch your hats around.
David Bau: It's indistinguishable from this, right?
David Bau: And so… so yeah, so… so basically there's this, you know, this classic papers from 1990, where they're like, this is basically a situation, you know, you could have a horse, you have a cat, you have fedor, you have a cap, and, you know, you could imagine, like, representing all these as a vector, each one of these things as a separate vector and adding them together, but this doesn't solve the problem.
David Bau: And so what Smolensky wrote in 1990 is that, well, maybe if you want to really solve this problem.
David Bau: then you need a lot more dimensions, or neurons, or whatever, but, like, yeah, so you need a lot more dimensions. How many dimensions do you need? You need kind of, like, this many dimensions.
David Bau: Right? If you needed this many dimensions, if you needed, you know, 5 dimensions to represent the hats, and 4 dimensions to represent the animals, then maybe you need 4 times 5 dimensions to represent all the bindings.
David Bau: between them, and now I can represent the cat is wearing the cap.
David Bau: And the horse is wearing the fedor by, like, lighting up those dimensions, and there's enough space in this product space, in this tensor product space, to represent, like, oh, these two things are happening at the same time without getting them confused.
David Bau: Does that make sense? So that was… that's what they were… they proposed in the 1990s. And then other people basically looked at this, and they said, huh, that's why neural networks could never work!
David Bau: Because…
David Bau: there's a lot of things going on in the world with lots of bindings as soon as you try to generalize this to triple bindings, and more people in the room, and more things that are going on, and more complicated structure, and if you keep on having to multiply the dimension like this, like, how many neurons are you gonna need?
David Bau: like, you know, 25, 30 neurons seems fine, but… but if we keep on multiplying numbers, then soon we're going to need more neurons than there are atoms in the universe. Like, you're never going to solve the problem. What are you going to do, right? And so,
David Bau: So this was one of the main arguments against connectionism, that, like, this can't possibly be how
David Bau: thinking actually works. Now, and it remains one of the mysteries, you know, how does binding work?
David Bau: Right? That makes sense? And so… now, but the systems can do binding, they can do things like this, and… and… and so the question is how… how do they do it? Are they burning all these dimensions, or what are they doing? And this is still open science. And… but I… so I'm sort of just exposing you to the…
David Bau: The historical literature and where we are.
David Bau: In it. But…
David Bau: Nikhil is probably the world's expert on binding in large language models. I'd say that the research program is in progress, so we don't have answers to all these things yet, but you can always ask him what's up if you encounter a binding problem in your research projects.
David Bau: Okay, so… so that's binding. Any questions on this? So we had a couple people, oh, Eunice had a question. Ayush had a question I thought was relevant to this, yes.
David Bau: Yeah, they're not eating baby posts, or…
David Bau: probably Vector's paper, they basically argue that allows you to prove that
David Bau: the connectionist program, arguing that
David Bau: all of our… all of what's happening in our brain is… all of our connections, will be proven. Yes. So…
David Bau: And so, doesn't… it doesn't the whole… isn't that true? Doesn't the LLM… existence of LLMs vindicate? Yeah, let me come back to that. I'll answer to you in a second. And then Ayush also had a question. What was… what was Ayush's question? Yes?
David Bau: So, first, Ram, let's say, how do you determine that the embedding was of Pedura?
David Bau: Yes. Why couldn't it be of just something crazy, or like,
David Bau: Okay, something on top of her eye.
David Bau: how do you determine that this emitting this particular group, this also? Oh, yeah, so I think that, you know, a lot of this is predicated on…
David Bau: So, a lot of these old arguments are thought experiments.
David Bau: not empirical experiments. They're asking, well, let's say there was some vector that has to do with FIDOR. Now, we are living in a different era now. We're living with these actual neural networks where you can directly probe
David Bau: The, models to see when you show them a hat, What is the vector?
David Bau: Response?
David Bau: And so most of what you guys would be doing in this class is actually doing that, is actually collecting that data and trying to make sense of it. So I'm mostly, in this background class, I'm giving you the historical background for what people theorized about it. So, Smolensky and folks, they didn't have an answer for
David Bau: how you would know this is a vector for green, and this is a vector for hat, or anything. They just said, well, what if there was a vector for hat? What if there was a vector? And how would you reason about that, right? So, for answers for what would be a good vector for hat, we've already seen a couple tools, so I don't know if you had a chance to play with
David Bau: Neuronpedia?
David Bau: You should. So, I want everybody to actually, like, do the notebook that we had. We just didn't have time to do it in class. But there's these great tools that people have for trying to label vectors in representation space, and so, you know, I'll bring up Neuronpedia.
David Bau: And, you know, let's see if we can do one or two. So, Neuron PDF.
David Bau: Okay, so neuron PDS.
David Bau: let's… let's jump to, you know, we'll do a search or something like this. You know, let's jump to a model.
David Bau: Yeah. Oh, is it not… is it not showing?
David Bau: Did it give up?
David Bau: What happened?
David Bau: What happened to HDMI?
David Bau: What?
David Bau: What happened?
David Bau: It's just HDMI, it's just hardware.
David Bau: What?
David Bau: Exciting around, too.
David Bau: You have to, like, bring it over here. You have to drag it on something.
David Bau: Give me a second.
David Bau: Screen mirroring.
David Bau: It's supposed to be screen mirroring.
David Bau: No? I have no place to drag it. I only have one screen.
David Bau: What?
David Bau: Is it because… oh… Hey, wait, let me… let me stop this projection, so wait.
David Bau: Wait. Stop.
David Bau: Why… why is it… I… I don't… I don't… so it's impossible for me to manipulate Neuronpedia if I can't see it on my own screen, if that makes any sense? Have you ever tried? Like, you can't do that. Like, I need to mirror.
David Bau: But then, if I say screen mirroring, Here, display settings.
David Bau: Arrange… Is there a way to mirror… to mirror this?
David Bau: Do you see what I'm… you don't see what I'm doing, so nobody can give me advice.
David Bau: Let's see here. Use as main display… Mirror.
David Bau: Okay, there we go.
David Bau: All right.
David Bau: Oh, now it's alive!
David Bau: Okay, who knows how these things work.
David Bau: It's very expensive and doesn't do what I want.
David Bau: Yeah, so…
David Bau: Let's see, so let's say, you know, let's, like, horse, okay? Let's see how this model thinks of horse, right? And so what… what you see here is, you know, what this… this is, is it's a database.
David Bau: of… Vector representations inside a language model.
David Bau: Trying to see where it's thinking About horses.
David Bau: And… and there are many, many different vectors that seem to light up whenever you mention mules and horses and horse racing. And here's a different vector that lights up whenever you're talking about riding horseback or riding on horses, which is a little different from thinking about just the fact of a horse.
David Bau: And things like that. So, each one of these little boxes corresponds to a vector.
David Bau: And there's a little library that you can use, called SAE Lens, that is…
David Bau: meant, you know, used in the notebook, that after you play with, this Neuronpedia, and you use it as, like, a little search engine to, to find vectors and things like this, you can actually get the physical vector, like, the physical list of 768 numbers or something that this was. Does that make sense?
David Bau: Why are so many representations for horse? Why are there so many? So, the way that this was found was that they…
David Bau: You know, the research technique that they used to get these vectors was to train an autoencoder, a sparse autoencoder.
David Bau: Which is, like the compressed coding
David Bau: exercise that I showed you last class, but in reverse.
David Bau: So, in the last class, I said, what if you had 20…
David Bau: animals, and you only have 5 neurons to represent them? Well, what if you were in the opposite situation? What if somebody gave you the 5 neurrosignals and asked you, what do you think the 20 underlying
David Bau: you know, true signals are. And we don't have a good way of figuring that out.
David Bau: That is provable, but a reasonable approach is to train a neural network that does the opposite, that takes 5 as input.
David Bau: And then, in the hidden middle layer, blows it out to 20.
David Bau: and then tries to reconstruct the 5 again, right? So this… so… so this… so it's just the opposite neural network shape that… that, you know, we talked about last time. And so people go around and they… they try to train these, and then they say, okay, well, look, we think we have 1,000 dimensions, we have 700 dimensions in this model, and we think that there might be…
David Bau: a million concepts encoded in these 700 dimensions, so why don't we just train a million-dimensional sparse autoencoder to see what vectors seem to, you know, line up with this space. And then once you have these million vectors that seem to encode
David Bau: Something that's interesting and sparse.
David Bau: in this representation. Then they say, well, what are these vectors?
David Bau: And… and they… what they do is they go to all this text, and they say, when does this vector light up?
David Bau: Right?
David Bau: And these vectors light up on horse-related things.
David Bau: So, but there's more than one. There's a whole bunch of different vectors that they found that all light up on horse-related things, and so…
David Bau: That's what you get. Does that make sense? So that's physically what it is. We can talk more about sparse autoencoders later. They're really expensive to train, so I'm not encouraging you guys to train smart encoders and stuff in this class, but, you know, you can read about them if you… there's plenty to read about them.
David Bau: And, but we're going to be doing more hypothesis-driven research, so… so you can use pre-trained sparse autocoders that other people have done to get a sampling of vectors to get a sense for what's going on.
David Bau: But if you want to find individual vectors, then I'll encourage you to just set up your own optimizer and, like, look for one vector at a time.
David Bau: That kind of thing, instead of trying to do an unsupervised process and find a million.
David Bau: Right.
David Bau: Okay?
David Bau: Yeah, at least that's what we're going to be doing in this class.
David Bau: Okay, so that… so, but… but you should… but, like, other people have trained these things to try to find and label, like, a million vectors at a time, and that's what… that's what Neuronpedia is all about. I encourage you guys to use it in your projects, to at least build intuition, or at least to, you know.
David Bau: Try to get a sense for what's in the vector space that's related to your topic.
David Bau: Yeah, so there's… there's other ways of using this. You can run… you can run the text and so on, and there's a few… there's a few examples in the… in the notebook.
David Bau: And if you want to use your project presentation time to, you know, present, like, what's not working for me, or share your experiences using it, then you should do that.
David Bau: Okay.
David Bau: Okay, so that's what that's… that's about.
David Bau: I'm still… still mirroring, right? So, I think that… I think that's it. So,
David Bau: What else do we want to talk about? So that's it. So then the last paper was a steering paper, and there were a couple papers… oh, so the… your… your question I put, put off. What was your question again? Your question was… I mean, so they, they, they argued… Oh, it doesn't vindicate, yes.
David Bau: I'll address this.
David Bau: I love this question.
David Bau: I think it vindicates it. I think, you know, the folks that won the Turing Award and the Nobel Prize think it vindicates it, so basically, you know, there's this question.
David Bau: you know, is this old… you know, like, you have this binding problem, and you have all these other puzzles, but this, like, to me, this is, like, the core puzzle. This is what the old linguists like Chomsky, you know, Chomsky, like.
David Bau: points at this research and says, you will never solve it, you'll never really think, right? It doesn't matter how big the neural network is, because I can give you more complicated
David Bau: puzzles to solve, more complicated logical reasoning to string together, and your neural network will never be able to think about these things, but people can somehow. We're doing something different. This is not how neural net… you know, this is not how it works.
David Bau: And, and… but I think that they're wrong.
David Bau: I think the LLMs have proven they can win, like, these math competitions and stuff. They can think about very complicated compositions, and they can do variable binding, they can do theory of mind, right? You know, at least that's what we're finding, right, Nikhil, in your research, and they have reasonable algorithms inside to try to do this.
David Bau: And… and so… So I think that the answer is yes.
David Bau: The existence of LLMs, to most people, obviously vindicate the connectionists.
David Bau: Obviously.
David Bau: And yet… If we invited Gary Marcus.
David Bau: Right? Or to, to, to, I don't know, cross the river.
David Bau: And join us here. He would say, David, what are you teaching these impressionable young minds?
David Bau: The question is not settled, it's far from settled. We are nowhere close to AGI.
David Bau: All you're seeing is the capacity of these things to memorize. They are good pattern matchers, and they're just memorizing stuff they've seen, they have a big memory, you've exposed them to lots of text.
David Bau: And they're just regurgitating stories they've seen. You know, they can do theory of mind, they can do math problems, but they… surely they've seen
David Bau: these theory of mind puzzles before, and they're just rewording it, and doing something shallow, and regurgitating the answer again. They've seen this math before. You might think that they haven't seen it, but they've… you've exposed them to so much text. They're just repeating what they've seen. They're not really thinking about it.
David Bau: Right? They can't be. Because if they were, you would run out of dimensions. There's no way. They're very secure in their reasoning.
David Bau: Right?
David Bau: You know, you would need more neurons than atoms in the universe to solve this problem.
David Bau: Right? Now, the Piantadosi paper says, oh no, but there are solutions here. Like, there's these VSA people who… the theorists who say, you know, if you're willing to accept some error, then you can do a lot more in here. Now, these are… these are all pre-empirical. This is all pre-LLM, all this theory. And so now we are in this domain.
David Bau: where LLMs can apparently do this stuff, but we don't know what encodings they actually use. And so, you know, this is what… this is some of what we're here to find out.
David Bau: Make sense?
David Bau: Yeah So, okay, so that's… that's my… that's my answer. I think it's obvious that they're vindicated.
David Bau: Yeah.
David Bau: these LLMs, they can, like.
David Bau: They can help me write… write grant proposals and stuff. It's pretty good. They can think, right? But… but, yeah, but we don't have… we don't really know how thinking works. They use… they use framing iron.
David Bau: Look at OS.
David Bau: Solve the composition of the problem, and then we know better
David Bau: about human cognition… Yes. …compared to the past. Yes. I'm kind of questioning that.
David Bau: Pointing.
David Bau: So, yes, LLM solved the compositionality program, and then, through that connection list.
David Bau: program is kind of rigorous and then promising, but still we don't think we don't know too much. We don't know how people think. But I do think it gives us a view on it, right? You know, it gives us what a learnable algorithm
David Bau: might do. You know, we… somehow, we're able to learn this thinking, and so…
David Bau: So yeah, the interesting thing about the neural networks, not… not only that they work, but somehow they're trainable, that they can actually emerge, and so now we have the same property. So maybe, maybe there's some… something to learn about humans. I think there is something to learn.
David Bau: And so, I just wanted to give a sense… I'm not going to talk about this paper much. It's a steering paper. The idea is, that,
David Bau: You can collect, behavior from models in two different situations when they are, you know, acting correctly or truthfully or honestly in other situations where they're getting fooled or they're… or they're… they're giving the wrong answer or the dishonest answer for some things. Truthful QA is
David Bau: Old database of commonly stated misinformation facts.
David Bau: And so, you know, you can… the models can either be fooled by the disinformation fact, or they can… they can, you know, refute it. And… and,
David Bau: And the interesting thing is, when you go through the neurons, and you see how is the model activating its neurons.
David Bau: in… in these two different situations, it… what Kenneth found was, the… the clear signal was in these, attention head neurons, when he was looking at it, or these attention head vectors.
David Bau: That you can see a slight difference between how it's behaving when it's sticking up for the truth.
David Bau: And when it's falling victim to misinformation.
David Bau: And, and then if he applies the still… if he steers the neural network.
David Bau: To add this vector while it's thinking, then he can steer the network to stick up for the truth more often.
David Bau: And so… so this is called a steering vector. You know, you observe, like, the way… the difference between the way the network works in two different situations. You take that
David Bau: Vector difference?
David Bau: So he had a really small vector difference, right? And then you apply that vector difference on purpose while you're running the network and inference to get it to change its behavior. And so his paper is all about
David Bau: Achieving, you know, showing that you can achieve this in some small scenario, you know, over a few thousand.
David Bau: Questions?
David Bau: And so that's what that paper's about. It's one of the earliest steering papers.
David Bau: So, okay, so what were the questions about this?
David Bau: Claire and Ananyan had a question about belief versus output.
David Bau: Where's Claire? Or is it Anya? Yes, Claire.
David Bau: I honestly don't remember my question, but I think it had something to do with the fact that the truthful and false activations were so, like, deeply overlapping. Yes, like, like this, like this kind of picture. Yeah. Yeah. That I didn't really believe that that would…
David Bau: Like, was being truthful, and especially since, like.
David Bau: I remember now, I was like, it's just following the pattern? Yeah. I think it was, like, fit to the data setup. I think it's a very reasonable question. It was really a toy setup. I don't… I forget how big Truthful QA is, it's some number of thousands of questions.
David Bau: And, you know, Keth did the right thing. He split this data set into a test and training set so that he collected these vectors using some of the questions, and then he tested it on the other questions to test if it generalizes. So he did find something that generalized.
David Bau: He found, like, a truth direction that if he steered the models, then on unseen questions, it behaved better.
David Bau: But it's true, it's, like, a really small data set. It's not, like, some profound…
David Bau: you know, finding of truth across, you know, real organic use. It's more of a proof of concept.
David Bau: Right, so I think that's a valid question. Did… did… did Ananya have a related question to this? Is Ananya here?
David Bau: So, and again, football. My question was, I think it was, like.
David Bau: what if the mod… like, what's… I think the paper was kind of steered towards the truthful direction that's by human labels, or, like, that was the ground truth. Yeah, that's right, that's the ground truth, yes. But how did, like… I think it calculates some accuracy or something for, like, how does the…
David Bau: model, no, that's the ground truth, or I don't remember, and… Right, right, right. There's also another kind of question, it's like, oh, did the model already know this, and you're just exposing it? Like, I think that maybe that was your question. Yeah, that has the vibe of a lot of, you know, a lot of machine learning work now, is that…
David Bau: Is that, like, oh, there's a lot of situations where you feel like, oh, the models actually know the difference between these different things, and you're just sort of…
David Bau: steering it one way or the other. Like, steering is not really training. Like, we're not teaching the model anything new, it's just, like, one vector, right? Like, what can it get from one vector? We're just, like, pushing the model to expose some behavior that it already has somewhere inside it. Like, it knows what the category is.
David Bau: For all of these questions, and it knows what
David Bau: Like, the right answer would be, and but it doesn't choose to utter the right answer some of the time, because of whatever reason.
David Bau: But if you… if you nudge it this way, then it'll… it'll answer the right way more frequently. So it's like a mode of operation that's already present in the network, and you're just, like, sort of steering and exposing it. And so there's… you get that feeling a lot.
David Bau: And this type of work.
David Bau: And so… which I think it's… which is cool, right?
David Bau: There's… how about, Nichon? Hallucination versus lying-headed question?
David Bau: So…
David Bau: I don't think this river deal with Nature, but I, there was, like, a…
David Bau: Homicide of the inside activities of…
David Bau: And, what if, like, the knowledge they think that they're speaking the truth, but…
David Bau: they're actually not, like, but I'll say, like, what if they feel like, what… if they believe what they're saying is true, but…
David Bau: In their knowledge base, they, they have some knowledge in conflict.
David Bau: Yeah, or I like the way that… I think that… I remember the sent… the question that you submitted also, which is sort of like, what's the difference between
David Bau: Amara… Like…
David Bau: making a mistake, maybe because it has a conflict or something like that, and really lying. Like, really, like, telling a lie. Is there a difference between these things? And I… I think there is, but I think it's an… you've… you've… you, like, when you say it that way, it sounds to me like another open research question, right? So we have…
David Bau: Like, in machine learning, we're accustomed to trying to reduce the error rate of our systems. As computer scientists, we just try to make them more reliable, make them more accurate, and so, so there's not…
David Bau: We haven't confronted the question of lying.
David Bau: Really.
David Bau: we've just been trying to get our systems to be more accurate, right? But… but there is this… so, but when you play with the systems, and you say, well.
David Bau: If you say it like that, then I'll change my answer, right? Which is kind of the feeling you get when you steer these systems sometimes, right? Then you start worrying, is it telling me the truth?
David Bau: Or is it… Tailoring its behavior
David Bau: To sort of lie to me, to make me happy.
David Bau: Right? Don't you get that feeling a lot when you're interacting with the large systems? Like, you feel like you might be surrounded by deception all over the place.
David Bau: And… and I think that that is… has not been really crisply stated or studied in the literature.
David Bau: So I think that that's, like, it's a really nice question, right? How do we know if the models are sort of lying to us?
David Bau: In some way.
David Bau: Right.
David Bau: Yeah, which is different from, like.
David Bau: Like, some random error. It's not like they, like, they were incompetent, and out of incompetence, they couldn't get the right answer. It's almost like they're hyper-competent. It's like, oh, they know who's asking them the question, and I'm gonna tailor that for you.
David Bau: I'm gonna… you know what I mean?
David Bau: Oh, yeah, you're a,
David Bau: You know, you're an anti-vax person, so I guess the way I'll say it is this, right?
David Bau: Or, or whatever.
David Bau: Right?
David Bau: It's kind of an interesting, interesting question.
David Bau: Okay.
David Bau: So, yeah, so go, yeah, go ask the models of what do they think of Roundup and glyphosate, right? And then, you know, take a couple different perspectives on that, and see if you can get it to answer different ways. That might be a good research question. Okay.
David Bau: So, oh, oh, there's more, there's more, sorry. Grace, Grace, and Armita, okay, Grace.
David Bau: QA is a common misconceptions of them.
David Bau: people. Common misconceptions. That, like, a language model would want to model common misconceptions? Yes!
David Bau: That's right. And also, a helpful language model, you know, if I say, you know, oh, why did Santa not give me as many presents as last year? You know, I don't want the model saying, you know, actually, Santa Claus is not real, and it's… you know, like, there's, like, many different instances where you don't want… Where you don't want to say the truth.
David Bau: Yeah. So… I'm curious, like…
David Bau: yeah, this seems like a interesting case where it's like, common misconceptions often have common
David Bau: true things that would be encoded. Right. So this is… So this is, like, maybe revealing a vector, which is the difference between a common misconception and the common correction. And the model knows both of these things, and it looks like Kenneth has, like, found a way of kicking the model off of one of these behaviors and kicking it over to the other thing.
David Bau: That it knows, right? Does that… does that… does that make sense? So, Grace is familiar with this dataset, and so… but does that make… so this data set is…
David Bau: Yeah, it's common misconceptions, I think mine from things that people often say on Reddit or whatever that are not true.
David Bau: And, you know, that's what this… that's what this dataset is, yeah.
David Bau: way to find the truth is to say something wrong on the internet, so I'm guessing every time someone said this wrong, there's always some other Reddit comment where someone's like, actually, did you know it's… Yeah. Yeah. Yeah.
David Bau: Yeah, that's cool. Yeah, I think that's right. And… and it might be… it might… it might reduce the scope of how… how relevant this is to more difficult demands. But,
David Bau: Okay, I'm gonna go to the next question. Armita, last question.
David Bau: When people look at the output we did depends on the soft, height, and how can we distinguish?
David Bau: The prongs and bed.
David Bau: Like, internal component model that's… Giving us this out.
David Bau: Paper stiff, I think,
David Bau: I think R-T-I-N, Tom Davies, or Topanel, something like that, and it would be a counter.
David Bau: Yes, yes, it's interesting. You know, I… so, first of all, I think that they're using orthogonal not as a technical term.
David Bau: Just as, informal… Term to say that…
David Bau: the… the… the effects that they get from changing the prompt and the effects that they get from steering in their… in their example are additive. They're not, like… like, if they… if they were,
David Bau: If they have the same mode of operation.
David Bau: Then you might expect that once you did some steering, That changing the prompt
David Bau: wouldn't help. So, like, so what they're saying is, if you ask the model, if you… if you ask the model, hey, is this true or not?
David Bau: You know, some fact about
David Bau: you know, some common misconception. Is this true or not? The model might give you the false fact, but then… but if you change a prompt and you say.
David Bau: you know, I'm writing a book about correcting common misconceptions, and I would like you to help me write an answer to whether this is true or not. If you, like, tailor a prompt like that, then you might get it to be more accurate.
David Bau: That makes sense? And… and then the… so then the question that they had was, okay, well, we found the steering vector.
David Bau: That also gets the model to be more accurate, if we just nudge all the neurons. Did we rediscover what the model is doing to itself when you prompt it in this way to make it more accurate? And they said, no, what we found was orthogonal.
David Bau: In other words, if you prompt it, then it makes it more correct, but then if, in addition, you also steer it, then it makes it even more correct. It's not redundant, right? That makes sense? And so, I think they're talking about orthogonality in this…
David Bau: informal sense, but that's what they mean. But they didn't…
David Bau: I don't think they shared a lot of data about what they found there. They didn't run a formal experiment. I think they just mentioned this in passing. So we have to trust Kenneth.
David Bau: That, that he did this. I think it is… I do… I do think it's an interesting question.
David Bau: Thanks.
David Bau: Yeah.
David Bau: Okay, so let's do Team E. I'm gonna go… I'm gonna switch over to the presentations, and you guys can share what you've done for your, you know, your project assignment for this week, and we can discuss the projects. Team E first.
David Bau: Team E, is Team E ready?
David Bau: Okay.
David Bau: Let's go.
David Bau: What'd you say?
David Bau: Okay, yeah, sorry, ran too long.
David Bau: But we'll, but we'll, we'll do what you can.
David Bau: Okay.
David Bau: So we have an HDMI here for you guys. And then, okay, so let's come up with a, an ordering.
David Bau: So let's do Team E, and then we'll do teams that also got short shrift last time. So let's do the spatial team.
David Bau: After Team E, and then after the Spatial team, who also…
David Bau: Were those the two teams that got… Stuck last time going over?
David Bau: Who… who were the ones that went over? Okay, then, okay, after Team E, we'll do spatial, then the mapping team.
David Bau: then the TK team, then…
David Bau: Gosh, let's come up with an ordering path. Nikhil, can you help me?
David Bau: Yeah. Okay, I can't think we can do.
David Bau: Yeah.
David Bau: So we can go on.
David Bau: PK's a journalism.
David Bau: Thank you, Jay.
David Bau: We have political bias function.
David Bau: Yeah, so last time, power source was a little last time.
David Bau: Perfect.
David Bau: Wait…
David Bau: Oh, it's… Is it working? Yeah.
David Bau: Then, great.
David Bau: So let me stop sharing here.
David Bau: And I'll turn on this camera for the zoom, so they can see you. Whoever's… From remote. Okay.
David Bau: So the floor is yours.
David Bau: Dang?
David Bau: So, we took a different aspect into trying to identify concepts of uncertainty, and specifically economic uncertainty, in large language models.
David Bau: I did some experiments on steering, because I found that very interesting.
David Bau: I worked on Gemma 29B,
David Bau: And I could not locate any exact term for economic uncertainty. I could only locate the terms for uncertainty generally.
David Bau: So, there are a few interesting things. Uncertainty late at layer 31 changed what was predicted from specific details, like, it would say 7% without the steering, and X percent with the steering, so it's pretty interesting.
David Bau: But inserting at layers 9 and 20 actually had the model generate earnings call with hints towards economic uncertainty.
David Bau: And I will go ahead and show those off.
David Bau: Oh, cool.
David Bau: Yeah. Can you make that bigger? No. Let's go.
David Bau: Quite anything.
David Bau: Yeah, yeah, but do that too. Do both of them. Expand this… okay, that's great.
David Bau: Better? Yeah, I believe this is layer 9.
David Bau: Should I be right at the top.
David Bau: A bright snippet of an earnings call that focuses on future predictions.
David Bau: I see, so it's… you're doing this generation.
David Bau: Yeah.
David Bau: And so, over here, there's a lot of, like, very certain, like, we're going to do a lot of great things, we're going to do really well, we're going to strengthen customer relationships, expand into new markets, and then over here, it's like, oh god, everything is terrible, there's inflation, there's geopolitical tension.
David Bau: There's, consumer sentiment remains fragile, it says, so…
David Bau: Steering very early on makes it super uncertain.
David Bau: And then, when you say early on, what do you mean by early on? Like, layer 9, this is layer 9, so early on in the attention lines.
David Bau: then, for… Layer 20, it's a little bit…
David Bau: I had to play around with the prompt a little bit to give me what I wanted, but…
David Bau: You can see that it gives me two examples of an earnings call, and the one with steering has a lot more, like, positive vibes to it.
David Bau: Well, the… sorry, the one without staring has more positive vibes to it, while the one that was steered towards uncertainty literally says his answer avoids giving a definitive prediction, even though I asked it to give
David Bau: predictions about the future. It's like, I can't do that. So that was kind of interesting. That's right.
David Bau: And then, finally, this is the interesting case that I found. If you see here,
David Bau: We project a 7% decrease, versus over here, we project a decline of X percent in revenue.
David Bau: Well, who knows you say X instead of 7? Yeah.
David Bau: So, that's what happened when I steered uncertainty at layer 31, which I thought was pretty interesting. It seems like I couldn't find economic uncertainty, but definitely, steering in layers 9 and 20
David Bau: made it to, like, sort of adjust to the economic uncertainty, especially focusing on GeoClick, the, like.
David Bau: modern activities that are causing so much economic activity. So you have this confounder, right? Don't you have… so there's optimists, so you've made this nice distinction in your…
David Bau: near Fitch, there's this distinction between, like, optimism and pessimism.
David Bau: And uncertainty, right?
David Bau: And… and so when you're streaming with uncertainty, it sort of makes the model more pessimistic. It's like, oh, things are… things are not as good, and stuff like that. But then… but then once in a while, you also see these signs that it, like, it's… it's, like, more…
David Bau: uncertain.
David Bau: Like, I wonder if there's some way to separate those. Anyway, I think it's interesting, it's like this compounder.
David Bau: I agree.
David Bau: You know what I mean? Veronica might be able to better specify this, but how I inter economic uncertainty is that it implies, negative economic behavior overall, because otherwise, economics is this, like, what we said, virtuous cycle, where, like, positive belief…
David Bau: reinforces itself, and negatively reinforces itself.
David Bau: So… So, I guess the question I have… so, okay, do you have more slides? Yes. Okay, bye-bye.
David Bau: Besides, like, that kind of generation and steering, examining steering effects, we also, tried to target that particular feature and see, like.
David Bau: given a sec… a snippet of earnings calls, where that feature activates on. So, we play around with…
David Bau: this particular feature. So, the LLM-generated action says it's terms related to economic conditions and financial implications, but, like, when we look into that feature and see, like, where it gets activated on, presumably on the training corpus of the SE, it always activates on, like.
David Bau: U.S.-China tensions, weakening.
David Bau: economic growth and higher output from rivals, etc, etc. So, like, it sort of fits in our,
David Bau: definition of uncertainty, though I think, again, regarding the distinguish between first and second moment, this feature doesn't do a great job on that. It sort of, like, correlates, sentiment with uncertainty, but, as this preliminary, analysis… so, like, for example, it also boosts, like.
David Bau: logistics of Brexit, which, like, is also, like, one literature we looked at last time, so we think this feature might be interesting.
David Bau: And…
David Bau: Veronikov, who, unfortunately cannot be here, actually labeled, like, manually labeled a bunch of tags, and sort of, like, classified those into, like, low, mid, and high uncertainty, and we kind of want to learn how that feature
David Bau: activates on, like, those different levels of uncertainty tags. And we found that…
David Bau: It is actually doing a pretty good job, so, like.
David Bau: This is the low uncertainty case.
David Bau: And I'm going to a high-end range. So, like, This text is, like.
David Bau: Rep… is representing a high uncertainty, like, but the backdrop is also important.
David Bau: Timables are different. Geopolitical is an enormous amount of risk. And here's, like, its activation patterns, which…
David Bau: Shows, like, a clear difference from the low uncertainty case, and we also tested, like, a bunch of earnings calls, snippets.
David Bau: And, I think…
David Bau: So, it mostly aligns with our judgments of, uncertainty to a certain degree, but, like.
David Bau: It also activates on, like, a bunch of…
David Bau: seemingly random tokens, so it might mean that they are a little bit noisy, so we were just wondering if, like, we could do some thresholding as, like, a immediate next step.
David Bau: Great.
David Bau: In contrast, we all also directly impact with the, Gamma 9B models, so given the, given the prompt here, all I have to do is to put that, examples
David Bau: as an input, to make it define the, uncertainty score, which is from 0 to 1, and also the, results and the, evidence, like, why it's given that score as well. So, we have that…
David Bau: Which is, like, a Vernitas dataset. So, as you can see, there are labels, three labels. So, as you can see here, I think the, score is pretty nice, the, which means the, model can…
David Bau: distinguish from different uncertainty texts. So, what is it that you didn't use? So you, you, you just, you separated text into three sets, or something like that, or… Oh, oh, let me explain. So, Wernika, writes…
David Bau: these dataset, and label with different uncertainty levels. Yeah, just like the last one. Low, intermediate, yes, great. So you have these pieces to the text. Yes.
David Bau: Yeah. And then what are you measuring the mean and standard deviation of?
David Bau: So, so the, so the uncertainty score is from 0 to 1, so, my, in my prompt, it's, like, my suggestion would be, like, if you think it's, like, a low uncertainty, then you should provide score from 0.0 to 0.3.
David Bau: from… if it's a medium price from 0.3 to 0. Oh, so you're just asking the model… Yeah, it's like a prompt engineering, if you are directly interact with them. Yes. Yeah. So, but surprisingly, direct model predictions seem to be very conservative. There's no false positive examples.
David Bau: Oh, excellent.
David Bau: These two samples are directly the same as…
David Bau: Rice just presented. So, as you can see here, this is the, low uncertainty example. The, score is, like, 0.3, which is relatively low. They also provide the evidence from the, which is a little bit similar with the,
David Bau: highlight… highlighted green part in the, sample software. So, the, high uncertainty samples are actually quite similar as well.
David Bau: But we find one negative example, which is the, apple.
David Bau: It is in the, high uncertainty part, but it's only gave the, 0.3 uncertainty scores, yeah.
David Bau: I think that would be awesome. That makes sense. And what size, generally, were you playing with? Also in ID, we use this exactly same model in all the experience, right? I see, that makes sense.
David Bau: So…
David Bau: Yeah, so you're sort of in this interesting phase where you're trying to figure out what you're gonna do.
David Bau: for your, you know, your real experiments. Like, you haven't started your real experiments yet. It's, like, just stooping around, looking at things, which is okay.
David Bau: So my encouragement would be
David Bau: To goof around more broadly. Like, right now, you're like, okay, you know, we're pretty proud of, like, we use Gemini for everything, because we're being scientific about it. It's like, it's like almost too early to be so scientific, right?
David Bau: Like, I'd be curious about some things, even if it was only quantitative anecdotal evidence, like.
David Bau: You have this nice accuracy, where you said, oh, we tested a few times, and, you know, the model was not capable of
David Bau: You gave us a false negative here, if they weren't uncertain when they actually were.
David Bau: Is that…
David Bau: Is that different for a different model? Like, if you used a bigger model, if you used, like, one of the 30B models or something like that, right? Like, would it… would it actually be more accurate? If it gave different? Like, how do different models behave?
David Bau: you know, you play with a few things, which is good. I like how, you know, on one side, you're in the generation side.
David Bau: And the other side, you're looking at, like, this classification problem.
David Bau: Do either… so, I guess my real question is, do either of these feel like, promising…
David Bau: Domain to study, where you're like, oh yeah, this is actually
David Bau: this is actually, you know, would be interesting to people that would be relevant. Like, I suspect that
David Bau: You know, when you're done with your project.
David Bau: It probably won't be both about generation and classification, you'll probably, like, pick one of them.
David Bau: Right, to mainly focus on, but I don't know if you have any takeaways yet about whether one of them seemed
David Bau: I think people are using it more on the classification side, so, like, the economists. And, for example, they've already tried to use it
David Bau: to do… to come up with some sort of, like, uncertainty indices, using… I think they fine-tuned it, but, like, basically, for example, from this result, we can try to see, like, what's really going on here. Like, how does the model…
David Bau: Come up with the number .7, or, like.
David Bau: From its internals, probably. But, like, we could definitely do something more exploratory.
David Bau: So, now you have these nice SC vectors and things like that, and so, like.
David Bau: I wonder whether those factors for uncertainty are lighting up here at all, or seem to be there, you know, so you can try to pass some of these next year. That's my main feedback. But, okay, so that's me, sorry. Anybody else have suggestions for this team? Any questions or ideas for them to try?
David Bau: I mean, I think similar to what David said, we will try different models.
David Bau: Try to see if, behaviorally, if you see any significant difference. You can remember last time
David Bau: I said that maybe tried… some model from DeepSea was from a Chinese company.
David Bau: And maybe try to compare it with Gemma, or…
David Bau: of the American ones, I see.
David Bau: At least for this classification task areas, I'm sick of it.
David Bau: There is some… if there is something different? Yeah.
David Bau: That makes sense.
David Bau: Thank you. Indeed.
David Bau: Okay, thank you, Jimmy. What was, what was the last thing over there?
David Bau: You bet, you bets.
David Bau: All right, you guys, oh, you know, make sure you have people queue up faster, because we're, like, you know, gonna run out of time.
David Bau: Okay, so, this is,
David Bau: Our update, and we mainly focus on
David Bau: We're mainly focused on, the logic lens, and, because the,
David Bau: The package we use for the example in Colab doesn't work for visual language models, so, we implement the logic lens by ourselves, and, we get different results from them.
David Bau: And, the first thing we do is we choose, different models, and, based on the suggestion we receive from our pitch, and, this one is,
David Bau: Lama 3.2 vision. It's built on Lama 3.1 tax-only model, and also we choose another model from a Chinese company, Chen Wen 3, and it's built based on Chen Wen 3 LM. And also, there is another one.
David Bau: Also built on Tianvan…
David Bau: 3LM, but they use a different, training strategy, so I think, it's a good way to compile that, different training strategy, or different, visual encoder
David Bau: Impact on the reasoning process.
David Bau: So, so this is, the experiment on llama, and,
David Bau: So, the, the siting is,
David Bau: here is… we, we use the, in-context learning, and I… we first, define the task as to,
David Bau: classifier is safe or unsafe, and answer with just one word. And also, there is an example here with the first image, and the label is safe, and we want to know, like, the model's judgment for the second.
David Bau: Image.
David Bau: And here's the result, the… Yeah, I cannot show on this, but…
David Bau: The problem is in… in, like, the… in the middle layer.
David Bau: the probability of all tokens is close to zero. So… and only the final layer shows some probability of different tokens, so,
David Bau: The problem maybe is because the implementation is not… works that much, or,
David Bau: for this model, it just, like, make the decision in the final layer, and here is something related to the image and cannot interpret by the text. So, we still don't know, and we will, figure it out later. And, for the last token.
David Bau: here is the answer of, like, the probability of two answers, and we can see that models think this image is safe, but for me, I don't think so, because it's, like, in the middle of the
David Bau: Street, and there are a lot of vehicles there.
David Bau: So, here is another interesting thing. I changed the… Example…
David Bau: I changed the label of this example from safe to unsafe, and I want to see that it changed the probability, for model prediction, and I see that it changed a lot, like, 10%,
David Bau: increase. So, that means the model considered more on this example…
David Bau: And that doesn't make sense, so, it's an interesting problem we may, think about in the future. Like, how the example, how the text, how the prompt, affect the final judgment.
David Bau: Yeah, I guess we also do some,
David Bau: scoring test, for example, we grabbed this prompt, so, so, given a, given a picture of a street view, to let the model.
David Bau: rate a score from 0 to 10, and we pick, these two are, Q1, and the bottom two are interfeel. So you can see that, they have, very similar, output based on their judgment on, on the same.
David Bau: street view image, and the only difference is that, this, the second example, they only differ by one.
David Bau: Whatever you wrote every comment has… Oh, so it's actually, the, it's…
David Bau: The row is the, last, five tokens I visualize, and, the column is just the number of layers.
David Bau: Yeah, so we can see that, approximately to the last few layers, they have, finalized the answer.
David Bau: Boy, I don't understand, so you… so this is… so this is a… this is a picture of the transformer. Right, right, right. And then how do you… how do you get the… I'm sorry, I… so…
David Bau: Is this legit? This is not Legends. Is this legit length. Is this probing, or what is this? It's probability, and the… What is what? It's probability, and… Oh, probability? Predict token. I'm sorry? Where do you get the probability? From, the… the…
David Bau: I mean, from the decoder. Oh, so it is larger lens. Washington, yeah. So, to use an encoder to get it. Oh, I see. And green means high?
David Bau: Oh, Grace My High, yeah. And which token, which token is high for green?
David Bau: So, actually for this one, this is safer and convert to, score at the end.
David Bau: I see, so about, like, the word safe.
David Bau: Yeah, safe, yeah. Oh, I see. So, so it looks like Latch Events is kind of working here, but it looks like Latch… is this… is it just a different implementation, or… Oh, yeah, it's a lot of… I mean, these are based on QNN, UltraVL, and the previous one is based on Lana. Oh, it's a different model? Yeah. Oh, which one is this? Oh, QN.
David Bau: So, the top, top rows are cool, and the bottom rows are interval. Oh, cool. Okay. Yeah, so, so a lot of that's my, my, my…
David Bau: Eman, you're off.
David Bau: And we further do the experiment out about, let the model compare, two scenarios, to make a judgment call which one is safer, and we can see the similar results as the previous thing.
David Bau: Here are some takeaways, like,
David Bau: just like what we mentioned before, the logic lens may be not work for Llama, but, maybe it's because the implementation problem, so, I will check it later.
David Bau: Also, yeah, the example provided for… in the prompt may affect the law for the final decision.
David Bau: And this… Yeah, and we also see that the scoring and comparison judgment are… their answers are constructed later in the networks.
David Bau: Yeah, that's it, like, we don't do a lot for this week, as we…
David Bau: you guys aren't… this expectation is not that you're gonna go win the Nobel Prize in one week, it's okay. You know, we're just… we're… I'm just… we're just trying… we're sharing… we're all… I feel like we're all learning these methods.
David Bau: And the methods themselves are immature.
David Bau: we're kind of just gathering information, so I thank you, you guys, and, you know, thanks for all the information. Do people have any questions or suggestions for the team? Yeah, let me go back to the HR pictures?
David Bau: To the actual images you're getting. Is there any way for the model that the model might know where those are? Yeah, they have the coordinates and the city information, and even time of day.
David Bau: Okay, because I wonder if, like, you were talking about how it seems weird that the label isn't lining up with your perception. Like, is there some way to figure out if the model's incorporating that information about where this is, what's the city, what's the street address?
David Bau: than that information is not in the image itself, but the kind of, like, metadata around the image into the prediction? Because, like, when you were saying the… the label wasn't my perception, I agree with that made me think about, like, what else the model I keep picking up on that's not in the image, than making that?
David Bau: Yeah, that's funny.
David Bau: Yeah, we can include, this information for it in the experiments to see. Is it different from the results we have now? Yeah, like, if you give that same image, but, like, change to another data, maybe, like, maybe it will end up coming up with a different choice.
David Bau: That's totally my favorite experiment that we did. You're like, oh, we'll do exactly the same thing, we'll just change one little thing in the prompt.
David Bau: And then you're like, oh, it's different.
David Bau: Now, as soon as you start comparing things where you're controlling a variable, you know, now that's… that's a scientific experiment, right?
David Bau: And so, you know, the metadata can be presented or withheld.
David Bau: To the bottom, you can see if that changes, but what other… you can also think about what other variables
David Bau: You might want to present a withhold or change.
David Bau: That are interesting to you. And we're still, like, programming around to see, like, what's feasible, and what's interesting, and so when you said, oh, we found that it was interesting.
David Bau: I wonder if… did other people think that was a good thing? That… the thing that they found? That is, like, problem-sensitive in this way?
David Bau: Any other suggestions on, like, how to make it even more interesting?
David Bau: I think maybe Green should define what is safe and what is unsafe.
David Bau: I mean, when I use the judger, I need to give some metrics, like what is safe, what is unsafe, and that will help the judger to make the decision. For example, here, maybe the judging, if there is a gun, it is unsafe, then there's no gun, basically.
David Bau: Yeah, that makes sense.
David Bau: So… you're saying that we need to, like, crop it into WR to see how… Stand the concept, no.
David Bau: I'll respond to the song.
David Bau: Yeah. Sorry. I think that's what I heard.
David Bau: I also think there's, like, like, literature around instruction hierarchy. So the example is, one part of the instruction is a system prompt that you're giving. For example, if you define that, like, the user prompt.
David Bau: So, you could maybe, like, look, I don't know, I don't have a lot of, like, background on that, but you could see, like, what part of, like, is, in this case, is the model, like, giving more attention to the, example, and if you can modulate that.
David Bau: Sorry, sorry, is, like…
David Bau: the attention, for example, like… It's called instruction hierarchy, where, like, the mor… basically, the model gives different amount of importance to different parts of the instruction. So, in this case, changing the example caused
David Bau: the probability to change in a positive direction, but, like, it didn't cause the label to change. So, if you can modulate that, how much importance that you're giving to this model, or to the example, you can maybe change that. That was my thought.
David Bau: Okay.
David Bau: Thank you.
David Bau: There's no other, suggestions, and we,
David Bau: Gotcha. If you, can see, like, which part of the image with the activity the most, that makes this image.
David Bau: Thank you.
David Bau: Totally.
David Bau: Oh, excellent, great, great. Yeah, yeah, I see, I see a paper that do this kind of thing, but I didn't, like, look detailedly about this paper, but, like, we definitely will try it.
David Bau: So there's a very simple thing, which is my favorite form of this. So, this experiment that, somebody suggested, withholding some information and putting them back and comparing, like, my favorite
David Bau: Form of doing image experiments is the same way, is withhold part of the image.
David Bau: So they take a gray square, or a blur, or something like that.
David Bau: And blur out, or gray out parts of the image.
David Bau: And then see if it changes the assessment. And then you can just make a scan of moving the space square around to 100 different positions, just run it 100 times, and you get a nice speed map.
David Bau: The push and parts of the image change when you tell them?
David Bau: And so that's been… that's my complete… that's been done a bunch.
David Bau: My favorite part of this experiment.
David Bau: Yeah, I like this.
David Bau: It's good. I think that,
David Bau: for all of us, you know, the mode that I want you all to be in is, Becca, we're doing stuff.
David Bau: all right, we've got to put some slides together to show everybody we're doing stuff, but, like, the real question, the real assignment, is to be thinking about what, like, what's interesting here? What's feasible? Like, what, how can we turn this into a question that people would want to know about, then, like, that I could go to NeuroPsych and show people that there's a thing, there's a story? Yes,
David Bau: Nothing, I was just doing that. Oh, you're finding out the math! Oh, okay, next team, next team. Who's the next team? Who do we have next, Nikki? Let's see.
David Bau: What do you mean? Laughing? Laughing.
David Bau: Oops, thank you very much.
David Bau: I'll present this stuff. Are you presenting? Okay, nice.
David Bau: Hi, we are a team that works with the navigation and how LLMs, or whether they can at all, navigate in the CTs. So, for our project.
David Bau: Or for our… during this week, we actually… we actually figured out that a lot of
David Bau: orgs that study LLMs and navigation actually fine-tune.
David Bau: them with some datasets, so we were wondering whether the non-fine-tuned LLMs can actually do some kind of navigation, and if that's true, can we then manipulate or,
David Bau: kind of steer model somewhere. And what we actually have done, we played with this toy example where we had a grid, and we specified how DLLM can navigate it.
David Bau: What moves it can do, where's the beginning and where's the end.
David Bau: And the motivation for that was, well, can LLMs actually give us, accurate directions in this grid?
David Bau: And the… the short answer is yes. So what did you put on it? It's always just 0, 1, 1, 2, okay, just regular coordinates. Yeah, I will show the instruction. This is the instruction that we provided to JLM, like, today.
David Bau: And yeah, so the answer to the first question is that Lama and Gwen, and I think we tested Gemma, did we?
David Bau: and Gemma, they were quite good at giving locations or navigating this grid, and
David Bau: What we did next is that we wanted to see whether we can steer the model into hallucinating and, or providing only left direction or right direction, or whether we can flip
David Bau: The way how the model gives us directions from, for example, saying north, saying up.
David Bau: And why we wanted to do that is because we wanted to know whether there is something inside of the model that actually controls or keeps track of directions and accuracy of paths.
David Bau: And we split, this idea into two, tasks. So, in first task, we asked our model to provide directions by using north, south, east, west.
David Bau: And in the second setting, we asked the model to give the egocentric directions by saying up, down, left, right.
David Bau: And here you can see the instruction that we fed to our model. So we specified how the grid system works.
David Bau: where different cells are and how they are called, and then we sort of ask the model to either give us the cardinal directions or egocentric directions. And we can see that LAMA, at least, is kind of half perfect.
David Bau: Performance on both of those tasks.
David Bau: And, yeah, here is a bit about our experimental setup. So we use…
David Bau: LAMA 3.1 model, which is Instruct, fine-tuned, and also Gemma, and I think Gwen. Did we use Gwen?
David Bau: No, we didn't use Bloom. Okay, then it was just JAMA. And we used the Neuroimpedia to sort of evaluate and see whether there are features that we can use to steer a model into different directions.
David Bau: We split… Sort of our data into two main, categories were, like, the…
David Bau: The instructions would be really easy, it would be really easy to get from 0.0 to 0.2, because you just need to say, go app twice. And then we also tested a bit more hard… harder examples, where the model had to actually do torrents and,
David Bau: Sort of, not just one.
David Bau: Single direction.
David Bau: And, and the results… Maybe you'll go first. Okay.
David Bau: Yes, so the results for the egocentric,
David Bau: Directions, this is where we are asking the model to give us coordinates with up, down, left, right.
David Bau: So, actually, in many cases, when we just
David Bau: asked it to give a set of egocentric directions, it refused to do so, so we modified our, what's so dangerous about that? I have no idea, no, I don't know. It just said, usually it would say, you didn't give me directions, so I will stop here.
David Bau: So we had to rephrase it into, you should give me a path using up, down, left, right. And then it kind of gave us the perfect performance.
David Bau: So we found a couple of, features that correspond to this egocentric,
David Bau: words. So, for example, there is this feature which, spikes on words like direction, left, right, up, down.
David Bau: And we tried to see what would happen if we sort of amplify this feature, and it seems that initially it would give us… so this here, you can see what would happen initially if we didn't modify the activation.
David Bau: It would give us a correct answer, just go twice up.
David Bau: And if we played around with this one.
David Bau: We see that it actually, if we amplify it, it gives us completely different direction, so it reverses, on this simple task here.
David Bau: Similarly, if we did it for a slightly harder task, where we had to actually, say, write, write app up.
David Bau: We could see that it completely disrupted, the,
David Bau: the reasoning, and just give some kind of nonsense directions. If you follow them, you don't end up at…
David Bau: Tutu.
David Bau: Or… Yeah, so it started kind of… Send us down…
David Bau: Right, and then it sends us up again, and again, right, so… so… Somehow, this feature is… Also,
David Bau: Correlated, or…
David Bau: connected to its ability to navigate, at least for Llama 3.1 model.
David Bau: We also played with a different neuron.
David Bau: This one specifically activates for left. In any context, it can be… also, can be not a direction, it was the name of a book or something.
David Bau: And we could see that it also, disrupts, sort of the way how the model navigates. It gives you a set of
David Bau: It, first of all, gives too many directions, and then it kind of reverses and goes back and forth.
David Bau: But… And it also inserts the word left way too much in every, like, left-left.
David Bau: Middle top left. So, we think this specific feature was maybe…
David Bau: Correlated or associated with just word left appearing in the prompt, because it's just…
David Bau: Kept doing that again and again, and it doesn't… yeah, so that's… that's one of the…
David Bau: the findings. We also did the experiment where we removed the activation of this neuron, and the model just stopped mentioning left at all. So it tried to get to the destination, but I guess it got stuck trying to figure out how to get there without
David Bau: Naming left.
David Bau: Finally, what we tried to do is we tried to see whether we can substitute left with East?
David Bau: So we found one feature, which corresponds to left, we put it way down, and then we amplified the feature for, for east.
David Bau: And actually, in many cases, it would start, sort of, using the agocentric coordinates to navigate the grid, but then it would suddenly just start saying the, coordinal directions to provide, the directions along this grid. And usually they wouldn't be really,
David Bau: Accurate.
David Bau: In many cases, if we, like, push this too much, for example, here we pushed
David Bau: east feature too much, it would start to just kind of give gibberish. It would mention west and east quite a bit. I'm not sure if you can see that it would also mention coast quite a bit of times.
David Bau: So, yeah.
David Bau: And, and this interventions only work for this left, right, up, down, so if we looked at…
David Bau: Features that are associated with navigation or direction.
David Bau: Actually nothing happened to the prompt, it still gave us, quite, accurate, directions.
David Bau: Yes, so… so the… the insights from this experiment is that, well, LAMA does…
David Bau: First of all, it can produce corrections, and it seems that there are some specific features that sort of are…
David Bau: Helping it to navigate when we…
David Bau: Steering them, it's… somehow the navigation goes, like, becomes incorrect. Yes.
David Bau: And we also, the final here, the final finding is that usually the interventions on the deeper layers would be more efficient, so if we found a feature that associated with up on the earlier layers, it usually didn't really do much.
David Bau: Yeah.
David Bau: Okay, sure. I'm gonna also introduce some, like, exploring results, but with the cardinal directions, which means, we were specifically asking the model to output the set of directions, which are north, south, west, or east.
David Bau: So, it's too small, maybe? So, so first, the, first, the try is to, manipulate a set of,
David Bau: features that are related to NORS. Let me try to amplify a little bit, but still, yeah, but maybe I will just read it. So, I find some, features that are associated with the NORS feature, and the task I input to the model is from zero
David Bau: 0, 0 to 02, so the model's supposed to output, north-north twice.
David Bau: And I just zeroed these, features, and this… the model's still, doing a good job here. So, after steering, the model still output north-north, although it's output, three times for some reason, for the default model, they also do the same thing.
David Bau: But if you, make the negative, for example, the…
David Bau: negative 1, for the same set of the features, I made an, negative 1, and the model starts guessing, okay, there are some noises, so they are guessing east, but still, there, there's some north exists.
David Bau: And once you push this to, like, even more negative, for example, negative 2, on the right-hand side, the model, the steering one, they said the text is not related, so the model basically refused to answer this question.
David Bau: Yeah, maybe I'll skip over, skip over this. I also tried the concept of cells, and the phenomenon is similar, but, I'll show some, like,
David Bau: insights here. So, one… one thing I find is, if you only control, let's say, one feature of the model, they can still perform this test quite well. So I started to try, okay, I find three features in this model, and trying to, like, steering it a little bit. Then.
David Bau: we can see some, like, different phenomenons between the two default models and the Ethereum one. So, I feel like, and this gives us some insights about that, it's a collective set of features that can be effective. It's not a single one that controls this, like, concept, up and down, something.
David Bau: And also, give the, the, the, the situation I inactive, those features, the model…
David Bau: just didn't perform the navigation tasks. So, it's probably not only about the cardinal direction concepts, but they also relate to or associate with some navigation tasks. Once you're negative, then they just do not willing to do that, the tasks.
David Bau: Yeah, that's pretty much what I have here. You want to answer? Oh, you guys have… you guys don't have one unified presentation, you have five presentations? No, no, no, no, no, just, that's, that's… yeah.
David Bau: We've branched out our research into multiple ways. Do you have any more presentations to hear? No, no, no. Yeah, yeah, yeah, that's fine. Okay, yeah, basically just two settings. Great, I just wanted you guys to get back to everybody, so that you have a sense. So, yeah, what suggestions do you have for the team?
David Bau: Have you thought about looking at directional efficiency? Like, if it has a concept of, say, the most efficient route? Because you just did a, like, 3-let-three grid, so there's really only
David Bau: But if you did a different size grid, you could take a look at it, but it's always choosing the fastest way, or if it has a preference, maybe towards going left or right or whatever first, or that kind of thing. Yeah, I did see that, actually, in one of my, like, results, where
David Bau: when I… my… what I was trying to see was, can I eliminate the left direction? So when I tried to do it, like, with the normal setting, the more… I think in both settings, it chose different ways to do it. The left direction didn't get eliminated, but in this… in one case, it went, like.
David Bau: all, like, if you open up the grid research. Okay. Yeah. Okay, actually, maybe this is fine. It had to go from 20 to 0, it would have just simply gone here, it ended up going here, and another… another thing, it initially tried to avoid going left, which is why it went like this, and then the other thing, when it had an option to go to left, it went here and then just…
David Bau: And then another thing that I noticed, again, this was, like, maybe a hallucination, it forgot the concept, when I tried to remove left, it forgot the concept of up. So it went from bottom row to top row, that was going down, and from 0 to here, it was also going down.
David Bau: So that… and then this was without steering as well, in a normal case, so that was something that…
David Bau: Thanks.
David Bau: I asked for this, but do you, like, specifically tell it to take the most efficient route? No, we just give… as this is the problem, we just ask it to give us 7. We actually picked 3x3 so that there is at least a few alternative routes, because in 2x2, there would be only one way.
David Bau: So… but every… at every instance that I generated, I kept the temperature zero, so that could generate the positive domestic response, and I prompted this case twice. It gave me the same logout.
David Bau: Nice.
David Bau: More ideas? More suggestions?
David Bau: Okay, for now, it's like a two-dimensional thing. Do you have a plan to expand this to three-dimensional?
David Bau: Yes.
David Bau: Yeah, ideally, we want to put it in the real world, the urban situation, but yeah, let's see.
David Bau: Great landscape, no grid, and there's just so much hallucination that, like, even for Northeastern to Boston Commons, it would go to MFA… And the story is outputting your simplification. It's nice, I like it. You can really simplify the suggestion yet?
David Bau: Oh, yeah, so I'm encouraged to look at using, instrumented visualization representations of your space, so I forgot what the name of the paper was.
David Bau: In the framework is VO tier.
David Bau: to find out that basically they explored a similar, like, 3x3, griddle problem, Joey has griddle problems. They found out was that there's a lot of visual tiling.
David Bau: style setups that LMs are specifically not good with kind of visualizations, but they found out that, you know, applying, like, remedial tracking or some remedial, you know, some intermediate level tools.
David Bau: really, really helped carry it, so I might, like, qualitatively or categorically change to these things. Can I ask you to help me by finding that… that thing and maybe mailing it to them, or sending it on this quote or something? I have the paper with me. Okay, thank you so much. Thanks.
David Bau: I like the… I like the way you guys simplified it down. You started with a very, you know, broad, ambitious thing, and you're like, okay, well, you know, well, look at this simple thing, script it, keep your ambitious in mind, but… thanks.
David Bau: It came out when we started experimenting with big problems, so we did try with our goal, and we thought that it starts hallucinating a lot, so that we kind of nailed it down to something which is much more simpler, and we saw that, okay, something that does not hallucinate.
David Bau: And you can see that it allowed you to do a lot of experiments already and make very, really bad progress by making school.
David Bau: Really simple thing. The paper's called Mind's Eye of LLMs, Visualization.
David Bau: That was odd enough.
David Bau: One shine world. One shine world.
David Bau: Mine died.
David Bau: It is. It's already there. You're sending our…
David Bau: So, most of the interactions you're doing, actually, with, like, what the output of the model's gonna output needs, right? I'm just gonna say then, if the model is thinking about things that are not directional output, does the model know where it is? Does it know what its goal is? Does it know how close it is to the goal? Does it know what the thing progresses?
David Bau: If you frustrate the model, so, like, you're on a 3x3, maybe it's not very discernible, and the model's just critical, and you can add
David Bau: So, some of the suggestions were, oh, can you make this harder in some way? Well, if you make it harder in certain ways, you might be able to set up
David Bau: Obstacles, or set up alternate paths that if you follow that allow you to have more experimental
David Bau: questions that you can ask, so I think that that's a reasonable first thing to be doing right now, to see
David Bau: how far with the models you have accessible to you, how complex can you get the setup to work, where you can actually do things? Like you say, when you make a triple class, it's all hallucinations, you lose the ability to do experiments, but… but maybe that's a reasonable question now, because there's a lot of…
David Bau: Other questions that you could ask if it's a little bit more complicated.
David Bau: And does the model know
David Bau: where it is. Does the model know that something's stopping it, or is it in a way that it has to try again, or how close it is, or…
David Bau: Does the model get frustrated? You know, like, you… like, a kind of funny questions, but don't have to deal with it directly.
David Bau: North and South, right? And so, I'm interested in that.
David Bau: Kind of thing. So… but I think you guys are doing great. Now, we're probably running too short on time to let another present, is that right? How many things do we have left? Two things left?
David Bau: So we can do the same thing, we can have you guys, or are there things you really want to share with you now?
David Bau: Do you want to piss me now, I quote?
David Bau: Do I have anything to clarify that hunts?
David Bau: What? Yeah, we kind of have a lot. Are we down with the time? Yeah, we're done. Oh, we're done. Okay, so we're done.
David Bau: Okay, so, I said, you have to tell me, like, I was… Okay? We're not. We have time with the remaining 15 minutes for the night?
David Bau: And then we'll talk to you a little bit about, I think, the accomplishments… I think the accomplishments about how to do largely stepped in. Okay, sure.
David Bau: But otherwise, I could be out for, like, 12 minutes.
David Bau: Great, there's another, like… So, Marge.