David Bau: Great.
David Bau: And then, do we have a classroom camera?
David Bau: Okay?
David Bau: Great.
David Bau: I don't know how this works.
David Bau: Alright, who knows?
David Bau: Anybody who's online is welcome to just…
David Bau: Let me know if there's any, audio problems or anything like that. Actually, I'm gonna turn… I'm gonna leave the audio on my computer.
Nikhil Prakash: When… no, I think now we can't hear you.
Nikhil Prakash: No.
David Bau: Hey, Nikhil, can you hear me?
Nikhil Prakash: Yeah, now I can hear you.
David Bau: You weren't able to hear me before?
Nikhil Prakash: No.
David Bau: Okay, let me see if I can fix this.
David Bau: Audio…
David Bau: Echo-canceling speaker for… can you hear me now?
Nikhil Prakash: Yeah. Can you hear me?
David Bau: Room at all?
Nikhil Prakash: I can hear you now.
David Bau: You can hear me now, but what about…
Nikhil Prakash: No, no.
Nikhil Prakash: And now you're on mute.
David Bau: Okay, here, we'll use my computer audio, sorry about this, this is…
Nikhil Prakash: Okay.
David Bau: It's what it is.
Nikhil Prakash: Okay.
David Bau: Alright, who knows?
David Bau: All right.
David Bau: Nobody needs this chair, so I guess I bet somebody could benefit from this chair. We should grab that. Okay.
David Bau: Okay, welcome.
David Bau: Day 2.
David Bau: Week 1.
David Bau: If it's zero indexed, I guess.
David Bau: Alright.
David Bau: Oh, and we don't have a clicker here, so I'm tied to my laptop. Oh, no, but I'm not using this?
David Bau: I'm using this, what do I…
David Bau: Where do I plug it in?
David Bau: Is there a USB thing somewhere?
David Bau: Oh, no.
David Bau: What have I done? I've used the wrong computer.
David Bau: I have no clicker.
David Bau: Do you know where the computer is on this thing? It's just a… it's just a phantom screen.
David Bau: Is there a USB plug?
David Bau: Oh.
David Bau: This high technology is just too high for me.
David Bau: Okay, well, we're stuck on this slide.
David Bau: Welcome.
David Bau: Because I'm stuck in the corner here.
David Bau: So, yep.
David Bau: Okay, so today… We're gonna start the routine, so on Tuesdays, I'll talk about something in class.
David Bau: Oh, gosh.
David Bau: Yeah, we have a lot of auditors.
David Bau: Sorry about that. Okay.
David Bau: So, on Tuesday, I'll talk about something in class.
David Bau: And, we'll have, like, a little lab…
David Bau: Experience. Cool lab notebooks and stuff. And then on Thursday will be your presentations, your project presentations, so that'll be the routine. So, welcome to the first Tuesday!
David Bau: Okay. And,
David Bau: And so we're going to talk about the first experimental method I want to talk about today, but before we start.
David Bau: I just want to check in on everybody else. You have a homework assignment due Thursday to, you know, do a little one or two-page write-up of what you're
David Bau: project proposal is, you know, project-based homework assignment, so the four of you work on a team to make this. How's that all going?
David Bau: Any questions? So there were 3 questions that came up from last night that had to do with research. I just call on people, and this is what I'll do.
David Bau: Through the… through the day today. This is the secret reason for me to ask you to send questions, because then I can call on you to ask your question.
David Bau: So, this, it's… I have Kabalia here? Kavalia? Kabalia, yes. What was your question?
David Bau: I was, reading the three papers that were mentioned for BC. Yeah, yeah, the research papers, yeah.
David Bau: It just opened up several questions in my mind, like, what do authors mean by
David Bau: important questions, it's kind of contextual in nature, and, very much weighty presented there, but it might even mean, like, Nobel Prize-winning questions, or, like, even small questions, like, just finding out what the allowance last layer does. Yeah. So,
David Bau: for our time period of, like, 4 months of a semester, what's most feasible for us to… Well, you could… you could win an elbow prize. No, that's all right. That's so… I think that's what Hamming is suggesting, is that you guys should be ambitious.
David Bau: And, you know, he says, do top-shelf research, do high quality, the best that you can, right? There's no reason to compromise. I think that it's doubly true in our field, where
David Bau: I, you know, I don't think that people are in academia, in computer science.
David Bau: For the money?
David Bau: There's, there's, like, there's other ways of making money in our field.
David Bau: Right? And so, you know, I don't know why you're hanging out in academia, you want to make money, you go through some other things.
David Bau: And so, So, but, so while you're hanging out here, you know, why not…
David Bau: why not be as ambitious, intellectually ambitious as you can? So what is high-quality research? I… you know, it's a taste thing, but I think that, one rule of thumb that I like…
David Bau: It's not universal, but I think one thing I like to ask is, Is this something…
David Bau: That you would… Want to teach?
David Bau: People. Like, 10 years from now.
David Bau: When you're teaching this field, and you're going over the history, or you're going over the basics of the field, and you're saying, this is what you need to know, is this piece of research that you're doing right now
David Bau: Is this something that would be on that list?
David Bau: Is this one of the top things that you've got to cover? Because it's so important for people to know?
David Bau: Right? If it is, it's probably pretty important research.
David Bau: It's probably pretty important. If it's not, if you're like, yeah, I don't know why anybody would talk about this in the class.
David Bau: Then maybe you're doing something more incremental.
David Bau: Which is okay, 99% of stuff is more incremental, so it's hard to find
David Bau: something foundational, like the kind of stuff that Hamming talks about.
David Bau: But that's… to me, that's…
David Bau: what it means. There's probably other ways of talking about it. There's other research that you would never teach in a class, but it's so impactful for some reason.
David Bau: That, you know, changes the world. Like, I don't know.
David Bau: if you discover something about how to make computer chips, or something like this, well, maybe it's too specialized, and you don't actually want to teach it in a class, but it could… it could change the world, because computer chips are so important, or something like that. So there's other reasons that research might be important, but I like the… I like imagining
David Bau: What a future syllabus looks like.
David Bau: In 10 years. And if your thing is.
David Bau: At the beginning of the chapter, that is probably pretty important.
David Bau: Make sense?
David Bau: Okay, so… oh, Jasmine had another question.
David Bau: Something about small worlds, something like Jasmine.
David Bau: Do you want to ask what you were asking? What were your thoughts there? You had a few thoughts. I guess the first was, like, what makes, like, a task feel, like, easier to evaluate than, like, or what makes a task, like, valuable? Like, if someone asked me, like, did you feed my plants? Or, like, did you… did you feed, like.
David Bau: a baby or something like that. Like, it's kind of like, you did it or you didn't do it. And then there's other, like, kinds of, like, research questions where it's, like, much more, like, abstract, where it's, like.
David Bau: what's, like, a sufficient definition of beauty that just feel, like, very intractable for some reason? Yeah. So I guess, like, what are some, like, intuitions you have for identifying, like, a question that is, like, tractable? Like, is… is, like, things you can take action on, I guess. Yeah, it's such a good question. I don't know if… what do people… I wonder if anybody else… we have so many PhD students here in the class.
David Bau: Anybody have any advice for Jasmine?
David Bau: Any thoughts on… How, how to…
David Bau: So, you asked a couple questions, but let me ask the tractability question. So, like.
David Bau: How do you get a sense for whether… yeah, Natalie, I love your answers. Thank you. Me too. Thank you, thank you. So, you ask how to quantify beauty, and it reminds me of one of the research projects that I was involved in in 2021.
David Bau: That we tried to generate, automatically humor.
David Bau: And how can we quantify if something is funny?
David Bau: And we actually have a challenge with this, and especially in our field that expects to have a ground truth. And in humor, there is no ground truth. Oh, there's some things that are not funny.
David Bau: You're right. Actually, we use the not funny, the clearly not funny, to track that,
David Bau: human that did not do their work. Yes, but it was challenging.
David Bau: This is part of the research. The research question, how to evaluate.
David Bau: Things that are ambiguous.
David Bau: Yeah.
David Bau: Yeah, sometimes that's the core of the research. Aruna, did you have a thing to add here? I was thinking about how this probably seems intractable, because I think in STEM and in ML research, like, quantitative evaluations are valued so much. Yeah. And, like, some of these are just qualitative, you know, like, what is humor or what is beauty, and I think the way that people generally evaluate it is calibrating it with
David Bau: what humans think is beautiful, or what humans think is funny. Right. And, like, seeing if you can
David Bau: Some, you know, if you can say, use…
David Bau: Like, a language model or something to give you a quantitative score, like, say, along a scale or something, instead of, like, a binary choice, and then, like, actually calibrating it with…
David Bau: a qualitative evaluation that humans were seeing if, you know… This is such a good question, and I really like this calibration question. I, I, you know what, I'm gonna… in a couple classes, I'm gonna try to put in some material where we talk a little bit about LLM evaluations and things like that.
David Bau: Because I think it's so relevant. And we have a few… social science, PhDs.
David Bau: In the room, and who wrestle with these questions of
David Bau: human evaluation every day, and so in your project groups, I'd encourage you to, you know, talk with people and… and things, and we can… we can talk more about these kind of things on Thursday. Is… is Yi Qian here? Is it Yi Chan, or… Yeah. Yeah, Yi Chan, yeah.
David Bau: Yeah, my question is, like,
David Bau: when we, at the very beginning of our research, do we first, do a hypothesis, or do we, sort of, like, experiment with a, base model first? Oh, yeah.
David Bau: What comes first in research, the chicken or the egg? Well, so what do they teach you in 6th grade when they teach you the scientific method? What's supposed to come first?
David Bau: The hypothesis is supposed to come first, that's what they teach you, right? You think that's how all science is done?
David Bau: You start with a hypothesis, And then, no, look at all these shaking heads, all these… Always…
David Bau: battle-scarred PhD students.
David Bau: So, if we stuck with that same hypothesis that we had on our first day of starting a PhD, we'd be nowhere.
David Bau: Yeah, it's like this iterative process, right? And so, actually, I think that's a little bit of the theme of what we'll do for these first few weeks, is give you some tools for this iterative process. I like this question a lot.
David Bau: I… so I feel like, you know, the… especially the early phase of research, but actually you can all back up and look at all the research that we do.
David Bau: It's this iterative process where we incrementally ask questions, and then we collect a little bit of incremental evidence about those questions, then it leads us to ask better questions next, and then those better questions are more answerable, and then that might give us insight that leads us to ask better questions.
David Bau: And so, at the beginning, we know nothing.
David Bau: And that's actually sort of where we are in our current field, where we have these amazing AI systems, we know almost nothing about how they work, and so we are asking very, very basic questions about them. We have very, very rudimentary methods.
David Bau: So I have to apologize for our very rudimentary methods. I'm going to start teaching you some of the rudimentary methods. If you take this class again in 10 years, I'm sure there'll be much more sophisticated methods, but we'll, you know, we'll do what we can.
David Bau: And, and so I think that the tool here is an exploratory experiment. You say, hey, is this… like, I have some interesting question.
David Bau: Humor. I want to know if my AI is, you know, has a sense of humor.
David Bau: Right?
David Bau: Does it really… or is it just regurgitating jokes that it read on the internet?
David Bau: Or does it really have a sense of humor? What does that mean? That's one… I think that's an interesting question, you know, maybe… maybe I would teach that in a class, that'd be great. But I'm not ready to do scale-up experiments yet. I have to ask if it's feasible?
David Bau: Right? And then I don't really know if it's feasible, maybe I have some idea of, like, something… maybe I'll do a little exploratory experiment.
David Bau: And then I'll take a look around and see how that little explo- that really small-scale experiment, how'd that work out, and look at the results. Oh, were the results interesting?
David Bau: Am I asking the right interesting question? Oh yeah, if things are looking good, then I take the off-ramp, and I say, okay, well, maybe time to scale up my experiments, be serious about this one. But if not, then I adjust my question.
David Bau: I might say, well, maybe there's a more feasible way of asking the question I loop around. And that's the phase that you guys are in. So right now, I'm forcing you
David Bau: As his homework to write down.
David Bau: what you think is an interesting question, but then next week, the week after, I'll ask you to do a little bit of exploratory experiments.
David Bau: And then maybe adjust your question accordingly, and then we'll go around the cycle a little bit until you feel like you have things that you can scale up, and then we'll talk about methods you can use to scale up your experiments.
David Bau: That make sense?
David Bau: Okay, so today… We will talk about… oh, wrong computer.
David Bau: Well, there… we'll talk about the exploratory experiments… talk about exploratory experiment methods. So, here's my goals for today. So, I'm gonna do… I'm gonna try to cover four things in the time that we have.
David Bau: Who here knows what a neural network is?
David Bau: Okay, who here is not quite 100% sure? Okay, everybody's pretty good. You know, a few, a few one or two, right? I'm going to give you a whirlwind review of neural networks, just in, like, 4 slides.
David Bau: Right? So that we're all talking the same language, and give you guys an opportunity to ask specific questions, get us all on the same page. Neural nets, transformers.
David Bau: And then… and then I'm gonna go from that straight into the first interpretability method, the logic lens. You guys had a lot of questions about the various papers that we had, so we'll stop and we'll talk along the way.
David Bau: And then, hopefully, we have a good half of the class to do some hands-on setup. We'll, like, run through some notebooks, and we'll look at some experiment reproductions that you can do with the setup that we have.
David Bau: Okay?
David Bau: Sound good? Ready?
David Bau: Here we go.
David Bau: What's a neural network?
David Bau: It's an old idea.
David Bau: This is a… this is a picture from Frank Rosenblatt's 1958 perceptron.
David Bau: Paper?
David Bau: It's still how we do it.
David Bau: Still how we do it today. It's the same. There are some inputs. These are numbers. This is like a vector of, like, a thousand numbers or something like this. Inputs will say X for input. People use these conventional letters a lot when they're talking about
David Bau: neural networks, maybe that you have some interior
David Bau: neurons that are… that… that, like, process those somehow, and they… they produce some numbers, some other vector of numbers you call Z, and then… and then, like, maybe, you know, you… you have… you have to describe, like, the process of how you get from here to there, and that process from how you get to the… from here to there is that… that's…
David Bau: Those are the parameters of your model, of your neural network. So these are weights in a modern neural network, in a perceptron-style network.
David Bau: These neurons?
David Bau: are just… Weighted sums of these inputs.
David Bau: These numbers are just weighted sums.
David Bau: Mediated by some.
David Bau: some nonlinearity. Like, with simple nonlinearity, it would be, like, drop all the negatives, turn them, you know, clamp them to zero.
David Bau: Right, so this… this sigma is, like, the nonlinearity. You could have a few different nonlinearities that you use, but this is the common one. So this is… this is the typical setup. This is called the perceptron sub. It was invented by Frank Rosenblatt.
David Bau: Or it was invented by some folks that came before. Rosenblatt, he was the first one to implement it.
David Bau: In a physical system, and… and it works. So, actually, what… the way it works is you'll… you'll… you, like, have these things.
David Bau: Take linear combinations of those things and plant the negatives.
David Bau: And then… and then you'll… and then you'll stack it up?
David Bau: With another layer of neurons. So, like, here's another layer of neurons. Why?
David Bau: And we have parameters that describe how to take linear combinations of Z to make Y, and we do the same thing, and we can stack them up as much as we want. If you stack it up deeper than about 3,
David Bau: then people… then Jeff Hinton will say that you're using a deep neural network, which sounds like deep thought, and it's a very good marketing term, and it also happens to be
David Bau: deeper than 2. But the interesting thing is that, you know, even if it's deep, you can train these systems by gradually adjusting the weights.
David Bau: to match, examples by reducing error, and so that whole process, gradient descent, could take all the classic machine learning classes on doing this. We're not going to do that in this class today.
David Bau: But, you know, so… but there's a very straightforward training procedure that tends to work if you apply the right tricks. And…
David Bau: And sometimes we'll take this, I don't know what happened to my fonts here.
David Bau: We'll take this whole, you know, details, and we'll just abstract that, and we'll say, hey.
David Bau: Z is just a function of X, mediated by its parameters.
David Bau: Right? And then Y is just a function of Z, just mediated by its parameters. And the reason that we will write that is because in 1989, Cybank approved
David Bau: that if you have two layers of Rosenblatt-style perceptrons or deeper, then it's what's called a universal function approximator. If the middle layer has enough neurons.
David Bau: Then you can get arbitrarily close to any function
David Bau: Like, Y can be any function of X with the appropriate settings of the parameters, if you have enough middle neurons. So…
David Bau: So, like, any function that you can imagine, any function in the universe, pretty much, right, you can… you can express as a neural network. So, that's why we're stuck with this funny old 1958 architecture, because it has this power.
David Bau: And, and this is also why some people will just write a neural network as F.
David Bau: Because they, they, they just drop it in as a universal function approximator.
David Bau: Okay, so that's the background. So that's…
David Bau: That's about 4 weeks of neural networks in a slide.
David Bau: Okay.
David Bau: So then… oh, and then when you have two layers.
David Bau: or more, then we call that an MLP. Actually, two layers is enough, says some rentals. You'll often see MLPs which are search two layers, but multi-layer is the general thing of what people say. So whenever you see MLP, that means, oh, you've got, like, a two-layer thing of neural networks.
David Bau: Okay?
David Bau: Alright, and that means you have an arbitrary function there that's learnable.
David Bau: Alright, so second thing to, to, to, review is what the heck is a language model?
David Bau: Okay, so a language model is…
David Bau: a way of… well, it's a way of modeling language, I guess. So,
David Bau: let's see. I'll just sort of skip to the modern formulation of what people use for a language model, and say, you know, if you have a language model.
David Bau: What you really have is you have a function.
David Bau: That tells you what the probability of the next word is gonna be.
David Bau: That makes sense? Now, that probability is not unconditional. I can't just tell you that V is the most common word, and, you know, the probability you're going to say V is, like, 2% or something like that, because that's not interesting. It's a contextual probability. Like, I'll tell you that you'll say V, or you'll say hyphen, or something like that.
David Bau: with a certain probability conditioned on what came before. And so a language model conditions its probability prediction on all the words that you said before. So that's what a language model is. So that's, that's, like, physically, that's what it is.
David Bau: It's a function that tells you what the probabilities are of the next word.
David Bau: Given the context of where you are in the text.
David Bau: That makes sense? And people have used language models for a long time for all sorts of things. People traditionally use language models to, like, smooth out computer-generated text to make it more fluent. They say, you know, if you want text to be a little bit more fluent, then you should match the statistics of
David Bau: of text, and they use these as statistical modelers. And the big surprise over the last
David Bau: few years has been that the… these language models are not just, like, statistical smoothers that make, like, language more fluent.
David Bau: But these are the things that are, like, getting up and talking and thinking and, like, programming for us and stuff like this. It's very weird. So the focus of our class is going to be these large-scale language models, which have very, very good models of language and behavior, but they all have this form. Give us some context.
David Bau: Give me some probabilities. So, there's a couple details here.
David Bau: The context is always provided.
David Bau: The naive way to think of it is that context is provided in words.
David Bau: And the output is provided in probabilities of the next word. That's what you'll say to all the laypeople. Of course, the detail is, it's not
David Bau: words, strictly speaking. It's tokens.
David Bau: So, there's a process called tokenization.
David Bau: Let's see this self-referential example here.
David Bau: There's a process called tokenization.
David Bau: And the way it works is, if you have a really uncommon word like tokenization, it might be broken up into littler pieces.
David Bau: So here is, like, a process called token, and you feed that to the language model, and it'll tell you that 99% of the time, or whatever, 90% of the time.
David Bau: what you really need to say next isization, the second half of the word as the next token. But then there's some other things, like tokenizing, you know, maybe that's sort of 9% chance you might say that, and then maybe there's a hyphen, the sort of 1% chance to say a hyphen, or something like that.
David Bau: And so…
David Bau: So this is sort of how the language modeling goes. And so what you're doing when you run the language model is… oh, wrong computer… is you're making a prediction for what you think is going to come next.
David Bau: You're never 100% sure, so it's always a probability distribution. And then, if you actually want to talk to this language model, if you want to do some dialogue, then you have to do sampling.
David Bau: to… to actually follow through on that prediction, pick one of the next words, say this is… if the language model we're speaking, it would say is Asian, because I guess 90% chance, right? Or you could roll the dice, and maybe 10% of the time you say it is icing or something like that. So there's different sampling methods.
David Bau: And then after you sample, you just take the token, you just jam it into the context, and then you run it again, and then there's this iterative process where you just keep on running the language model over and over, and then it just generates this beautiful, fluent text and has it valid with you.
David Bau: And and then, you know, does your computer programming for you, or whatever.
David Bau: Isn't that crazy? It's crazy. It's crazy that this works at all.
David Bau: But that's how… that's what a language model is.
David Bau: Any questions? Oh, yes, actually, Ayush had a question about language models. I forget what the question was. Is Ayush here?
David Bau: Yes, Ayush, what was your question?
David Bau: Yes. And, currently, like, the management will be on the past 11, but there must be something that's going on.
David Bau: by tokenizing, let's say, project-centric, connect. Yes, yes, yes, yes, yes. So, this whole idea of tokenizing text.
David Bau: Came from the text…
David Bau: arena. Like, you know, what the text modelers really wanted to do is they wanted to have one word.
David Bau: for input, except that words are so weird, people misspell words, and there's very obscure words, and there's little phrases, like, that maybe you should treat the whole phrase as a word, and there's all sorts of weird situations, so that's when they said, okay, you know what?
David Bau: we're not going to stick to words, we're going to introduce tokens. Tokens are going to be some chunking of text that is just, like, statistically useful, and we'll use tokens instead of words. And then after they introduce that.
David Bau: then people coming up and wanting to use this class of models on other types of data, like images, or sound, or physical experiences, or things like that, they said, how do we, you know, these language models are working so well for the NLP people, can we use these things
David Bau: on other types of data? And the answer is yes, absolutely. You can break up images into tokens.
David Bau: Right? You can break up
David Bau: physical data into tokens. You can break up biological data into tokens. There's different ways of doing it, like you could say, you could represent graphs as tokens, you can represent other things as tokens, you could… you can quantize, you can quantize, image data in a bunch of different ways.
David Bau: Or you can have continuous tokens that aren't even quantized, things like that. So there's a lot of different techniques.
David Bau: I'm not gonna go into them.
David Bau: But, but, you know, there's… but there's… but the general… Idea is… That, at least with current
David Bau: neural network architectures, there's some overhead for processing
David Bau: like a token. And so, you want the tokens to be chunky enough
David Bau: That when you spend all that electricity to process one token, you make some appreciable like, progress.
David Bau: In your… in your… in your computation. So, if you make the tokens too small, it tends to use too much electricity. If you make the tokens too big, it causes other problems, because they're too coarse, to capture the details.
David Bau: And so… so there's some trade-off of, like, how you tokenize things. And so the image people, probably debate a lot about how big a token should be. Should it represent a little patch of an image, or a big patch of an image, or how should we do it?
David Bau: But we won't get into those things. Take a regular machine learning class on computer vision to see those debates. That make sense? So in this class, we'll focus on language. Some of the tokens will be approximately a word.
David Bau: Sometimes a little more chunky, sometimes less chunky than a word.
David Bau: Make sense?
David Bau: Okay, so, next thing.
David Bau: All right, so
David Bau: That's a general language model, that's a general neural network, and so what is everybody using now? They're using these transformers, so who here is very familiar with, like, what a transformer is?
David Bau: Okay, great. And who's, who's, like, could use a refresher?
David Bau: Okay, a few people. So, what was covering this?
David Bau: Alright.
David Bau: So, this is what the neural network architecture of a transformer looks like.
David Bau: The input comes in on the left.
David Bau: And the output comes out on the right.
David Bau: It is a language model. So the input is all these tokens for the context, for all the words that came before.
David Bau: And the output, even though I wrote this as a single word output, it's actually… the output is this probability distribution telling you what the likelihood is of all the next tokens is going to be. That make sense? And so, it has three phases.
David Bau: Right? First.
David Bau: to have a neural network, you need neurons, and so the neurons all have to be set to numbers. It's a numerical network, so these tokens are no good, so you need to encode them, so basically you take every token, and you turn it into a vector of numbers.
David Bau: for all, like, the neural inputs that are going to be provided on the input side. So, miles, Davis, each one of these things
David Bau: turns into a whole bunch of numbers, about a thousand numbers. So there's, like, a thousand numbers that correspond to miles, and there's a different thousand numbers that correspond to David's, and so on, right? And then each of those numbers gets fed into the input of the neural network. So the… the… how do you get those numbers?
David Bau: It's called the encoder layer. It's literally a lookup table. There's a vocabulary of something like 50,000 or 100,000, tokens, depending on
David Bau: what vocabulary you're using for your neural network, and the encoder is just a list of 50,000 vectors of a thousand numbers. It's like a big matrix of 50,000 by 1,000.
David Bau: Where you just say, oh, Davis is the 300th line in the vocabulary. I'll go look at the 300th line of this big dictionary, this matrix, and that tells me what thousand numbers to write down. I light up these thousand numbers, and that
David Bau: that… that form, and then that gets the network started. That makes sense? So that's the first phase.
David Bau: is we… we do… we do that, and then there's a series of neural network layers that are… it's very deep. It's like, you know, hundreds of layers deep, not just two, but there's many, many neural network layers that take these inputs, and they mix them in some clever ways, which I'll talk about in a minute.
David Bau: Right? They transform all these vectors.
David Bau: all the way out to a similar… so, like, if you had 4 vectors, each corresponding to a token coming in, then you had the same number of neuron outputs on the output side of a transformer. So a transformer is rectangular.
David Bau: It takes an arbitrary number of tokens on the input, and then it outputs, like, the same number of vector outputs on the output side. So each one of these little purple dots represents
David Bau: A vector of about a thousand… thousand neurons, 1,000 numbers.
David Bau: That makes sense. So I'll talk about this internal architecture in a minute, but it just does all this stuff. After it goes through several rounds of this computation, you get to the end, and then…
David Bau: you decode it back out into tokens again. So you have this thousand numbers, and then it goes through a little neural network layer that turns these thousand numbers into
David Bau: 50,000 probabilities of what the probabilities for each possible next open is. And that's… that's just another matrix, just a linear operation to do that, but then we use a softmax to make sure that all the output numbers
David Bau: Add up to 1.
David Bau: So… so, but there is just an output layer. Okay, so that's the overall… it's, like, just three phases, that's what a transformer is. I'll talk about a couple things in details in a minute. Any questions at this stage? Oh, hi, Yuki? And…
David Bau: Ananya, how you?
David Bau: Yes. Yeah, so my question is, so as an encoder, so we convert, vocabulary to, embedded? Yes. We have some sort of operation at,
David Bau: very meaningful, representation. Yes, yes. For the next step generation. Yes. My question is, can we just leverage that embedding?
David Bau: I mean, without converting into a token, just to set as an embedding for the next, but for the next version.
David Bau: So, let's see. There's, there's some… like, I think the adapting from…
David Bau: the token vocabulary to the embedding. I'm not sure if this answers your question, so ask again if I'm not quite hitting it.
David Bau: But some of these numbers are adjusted to make them line up with what worked well in practice. So,
David Bau: So what ends up working well in practice is token vocabulary, somewhere around 100,000. So there's a couple thousand words in English that are commonly used words in everyday speech.
David Bau: But then there's a lot of languages to handle, and there's a lot of technical texts, and a lot of other things, and so people for a long time used token vocabularies around 50,000.
David Bau: Right? To capture a lot of this subtlety. And then they realized, oh, actually, they can get better performance out of the network if they go up to about 100,000, 150,000 token vocabulary. So you'll see this pretty large.
David Bau: Token vocabularies, but… and that… but then… and the natural way to represent a 50,000-way choice in a neural network
David Bau: Is to have 50,000 neurons, All of them off.
David Bau: Except for the one that's your choice. So, we call that a one-hot encoding, with all the neurons off, except for the one thing.
David Bau: But it turns out that's kind of inefficient for neural networks. It's not… it's not dense enough. As a computation, it wastes a lot of electricity to do that. Like, the widest that a neural network seems to perform with modern hardware and modern architectures is around 1,000, something like 1,000, or 5,000, or 10,000.
David Bau: neurons wide, one of these representations. So you'll see it's pretty frequent to have a thousand or 10,000 neurons working as a bank.
David Bau: As a vector, as a neural representation.
David Bau: Instead of, like, 50,000 or 100,000. So… so one way of thinking of what an encoder is doing is it's… it's taking one…
David Bau: Numeracy, like, the number that people thought was, like, a good engineering number on one side, the vocabulary side is about 100,000.
David Bau: And it's reducing it down to some other numeracy where some other engineers are saying, oh, no, this is about the right dimensionality for this other problem, which is, like, 10,000. So, like, this tenfold decrease in size means that
David Bau: That, you know, we're taking this pretty large vocabulary and we're, like, compacting it down.
David Bau: To, to, to, you know, vector choices in it.
David Bau: smaller.
David Bau: a smaller…
David Bau: dimensional space, 10,000 dimensions is still a lot. I don't know, did that sort of answer your question? Not really.
David Bau: So… So, I mean, as a final…
David Bau: Yeah, sorry. Yeah, we have that embedded. Yeah, so it's, like, 10,000 numbers, or 1,000 numbers. Let's say it's 1,000. There's, like, a thousand numbers here.
David Bau: So, can we just use it as a support for the generating expert?
David Bau: Oh, can you generate a next word straight from this thousand? No, I mean, for the next…
David Bau: Oh, like, could you take this thousand, and then just feed it in here for the thing?
David Bau: Yeah, yeah, yeah, could you do that?
David Bau: Could you do that? Yeah, I don't know. I think that there's something…
David Bau: It's… it's a… I think it's a good question.
David Bau: It's a good question. So, if you do that.
David Bau: You're not making any choices for the network. You're not, like, collapsing
David Bau: its probability distribution down to a choice. You're just saying, hey, you know what? I'll take this blurry probability distribution you have, and I'll just take this blur thing, and I'll just feed it in as the input for the next thing. Now, obviously, the networks are not trained to do that, so physically, you can't do that, but you could conceivably try to train a network
David Bau: to do that. Why… why don't we do that?
David Bau: My guess is that There's something special about, actually, the sampling procedure.
David Bau: About actually forcing the network to make a choice. It says, oh, these two things are equally likely.
David Bau: Well…
David Bau: like, we need some way of encoding the fact that in the real world, you're going to do one of those things. You can't continue to think about both of those things on an equal basis.
David Bau: forever. You've got to sort of make an observation of the thing that you're going to actually do. You're going to have to utter this word, and make a choice, and then go on. And so I think that there's… so… so I think that's actually important for these language models to work well, but I think it's a really reasonable question, and you could do research to prove me wrong.
David Bau: Right. Yes? Yeah, I'm not a social scientist, so I apologize for… Sure. Yeah, it's okay. Yes. This is a decoder-only transformer, so, okay, so there's,
David Bau: All right. So, historically, the way that transformers were invented, there was a Faswani paper called Attention is All You Need. Who is familiar with that paper title? Oh my gosh, what a paper title. Attention is All You Need.
David Bau: I don't know. But then they had several things in that paper, so they said that this general architecture
David Bau: they set it up in two ways. They set it up as an encoder architecture, and they also set it up as a decoder-only architecture. So,
David Bau: Let me see if I can digest what the difference is for you really briefly.
David Bau: So that… Idea is… that… The encoder's job is just to read text.
David Bau: It doesn't need to write text.
David Bau: So, it never predicts the next word.
David Bau: It just tells you what the words that you have.
David Bau: meet. And the way that you set up a transformer
David Bau: If it was set up as an encoder.
David Bau: Is that this little vector, which captures some meaning of, like, what this word means.
David Bau: By the time you get to the end.
David Bau: This vector still is talking about that same word.
David Bau: Right? Except that it has a lot more context.
David Bau: And so by the time you get to the end, these vectors that you get to the end have a really meaningful representation of what that word means in context, and what the whole context means, and everything.
David Bau: Right?
David Bau: And… and what do you use those vectors for? Well, you could use them for a lot of other things. You could use them for
David Bau: you know, classifier for drawing an image, or something like that. Like, so, like, if you… if you want to, draw a picture from a piece of text, it would be very reasonable to take, like, a text encoder.
David Bau: and then take the output of the text encoder and then feed that into an image generator, and the image generator would understand all the subtleties of language that you use to generate its image, right? So that's what an encoder architecture is. But it doesn't need to speak. It doesn't need to generate text.
David Bau: What a decoder architecture is, is it shifts that role
David Bau: Buy one, and changes the job.
David Bau: So decoder architecture is very weird, because
David Bau: Even though this is representing the word the at the beginning, by the time you get to the end.
David Bau: what this job is, what the network is trained to do, is it's trained to tell you what it thinks the next word is going to be. So, you know, the next word, you know, it belongs down here, strictly speaking, right? The representation of the next word would be sort of the next neural line, except that if you have to speak
David Bau: If you have to be a language model and predict the next word.
David Bau: you can't input the next word over here and have a line here, right? Right, so what they do is they just shift all the roles by one.
David Bau: And they say, if you inputted one word, then your job is
David Bau: On that neural line to predict the next word.
David Bau: Does that make sense? And so this, this shifted thing
David Bau: Where you can't… you don't get to see the future.
David Bau: Right? Is called… an output only Transformer.
David Bau: And… and so you have these input… you have these encoder transformers that can only read.
David Bau: And you have these output transformers that aren't really designed to read. Of course, one of the secrets is they do read. They read themselves, they read their former past.
David Bau: But they're completely trained to say the next thing, that's their job, right?
David Bau: And… and in the original Vaswani paper.
David Bau: They had this funny hybrid architecture where half of the neural network was an encoder, That only read text?
David Bau: And then half of the neural network was a decoder that only uttered
David Bau: text, and ran… this is called autoregressively. After you pick a word, then you feed it back into the input, and then you run it again to, like, get the next word. So this… this… this loop is just called the autoregressive loop. And,
David Bau: And so they had this reader and this writer, and they said, when we've set up a reader to read French.
David Bau: And we set up a writer to write English, and we hook them up in a certain way.
David Bau: Then we can get better Natural language translation results than have ever been achieved in history.
David Bau: Therefore, attention is all you need. Mic drop.
David Bau: Right? And so… and why… why was that such a moment? It's because natural language translation is one of the oldest problems in computer science. A lot of the original funding for computer science in the 1950s and the 1960s
David Bau: came from military sources where they're like, you know, I think that World War II would have worked out a lot better if we could read German.
David Bau: or Japanese.
David Bau: better, we should have computerized natural language translation systems, and people, like, made these things, and none of them worked very well. It was… it's been this holy grail for decades, and… and then, you know, we all live in a world where there's been Google Translate for a while, which has been considered this amazing, you know, engineering triumph, but actually.
David Bau: These attention-is-all-you-need transformers outperformed everything that the statistical machine language, translation people did.
David Bau: By quite a big margin.
David Bau: And so… so that's why.
David Bau: Like, they invented the thing.
David Bau: Little did they know, This architecture would take over the entire field.
David Bau: And, sort of, you know, encapsulate, like, basically every cognitive task.
David Bau: That we set in front of them.
David Bau: That make sense?
David Bau: Okay, so that's… so that's the answer. And… and the other… the other historical note is that
David Bau: Google invented this encoder-decoder architecture that was from their paper, so they had a lot of encoders. They actually said, hey, you know, for a search engine, which is all about reading the internet, we just need encoder-only architecture, so they threw away the part that speaks, and they only used the encoders to go index the internet, and they tried to get everybody to study these and make them better, and they got a lot of academics excited about making these systems like BERT,
David Bau: or whatever, right? That they're encoder-only things. They're read-only neural networks.
David Bau: Oh.
David Bau: And then, and then the OpenAI folks were like, no, we want to… we want to, like, pass the training test. We want… we want you to have a dialogue.
David Bau: with our system, we wanted to talk, and so Alex Radford and folks over there said, we're going to make the opposite choice. We're going to throw away the encoder, and we will just focus on getting the engineering working for a decoder-only thing that can speak without focusing on the reading part.
David Bau: Right? That makes sense. So that's the historical note. So… so there was this split between these two AI labs, what their focus was, and the OpenAI guys made these decoder… they're really large decoder-only systems, and that's… and that's how they released ChatGPT.
David Bau: And, and then ever since then, people are like, you know what, we don't need these encoder things. The decoder things can also read, they read themselves. Yes, go ahead.
David Bau: Yes.
David Bau: Yes, that's right. Here's if most of the techniques covered in this class are agnostic to whether it's literary.
David Bau: So, like, some of them, there's a mix. Some of them will be work… the work on both, and some of them are really very focused on the decoder architecture. So, there's a funny thing, which is…
David Bau: So, I suspect that there are certain aspects of the decoder-only architectures which are more amenable to interpretation.
David Bau: than the encoder architectures. Decoder architectures are forced to do something which humans have to do all the time. They are forced to make decisions constantly.
David Bau: so this thing that I was trying to make a big deal about, like, there's going to be a sampling thing. You off with this distribution, and then at that moment, like.
David Bau: the sampler is going to make a choice. It's going to say that you chose this word, you're going to say trumpet. There might be another word that you could have said, you could have said on, but you're not allowed to think about both these things at once. You're going to say one of them, and you're going to, you know, put that back autoregressively, and you're going to continue. So this constant making of choices, this bottleneck moment.
David Bau: of the present, where you're going to do one thing in the future, is… is this… is this thing that the autoregressive, output-only neural networks are forced to do, and it feels… like, my intuition is it feels very human-like. It's just, like, the choices that we constantly face as humans of, like, what is the next action we're going to take?
David Bau: And as a result, the internal neural pathways inside these autoregressive, like, these speaking output-only models, like, feels like they're organized in a way that's kind of comprehensible.
David Bau: to us.
David Bau: They seem to make decisions, they seem to try to collect together the data. They need to make a decision and make the decision.
David Bau: Whereas encoder-only architectures, Are a lot more diffuse.
David Bau: they never have to make a decision, they're just trying to get this gestalt.
David Bau: And and and so it can be harder to get intuition about what they're doing inside when you look inside.
David Bau: That make sense?
David Bau: So, right.
David Bau: So, but, some of the methods will work both ways.
David Bau: But you'll see some methods that are very focused on this.
David Bau: And this is a debate, because a lot of people are like, these encoder-only architectures, they should be more powerful than the decoder ones.
David Bau: But I don't think that that's been shown to be true. I think people have studied the decoder-only architectures, and they've found that they're incredibly powerful.
David Bau: And there are tricks that you can use. So some of the folks here are in labs where they
David Bau: Like, use… they train, decoder-only models on, like, computer code and things like that, or editing tasks where, you know, when you're editing a document, you don't edit things always at the end, you might edit things in the middle, but there's ways of getting
David Bau: the…
David Bau: autoregressive architecture to work on things out of order. Think of the way that you work with an editor, like a word processor. Like, you'll scroll up, and you move the mouse, and you click on things. You know, you have, like, this forward-in-time sequence of things that you do. You can train an autoregressive model to
David Bau: Do these types of things, and…
David Bau: Arguably, it might even be smarter, then.
David Bau: One of these encoder.
David Bau: Awesome. Make sense?
David Bau: Okay.
David Bau: So, oh, that's pretty deep. How about Ananya? Did you have a question?
Ananya Malik: Yeah, hi, I'm actually online.
David Bau: Hi, nice to see.
Ananya Malik: Hi.
Ananya Malik: Yeah, yeah, I had a question. I guess I was having a difficult time forming an intuition why, like, features encoded within the representation spaces may be linearly separable.
David Bau: Yes.
Ananya Malik: Like, I would imagine, like, yeah, sorry.
David Bau: Yes.
David Bau: So… so you're asking a question that is, you know, has puzzled machine learning scientists for a while. Like, what makes a deep neural network effective? When you make it deeper, what's the… what's the benefit of depth?
David Bau: And, and so I'm not gonna give you a total answer, but the intuition is that
David Bau: It's sort of like it organizes information better and better the deeper you go, and it represents better and better useful abstractions. So, at the very shallow layers.
David Bau: You have a word like plays.
David Bau: That is just… it can only be represented as the word plays. Like, all… the first token that comes out, the first vector that comes out of the word plays here, just means that the word plays appeared in the text. But we have no idea if that's a verb.
David Bau: or a noun.
David Bau: Or, you know, in one… is it an instrument is being played, which is what it means here? Or is it a child who's playing? Is it a… is it a… is it an act? A stage act, a play? Right? Is it… there's so many different ways that, like, the word play could be represented. And in the very first layer.
David Bau: There's no linear separability, there's no way of looking at that vector and telling which meaning you're talking about.
David Bau: Right? There's no… there's no, like… if you… if you try to separate childhood concepts from adult concepts or something, like, you can't separate plays
David Bau: At that layer. But as you go deeper into the network, then you might be able to separate all the childhood concepts from all the adult concepts, even though plays is this hominin. It, like, it's both. But by the time you get deeper into the network.
David Bau: you know, it may have separated those two types of meaning into two different parts of the vector space. So that's a little bit of the intuition, but, you know, there's a lot of deeper theory that goes
David Bau: do that. Is that… does that help at all?
Ananya Malik: Kind of, yeah, I think it helps with understanding. I do have other questions, but I can definitely, like, if you have any links to read up on this.
David Bau: Okay, I'm gonna keep on going.
David Bau: Yeah. Okay, thank you. We're gonna… we're gonna… we're gonna forge ahead. There's much to talk about.
David Bau: One of the… one of the funny things about transformers is because of the way they're trained.
David Bau: when you're using them autoregressively as a language model, you just need to know the probability distribution that's coming out of this last thing. But because they're trained in parallel, you actually get
David Bau: a probability distribution, like, at every single token. So, even though it, like, if you say miles, then it says, oh, you know what? I think that the next word is per, like, miles per gallon, I guess, right?
David Bau: you know, even though the next word… it knows the next word is Davis, because you just fed it, right? But… but, like, you know, this neuron doesn't get to see what the next word is. It… it was forced to make the…
David Bau: guess for what comes next, and you can read these guesses out. It's a little bit funny, you can see how the evolution of all, you know, of this incremental
David Bau: predictions are going. And all the transformers now, they all do this today, all these autograph things. And that's why, people had this intuition that, oh, it's really wasteful, like, why am I making all these irrelevant, you know, predictions that I don't need to make? Like, I'm wasting all these neurons on doing this. It must not be as good as
David Bau: Like, an encoder-only model that doesn't do this type of stuff.
David Bau: But I don't think that the science backs this up. I think the science has, you know, been quite happy with these decoder-only models. They perform very well, maybe even better than the encoder ones.
David Bau: Even though they're doing this weird, useless thing.
David Bau: Okay.
David Bau: So… Important transformer pieces to know. We talked about the encoder.
David Bau: We talked about the decoder, and then there's two main other pieces.
David Bau: There's attention, which is all you need, they say.
David Bau: Right? And what it is, it's a kind of neural network architecture, it's like a way of hooking up neurons that remembers recent information from other tokens.
David Bau: Right? So, it's these red lines that reach back to previous tokens in the context, and
David Bau: And, and it… there's this… there's all this detail of how attention works. It makes this query vector, matches key vectors, and then uses the query key matches to recognize
David Bau: Which previous tokens were the most important, and then it gathers vector contributions from those most important ones together, and then brings all that information forward.
David Bau: to now. So that's what intention is. We… if people are interested, we can talk about that more. And then MLP is what we… this is… this is Frank Rosenblatt's architecture. It's just two-layer neural networks that match and modify single-token feature vectors. They just have some input neurons, they… they put them through some neural network magic.
David Bau: To get a universal function approximator, you know, and then they output
David Bau: something to add into the… add into the, add into the vector. So… so… so attention, it turns out, is not all you need. You also need MLPs.
David Bau: Right, there are actually more MLP neurons and weights in a transformer than there are attentions, so it was a lie, I'm sorry.
David Bau: You need MLPs also. And then, other than that, though, there's very little. You have encoders and decoders, there's some normalization layers that just keep everything from exploding, you know, keep everything normalized to about size 1, and that's… that's it. That's pretty much it inside a transformer.
David Bau: And so the other… there's another trick for… oh yeah, so I guess what I… the way I think of it is…
David Bau: I think of attention as… like, I put this little thing there that's not quite science, so it's diagonal. I think of attention as the implementation of short-term memory.
David Bau: Amy thinks.
David Bau: So, if you go to a person and you say.
David Bau: What kind of memory did I just talk about?
David Bau: You would say, short-term memory.
David Bau: And then, how do you remember that I said something about short-term memory just a second ago?
David Bau: Is it because… You rewired… when I said short-term memory, did you rewire your neurons inside your brain?
David Bau: to, like, store this memory that I said short-term memory just a second ago, so that you could regurgitate this when I ask you, what kind of memory am I talking about?
David Bau: I doubt it. There's actually a lot of neuroscientists who have been going around peddling this idea that, yeah, as soon as I utter short-term memory, that the neurons are instantly wiring themselves up so that you remember this instance, how short-term memory works. I don't believe it at all.
David Bau: I think short-term memory is just… Transient electrical signals.
David Bau: They're circulating around in your brain that you can regurgitate and remember something without having to make new neural connections, and that's what attention is.
David Bau: Attention is a way of remembering
David Bau: what happened recently by looking at the context from all of the neural activations, from all of these numbers floating around, without rewiring permanently the
David Bau: the architecture of the computer. Now, the reason attention works so well is because it sort of has this temporary rewiring that does this query key business where it says, hey, I'm going to only pay attention to certain neurons that are the most relevant, whatever, right? It's sort of a way of it
David Bau: sort of pretending that it has a neural connection over there, just for a moment. Like, I'm gonna run a query and then listen to specific neurons.
David Bau: And,
David Bau: And so, but, that's not a permanent connection, like, that goes away as soon as the context changes.
David Bau: And so… so I think of attention as short-term memory, and then I think of MLP as long-term memory. There's a lot of parameters. The bulk of parameters, most of the parameters in a transformer are in MLP layers. And where do these parameters come from, these weights and these biases? They come from all this gradient descent training from, like, you're exposed to all this data, and you try to match it.
David Bau: And then… and then most of that data turns into encodings.
David Bau: of weights in these MLPs, and so when you remember something like the Eiffel Towers in Paris.
David Bau: It's not because it was mentioned recently, it's because we remember this association of Eiffel Tower in Paris, and then stored in an MOP somewhere.
David Bau: I mean, essentially, I think of MLP as long-term memory.
David Bau: Alright, not very scientific.
David Bau: You know, some neuroscientist is gonna kill me.
David Bau: Okay, so… but… but that's… that's the way I think of it.
David Bau: Okay.
David Bau: Yeah, so there's… so I won't talk about the details of self-attention, but unless… oh, oh, we… but we do have a question from Luzi, which is why I put this slide in here. Is Luz… Luz or Lucy here?
David Bau: No. Okay, that's okay. Maybe somebody came in the first class and not here. That's all right. So there's just a question about how attention works, and I was going to use this slide to answer it. But that's all right. So, but if you want to know more about attention, there's lots of YouTube videos and things like that.
David Bau: So, I encourage you to read them, or watch them. There's a concept that you need to know to understand how logic lens works, which is… it's called the residual stream.
David Bau: So it's a very clever way of hooking up deep neural networks.
David Bau: So instead of taking X
David Bau: the Y, the Z, you know, whatever, right? And then taking the output of one neural network layer, and then making that the input of the next neural network layer, and then just feed… stacking them like this. What we actually do is we use what's called a residual architecture.
David Bau: Which is very weird. So, okay, so in a transformer, you do have layers.
David Bau: like in… like in the transformers we'll play with in the lab today, there's, like, 80 layers.
David Bau: Right, a lot of layers.
David Bau: Right? And each layer has some input, which is like a vector, like, for a given token, stream.
David Bau: The… it'll have, like, a layer input, which is, like, a thousand neurons, or 10,000.
David Bau: And… and a layer output, which is the same number of neurons, right? And then there'll be a little attention thing and an MLP thing that go… go between them. But look at the way that I drew MLP and attention here.
David Bau: They're, they're, like, to the side.
David Bau: Because what's going on is, every layer is not taking this input and then producing this output. Every layer's calculating what's called a small residual vector.
David Bau: To add to the spin.
David Bau: Does that make sense? So, if I ripped out these neural network components, and I just zeroed them all out computationally.
David Bau: Then, this vector would still pass through to this one and be exactly the same.
David Bau: So that's called the residual stream.
David Bau: So, without any special neural network connections, the vectors just remain the same.
David Bau: And so the bias for a transformer is for it to gradually transform vectors from layer to layer.
David Bau: you know, sort of slowly, and add little residuals at each layer. And what that means is that if we want to know
David Bau: What the deal with this vector is, and how to interpret it.
David Bau: One clue is, if we already know how to interpret this vector, it's probably not that different.
David Bau: Like, most of, like, what you need to do to understand this vector is, like, is the same as what you have to do to understand this vector.
David Bau: Except that these components might have added something.
David Bau: So there's just a little bit of a gap, and so you could… you could try to understand what these components are doing, but they typically will add, like, less than 10%
David Bau: Of, like, vector-ness, vector size, or something like that, to the vector.
David Bau: Right? It's just that gradually changes it from layer to layer.
David Bau: Make sense? So that's a… so this is called the residual stream architecture.
David Bau: And it means that it has some interesting implications. So, 2016 and 2020, people looked at these residual network architectures, and they realized that there was this cool thing that you could do called early exit decoding.
David Bau: So, early access… so, it takes a lot of electricity to run one of these things, and time.
David Bau: And… and what people notice is that after you go through, like, 70 layers, and you've got 10 more layers to go.
David Bau: You know, when you get to the last thing, you're gonna decode this thing, and it's gonna be a private distribution, it's gonna say trumpet.
David Bau: But these last bunch of, like, vectors are pretty similar to each other, often.
David Bau: They're pretty similar. So what Panda and Avaya, you know, proposed was maybe we can just skip the last few layers. Maybe, like, we can guess
David Bau: Close enough.
David Bau: Let's, like, save the electricity.
David Bau: We'll just call an end to this game, and we'll just decode right now. So if you skip the last layer and you decode.
David Bau: It often gives you the same prediction.
David Bau: Trumpet.
David Bau: And you could actually skip a few layers, and it would keep on giving you the same prediction, right? And so this is called early exit decoding. And the funny thing is that if you skip too many layers.
David Bau: You'll get, like, slightly different things. You peel off from a…
David Bau: And it'll start saying other things like corn.
David Bau: And if you peel off layers, right, you'll start doing other things, like miles.
David Bau: or Stein, or whatever, right? It's just, like, say other random things, as you peel a lot more.
David Bau: But this, this is interesting. So what… what happened is around,
David Bau: 2020 people, a blogger, nostalgabrist,
David Bau: Suggested, hey, we should do this on purpose.
David Bau: to get a little insight on what's going on inside the network, and they call this the logic lens. This is applying early access decoding
David Bau: To get a little bit of insight about when the model is prepared to… Make certain predictions.
David Bau: Sometimes earlier, and sometimes later.
David Bau: But a little… little window in the inside. So this is called the Logic Lens. So, let's… so there's a lot of questions about logic lens. Let me… is Isaac here?
David Bau: Oh, Isaac, hey, what did you want to ask about Logilens?
David Bau: Yeah, so I made the… so in the blog post, the author suggested that, early on in the model.
David Bau: Yes! And then, if the…
David Bau: other piece about the internal language that suggests that actually it's still working on… Yeah.
David Bau: Yeah, yeah.
David Bau: left to, that's the hot button. Yeah.
David Bau: So, I was wondering about…
David Bau: If I'm understanding correctly, and then if so… Yeah, I think you're… I think you're understanding correctly. So basically, you know, the story is that
David Bau: because of this residual architecture, the models frequently, especially the shallower ones, like GPT-2 is a smaller model. It's a little less true when you get to the very deep ones that have, like, 80 layers, like Mama, that we'll see. But even at the early layers, especially for the smaller ones, it's already prepared
David Bau: To kind of guess what the next word is going to be.
David Bau: And… and you get what language model people call bigram statistics.
David Bau: Like, what is, like… Like, without knowledge of any context.
David Bau: Like, what's the next word?
David Bau: That comes after… Miles.
David Bau: Right? Without any context, the next word is going to be per, like miles per gallon, something like that.
David Bau: And… and so…
David Bau: So if you go back to really early layers of, like, GPT-2, and you… you decode, what do you want to say now, right after miles, at, like, layer 1 and Layer 2? It'll probably say some word, like, per. It, like, already knows. It already knows, like, what his baseline guesses.
David Bau: So it'll know that very early, but… but then as you get later in the network,
David Bau: It absorbs more of the contextual information, and it'll change its guess.
David Bau: And… and it'll have more sophisticated guests later on. And so… so both of these viewpoints are, I think, correct.
David Bau: That, you know, especially in smaller networks, the early layers do…
David Bau: I mean, like, what initial guess would be?
David Bau: When the networks get really big, when you get, like, 80-layer networks, then it's less clear what the role of the early layers is.
David Bau: you can kind of think of it as, like, yeah, the networks kind of know that they have a lot of time before they're going to have to make a decision, and maybe even making that first guess is, like, a waste. Like, why guess now? I don't have to give you bigram statistics now, I should, like, do some other computation for a while. I've got, like, another 50 layers before I'm going to start running out of time.
David Bau: And so what are, like, what are they doing in this mysterious interior 50 layers? Well, we're not sure. I, you know, that's what a lot of the research is going to be about. But still, you know, we can look at it with LogicLens anyway and see, but it might be a little less meaningful.
David Bau: Okay, so, and then Avery had a question. Is Avery here? Yeah, it's kind of like…
David Bau: mentioning, is it appropriate to kind of apply the loading lines to earlier there, especially for transporters, when it's not really… Yeah. You know, coming up with the old guesses, it's time to use it.
David Bau: Yeah, it's probably not that appropriate. It's okay. I mean, it's a thing, you can just do it, but it comes with this caveat that it might not correlate with any real prediction, it might not be that meaningful. There's a series of papers
David Bau: that I think that I linked somewhere in the reading on the website called, like, Tuned Lens and other types of lenses, so, like, there's variations on logic lens. They try to correct for this. They say, hey, you know what? When you get to the early layers.
David Bau: the thing to do is not to decode them directly, but if you… but there's, like, an average transformation that the early layers tend to go through by the time they get to the end, and you can learn what that average transformation is, and if you just apply that, then that can give you more accurate decoding of what's going on, so that's the two-in-lens idea.
David Bau: And then there's other things. I don't know if… Sheridan is on the West Coast, otherwise I'd have Sheridan give a lecture about the concept lens. Maybe I'll… maybe I'll give that talk at some point. But there's other ways of decoding earlier layers.
David Bau: Okay, so… but yeah, but logic lens is the most straightforward thing to do, and it might not be totally valid, but I'll point you to other research to look for that is, like, different… different nouns before the word lens.
David Bau: tuned lens, con… you know, contact lens, future lens, attribute lens. There's a bunch of people who've, like, come up with different lenses.
David Bau: Any sense?
David Bau: Okay.
David Bau: Alright.
David Bau: And, so we're gonna do this UI thing. What I'm gonna ask you to do in a minute, and you can do this now if you want, is if you go to workbench.ndif.us, then you can actually play with LogicLens, and I'll step you through an example,
David Bau: experiment.
David Bau: To do with LogitLens, right now. So this is the Logit Lens grid, and it lets you view, like, a transformer grid as next token predictions. It's just, like, you know, that big grid that we had. If you just go to any one of these purple dots and you stick the transformer decoder on it, this is what the number one, token prediction would have been at that, at that point.
David Bau: And so, I guess this is… Miles Davis plays the… and if you just go to, like, layer 30, and you stick this thing on, like, a little bit early, then it says horn.
David Bau: Right? And so you can see that before it was going to say trumpet, it was going to say a form.
David Bau: And then before it was going to say Horn, it was going to say Miles. And before that, it was going to say Stein, and before that, it was going to say some other things. And then it has some CSS, something like that. It's going to say, I don't know what, right? But it has all this other stuff that you can figure out. And so you can kind of… you can kind of do this.
David Bau: And so, so okay, so that's… so that's what this logic lens grid looks like. And then there's a few questions about logic lens. So is Eunice here?
David Bau: Yes, Innes, what was your question?
David Bau: Cool.
David Bau: The paper kind of Differentiates those layers.
David Bau: cluster layers into two groups, M group, TOSEP, Old Group. Yes.
David Bau: What they label the middle layer as concept layer looks very kind of different.
David Bau: It's just label… labeling to unidentified phenomena. Yeah. Just to assertive saying that this isn't sexualization. As a social scientist.
David Bau: I kind of…
David Bau: hard to understand how they label it as a concept. Yeah, yeah, like, how come they're so assertive about this? Yes, I agree. It's like this circular thing. It's like, you know, how do you know what's the reality of what you're seeing at Vigilance?
David Bau: I… I think that you're… you're right. Larger lens is a funny way of looking at things. It's this accidental…
David Bau: Do that we have, and it's not clear what the insight is.
David Bau: that you give, and I feel like when…
David Bau: When the… when different people write about
David Bau: You know, the early layers tend to give you insights about the input.
David Bau: The la- the final layers tend to give you input… insights about the output, and the middle layers tend to give you input, you know, insights about, like, the abstract concepts.
David Bau: you know, this sort of assertion is… it's very qualitative. A lot of these assertions are not backed up.
David Bau: By quantitative experiments yet, although some people attempt to do this, but, like, you know,
David Bau: there's a couple papers, maybe I'll… I can point you to try to… try to quantify these things. But the science is not…
David Bau: Very locked down yet.
David Bau: But there definitely are patterns, and people are writing in this maybe overly assertive way to share their qualitative experience of others.
David Bau: And, so that's… I think that's the real estate of what's going on.
David Bau: Yeah, that is highly… people are not being maybe careful enough.
David Bau: In the words, none of them are social scientists, so… Yeah, they're not social scientists, they're computer scientists waiting… actually, so computer science is almost a misnomer.
David Bau: computer scientists are all… computer scientists, we… we're in this very weird position where we're all trained as mathematicians and engineers. None of us are trained as scientists.
David Bau: And so, as a field, we tend to not be very careful writers.
David Bau: You know, we say, this is how it is, because we're used to engineering systems, and when we make assertions about systems that we've engineered, we can speak confidently because we put that into the system.
David Bau: And that's how we write.
David Bau: This… we're in a very weird situation with these neural networks, where we didn't put it in.
David Bau: So we should be more cautious about what we say about these things, but…
David Bau: The sort of the culture and the style of the field is.
David Bau: you know, maybe not adjust them today. So that's the way. I'll apologize for the computer scientist.
David Bau: And and so it was just sort of the way that we write.
David Bau: So,
David Bau: Yes, but I like the question a lot. I, I, I agree with you. A lot of the, a lot of the writing is a little bit over-assertive.
David Bau: But on the other hand, if you don't write over a survey, like, it can also be hard to communicate to other computer scientists. Like, it's… yeah, it's hard.
David Bau: So, you know, we're in this, we're in this funny, funny, funny period. Claire had a question, or maybe a similar question. Is Claire, who's Claire?
David Bau: Oh, yes, Claire.
David Bau: Oh, I thought that maybe the regular basis is, like.
David Bau: irrational. Yeah. We're saying that the space of which the decoder works on the last layer is the same as it works on every layer, and I don't know if that's true. Right.
David Bau: That's right, it's definitely not true.
David Bau: At least strictly speaking, so the decoder was only trained to be able to decode the last layer. It was not trained to decode any other layer.
David Bau: So the fact that it does anything that's interesting at all on other layers, It's some weird accident.
David Bau: it wasn't something that was put in there intentionally. But…
David Bau: It is a residual network, which does have this mathematical accident, that…
David Bau: the layers do tend to resemble each other from one layer to the next. So it's not completely unfounded, it's not completely accidental.
David Bau: And I think that the structure might be such that at some point.
David Bau: like, careful proofs or careful empirical work, careful reasoning might be able to really reason through what it is that we're doing when we do logic lands. So I don't think that
David Bau: It's completely vacuous.
David Bau: what we're doing, but I… but I would agree with Eunice that maybe, you know, people who use it maybe are a little bit over-assertive about what it tells… what it tells you, because we don't… we're not quite sure what it means yet.
David Bau: That make sense?
David Bau: Yeah, so I, you know, I agree with you, but… but I push back a little bit.
David Bau: It's not completely vacuous. It's not completely vacuous. There's something going on.
David Bau: Yeah. And so, here, I'll give you an analogy. Like, psychologists will do this.
David Bau: They'll say, hey, you know what, I want you to, like, look at some stimulus, and I want you to tell me what you see.
David Bau: Oh, but I want to try to get insight on the internals of your brain, so I'm going to put it… I'm going to give it… use a stopwatch. I'm going to flip the lights on and flip the lights off so that you can't see it after, like, 100 milliseconds, and then I'll want you to tell me what you see, right? So they're trying to, like, interrupt
David Bau: the computational process that you have inside your brain to just, like, a really short, like, period of time. They'll limit your stimulus, they'll, like, make you… make you utter something early, they'll let you see something for a really short amount of time.
David Bau: And so, this is a little bit in that spirit. It's like, this is not the normal way that you go around viewing the world. Like, you normally have, like, plenty of time to see the thing. You don't, like, see things in, like, 100 millisecond bursts under a strobe light, right? But the neuroscientists, the cognitive scientists, believe that that tells you something.
David Bau: about the thinking process, and they try to map out the stages of thinking by doing these kinds of experiments, these stopwatch experiments. And this is a little bit the analogy of it. It's not quite an exact analogy, but it's like, hey, you know.
David Bau: you were expecting to have 80 layers of computation to do this, but look, I'm gonna… I'm gonna stop you after 70 layers, just a little early, and I'm gonna make you, like, utter the thing. And then, does that give you any insight? I'm not sure, this is not the way that the model's trained to work.
David Bau: You know, it's a little accidental that does anything at all, but yeah, maybe it gives you some insight, right? And it's kind of in that… it's kind of in that spirit. It's a little… it's a loose analogy.
David Bau: Does that make sense?
David Bau: Yeah.
David Bau: Yes.
David Bau: We don't entirely know, like, what's going on in the early layers, right? And, you know, when you look at logic lens, or at least, like, generally what people believe is that the early layers are doing some kind of encoding, so I think it's like, when you say, we don't entirely know what's going, like, what
David Bau: What other miscase do you think are there? So, I… maybe I'm understanding it. I'd say that LogicLens is not a very good tool for understanding what's going on in the early layers.
David Bau: There are other experimental methods that you can use to try to get insights for what's going on in early layers, and we will study some of those methods. So, maybe that makes you feel better? This is not the only method we're going to use. We're going to learn probing.
David Bau: We're gonna learn causal mediation analysis, we're gonna learn some other tools that we can use, that we can use to get insight on what the early layers are doing.
David Bau: Yes.
David Bau: Oh, this might be, like, a silly question, but, like, do we have to have early layers? Like, could you make them more comfortable if you maybe, like.
David Bau: you could switch the order of the layers during training, and maybe they'd, I don't know, become more generalist? Yeah, wow, wow. You're gonna invent, like, the universal transformer architecture. Yeah, no, it's good. I think that, yeah, so I think these are, like.
David Bau: sort of engineering-ish questions.
David Bau: Can you make it so that the layers are more transportable? So there are funny things that if you swap the orders of layers in a pre-trained transformer, it doesn't damage.
David Bau: the,
David Bau: the capabilities of the transformer that much. It doesn't really notice that much that the layers are moved around. Unless you start swapping layers really far, like.
David Bau: the early layers with late layers, then it starts to confuse things. But if you, like, go to adjacent layers and you mix them up, it doesn't change things too much. So… so there's some… there's some spirit in which, like, the residual stream is already giving us this ability that, you know, to say layers are not that different from each other.
David Bau: But, but yeah, but… but people don't…
David Bau: Training them with that as the goal.
David Bau: Right? Early layers are allowed to be very different from late layers, and then they… and for the earliest layers and late layers, they tend to be different.
David Bau: So… and then Guangyan had a question about thinking yes.
David Bau: Okay, another question, yes. So, I was reading about logical answer, and I was just wondering if it has anything to do with, the dimensions. So, so, I think…
David Bau: most of the representations inside the model are in one dimension? Yes. Could it be possible that… Most of the representations are in, like, a thousand dimensions, not one. Say again? So, could it be possible that these representations are, like.
David Bau: More complex, and we need to, like.
David Bau: Interpret them in, like, in more dimensions or something?
David Bau: Okay.
David Bau: So… So the… so the decoder… Definitely doesn't.
David Bau: Reveal everything about…
David Bau: the representation. So there's… like, so is Eric in the room? Eric Todd? Yeah, Eric did this nice experiment in his paper where he asked the question… he had found some representations that we'll talk about in a few weeks.
David Bau: Inside the neural networks, which don't really seem to correspond to words very well.
David Bau: And… and then… and then, so then the question is, if you… if you try to… so what is in those thousand neurons?
David Bau: If it's not, that'll root prediction.
David Bau: Right? So the thing about this logic lens thing is you can think about a lot of
David Bau: Thinking in a neural network.
David Bau: Especially as a language model, as, like, achieving the goal of doing a word prediction.
David Bau: It's like, all the thinking, like, the whole thing is only trained on the job doing a word prediction. So if you look at, like, an attention module, an MLP module, and you look at its output.
David Bau: This paper is written about how you can interpret the output of any neural network module as adjusting the word predictions.
David Bau: It's like, if you look at, like, what the MLP outputs, and you say, hey, where's the Eiffel Tower? And you look at the MLPs that, like, seem to be active when you're asking, where's the Eiffel Tower, you can look at the outputs of the MLPs as just boosting the word prediction probabilities of Paris.
David Bau: it's injecting some vector into the residual stream, and then you think, what is… you can do all the decoding to figure out, like, what is that vector doing? You can actually take the decoder ahead, and instead of sticking out in the residual stream, you can stick it directly on the MLP,
David Bau: to say, what is this MLP like adding to the residual stream? What's it doing? And you'll see these interesting things. You'll say, see, this MLP, which is very active.
David Bau: It's like, when you decode what it's injecting into the stream, it's like.
David Bau: It's encoding this thing that says.
David Bau: triple the probability of Paris. My god, it's Paris! You know, don't forget, it's Paris. Eiffel Tower, Paris, Paris, Paris, Paris! That's what it's saying in this MLP, right? That make sense?
David Bau: And, you know, it's fun. Or, or London, or whatever, whenever you… there's… there's a lot of knowledge that you can see. You can kind of see the…
David Bau: this word prediction thing. But is everything in terms of word predictions? So there's some papers which, like, talked about how you can, like, scan over all the components of the neural network and understand how everything is done in terms of word predictions, but then we do see some representations.
David Bau: Where they don't seem to correspond to word predictions really well. If you put the decoder head on them, you get this garbage neuron… you get these garbage token, you know, predictions that don't seem to correspond to anything that we understand.
David Bau: And… and so the experiment that Eric did is he, eric, you can raise your hand. Okay, who's Eric? Yeah, I'm gonna describe your experiment.
David Bau: Is he said, what if we take a representation vector.
David Bau: And we do a reconstruction to find the smallest or the more similar, you know, like the, you know, representation vector, you know, the simplest unit representation vector that
David Bau: decodes to the same vocabulary. So you take the top 100 words and their probabilities that are decoded
David Bau: When you decode this, this concept vector.
David Bau: and you find another concept vector that decodes to exactly the same 100 words, in exactly… with the same… exactly the same probability, but maybe throws away the rest of the junk, right? So if… if you think that the only thing that's going on is, like, word predictions.
David Bau: Then, by the time you get to the top 100… the bottom of the top 100 words, then you're in the noise, you're in the zeros, like, there's, like, zero word prediction, you know, power there.
David Bau: Right? So if you take the top 100 things and you get another vector, which throws away all the junk and just keeps the top 100 things, and that's where all the signal is, then that vector should do the same thing as your original vector. And so he went around to these interesting vectors that he had, these interesting neural signals, says, what if I just do that?
David Bau: And he's like, it loses this explanatory power for the phenomena that he had. So there's definitely concepts, which we'll talk about in a few weeks.
David Bau: which don't correspond to just word prediction things in these models. And so… so LogicLens, like.
David Bau: On that… by that note, LogicLens would be blind to those things.
David Bau: Right, so there are definitely, like, we know that there are things that LogiteLens would be blind to.
David Bau: That make sense? So, as… that maybe asks… answers some of the skepticism questions here. It's like, you are right to be skeptical, but in this very concrete way.
David Bau: Make sense? But, and yet, there's so much word prediction going on that it's, like, a good thing to look at.
David Bau: Okay.
David Bau: Okay, so… oh, and then Guan Yang, is Guan Yang here? Yup. Yeah, what was your question? I guess the… Did you get some answers already? You've got your answers.
David Bau: Okay, cool. I just want to make sure people get their answers. There's more questions. Oh, Courtney, Aria, Yuchi, Christopher, Jacobia. Okay, there's so many questions. So, so, okay, so there's… so there's this experiment that I want you to run.
David Bau: Which is the natural language translation experiment. If you go… to,
David Bau: If you, if you say something like this, if you say.
David Bau: French, and then put a French word.
David Bau: and you say another language, like Deutsch, or Chinese.
David Bau: Right, and then you ask the model, like, predict the next word. All these models, they know how to do translation.
David Bau: And so, what I really want you to do is, like, run this.
David Bau: reproduce the Chris Wendler experiment, and then look inside.
David Bau: What's going on in the neural network. And what you'll find… is…
David Bau: You know, on the way from translating from French brewer to Chinese.
David Bau: the same word. It goes through the English word.
David Bau: flower in the middle, even though there's, like, no English in sight anywhere.
David Bau: In the… in the input text, which I thought… thought, you know, like, so Chris thought this was interesting, a lot of people have observed this, and… and it… and it suggests certain things. I don't know what you think it teaches us.
David Bau: Let's… actually, let's… Courtney.
David Bau: Yes, what do you think of this experiment?
David Bau: Maybe I'm not understanding this statement. Yes.
David Bau: confused and make this claim that we pivot into, like, an English concept space, but not necessarily, like… Yeah. It's translating to English and translating back, it's more of, like, a conceptual English space, and I was wondering if that's really an artifact of how we're decoding. Yeah. Like, the launch process itself is…
David Bau: trained in English, so that's what we're coming up with these decoded English. Well, you know, the logic process is trained predominantly in English.
David Bau: That's true. But it's also trained in all the other languages, because, like, if you take it… if you decode the very last thing, it'll decode it in Chinese, decode it in French, right? And so…
David Bau: But you're… but you're right, there's definitely a bias in the training for these models that are trained on the West Coast, you know, they're, like, trained mostly in English.
David Bau: And, So… I think the interesting thing is that
David Bau: Not the fact that it's English, maybe it's a little interesting that it's English, right?
David Bau: But to me, the interesting fact, then, is if you go from…
David Bau: French to Chinese, if you go from, like, German to… you know…
David Bau: some other language, to Japanese, or whatever,
David Bau: If you go from Spanish to Italian.
David Bau: you know, like, the example I like to use is in, like, Spanish, you say, amor.
David Bau: And in Italian, you say amor. And so it's just kind of, like.
David Bau: translate from a more to a more. It's like a gotta add a U or something. It's, like, almost the same tokens, right? But the… but it goes through, like, love.
David Bau: In the middle, completely spelled differently, completely like, you know, morphologically different tokens. And,
David Bau: And so, and so, to me, the interesting thing is…
David Bau: That no matter what the language pair is, the thing that goes through in the middle is, like, decodable the same way. It's, like, decodable in English.
David Bau: And so whether…
David Bau: the fact that it's English is important or not, I think that the interesting thing is it's, like, always the same.
David Bau: Right? It's like some shared representation.
David Bau: That, that sort of answer? Yeah. Yeah. And Aria, you had a question.
David Bau: about this setup is… do you have a similar question, Aria?
David Bau: His, violence.
David Bau: Have, you know, and the paper is the design experiments so that
David Bau: your next word as a single… Oh, I love this. So what do you do?
David Bau: Yeah, look at these garbage tokens that we have. I don't know if we have any here, right? So there's, like, like, there's this, like, little token, duh, or there's, like, this token, ROID. What is this ROID? I don't know what this roID is for, right? Or la. Like, what is la… like, is la the beginning of something? You know, Francois la… like, I don't know what La means.
David Bau: Right, laws, like… like, and you'll see this in some of the other examples I show later, right? But, like, some… like, a lot of times you see, like.
David Bau: One of the examples I'll see, I don't know if we'll have time to show you, I'll show you one that has to do with AI safety and, like, making bombs, and you can see that when it's thinking about, like, explosive power of a bomb, it… it has this Logic Lens thing where it says, EXP. It's gonna, like, talk about explosions.
David Bau: But eXp is, like, it doesn't say explosion, it's just, like, the first token in Explosion. It could be the first token in something like Experiment.
David Bau: Like something really innocent.
David Bau: You know, experience… Right? You know… But…
David Bau: And so there's this question, like, how do we get insight on whether it's talking about, like, Experiences…
David Bau: Or explosions! Like, it just says X.
David Bau: So I think this is a really good question. You should look up… look up Koyena's old paper, Future Lens is a nice one, and then… and then Concept Lens is another one. So some of the lenses wrestle with this a little bit.
David Bau: To see if there's a way of getting it. But we're gonna just play with regular logic lens, which is a single token prediction error, but I like the question a lot.
David Bau: Okay, so then… but if you want to use, like, multi-token stuff in one of your papers, you should, like, read some of these other things. You can extend the method. There's things you can do.
David Bau: Okay.
David Bau: Yuchi, Chris, Giseba, who are you? Raise your hands. Yeah, go ahead.
David Bau: My question? Yes. Well, so my question is, one of the things the paper touched on in the introduction here was that it said, oh, well, if we're…
David Bau: If we're learning from our sources, we may be encoding, anglocentric biases into the model itself. Right. So, one question is, you know, the model set… the paper gave a method saying, well, here's how you can surface, like.
David Bau: The tokens it's thinking about, but as far as actually making a…
David Bau: A rigorous cleanup, qualitatively as a model.
David Bau: actually doing things like that, you know, what, like, how would that actually work, right? Are you going to do qualitative analysis on the tokens, but then it's complicated, you know, by the fact that a bunch of them's gonna be garbage, too. Yeah. Yeah. So that was kind of… so that was kind of one… that was kind of one direction, because, like.
David Bau: well, there might, you know, there's going to be naturally, you know, representation of the things it's trained to recognize more, but it's just letting me know that there is a coherent
David Bau: bias in the, you know, in the human sense, right? Yeah. I think that's kind of a very difficult claim to substantiate. Yeah, that's right. Right, so there's got… there's, like, all sorts of problems with logic lens as an experimental
David Bau: method. And, so, so…
David Bau: I'll leave your question open, if that's okay, because I think that it's worth thinking about what the limits are as we look at different experimental methods and see if… I think that the real answer is, oh.
David Bau: Each one of these methods has some imperfections, it gives you a different view on what's going on, and you end up trying to triangulate.
David Bau: Kind of follow up, because I got this conversation, actually, yesterday, it was…
David Bau: Even hypothetically, like, let's… like, let's say it does seem to be, at least in human eyes, some troubling pattern or bias. Yes. …tokines that appear in intermediate ways, but if the… but in terms of the end user, what they experience, either by prompt engineering or things like that, I mean… Yes.
David Bau: Is that still even a problem? Do we care, right? I don't know, right? Does it matter? Does it matter that my language model is lying to me? What if it hates me, but then it pretends to like me? Is that okay?
David Bau: I don't know.
David Bau: I think that's a good question.
David Bau: It depends on how good an actor it is. Maybe by the time I die, I don't care anymore. They hated me.
David Bau: But… but… but if… but if that… if it… if it might, like, sneak out in some… some behavior, maybe I do care. Maybe if it only reveals that it hates me when I'm not around, then…
David Bau: Maybe I care.
David Bau: Right.
David Bau: So I think… but I think it's a reasonable question.
David Bau: I, you know, I think that intuitively, people are like, oh yeah, I want to know what it really thinks.
David Bau: So we're sort of chasing that intuition.
David Bau: And… but there may be reasons that this is limited.
David Bau: Oh, more, more questions, more questions! Who else? Just have a… Yes.
David Bau: Do you wins.
David Bau: it's, I guess… Internal computation. Our internal computation is something that we can understand. Yeah. But then, when we're trying to interpret it, it's just going through our own lens, which is not super scientific. Like, so if I was thinking, what if you used
David Bau: an SAE, and, like, trained it out the middle layers. Right. To kind of have a better rate of, like.
David Bau: what's… is it actually English, like, Ness? Right. Right.
David Bau: Right, right, right, right. So my… I guess my feeling is that the essays are a little bit of a different technique. We'll talk about them next week, and… and they give you a different view.
David Bau: on what's going on. Maybe it could give you a view into vectors that don't correspond to language that well, so it may be thought of as a good supplement.
David Bau: To this… to this larger landscape, which is so language-oriented.
David Bau: Just a way of looking at other types of vectors that aren't necessarily as language-oriented, maybe Wildcheck.
David Bau: So yeah, it… But I, by… I would tend to not… Combine the techniques too much?
David Bau: Dude, but look at them as complimentary.
David Bau: Right? But, you know, I'm not sure. Probably some people combine them as well.
David Bau: But that's just my intuition, how to use these things. So, essays and these vectors, we'll talk about next week.
David Bau: And then… and then, okay, did everybody on the list get a chance to ask questions? Yes, Yuji? Yeah, it's not, like, a real question, but it's more focused on the late language paper. Oh, yeah! Yeah, it's like,
David Bau: I ask this because of my background, like, I'm bilingual, so… Yeah. Most of the time, I've been in Chinese, but now I'm also in English. Yeah, it's, like, weird. They, like, speak a big… and there's, like, bilingual models, they plan, so they're, like, equally trained in English and Chinese.
David Bau: No, like, of course I'm more for Chinese. Okay. But, like, for the logo, I think, like.
David Bau: about 90% of the trimmed beta is in ingots. For this model, for Lana, yeah. Yeah, so… but that's not true for the Quan.
David Bau: Well, we don't know, they don't tell us what it's trained on.
David Bau: But… But, so I think, like, maybe it becomes, like.
David Bau: there are more translated in English, so it can, in the middle layer, it's more, like, English-spoken rather than other language.
David Bau: But I think, and, bills of use,
David Bau: concept called token energy to say, those tokens are not really English tokens. Right. Different. I think it's maybe because it's multi-language, so… Right. Maybe the model thinks in, like, different language, so the token… the token in the middle layer should be…
David Bau: like, a body of different countries? Yeah. There are different…
David Bau: So, I have a question to ask you back that has to do with this, and I would encourage you to do this in some of your experimental work, is if you use LogitLens on a model like Quen.
David Bau: that was trained with different language biases, how does that change its output? I feel like there's been a little bit of work that looks at this, but maybe not enough.
David Bau: I think that some of these experiments, like the multilingual experiments, does it go through Chinese in the middle instead of going through English? Like, I don't know the answer to that. I don't know if anybody's ever published the answer to that. I think it would be very interesting to know. And so I would hold on to some of your questions and, like, use that as an opportunity to
David Bau: Like, ask… it might be obvious questions to ask, but, like, nobody's asked them. We should look.
David Bau: Right, does that make sense?
David Bau: Okay, so are we out of time?
David Bau: How are we doing on time? Am I out of time here?
David Bau: I'm… I am, I'm out of time.
David Bau: Oh, no!
David Bau: I, like, okay, let me… Yeah, let's plug it in.
David Bau: I have 5 minutes. Okay, so screw all this stuff. Okay, so, alright.
David Bau: And do I have 5 minutes, or am I out?
David Bau: Am I over?
David Bau: I'm 5 minutes over.
David Bau: Okay. I'm exactly, I'm exactly at the end. So maybe, maybe on Thursday, I'll,
David Bau: I'll carve out a little time for people to try some of these things, or you can try them at home, but I'd like to actually show you some of these experiments. I'll flip through some slides here so you get a little sense, but then… and then you can tell me at the beginning of Thursday if you'd like to actually, like, go through them. But there's this nice notebook that I prepared.
David Bau: Oh.
David Bau: there's not enough time in a class. And then, to do, like, example research, so, like, if you ask it, what's the capital of France?
David Bau: then it says, oh, the prediction is A,
David Bau: the capital of France is A, a famous city, it's gonna say, or something, I don't know what it's gonna say, right? But you can see right before the last name, before it says A, he's thinking, Paris, Paris, Paris, Paris, Paris, Paris, Paris, and then at the last moment, it says, no, no, no, I have something more grammatical to say, I'm gonna say A.
David Bau: Right? I don't know why. Here's… here's, like, Amor, going to Amor, and then going through love, right? This is all these interesting… here's a pun. You know, why do electricians make good swimmers?
David Bau: like, does it… like, does it… when does it get the idea that maybe it should stay current? You know, it's like, does it know a joke? So there's this… you can take these kind of things, like neutral versus funny contexts, where you say, I used to be a banker, but I lost my…
David Bau: What comes next?
David Bau: Lost my job. That's dire.
David Bau: But what if… what if it was in this context? What if it says, I used to be a tailor, but the job didn't suit me?
David Bau: the tongue.
David Bau: I used to be a banker, but I lost my… And then you say, interest!
David Bau: Right? Get it? Get it? Get it? Get it? Right, so you can run these through the model, you can see its internal thinking.
David Bau: For when it gets like that.
David Bau: The punchline?
David Bau: Right? And,
David Bau: So, okay, so there's… there's a bunch of things like this. There's, like, there's also a safety example that has to do with… with bombs and stuff. So if you're… if you're interested, think about it. On Friday, we could spend a few minutes going through this at the beginning and playing with them, but then… but then, remember, on Thursday,
David Bau: Get your… get your, like, proposal for what you want to study.
David Bau: Any research projects written down?
David Bau: with your team, and then get ready to do a 5-10 minute presentation. My request to you is to… Do the eye in finer. Tell us why it's interesting. Tell us why it's interesting. Tell us, like, point us to some background literature. Like, why do people in your field think it's interesting, or… why is it… teach us why.
David Bau: Does that make sense?
David Bau: Okay. Thanks, guys. See you on Thursday.
David Bau: What? Yeah, yeah, slides, like, a few slides, a few slides. Yeah, yeah, yeah, a few slides is okay.
David Bau: Yeah.