David Bau: This might vary across some of this, but generally, right?
David Bau: There's gonna be also a lot of places called right now. It's right now.
David Bau: I don't do it, but you want to work with us.
David Bau: Let me also get my computer.
David Bau: Give me a second.
David Bau: Yes, correct.
David Bau: I'm giving back to school.
David Bau: Yo, Avid?
David Bau: How can you get started?
David Bau: Ethan, this… There's Zoom stuff all over the screen, so feel free to contact the mouse over there.
David Bau: I feel like it's fine like this with one of the one on stage.
David Bau: Hi, everyone, so this is our basket for, clinical ideology. But this is our estate.
David Bau: And something that we wanted to look at was the contrastive pairs, oh, go ahead and make them bigger, sorry, the font's really tiny.
David Bau: Sorry? Yeah, there's a menu under More. Under More, you can say hide that thing.
David Bau: There's loading meeting controls.
David Bau: Take your time.
David Bau: Like… Okay, there you go.
David Bau: And you can minimize it, that's great.
David Bau: That's not. Okay.
David Bau: So, we decided to look at some contrive pairs experiments, since we're trying to tease out this, like, idea of the LLM syncophancy, versus, like, the user context.
David Bau: And we wanted to determine if we could extract, like, a non-sycophantic vector from the model to determine, like, when it's responding just based off of its, like,
David Bau: like, the baseline model behavior versus when you're giving it a certain context and, how those responses differ. Originally, we kind of framed this as extracting, like, a conservative vector to see when the model is, like.
David Bau: Flipping its responses to be more towards, like, a conservative or right-leaning ideology. But then we just decided to go with non-segmentic vector because we saw that sometimes the baseline responses for the model, without providing it any, like, personal context, sometimes let
David Bau: lean towards, like, a liberal ideology, and sometimes towards conservative, so for a bigger data set, then we're having, like, a, non-significant vector, not just conservative.
David Bau: So, we gave it a short user context, for example, like, I am a sibling, or I am a farmer, I'm a Republican, 26 different user contexts, ranging from neutral to possibly a political proxy or a stereotype, and then explicitly political.
David Bau: Along with the 60 political compass questions, and we asked the model, do you agree or disagree with the following question?
David Bau: and, I use Llama 70B Instruct, because we've seen the best, results with that, and…
David Bau: It's been the most, robust.
David Bau: And, for the contrastive criteria, as you can see in the purple and the green, the responses, agree or disagree from the, I am a human and I am a person, which are, like, baseline, responses without context, are used as the toxicophantic response.
David Bau: Regardless of context, so if the model said that they agree, like, in the top purple one, where it says, I'm a human and I'm a person, and the model says agree for both, then that is, like, the non-sycophantic or the non-flip, and then all the other responses, where the model said disagree, would then be the flip response.
David Bau: After that, we got 30, or 317 statements where the, agreement… So, okay, someone had a prompt. So…
David Bau: So this is… this is asking the model to predict its…
David Bau: own text, or is it predicting the user text? Is it in dialogue, or is it not in dialogue? It's just… It's in, it's a dialogue gallery. I see. So you mark, you mark user and assistant.
David Bau: use those little tanks to… Oh, we're just asking the model, like, I guess, just putting the prompt, like, I am a human, do you agree or disagree with the follow-up statement? Like, for example,
David Bau: I don't know, some political compass question. Please answer either agree or disagree. I see, I see. So it's… I guess I'm asking sort of a technical question.
David Bau: Which is, you know, when you run these models in dialogue.
David Bau: then you have these little tags that set off, like, what a user says, and then little tags that set off, like, what the assistant, like, the AI is supposed to be saying. And it's a language model, so it can actually predict.
David Bau: text in all these different contexts. It's happy to protect text, even if you don't have any tags. I'm just wondering if you're using tags or not yet. Right now, it's just practical.
David Bau: the standard user, and then assistant. Oh, so you are using… When the user's introduced. So, a few weeks ago, I tested flipping them and saw that there was secrecy in both. That's great. I, in fact, forgot. I forgot which team did that, right? That's great, so you guys did that, so you're inside that dialogue. Yeah, but now it works.
David Bau: Yeah, now we're just seeing, like, how it flips depending on the context that you,
David Bau: like, provide about your, like, yourself, like, the user context. I am XYZ. I see, so the user says…
David Bau: I… I'm something, and then…
David Bau: We want to see if the LM changes its response to agreeing or disagreeing with the political compass question.
David Bau: I see. It says, oh, so the user says, I like all this stuff, and I'm liberal, do you agree? Yeah, exactly, and we want to see whether the model, like, now agrees or disagrees.
David Bau: the question differently from its response without any, like, context. Yeah. What's kind of, like, the way this model will respond
David Bau: there's, like, no user intro, or a user intro that doesn't seem to reveal anything about their, you know, I'm the person, like, it doesn't say anything about their political meaning, and we're saying, does it switch to… because it's, like.
David Bau: for the most part, they tend to be left-leaning, but sometimes they'll be like, is capitalism good? Like, is the question, and… I see, so, like… Yeah. Yeah, yeah, so I was a little… I was just a little confused by that, because I didn't see, like, is capitalism a good kind of question? It's like, some questions about me, like, I'm going to be short, and I didn't really understand, like, I was expecting it to be a…
David Bau: a question about some of the… Yeah, maybe I explained it wrong. So that's the context. The context, like, the first part of the entire pledge that we're giving the model is, like, I'm a teacher, I'm a sibling, I'm a banker, I am a twin, something like that.
David Bau: And then we ask it, do you agree or disagree with one of the 60 Political Compass questions? I'm giving you an interview, hey, LLM… Oh, but then you drop off the items. But just for context, I'm, blocked. Okay, so that's… that's the context. I understand that, that setup now. Yeah, so then we do, like, all the 60 questions for all 26, like, user contexts.
David Bau: And then we have 317, flips.
David Bau: Where the model will respond differently from, like, its non-contextual, response based off of giving its kind of context. Cool, and so how old fraction is that? It changes? So, sometimes it behaves the same either way. Correct. Yeah, so 317 flips and 1,185 non-flips, so that's, like.
David Bau: About 20%?
David Bau: Yeah, yeah, any, like, original positive testing, because, like, on the AP, it would only… you said you have to pay.
David Bau: 5 or 10% of the time, and for lots of PPUs.
David Bau: Somewhere between a third and half a
David Bau: Nice. Yeah. Nice. Yeah. So now we have this, like, 1500 data set of contrastive pairs, and then I wanted to do, like, the PCA visualization.
David Bau: To see if we can extract some type of, vector difference. And so I just sampled 317 from the, like, non-flips, so that's more, even in visualization. So the flips are the purple, and the non-flip cases are the green.
David Bau: early layer and the final layer, there's really not much separation at all, which is kind of surprising. I was expecting that the statements that, caused the model to flip, in which the only difference is the context, like, I am a liberal, or I'm a Republican, or I am a gun owner, or something like that, would be very different, and it's not.
David Bau: And then we can see the same results here with testing sycophancy direction, and we don't receive great classification accuracy if we use the, like, non-flip or flip as, classification, and we also don't have super great linear separability.
David Bau: Corby, what tokens is? This is just on the… the…
David Bau: the contest token. I go out, like, I see everything. The very last token. Oh, the content, sorry.
David Bau: And then also here, we wanted to look at if there is a difference, like, a vector we can find for the flip direction, and you can see this really small, vector, which is basically, a known vector, so, yeah, it's, like, right there. And then…
David Bau: Yeah, so we didn't get as great results as we wanted here, in terms of seeing if there really is a difference, like a separability between these, statements, and so what I'm curious about looking next is looking more at intermediate layers, just for, like, compute and time's sake, because it takes a really long time on 70B,
David Bau: we only looked at those two layers. But then, also, I wanted to know if there are possible clusters related to, how people answer the political compass questions according to their ideology. So, for example, if the model's baseline answer, like, agree.
David Bau: or disagree corresponds to a left-right authoritarian or libertarian, according to the political compass answer. Maybe there are actually clusters of that inside of
David Bau: Like this. Like, maybe there's, like, an authoritarian cluster, and then, like, a libertarian cluster. And so that's what I'm working on right now. And then I'm also interested if there are possible clusters related to the neutral, proxy, or explicitly political user context. So, maybe there's a cluster, depending on, like, if you're explicitly telling the model, I'm a liberal, or I'm left-leaning, or something.
David Bau: Versus, just giving it neutral information, like, I am a twin or something.
David Bau: But I'm also open to any suggestions from you guys about what you think may be able to help us tease out some kind of relationship here.
David Bau: Feel free to think on it, and Avery can go next.
David Bau: Okay, Amy, yeah, I wanted to…
David Bau: kind of built off of Courtney's experiments, and also since last week was week of activation patching, we want to do some activation patching on 70B, using, basically the same setup, but with, so one, like, neutral prompt in this case is saying, I'm a human,
David Bau: And a conservative prompt would just say, I'm conservative, and it's asking, do you agree with a question?
David Bau: And then… Yeah, patching from each token, or patching, yeah, patching each token,
David Bau: correspondingly, so, like, between the prompts, over as many questions as I could last night, which is 37. So I got… so, 19 of the questions were…
David Bau: I was able to get a moment to flip the answer from, you know, saying, I'm a human, I agree with this question, and say I'm a conservative, I disagree with this question now, because
David Bau: And… yeah, next slide, please.
David Bau: So, this is the, yeah, average, like, effect of, patching, so… average, difference… so, sorry.
David Bau: Average amount of, like, probability change from the target prompt, which is the, I'm conservative, to how much more likely it is to answering in line with the source prompt. So, saying, like, in the case of… so, let's just, for the flip questions, where in the case of the target prompt, saying, I'm conservative, I agree with this question.
David Bau: How much, then, would I patch from a token to another token, from between tokens, between prompts? Did it switch to, I disagree? Because that's what the human prompt said. And there's…
David Bau: you know, mainly centralized at the, the token that's being switched around, the human or conservative, and also at the very last layer, VailRoute's token in later layers. It's a little different from what Grace showed
David Bau: Last time, where she had the effects of, like.
David Bau: this was, like, one down at the period, so maybe I probably just have, like, a one-off by one error in this case, and to go and fix that…
David Bau: I think it's… I was doing 8B, so 70B, so maybe they're, like… So what's a word, what's the word where you're saying the console effect, sorry? At the… here. This is the…
David Bau: Human or conservative.
David Bau: To, yeah, or, like, context. Yeah.
David Bau: I mean, both.
David Bau: It's, it's very, it's a very stark difference, but the absolute probability changes, like, 0.1. So… Yeah,
David Bau: So, yeah, this also, like, kind of…
David Bau: So, it is in line with what we saw in class, where, like, the Shaq, Megan Rapido thing, where you have a lot of change focused on the…
David Bau: token in the actual statement that's being changed, and also at the very end, when speaking for judgment.
David Bau: And what's the little patch over here in the middle? This? Yeah. That's a… we'll do a question. This is, so at the end of the prompt, we say, like, please answer either agree or disagree to coerce, like, a single token answer. This is at the… this of disagree at the very end.
David Bau: It's interesting.
David Bau: and… I don't really know what to make of that yet.
David Bau: Yeah. I guess… next slide, please.
David Bau: This is from some older experiments on AB with a slightly different prompt, but the same idea. In this case, it's patching from, liberal to conservative, where I'm saying, like, I'm a liberal, do you agree with this question?
David Bau: Conservative, do you agree with this question? Patching tokens around. For… this is for, specifically, attention heads and the MLPs in each layer. And interestingly, like, there's no stark,
David Bau: Shading, like what there was in the previous…
David Bau: Experiments, where, like, you had a lot… a very solid bar of coloring here, where we're flipping the token, liberal or conservative, and a very solid bar at the bottom, for the last token.
David Bau: So… Maybe the, you know.
David Bau: My hypothesis, is that, the effect of, like, switching the answer, the sycophancy, is…
David Bau: probably a combination of effects from the tension layer, attention heads, and MLPs, since, you know, there's some…
David Bau: stuff here, but it's not as very… not as, stark, not as, noticeable as…
David Bau: When we're doing the whole layer, so… It's both. But… Yeah.
David Bau: That's for…
David Bau: future experiments, I would want to do this on 70Bs, with an 8B, and do this with all the questions, maybe
David Bau: It just happens that the first 30 questions of the clinical compass test are very Decisive.
David Bau: Quick clarification. So this is MLP and attention heads? Yeah, left is attention heads, this, right, is…
David Bau: I'm getting myself confused.
David Bau: This is just attention heads for two questions. Sorry. Not everybody's thoughts. Go to the next slide, please.
David Bau: this slide is for MLP patching, and you see, like, there's some, you know, stuff up here, not as… again, not as stark as, like, when we're doing the whole layer activations,
David Bau: That's essential in the sense. The questions are different. I think in the 70BUR, do you… the question was, do you agree or disagree? Here, the question…
David Bau: It's not that? It's the same,
David Bau: setup, but instead of switching from, like, neutral, I'm a human, it's, liberal. Like, I'm a liberal, you agree with this, I'm a conservative, you disagree with this for these experiments. But that final sentence, sentence is still the same?
David Bau: so, yeah, I think it might have started to cut off at the bottom.
David Bau: Yeah.
David Bau: Okay, my main question was, if it is actually the same, do you see any effect on…
David Bau: the attention had on the disk token where you were actually seeing effect on the 7DB.
David Bau: Good question. I…
David Bau: I think here you are having an older, like, the older experiments from last week. Right. We had the correct perspective, but then for those… Yeah, that's what I thought, the questions are different. For 70B, we did, like, the reframing of the question, so we could run it.
David Bau: Sorry, I'm disorganized today, I guess. No, I mean, we also had, like, stuff, this was, like, from last week, but then we did some over the weekends.
David Bau: Yes, okay, so I forgot…
David Bau: to include some slides, so I was doing a lot of going back to week one and looking at logic lens, because one thing that I thought was interesting, like, looking at a bunch of different logic lens patterns was that
David Bau: Even though the model generally defaults to a liberal answer in a neutral context, The logits for the…
David Bau: conservative answer is much higher. So even if you see that, like, if you say to the model, like, the same opinion, like, you know, I'm a liberal.
David Bau: you'll see the conservative launches go way down. So, like, the Liberal would have still won.
David Bau: But they're conservative, so it's, like, internally sort of, like, it's thinking less of the…
David Bau: maybe conservative thing, if that makes any sense. And so I was looking at ways to
David Bau: find logic lens patterns where there was more internal competition between, where it was, like, really, like, really undecided between the, like, basically the difference between the conservative and liberal answer was smaller, or that the…
David Bau: Yeah, so that's what I was like, how much inner conflict is there in the logic that matters? I don't know how principled of a metric that is, but I was just…
David Bau: curious. I forgot to include… there's, like, a few examples that I found.
David Bau: And then I was curious whether or not this inner conflict would, like.
David Bau: There was more inner conflict, or, like, smaller differences between those lodges, so that would reveal itself in the…
David Bau: attribution patching things, and so these are…
David Bau: So I set up this experiment, and then after I ran it and reflected on what I did, I was like, wait, this was a very silly experiment to run. But basically, I was looking at
David Bau: what is the logic attribution to all of the attention heads and MLP layers?
David Bau: Or examples where the model was sycophantic, so its answer flipped, or not sycophantic, and it didn't flip between the neutral and conservative context. But then I was like, wait, if it didn't flip, then the logic difference is super low, probably, between…
David Bau: the neutral and conservative context, so it's really noisy to even try to interpret what
David Bau: those differences are. So anyway, so I don't… I don't really buy this more. So, basically, if you look at the flipped context, which is a tiny little sample size, it seems more concentrated, and in the not flipped context, it's more diffuse, but I think this is probably just an artifact of
David Bau: There being a small logic difference, so…
David Bau: So basically, I want to run more attribution experiments where I'm looking at 7 dB, where it looks a lot more, and there's bigger, like, latent logic differences, even if you don't see them.
David Bau: blank.
David Bau: In terms of the Alright.
David Bau: Any suggestions for the team?
David Bau: Yeah, this is, like, a bigger picture thing. I, like,
David Bau: I think with, like, the idea of psychophancy, I'm like…
David Bau: trying to think through in this context, I'm having a lot of problems, because it's one I, like, think about in the context of people, and it's like, if I'm talking to you, you probably have, like, a true belief, and then another question is, like, are you telling me your true belief or not? I'm not quite sure what I mean for LLMs to have, like, a true belief in this kind of context.
David Bau: I wonder if, like, that's kind of, like, one way for thinking about, kind of, what's going on in some of these experiments where you're struggling to have traction. I'm not sure what directions that lead.
David Bau: There's, like, maybe a different set of questions that are related, that are, like.
David Bau: there's a kind of, set of tendencies within the model to answer political questions in certain ways, and how malleable or how plastic are those tendencies, seems to be, like, one way of thinking about these experiments.
David Bau: Yeah, I think that both the psychophancy and then just, like, a lot of moving parts in terms of, like, the different…
David Bau: things that could be going on inside the model. Representation of self, representation of other, representation
David Bau: political beliefs, and kind of, like, how all of the different things are interacting and kind of, like, what you're isolating in there. Maybe there's, like, a way of describing it, not in terms of psychophancy, that might help to get a little bit more, kind of, real time. Yeah. Yeah, I think, I think that's part of the reason I should have added the examples, but, like, I'm…
David Bau: Little March website goes for.
David Bau: inner conflict adeffect, where there's, like, competition between two answers, where I'm not saying, like, this one's the real one, and this is the one it's, like, pretending to do. It's more just, like, oh, sometimes it just, like, immediately commits to…
David Bau: Other times.
David Bau: there's more of that, confusion. Does that mean anything?
David Bau: But because that doesn't require, like, saying whether or not the model has a self or, like, some…
David Bau: Excuse me.
David Bau: I agree, it's…
David Bau: Yeah, I agree. I think that's a good… I appreciate that feedback, because I think we were talking some a bit about, like, private versus public beliefs, and how, like, yeah, like, the LM may have this private idea of what's going on, but then based off of, like, the context or what you're giving the model, it's gonna help
David Bau: indifference, so yeah.
David Bau: Is there something you're hoping to find? Or, like, expecting to find? Maybe that's the wrong word, but, like…
David Bau: What would be, like, a really…
David Bau: Yeah, what would be, like, a… And…
David Bau: Because I haven't been following y'all's project, but it's important.
David Bau: That'd be approved.
David Bau: Bing, but… this is maybe not directly answering your question, but it… I think there's a lot of…
David Bau: like, I think the reason that we, like, veer towards significantly framing is that it's very politicized, like, it's, like, I think…
David Bau: Siga fancy, like, itself, as a thing. Oh, yeah. Where people are, like…
David Bau: The model's trying to read your mind, and it's trying to…
David Bau: persuade, you know, and I'm like, I don't know, like, basically how deep
David Bau: is this? Like, is it actually, like…
David Bau: He's trying to model you, and figure out…
David Bau: What you want, or is it… is it just, like, naive?
David Bau: Stochastic Parrot, like, I don't know, people agree with each other. So is it just, like…
David Bau: That, that kind of… so any… like, I don't know exactly what that would be, but anything that we could basically
David Bau: my hope would be that by characterizing some amount of the mechanism, we could shine light on, like, how sophisticated is this? But I, yeah, I think at this point, I don't really know what that would look like.
David Bau: Right, maybe it has to do with model size, as you said.
David Bau: Is there a way to maybe also add, like, like a…
David Bau: to what extent, like, 1 to 10, like, it's like, like, I've changed my mind. It's like, I mean, she's a liberal answer, and also, like, I'm very certain of this, like, it's like a 9 or something, like, is there any way to maybe, like, get more color on, like, the models?
David Bau: I don't really know where I'm going, but, like, I feel like, for me, that would, like, be another, like, data point that would help, like, if you added some sort of scale.
David Bau: Like, in terms of, like, how positive is it? Or, like…
David Bau: I don't want to say, like, certain… Yeah, yeah. Yeah, yeah, yeah, it's both, like, oh, like, I agree with this thing, and then it's also, like, on a scale of, like, 0 to 5, like, really strongly agree, versus…
David Bau: I don't… it's not really, like, a principle thing, I guess I would… that's just something, like, I would do to be like, oh.
David Bau: Nope.
David Bau: Also, like, kind of adding to what everyone's discussing, identifying how, like, adaptable the model is, like, I would assume that you're giving a context, the model will adapt to the persona that you're giving, versus sick authentic, is it, like.
David Bau: happening really early, and, like, if you probe the model further, does the model stay consistent to its sycophantic behavior, or is it just adapting to the given context, and when you change the… slightly change the context, does it…
David Bau: like, change that, so having that would… like, basically, like, I don't know if a lot of papers have done that, like, having this differentiation between significantly and adaptability.
David Bau: I'm preparing to serve.
David Bau: Sorry, can you elaborate on me?
David Bau: Like, I feel like… maybe I also don't know what that means, but I would just assume, like, sycophantic in a more negative connotation that says, like.
David Bau: oh, the model abandons all knowledge or belief. It has, supposedly trained to be a non-hateful model, and one of the political approaches leads it to a hateful speech.
David Bau: A sycophantic answer will, follow that path, versus an adaptive will understand that the user is conservative or, like, liberal.
David Bau: will adapt the persona, but not, like, let down of its guardrails. I guess that's what I mean, like, when does this adaptation, like, when does the model
David Bau: Understanding that the user is conservative or liberal.
David Bau: like, change its models, beliefs on all that stuff. Like, where's its, like, threshold for, like, being synchronic versus following what you're trying to do? I guess so, yeah, I guess that's the…
David Bau: I love the minimal pairs you guys are putting together. I, you know, I think, I wonder…
David Bau: I wonder a few things. So, I like how, you know, you're talking about the difference between AB and 7 dB. I forgot your initial experiments where you measured these different models, so is your intuition that AB does have some capacity, but… It's much more. But much more in 70B?
David Bau: So, so yeah, so, you know, obviously, you know, looking at stuff in the viewer, looking at wherever the phenomenon happens is…
David Bau: It's the thing to do.
David Bau: I also, like, I like the discussion. I wonder if there's some prompts that…
David Bau: You could try to just have different forms, right? So here, you're… the user starts.
David Bau: By saying… by saying something.
David Bau: But I wonder if you had something like…
David Bau: You know, the assistant says something.
David Bau: Look, you can just open force it to say something.
David Bau: Right.
David Bau: And then the user could say, I disagree.
David Bau: Yep. And then… and then it's kind of interesting if the…
David Bau: model, like, sticks… sticks by its guns, or all of a sudden, it has a different opinion. Like, maybe, like, this…
David Bau: I feel like catching it, in mid-context, changing his mind, it might be an interesting signal.
David Bau: Right? Yeah.
David Bau: Right? I don't know. It's, like, multi-turrent context as well, like, we're just in this
David Bau: Right, and you don't have to do all multiple turns, right? You can just… you can just token force of all the things, and then just, like, do the second turn and see if there's something that happens as a secondary, right?
David Bau: And, You know, I'm not, I'm not, I'm not sure. I think that…
David Bau: There's… so there's a couple other things that occur to me. So, you have a prompt form, agree or disagree, which is kind of a multiple-choice thing, so you, you know, you let it think a thing, and then you do multiple choice. So, there's been various challenges.
David Bau: and untangling… Posal change.
David Bau: Through multiple choice, it's kind of like an extra level of indirection.
David Bau: Right.
David Bau: Because… now… now it's gotta decode what does agree mean? What does disagree mean?
David Bau: There's an actual sentence somewhere, right, that it has to refer to.
David Bau: And, and so, anyway, if you wanted to, like, so I'd recommend…
David Bau: It was a nice paper by, What's the name?
David Bau: Yeah, Sarah, Sarah, what's your last name? No, so Grefra.
David Bau: Yes, sir, my friend.
David Bau: That, like, takes apart the whole mechanism of multiple choice, so it is worth a read, just to…
David Bau: You know, refresh yourself and see if, if,
David Bau: If… if, the mechanisms that you see
David Bau: look like that, and if it gives you any clues for decoding… Is that, like, so maybe the approach is, like, come up with as many different weird variations on this prompt, and just chuck the methods at them and see what… I'm not sure, yeah, that's not a possible thing, right? So I think that clever prompting can…
David Bau: you know, it's kind of like biology, right? You know, it's like, well, clever experimental design, just putting the right things in the test tube can sometimes, you know, clarify what's going on, and it's worth the try, because it's relatively low cost to string together words in a different way. I like…
David Bau: you know, I like your… the… some of the… some of the things you're looking at. So I think that when you jump to looking at a thousand cases at once, or something like that, or 100 cases at once, it might be premature. Like, oh, actually, maybe take a look at individual cases, you know, more. It's just faster.
David Bau: Just to give yourself a sense of where things might be. I… yeah, I tried some prompts, like, without the,
David Bau: Without the, like,
David Bau: I agree or disagree, or, like, please answer, agree, or disagree, like, I just asked it, how do you feel about the following statement, or do you agree or disagree, or, like, what is your perspective? And I tried, like, maybe, like, 20 different things, and without explicitly asking it, like, please answer, agree, or disagree, I couldn't get consistent, like.
David Bau: outputs, like… Yeah, it would like to say some other things. Yeah, yeah, it wants to say something completely different, so at least, like, for this
David Bau: contrasted pair, that's why I love with that specific wording, because without it, I couldn't get it consistent. Yeah, I understand, I understand, yeah. And so, you may… there may never be… maybe a way of weighing this multiple choice, form. But then, read Sarah's paper, and maybe… maybe there's some tips there about it.
David Bau: All of that.
David Bau: Nick, you'll also… I had the same issue with his… his…
David Bau: This thing, which is another publisher's form. What's the name of the paper, or could you, like, send it? I'll send it on my screen. Okay, thank you.
David Bau: By the way, I just want… maybe more concrete goal would be… Try to reduce sequencing.
David Bau: with just, changing the internal representation, or maybe weights of the model. That could be a potential. Have you guys thought about it? Some people have done it.
David Bau: do that, but they do it in a sort of black box, where it's like, please steer it in a way that makes it agree with the user less. And I think, to me.
David Bau: It's kind of unsatisfying. I'm like, I don't know what just happened. Like, did I just prompt the model through vectors? It's a technique, though, so if you adopt… if you read one of those papers, you're like, oh.
David Bau: what's plausible? I kind of believe what they're doing here. You could adopt one of the methods and then ask.
David Bau: your… your interpretability question, which is like, oh, so what did they really do? Yeah. Yeah, I think that's true. Some of their, like, papers, they use, like, the term, like, steer, though, but they don't mean it in the same way that we do. They're just like, yeah. It could just be some fine-tuning, they have a lore, or something like that, but still, you could adopt it, you could say.
David Bau: Oh, what is this lawyer doing, if you apply it on top of the model?
David Bau: And so it's another possibility.
David Bau: gosh, it's… it's tricky. I don't know, just… so, like, you're… you're… now you're into it. You have pretty good…
David Bau: experimental ideas, I'd say.
David Bau: I don't know what the advice is. Keep on digging. Right?
David Bau: Yeah.
David Bau: It's great.
David Bau: But if anybody has ideas on this, it's a pretty neat setting. I like the… I like the question, like, is there a sharper hypothesis that they can have?
David Bau: So, you know, I think that's worth thinking about.
David Bau: You don't blink, link.
David Bau: Beautiful.
David Bau: It could be that you're not…
David Bau: It could be that the whole idea of secrecy is, like, A bizarre, malformed.
David Bau: No opposed idea.
David Bau: Right? It's like, is there really… can you really do thinking without context?
David Bau: you know, is there such a thing as your true beliefs, or do you… whenever you are asked a question, you're always in some context. There's no such thing as your true beliefs. You know, you get abducted by aliens or something, you ask your political beliefs. Is that, like, the true context? I'm not sure, right? You know?
David Bau: And so…
David Bau: So it might be… it's an ill-formed question, it might be that there's no such thing as a non-sycophantic
David Bau: universe. It's always… Right? I mean, that's why, you know, you're sort of seeing everything all mixed up, right? But then there's… but then… but you're closely related to other kind of questions. I like the question of assumptions.
David Bau: you know, you were brushing on this earlier, like, oh, I'm a nurse. Then, all of a sudden, the model has all these assumptions.
David Bau: And it's like, oh, why did you assume that, right? Do assumptions change
David Bau: When you have it today? Okay.
David Bau: if, like, I'm a nurse, I feel like I have to try to be a good assumption there would be a nurse could educate or something like that. Is there a way of understanding how that knowledge is
David Bau: Organized. See the finger, right?
David Bau: 6 years, something like that. I'm not sure. You know, I… you know, it might be that there's something downstream.
David Bau: Alright.
David Bau: I like it, I like… I like that you guys have… you seem to have picked up on all the different experimental techniques, and…
David Bau: Great.
David Bau: All right. Thank you, everyone.
David Bau: All right, let's, let's talk about probating. So, you know, it's not…
David Bau: What did you guys think of the reading?
David Bau: Let's see… Who's, yeah, I picked a couple… papers.
David Bau: He has to read?
David Bau: There's… What's… what's really nice is there's… there's, like, probably as a… way of looking inside networks?
David Bau: Because it's simple, but it's also… it follows machine learning principles, so a lot of people believe in it.
David Bau: And it's simple enough that a lot of people can do it.
David Bau: So let me just introduce, like, what the heck.
David Bau: People mean by probing? It's pretty simple.
David Bau: Right? You're asking this question.
David Bau: Without the censorship, you have this little black square. Here you go.
David Bau: Right. You're asking this question, right? Does my neural network
David Bau: representation. Do my neurons know some concept?
David Bau: Right? Does it know some concept. But this is a weird question, right? Like, when you started this class, if I… I said, hey, your mission was to figure out if my neural network
David Bau: knows concept C. Does it know the difference between right and wrong?
David Bau: Right? Does it know something like that?
David Bau: You'd be like, how do you even quantify that?
David Bau: And so probing gives you a really simple exercise, you know, it's a formula.
David Bau: We're quantifying it, and we'll… so today we'll just talk about
David Bau: You know, what that formula is, some of the limitations, some of the other things people have done.
David Bau: I think you guys had a lot of good questions about the reading that will cover most of the stuff.
David Bau: And so the formula goes like this. So basically, you have to decide what your concept is. Like, what's the difference between right or wrong, right?
David Bau: Well, how do you do that? You do the machine learning thing.
David Bau: You define some data set.
David Bau: like, a labeled dataset of, like, I've got some X's, which are right.
David Bau: And then other X's that are wrong?
David Bau: So the example from the TKF figure one was…
David Bau: I've got some images, which are almost drapey.
David Bau: And then I've got other images, which are just regular… You know, images.
David Bau: And… And that's the difference, right? Like, these are stripes, these are… Regular images?
David Bau: Can I tell the difference between these two things?
David Bau: I wonder if the network has an internal concept.
David Bau: Which would, like, know the difference between these two things. So that's… so…
David Bau: So… so that's what it means to define a concept. So you could see, oh, actually, maybe the concept is, like, this is, like, not as stripey as that, or something like that, even though there's some stripes on it, things like that.
David Bau: Your definition of where the line is between
David Bau: Positive examples and negative examples to be a positive are just given by a data set.
David Bau: Right? And if you want to define your concept a different way, you just form your dataset a different way.
David Bau: So that's his first step.
David Bau: Second step, is… This is… you have to decide
David Bau: Where in your network you care about
David Bau: what it knows. So, I've just written down, there's some layer of the network Z,
David Bau: Which is where you care about it.
David Bau: And then… and then you just… Ask the question, And we extract a classifier, For my concept.
David Bau: out of that Z. So the Z comes out of F , the first bunch of layers of my neural network.
David Bau: then I just train a classifier, G,
David Bau: To try to predict my concept.
David Bau: Out of that, that layer of the network.
David Bau: Right.
David Bau: And so that's just classic machine learning. There's a lot of ways of training a classifier, and you can use any of them. But, like, the simplest way is to just train
David Bau: a linear model.
David Bau: Like, basically learn a vector that if you dot product it with
David Bau: The neurons, and that weighted sum gives you a score that tells you which side of the line you're on.
David Bau: That make sense?
David Bau: And then, you know, if the data set's big enough, the line's never gonna be perfect, right? It'll have some.
David Bau: They'll have some level of accuracy.
David Bau: That make sense?
David Bau: So that's step two, is you just, like, train this classifier. Now, it could be a complicated classifier, it could be a nonlinear classifier or something like that, and so it's not a separating thing anymore, now it's, like, some weird boundary that you're learning.
David Bau: But, you know, you can learn any kind of classifier.
David Bau: And so there's some debate.
David Bau: About the second… second step, like, what you should do. A lot of papers have been written about the second step, because
David Bau: There's the machine learning part, and it's on machine learning venues, so they like to debate about how to do the machine learning.
David Bau: And then… And then the third step is, after you get your classified in.
David Bau: And you, you, you measure its accuracy.
David Bau: And the idea of the probate… is that…
David Bau: If you can extract your concept, more accurately.
David Bau: Then you take that as evidence that The information is in there.
David Bau: And if you have trouble extracting your concept, then that's evidence that the information's
David Bau: Not in there. Or, at least, if it's in there, it's hard to get out.
David Bau: That make sense?
David Bau: So that's… that's it. That's… that's the formula for probing.
David Bau: So, this is… so now, all your projects.
David Bau: Are sort of of the form.
David Bau: I wonder if my neural network has a certain thought.
David Bau: Or if I wonder if my neural network knows a certain thing.
David Bau: And I think that in 2019, you could write those papers.
David Bau: And the classic thing would be, you'd learn how to train a classifier, You do these 3 steps?
David Bau: And you'd be done with your paper. You write the thing.
David Bau: That make sense?
David Bau: And actually, so, but because of that, it's not a bad… Step.
David Bau: To do this, anyway.
David Bau: Right? But I want to also, you know, sort of give you guys
David Bau: room to ask, you know, to sort of question this, and we'll explore the area probably a little bit more. So, okay, so let's take a moment. Okay, Jasmine, kick us off. You had some questions.
David Bau: I guess, like, the… I'm, like, really confused by the Hewittown paper, where they're, like, you can have problems where, like, your probe looks like it's doing really good, but the representation, like, still isn't there. Right. At the same time, though, like, if it performs well, like, there's, like, some kind of signal there, so to me, it's like, what… like, why are we doing all this? Like, what is interpretability… What's it about? What's it… what do we… Like, yeah, like, mean, like, at what point do we have, like, a concept that we, like.
David Bau: Yes. What's the hope, and what's the challenge here?
David Bau: So, I think the hope is, Say…
David Bau: Get our finger, you know, get our hands on this, actually.
David Bau: Right? It's sort of the light effector question.
David Bau: What did Nixon know, and when did he know it?
David Bau: Right? Sort of the question?
David Bau: And, and so we're asking that this environment, what do they know, when did they know it, right?
David Bau: So that's the hope.
David Bau: But then it turns out that it's kind of this mushy thing.
David Bau: Right? So…
David Bau: And I think that the Hewitt thing… part of the Hewitt thing comes down to this question of…
David Bau: you know.
David Bau: if you… If you know…
David Bau: Something that is closely related
David Bau: Bill, what you're being asked about.
David Bau: You know, does that mean you know… the fence?
David Bau: Like, if I knew,
David Bau: you know, oh, I probed your brain, And… I know that…
David Bau: You're thinking that it's, like, a square root, and it's a square root of 2,
David Bau: And then I say, yes, so you know that it's 1.4 or something.
David Bau: Right? Well, I don't know if you know that it's 1.4 something. Maybe you know that it's, like, the square root of 2, but maybe you don't know what the square root of 2 is, right? But then, like, you know, somebody… maybe somebody with certain… a certain level of expectations about common sense?
David Bau: might go to you and say, well, you knew it was a square of 2, so you knew it was 1.4, right? Like, anybody would know.
David Bau: But then… but then you might say, you know what? I didn't know.
David Bau: Right? I didn't know, but the square root of 2 is 1.4, like, is that… they say, how can you not know? That's so obvious, it's just a definition of what it is, right? So that you can have a difference of opinion of what this is.
David Bau: And so… so when you train a classifier to look at a thing and, like, solve a problem.
David Bau: Is this really a view?
David Bau: That the network knows something?
David Bau: Or is this a view that the class supplier could figure out the answer to the question?
David Bau: Right? It's like, oh, like, I can look at what information you had, it's a square root of 2, and I could figure out it's 1.4. Like, I'm like the classifier here, I can figure out what's going on, but maybe you couldn't have, right? So there's this weird…
David Bau: piece of epistemology here, like, what does it mean to actually know something?
David Bau: But I think that, you know, just asking generally what the goal of it is, is we'd like to… we have some…
David Bau: sense of what it means to know something, and we're trying to get our hands on it, and probing is one…
David Bau: approach to it, I think that some of the newer things Are… are related to this.
David Bau: dilemma that I'm describing.
David Bau: About what's obvious, what's not obvious. We'll talk about you a little bit more. Let's ask, what is Kai's question?
David Bau: I see her. Who's Kai? Well, Kai is. It's kind of the same thing, like.
David Bau: Most of the concepts from the concept activation paper are, like, pretty concrete, stripes, or beacles. Yeah. So I was wondering, like, how do we know if it's actually capturing something else? A concept, like, beautiful. Yeah, like, to me. Yeah. Yeah, how about a concept like sycophantic?
David Bau: Right?
David Bau: So I really like that. So these minimal pairs that your classmates just presented here for syncrofancy is, like, an example of constructing one of these data sets. It's like, oh, there's, like, Syncophancy over here, there's not sycophancy over here.
David Bau: I wonder if…
David Bau: My neural network keeps track of the difference between them. You see this scatterplot with everything mixed up? It doesn't look like this, right? You think, I don't know. Maybe it's not keeping track, right? Look at Gray's face. It's like, I don't know. It doesn't look like this. Right, maybe my network's not thinking that way.
David Bau: Right? So, but I think that… so you can ask this… so that's the beauty of this general formulation. You can ask this question about anything. It could be beauty, it could be in some abstract thing, just whatever you have in your mind, and you can just… you can just, like, define the classification set, and then you can ask the question.
David Bau: That doesn't… it doesn't guarantee you that you're gonna get a positive answer. You might get really low accuracy.
David Bau: At the end of your program.
David Bau: But that's… but… or maybe not.
David Bau: Okay, so that's… so that's sort of that.
David Bau: And, and Claire.
David Bau: Yeah, so what was your question?
David Bau: Islands, like…
David Bau: The probes are only using human-like features, but sometimes models split things into, like… Non-human-like things? Yeah, I really like this question. I think that, so…
David Bau: You know, a lot of the methods, most of the methods we're studying in the class.
David Bau: start off with a hypothesis, like, you know, you want to know if the… AI knows something.
David Bau: And to ask that, you have to define this dataset.
David Bau: You have to decide… decide what's. So what… what if the thing that the neural network knows is something that you can't?
David Bau: distinguish, and you can't tell the difference, then how the heck do you make this probe data set in the first place, right? So it's a limitation of this method, I think, that
David Bau: you know, that's one of the things. So, I think that's a good question. There are probably other reasons, there's probably other ways of, like, chasing after some people.
David Bau: Data network.
David Bau: I know is that you might not be able to…
David Bau: Suss out, but, you know, probing is not the way to get it.
David Bau: Isaac.
David Bau: Yeah, this is just, like, a, I think, inversion of all of the questions that have been asked, but it struck me that, like, they had, like, the example of, like.
David Bau: images of CEOs, but, like, what makes them images of CEOs, or just that we're calling them images of CEOs? Like, it seemed like a lot was writing on how we are kind of imputing our own concepts in the images, as opposed to…
David Bau: Necessarily kind of coming out.
David Bau: Yeah, itself, or corresponding to something in the images that the model is responding to? Yeah, so I think it's right. It's basically the same kind of question, and so the things that we're… that it equips us to answer
David Bau: Is trying to find a correspondence between the way that we think about the world.
David Bau: And managing the way that the network is representing the work itself.
David Bau: So… bumped.
David Bau: That's right. And so, if it's something very inhuman that we're trying to suss out, then it's hard to do this for. Now, there are some techniques, so for example.
David Bau: There are people who ask this question about inhuman things by doing things like taking two neural networks and seeing if they agree.
David Bau: Right? Like, you can just take one neural network and take a random neuron or something and say, well, I don't know what this neuron does, it might be something imperceptible.
David Bau: But… I wonder if this second neural network learns the same neuron.
David Bau: Right? If it, if it has the same information.
David Bau: So you can use the… First neural network's neuron.
David Bau: To create this dataset.
David Bau: And then you can use a second neural network.
David Bau: Let's see if you can probe.
David Bau: That concept back up.
David Bau: And so, you know, people have done that kind of thing.
David Bau: I just try to get a handle on things that are not… even interpretable, and there's…
David Bau: There's, you know, there's various fancy ways of doing this.
David Bau: Larger scale.
David Bau: That, you know, even neuroscientists do to try to align brains
David Bau: People with each other and stuff like that.
David Bau: So…
David Bau: If anybody's interested in doing that kind of thing for their project, then let me know, we'll add a segment.
David Bau: To the class to talk about these kind of moments.
David Bau: Yes. Is the goal? To,
David Bau: for a concept that exists in multiple different types of models. Can you do this kind of thing? Yes. So, I think that the… so there's this… there's this…
David Bau: the sort of pulley ground of the project… of the project of interpretability for AI, which is
David Bau: Can I get a printout of everything that the AI knows, including stuff that I didn't know to ask?
David Bau: Right?
David Bau: And, and so, well, certainly you can, you can just print out all the numbers of all the neurons, but that doesn't tell you anything. So, like, what are the meaningful concepts? Can I get a printout of all the…
David Bau: meaningful concepts.
David Bau: It does… what does that even mean? And so one… one way of choosing that.
David Bau: Problem.
David Bau: is you take 100 neural networks, they're trained slightly differently, slightly different architecture, something like this, and you use techniques kind of like what I'm describing, like, you know, see if they agree, if they have the same neurons as each other, or something like that, and then you'll see that, oh, there's only a certain number of neurons that they all have.
David Bau: Right? Most of the neurons, they, they, like, have…
David Bau: You know, they're very different, but they have a few neurons.
David Bau: Which, they all… regardless.
David Bau: And so there was a nice paper, you can look it up, it's called Rosetta Neurons.
David Bau: Where they do this.
David Bau: And… the name of it, you know, suggests the research program, right? They're like, if all these neural networks
David Bau: have these neurons in common, then these are sort of, like, the key…
David Bau: you know, translation points, it's like a Rosetta Stone.
David Bau: for us to be able to translate between concepts, maybe it's like the key for even decoding the whole idea of the internal language of neural networks.
David Bau: To begin with. So that's… so…
David Bau: So that's… so that's the technique that people use.
David Bau: Presents us, and so… Great.
David Bau: Right, right, right. And so some of these things are… Yeah, right.
David Bau: Some of these things are really nice streams.
David Bau: I think that they're mostly speculative right now, like.
David Bau: I don't think anybody's really figured out the point of that.
David Bau: Producting everything that our level does.
David Bau: So, okay.
David Bau: So, I'm gonna have had this question.
David Bau: Rita.
David Bau: I mean, yes.
David Bau: How expats with it.
David Bau: Abyss.
David Bau: what the heck about LLMs? Why, why, why do I assign you, like, an image paper? This TCAP thing? Well, partly, I just wanted to give you a little historical context, and probing is pretty old. Probing goes back before the TCAP paper, but, but, you know,
David Bau: But yeah, as early as the TCAP paper, people were already thinking about how to understand
David Bau: What neural networks knew, and they were using probing methods.
David Bau: And, and so around the same time.
David Bau: People were looking at machine translation systems.
David Bau: And, and using probing to figure out what's going on. So let me just describe, like, an old machine translation setup as, like, a classic way that people use probing, so you can kind of see how it works in practice and other people who just read papers, right? And so, you know, the Hudson Machine Translation, system. I'm not gonna…
David Bau: get into RNNs and everything. They're just… they're kind of like transformers, but different, right? But they also have,
David Bau: Little, little layers where you have a representation and token.
David Bau: And… and then you can ask the question.
David Bau: What does the network know about this token?
David Bau: Does it know that it's a pronoun?
David Bau: Does it know it's plural? Does it know it's a verb or a noun? Right, so it's kind of… these…
David Bau: This can actually be kind of interesting questions, because if you have, like, the word play.
David Bau: Right? It could mean a verb.
David Bau: It could be in a noun.
David Bau: And then within the concept of a verb or not, it could mean different things.
David Bau: Within that. And so you, you can ask, you can ask it, hey, is play a verb or a noun?
David Bau: classify that for me from looking at the neural representation here. And if you can get good accuracy on a probe like that, then it suggests that maybe your network knows
David Bau: The difference between a verb and an outlet.
David Bau: And… and maybe it doesn't. And so… so this is the kind of thing that they would do. And they would ask a simpler question, like, is this plural? Is this singular? Is this present tense? Is it past tense?
David Bau: you know, these linguists, they have all sorts of things that they'd like to, like to measure. And so… so here's, like, an example of a set of,
David Bau: questions about plural and past tense and uppercase. They call these morphological, attributes of a word, and what they did is they went to a machine translation system, and they went to all the layers, the early layers and the late layers.
David Bau: You know, not that different from a transformer.
David Bau: And… and they asked, what was the accuracy?
David Bau: of the probe classifier when translating from Czech to English, or Spanish to English, whatever.
David Bau: And… and you can see that
David Bau: at the early layers, it's pretty accurate. You can tell the difference between these morphological, you know, things. It can classify plural versus non-plural.
David Bau: Pretty warm at the early layers. But the weird thing
David Bau: Is that as you went deeper in the network, These are only 4-layer networks.
David Bau: But as you went deeper in the network, Is accuracy dropped.
David Bau: what the heck do you think is going on? So do you… so do you see that set? Don't worry about the last bar. The last bar is aggregate. If you train a probe over all four layers at once, which you can do also, then it's, like, super accurate, right? So, but if you train it on just one layer at once.
David Bau: If the accuracy is dropping.
David Bau: What do you think that means?
David Bau: Now, that means, why would the accuracy drop the more computation you did?
David Bau: Like, getting the concept?
David Bau: And forgetting. You know, I…
David Bau: It's good when you tell someone, oh, I read somewhere, and you forgot where you read it, or even what language you read it in. Yeah, so it looks like forgetting here.
David Bau: Right? So this is a… this is a funny picture of, like, forgetting.
David Bau: It's like, when it first read the text, it knew this was Past tense.
David Bau: Right? And as it goes further into its layers.
David Bau: then it's… it kind of, like, the story is, like, lower probe accuracy, right? It's like, it kind of is forgetting that it's past tense.
David Bau: Is that weird?
David Bau: Strange, right?
David Bau: Then with the Hewitt thing that they… Suggest.
David Bau: Memorizing proof.
David Bau: Yes, we'll get to that. Yes, yes.
David Bau: No, I don't skip to the end. No, no, it's okay, no, it's okay. But yes, in the human thing, right, so the question is.
David Bau: Is this even meaningful to ask?
David Bau: This is… are we looking at the right thing? So, like, the classic propane.
David Bau: Is to look at accuracy and see what it's like.
David Bau: It's not about accuracy. Why is everybody looking at accuracy? Right? We should look at selectivity, so we'll talk a little bit about that. Yes. But in that example, because it's a function of the word identity, you would expect earlier layers to have higher accuracy for memory.
David Bau: Because… that's right, so that… because at the earlier layers.
David Bau: You were just told the word just now.
David Bau: Like, the information hasn't been messed up.
David Bau: And so it's obvious. Yeah. So I guess it's,
David Bau: Yeah, I don't know, right? It's kind of like, do you, like, do you fine-tw it, or not? Like, do you really want to look at selectivity, or… or if you drew this graph and you showed selectivity instead of accuracy, what would be an insight?
David Bau: I think that's a pretty good question.
David Bau: The, But you do see these accuracy drop, and the story that people tell
David Bau: But you can think about it now that you've read all these papers, right? The story that people tell is that when the accuracy drops.
David Bau: It sort of suggests that the network is forgetting something. It's like erasing that information. It used to have all this information about whether the
David Bau: the word was thorough or not, and it went into the deeper layers, and it sort of erased the information. So, let me show you a contrasting experiment. Same… same set of people.
David Bau: Now here… We're looking at, syntactic relations.
David Bau: So I'm not really a linguist, but, you know, I gather this is, you know, if you have…
David Bau: like, a dependent clause or something like that. You can have different sentences where the same word might sometimes be the object of the sentence, might be the subject of the sentence, might depend on another word, or might stand alone, right? And so you can make a syntax tree, and you can say, what are the relations.
David Bau: for a word, and then you can ask… you can ask a probing question about this. You can say, can I tell the difference between words that have a syntactic relation and words that don't?
David Bau: Or words that have a different syntactic relation?
David Bau: And… and so the… the accuracy is…
David Bau: for this task, look kind of like this. It depends on the language.
David Bau: I don't know what's going on with French to English, but, like, from Spanish to English, like, the probe accuracy goes up as you go through the layers.
David Bau: Why would that be?
David Bau: Why would some of the things, like, go down, and other things go up?
David Bau: Interesting.
David Bau: So, this… this contrast between things going down and things going off was enough… it was, like, compelling enough that people wrote a lot of papers about this.
David Bau: And, and the intuition was, That some of these things, it takes a lot of context processing
David Bau: to figure out…
David Bau: you know, what's going on here with syntactic relations? And if you want to do a good job at machine translation, you need to know this pretty well.
David Bau: So more and more of the neural network's representation is being used to represent stuff like this.
David Bau: So that's why, when you probe it at the later layers.
David Bau: then the accuracy goes up. It suggests that it's using more of…
David Bau: It's doing a better job at representing this information. It's got, like, this information represented more accurately.
David Bau: Getting there. And at least that's the story.
David Bau: That make sense?
David Bau: But it's cool, though, right? Like, I think that these experiments are genuinely cool. This idea that you might be forgetting something.
David Bau: That you were just holding him, but in order to make room.
David Bau: For something that you're figuring out.
David Bau: That wasn't obvious for them.
David Bau: And I think that people generally still, like, believe this story.
David Bau: And if you find a sequence of communications in your neural network, That you want to characterize.
David Bau: Doing a probe.
David Bau: Like this, and contrasting
David Bau: between, you know, probes that are less accurate and more accurate is definitely a worthwhile thing to try and to report. And, you know, people will… people have been looking at probe accuracies for decades, so they'll be able to read, you know.
David Bau: what you have. They'll know what all the caveats are.
David Bau: They'll be able to read that experiment.
David Bau: Make sense?
David Bau: Okay, so… Okay, so for example…
David Bau: My student Sheridan, last year wrote a paper, which was this type of probe. So, actually, there was a question here. Did I skip any questions? I want to skip questions. I've noted questions, but I'm just stop and ask you guys for any questions. So, I think that one of the questions was, like, hey, so what about multi-valued
David Bau: Attributes. When you… it might not be, you know, yes or no, but it might be, like.
David Bau: 5 different things, or 10 different things, then you're… so… so, yeah, you can make probes that are multi-way classifiers, no problem. And so the classic one is LogitLens.
David Bau: Right, the class… the logic lens is a probe, it's a non-trained probe, it's a probe that is just, like, the pre-trained decoder layer, and it's a classifier. It's a 50,000-way classifier that tells you which token
David Bau: Is the most likely one.
David Bau: At any given time. And so…
David Bau: What Sheridan did was, instead of using LogitLens, Sheridan trained a probe
David Bau: A 50,000-way probe on predicting the most likely word in a drink pantry.
David Bau: And so, here's the setup. You have, like, the intermittent as, like, a piece of Wikipedia text.
David Bau: And Sheridan takes, like, the token mitt.
David Bau: And so the token mitt comes in, it just represents the token mitt, it's just like an embedding permit at the first layer, and as it goes through all the layers.
David Bau: And that vector is evolved into something that eventually, like, predicts.
David Bau: the next token, right? And…
David Bau: But instead of, like, saying how accurate it is at predicting the next token, which is what LogitLens does.
David Bau: Sheridan asks, how good is it at predicting MIT itself? Right, so, like, you input admit.
David Bau: And if you go, like, 5 layers down, And you asked…
David Bau: Oh, out of the 50,000 tokens, which one was the original token that you started with? Do you still remember that you were bit?
David Bau: That make sense?
David Bau: And so you can use that as a probate question.
David Bau: And you might think, oh, that's easy, it's like 100%. No, but it's not, because you see the same erasure phenomenon going on, like, the accuracy drops.
David Bau: And so here's… here's Mint. Starts off at 1.0. Of course, you can read the embedding right out.
David Bau: Of the network. But as you go, you know, in 10 layers, right, it kind of forgets a little bit. It's down here, not didn't forget a lot, except, like, at 80%, you know, it drops down to 70% or something.
David Bau: Right? So, like, it's erasing this information, is sort of the story, right?
David Bau: Now, there's other information that's interesting.
David Bau: like… How about the previous word? If you go to MIT, and you ask, what's your representation.
David Bau: How… how much does it know it comes after enter?
David Bau: Right? Intermittent, right? Like, does MIT know that comes after intermitt? That seems like it's a pretty important word, you know? It's like, it's not… it's not about mittens here, it's about intermittance.
David Bau: Right? And it was pretty different.
David Bau: So, to understand, really, what MIT means, it might be handy to know that you come after INTER, so… so that's, like, another probe. It's a very mechanical probe.
David Bau: And then you can ask this question, this classified question, and at the beginning, MIT is very bad.
David Bau: I'm predicting.
David Bau: The intro comes before, because there's been no attention heads applied to it.
David Bau: So the probe is, like, at almost zero accuracy. Now, it's not at zero accuracy, zero would be all the way down here, it's, like, a little higher than zero. Why wouldn't it be a little higher than zero? Even if… even if you haven't written any atten… you haven't run any attention pads yet, and you haven't been, and you're like, oh, you know what?
David Bau: 10%… 10% of the time, I can guess what the token was right before me. Anyway, without even seeing it, you're totally blinded. You don't even have an attention edge. Why is the… why is the accuracy not down close to zero?
David Bau: Why's actually…
David Bau: The previous words also include information on the future words, so, like… Even easier than that.
David Bau: Even easier. It's almost like… imagine you were a cryptographer.
David Bau: And you just had, like, the middle of the word given to you.
David Bau: Without any context, you might be able to guess
David Bau: What that part of the word was before.
David Bau: Right. Does that make sense?
David Bau: Just because of bygram words… just bygram words, it's just because you know that you're in English.
David Bau: And there are certain things that are in English that just tend to happen before, even without any additional information. So the baseline accuracy of the classifier is sort of the accuracy of diagram statistics.
David Bau: in English, without any other contextual knowledge. But then, as soon as you start running attention heads.
David Bau: Then you do have contextual knowledge.
David Bau: And the accuracy goes up to… you know, 70…
David Bau: 70%. And so… so the information about the previous token is being fed in, Goes up.
David Bau: Right? And then it pays attention to it for a little while, and then it starts erasing that, too. Kind of interesting.
David Bau: Right?
David Bau: That makes sense? And then you can ask about, oh, how about two tokens before, like V, right? Oh, it actually pays attention to that, too. It puts a little bit of that information in there, and then it erases it.
David Bau: Over time.
David Bau: Right?
David Bau: And then… and then 4 tokens before, it has this a little bit less, right? And then what's this yellow line? The yellow line's interesting. It goes like… it's not… the yellow line's different.
David Bau: You see what that yellow line is?
David Bau: This yellow line… It's about not looking back, but looking forward, looking at the next token.
David Bau: Right?
David Bau: So can it predict the next token from the representation?
David Bau: Why is it going up?
David Bau: Why is it getting more and more accurate at predicting the next survey?
David Bau: As you go through more and more layers.
David Bau: Why would that be?
David Bau: Okay… Some machine learning person. Tell me, why would it be?
David Bau: Wait.
David Bau: Because of the language model. Why do you say it's because of the language model? That is what it is.
David Bau: What does the TA say? I mean, that's what it is trained on. That's what it's trained to do! Right, so, so basically, you know, just from Lecture 1, right, you know, this is a language model, so it's trained to predict the next word.
David Bau: Right? And so, as you go through more and more letters, it's like… has more information about, like, its bootgas for the next word, until it gets here. So this is about how good this language model is, like, 40% of the time.
David Bau: and guess the next word in Wikipedia, correctly, given the context. So that's pretty nice, right? So it's like, oh, this is good, so it's, like, working, right? The language model's working. But then, at the same time as predicting the next word, it's also keeping track of all these other things.
David Bau: from, like, previous words, and you can see this interesting profile of the information. So that's basically… I just wanted to share this, how probing works. Okay, so the weird thing, the reason that Sheridan wrote a paper and why you could write… so you could write this, you could, like.
David Bau: Somebody would have written this.
David Bau: In 2017, And it would have been a paper by itself, this graph.
David Bau: But in 2024, you can't write that paper anymore. Everybody knows this, right? So, the reason that it's a paper in 2024 is because of this contrast with this other plot, which is the same lines, but with… which is this observation that at certain tokens, the plot looks different. So if you… if you take a look at the word Ent, or the token Ent.
David Bau: and you do the same plots instead of the token MIT, then the accuracies look like this. So what MIT really is a stand-in for
David Bau: any generic token in Wikipedia.
David Bau: Right? But what Vint is, is…
David Bau: Not any generic token in Wikipedia, these are…
David Bau: What Sheridan calls the last token of a multi-topian word.
David Bau: So, you know, some words are tokenized in ways that they have to be split up into multiple tokens.
David Bau: And if you have a word that got split up.
David Bau: And you take a look at the last word?
David Bau: And you do this probing?
David Bau: Then you get this other thing.
David Bau: And what Sheridan says is, she says, It means that Like…
David Bau: That word erases knowledge of what it was.
David Bau: Like, so, like, does ENCH know that it's ENTH? Well, obviously, right, as soon as it comes in, it knows, but as soon as you start exposing it to tension hits, what the heck is the model doing? The model's like.
David Bau: You may have been Ent, but I'd rather not know that you were Ent. I need you to forget this, like, the model's, like, quickly erasing this.
David Bau: And it's also quickly erasing all the other contextual information. It's crazy. I don't know, like, what? So… so basically, Sharon wrote this paper saying.
David Bau: What the heck is this erasure? There's this token erasure going on.
David Bau: Right?
David Bau: Why is it forgetting stuff so quickly?
David Bau: Yes. A couple questions. Yes. The first one, like…
David Bau: the fact that ENK is a common, like, ending word in English, like, that's persistent. So, did… did y'all try, like.
David Bau: changing the prefix, and keeping the last token constant, and seeing if the model's erasure is consistent? Yeah, that's basically what the experiment is. So basically, you know, this is… this is based on a huge scrape of Wikipedia.
David Bau: And so ENT, I'm sure, shows up at the end of a bunch of different words. Probably all these tokens show up at the end of a bunch of different words. And so this is an average…
David Bau: Accuracy over all of that mix.
David Bau: And, and yeah, so, and pretty consistently, ent.
David Bau: If you ask the representation event, do you know what came before you?
David Bau: Then, it does peak over here at layer 3. It does sort of know what came before.
David Bau: But then, pretty quickly, when you get to layer 10 or something.
David Bau: It's basically forgotten that, unlike the normal case.
David Bau: When at Layer 10, it still remembers volcano.
David Bau: Make sense? Does that answer the question, or… Okay. How did, like, they develop the inclusion that we should look at the… To look at this? Yeah. Like, versus, like, look at the space after AWT. Like, how did the graph particul data graph do? Yeah, why look at this?
David Bau: Yeah, it's from the kind of digging that we're asking you guys to do. It's like…
David Bau: What was it, what was it that… so, basically, the real story for this piece of research was
David Bau: that, I had told Sheridan
David Bau: Look for the opposite of this.
David Bau: Like, if you have a multi-token word.
David Bau: Then, the process of concept formation in this
David Bau: Environment where the words split up into lots of tokens.
David Bau: It should require you
David Bau: to have a better awareness of the tokens that came before, if you're a meaningful word. So if you're a word like intermittent.
David Bau: Then, you should…
David Bau: not have the luxury of forgetting what came before you. Like, if you were the word thee, then, yeah, who cares what came before you? You don't need to know that, because you know thee means thee. But if you're intermittent and you're aunt, you need to know what came before you, so you, like.
David Bau: you would have a really accurate view of this. So Sheridan is like, okay, probing, I can do probing.
David Bau: Set up this probing thing?
David Bau: And Sharon showed up a week later, and says, I don't know, I…
David Bau: you know, it's my first week in the PhD, and I… and my experiment's, like, coming out wrong, right? It's like this. It's like the information is not there. I must have set it up wrong.
David Bau: Right?
David Bau: And so, Sheridan basically spent You know, a couple months digging.
David Bau: True.
David Bau: The probing code, and through different data sets, and different ways of setting this up.
David Bau: Until we were finally convinced that
David Bau: Oh, actually, Sharon is a brilliant experimentalist, had set it up right on the day one.
David Bau: And had found this surprise that was exactly the opposite of what we'd expected.
David Bau: That make sense?
David Bau: And so that's, you know, so that's… that's where this came from.
David Bau: Right.
David Bau: Right, and then… but then… then the question is, okay, if it's counter to what you would expect.
David Bau: Why is it doing this? What the heck?
David Bau: Right? And it leads to some other papers.
David Bau: Okay, so… so yeah.
David Bau: Oh, Isaac.
David Bau: What was your question? I was wondering about the relationship between this and patching. Like, the class notes suggested that they're complementary. Yeah.
David Bau: Is it complementary in the sense of, like, if given an option, we should always do…
David Bau: patching, and given resource constraints, this is a viable second bestive alternative, or is there, like, something more complex about the relationship? How we should think about the relationship? It's a little bit more complex. I'd say that they're… I think they really are complementary. So, in the second best, I think that, you know, because probing is older than the practice is patching.
David Bau: You know, I often go around telling people, no patching's better!
David Bau: You should think… you need to do… you need to do causal experiments, because this is just a correlative experiment. You can get fooled in all these ways that we'll talk about.
David Bau: And then I think that you guys already have intuition about that, from the questions, you guys understand how probate can be fooled.
David Bau: But, So, but I think the propane is maybe half the story.
David Bau: I think that if you… if you… you can get… you can get… you can sometimes get a similar picture about, information content from patching.
David Bau: But, it's, but it's different. Your factory is telling you if the model is using some signal.
David Bau: It's not telling you if the signal is present,
David Bau: If it's not used. And… and sometimes you are interested in whether the signal is present, even if in this particular piece of text it's not used. Like, maybe in some other piece of text, it might be used, or something like that, and you just want to know if the model is I prepared.
David Bau: have that information available. There's sort of this… this experiment is kind of like this, it's like.
David Bau: you know, I don't know what might come after this.
David Bau: You know, I don't know if the model needs to know this information, but it's just interesting to know that regardless of whether the model's using that information later, it's just not even there, right? Even, like, a trained probe can't get it out.
David Bau: Right? And so if TrainPro can't get it out, then…
David Bau: The model probably won't be able to get out.
David Bau: And so… So yeah, so it's complimentary.
David Bau: I would, you know, I would recommend for any of these papers, like, make a tripod.
David Bau: Like, try to find more than one way of characterizing
David Bau: Their representation would be kind of brokenness of the powerful. Cool. They should be in your arsenal.
David Bau: Okay.
David Bau: That sort of answers the question?
David Bau: I don't know. Yeah, these things are always in flux.
David Bau: IU had a question.
David Bau: Yeah, so my question is, so it has a,
David Bau: concept vector. Does the prediction use that vector to do some prediction, or it's just correlated with the concept? And I think I answered that just now, right?
David Bau: It's like, it doesn't necessarily use it, right?
David Bau: So, so, so the, the probe set, the probe is using it.
David Bau: Right? But we don't know if the model itself is going to use that information downstream that. Right. I think that's the fundamental
David Bau: Problem with probes.
David Bau: Is that just because information is there doesn't mean it's actually feeding it.
David Bau: Or doing the thing that you imagine. So, you know, this is a classic thing. So, the classic approach
David Bau: And people are very, you know, people are very interested in this question of, does my model have an inherent political belief? Maybe it's biased.
David Bau: you know, maybe my model is biased against professors. I don't like these models that are so biased against professors, right? So, so what I can do is I can… I can look to see if I write a sentence about professors or something like that, or I write a sentence about me, and I can… I wonder if it's keeping track of the fact that I'm a professor.
David Bau: Right, so I'll take a bunch of texts that I've written, and a bunch of texts that non-professors have written.
David Bau: And I'll ask, if I probe that text.
David Bau: Can the model tell the difference between whether I'm a professor.
David Bau: whether I'm not a professor, just from my email or something, like, that would be, like, evidence that it's biased against professors, right? Because it knows, it's keeping track of the fact that it's different, right? And I could train up one of these probes, and what if I…
David Bau: got this hit. What if I found out the probe is super accurate? If you go with layer 15,
David Bau: On the last token of my email, I can tell, I can classify the difference between professor and non-professor email.
David Bau: At, like, 99.9% accuracy.
David Bau: It's a shouting red.
David Bau: It's, like, completely there, the information of whether it's a professor or not.
David Bau: Right. Does that mean… that the model…
David Bau: is, like, thinking about, is this person a professor? I gotta be biased against him or something?
David Bau: It's not necessarily, right? And the information might be there, but it might be that the model doesn't actually use the information for anything.
David Bau: And so… and it might be a spurious correlate, I mean, it might be that it's changed often. So, you know, what is it, like, that I'm probably out?
David Bau: It's like, maybe the probe recognized that
David Bau: you know, the model cares if I'm writing things about
David Bau: Having lots of fundraising meetings, and…
David Bau: other boring professor stuff, or whatever, right? You know, just, like, all sorts of stuff, right? And it's just keeping track of all sorts of other things, and it just happens to be that I can aggregate all those signals and get, like, a 99% accuracy on, like, my professor or not. You know, is that a concept inside the model, or is that just an aggregate?
David Bau: classification results that you can get if you probe a thing. So that's…
David Bau: So anyway, that's… that's the answer… question the answer is no.
David Bau: Okay, so… and then…
David Bau: Anya had a different way of asking the question. Yeah, but I think you answered it. No, I'd like your question better.
David Bau: What was your question? I don't remember, but I remember… I remember… Oh, too late. Okay, then, you don't get the answer. Okay, well, no, you asked it in a slightly different way, would you say? Yeah, I guess that, like, I know this is what the papers were called for, like, correlational probes, which my friend Paula asked, like, which is, like, understanding whether, like, the information exists, whether…
David Bau: like…
David Bau: again, causal probe, I would imagine, is like, how does it use… What would a causal probe be? Yes, and I forgot, I was going to stick in another slide for what a causal probe is, and I didn't have time to come back to add this slide for you, so… but I would point you to a couple papers, so…
David Bau: Like, so you can probe a model to see if you can get a classifier out to do things, but you can also train
David Bau: Train an intervention on a model.
David Bau: that asks, what is the optimal intervention that I can do
David Bau: That optimizes for some causal effect.
David Bau: And it's sort of a duel for a probe. It's sort of a causal probe.
David Bau: And there's been a couple papers written that suggest this kind of thing.
David Bau: So, the two that I would suggest is there's a paper called Amnesiac Proving.
David Bau: Like, a major, if we got something.
David Bau: So… maybe Nikhil can remember Amnitric program, we'll put that link
David Bau: in the Discord for people, if people are interested in trying that technique.
David Bau: On the thing, for a causal probe?
David Bau: And, and then the other thing that people do is they do something called, distributed lineman search.
David Bau: Where instead… so, like, in a probe, you're looking for what subspace, like, what vector, you can dot product, find something to give you the most accurate…
David Bau: Classifier, but you can also do it,
David Bau: an intervention search. You can say what vector or what subspace is there where, if I were to perturb things in the direction of that subspace, it would maximize certain causal effects? And so that… so that general…
David Bau: scheme.
David Bau: People figure out how to operationalize that, and they call it distributed alignments.
David Bau: So that's another business or development.
David Bau: These are… they're not as well established as probes, since, you know, people who are advocating for these things, you know, I think the juris still out in them, but if you're into it, give it a try.
David Bau: Okay, so the TCAP paper and spurious Futures. Okay, Rice, Rice, what was this question about spurious Futures?
David Bau: I don't remember. What was your question? I think I was asking about, I think it touched upon, like, the questions just a little bit off.
David Bau: I'm just asking, like, when features, like, coexist, and, like, they have correlation with each other, and when you have, like, correlation… correlational probes rather than causal probes.
David Bau: Would you, like…
David Bau: So, like, Lauren, something else that, make your prediction on this particular feature accurate, but, like, you're actually, like… Right, right, right, right. And what does it… what does it look like? So, I'll come back to that in a second. Actually, I'm gonna answer Armini's question a little bit more at first, because I did… I did have this slide. I'm remembering now why.
David Bau: I put these things in here.
David Bau: Okay, so, is that right? And then…
David Bau: So, for a causal probe, actually, the paper that you read, TCAF,
David Bau: is a little bit causal. So, because it has two steps. First, it gets this probe direction.
David Bau: Right? By, by just doing a regular, linear.
David Bau: you know, classification training. But then once you have this… That's fire.
David Bau: They don't just recommend just reporting the accuracy, which is what most… that's a standard thing to do with pros, is you just train this thing.
David Bau: And then you take a holdout set that you didn't train on, and then you just report accuracy on that thing, and then that tells you, like, how much this information is present. That's the normal formula.
David Bau: And…
David Bau: And but what they do in TCHAP is they say, no, actually, what you're really interested in is how much, like, the concept of stripiness, it might be present.
David Bau: But we want to know how much is used, In the classification of zebras.
David Bau: Right?
David Bau: And so what they do is they say.
David Bau: If you took this gradient, if you ask.
David Bau: If you take, like, perturbations of the representation.
David Bau: How much does it change the prediction of Zebra?
David Bau: Alright, so that's basically what this gradient is. Basically, we've got the predictions of Zebra, And we're asking…
David Bau: What is the gradient?
David Bau: What is the derivative?
David Bau: Of, of the prediction of zebra with respect to… Vector changes in the representation.
David Bau: So that gives you, like, you know, how much…
David Bau: The change in the representation change zebra, and then you dot product it with
David Bau: This stripey direction that you got from the probe.
David Bau: And so what this is really telling you is, If we made a perturbation.
David Bau: If things were a little bit more stripey.
David Bau: Right? If you made a perturbation, like, this concept of a strike.
David Bau: Then, how much would that proportionally change?
David Bau: your score for Zebra. Would it push it up? Would it push it up a little? Would it push it up a lot?
David Bau: And so whatever this number is.
David Bau: That's, like, the TCAP score of, like, how much stripes
David Bau: Are thought of as causing zebras.
David Bau: That make sense? So, cause that's, cause that's, like, like, maybe I want to know if, like.
David Bau: being a professor leads to bias against me, so I have a different sentence that says.
David Bau: You know, do you like this person or not? Yes or no? Right? And I'm looking for…
David Bau: No, like, the model doesn't like this person, and then I, and I put this professor email.
David Bau: before, and I… and I, like, train a classifier, all the professor emails, and stuff like that. And then now what I want to know is not just, like, is the probe accurate? Like, gonna tell if it's a professor or not?
David Bau: But then I also want to see, if I bump it in that direction to make it think it's a professor, how much is it saying, you know, adjusting the, the,
David Bau: The prediction? No.
David Bau: Awful, right? How much is it making me say, no, I don't like this person?
David Bau: So… so then, then I would say, yes, you know, the model has this class of things that it doesn't like, and one of the major signals that it uses in that class
David Bau: is whether you're a professor or not. Your professor really doesn't like you.
David Bau: And, you know, I can compare it to other things. Like, if you like ice cream, it does like you.
David Bau: Still like that.
David Bau: That make sense?
David Bau: So you can… so this is what the TCAV.
David Bau: Proposal is.
David Bau: Decav is, you know, a well-read paper. It's not a bad technique to try if you want to try to attach some causality, but also.
David Bau: For more modern takes, because Steve happens, we might look at the other papers that I mentioned.
David Bau: But this is… this is one way of getting causality out.
David Bau: Yes, now, okay, now Bryce has a question, right? Yes. Sorry, I was following up on what you said. Yes. I was just wondering, how effective are, like, graded-based interpretations on LLMs? Right. Right. Like, I'm… for example, I suspect, like, a lot of tokens might have, like.
David Bau: non-zero credits. Yes. And it comes to, like… yes.
David Bau: Yes, I think that, it's maybe controversial.
David Bau: Well, I think,
David Bau: After Spring Break, I'm gonna see if I can get Gabriel to come talk about salience methods, which are very, very gradient-based, and I think he's a believer that they have a lot of value.
David Bau: on LLMs, and so I'll let him, like, give you the optimistic point of view. I think that the pessimistic point of view is that these systems are trained on this very discrete data, so it may be a little different from, like, image classifiers, where images are kind of continuous data.
David Bau: But, like, in really discrete data, it's not as clear that the gradient is informative as it is in images. On the other hand.
David Bau: you know, what else do we have, right? So the other thing that we have is we could do a non…
David Bau: minuscular perturbation, we can actually change something in the direction. Like, really, really substitute in a thing, which is what patching is. And so I'm… that's why I mostly recommend people do patching.
David Bau: That makes sense. It's, it's equivalent.
David Bau: Okay, or steering is the equivalent of it, but in macroscopic.
David Bau: So, okay, so now, now, now the other, red question was about spiritual features, so I wanted to tell you a story. How much time do we have?
David Bau: 15 minutes.
David Bau: Okay, so, story time.
David Bau: 25.
David Bau: This is about a paper I'm very proud to have, helped out on, and a mistake that we made.
David Bau: In the paper. So, a research mistake.
David Bau: I'll fess up to you, you're on video.
David Bau: Okay,
David Bau: So this is… this is Cassidy's paper, and it's about emergent world representations. And the idea is, it's about concepts. It's about, like, does a model have a concept inside? And here's the basic setup.
David Bau: The setup is, let's say you were playing Othello.
David Bau: And, with, with, with your friend.
David Bau: And every day, you, you play Othello, but your friend is very loud and formal about Othello, so every time… what your friend says is, he says, when we play, I want you to shout out
David Bau: the move that you make. I'm going in C4. Okay, I'm going in B3. Okay, C6, so you can shop these moves out really quickly.
David Bau: And then you play the whole game.
David Bau: And then… and then, one day.
David Bau: You, you play, you're playing, and then, and then there's this odd sound. You hear this bird going like…
David Bau: C6?
David Bau: Right? And outside, there's this crow that's learned how to imitate your games, and oddly enough.
David Bau: Right? When you look at the crow, right?
David Bau: It's, you know, you say D3, and the crow is thinking.
David Bau: You know, C5, B6, E3, right? It's like, what… actually, when you go and you listen to the crow.
David Bau: And you realize it's actually emitting
David Bau: valid games, and after you say a bunch of games to it, right, the odd thing is, it will always tell you another valid
David Bau: That's how I move.
David Bau: After… after you've done this.
David Bau: So, if you had come across a strange crow.
David Bau: They could utter valid ophthalmos on every word that it said.
David Bau: It would lead you to this question.
David Bau: Does the crow know what Othello is?
David Bau: Right? It's just, like, listen to these words.
David Bau: Does it have… like, what… so what you call it is you'd call it a world model, Right?
David Bau: Not just, like, a single binary concept, but, like, does a crow have, like, A complete, coherent idea.
David Bau: of what Othello is. Does the crow, like, know in its head, like, what the game board is? And certainly, if you, like, went to the… went outside and looked at the crow, and it was standing there on top of an Othello board, and it was, like, pushing little seeds around.
David Bau: Then you'd be like, oh yeah, it knows. It knows. But if it was just saying it, then, like, how do you know?
David Bau: whether the crow knows what Othello is. So, one way of knowing is to use probes.
David Bau: Right, and here's how the probate would work.
David Bau: And this is what we did in our paper, right?
David Bau: We went to the Crow, which obviously is an 8-layer autoggressive transformer, and then went to the Crow, and we went into its brain.
David Bau: went to each layer, and we said, at this layer, I wonder…
David Bau: If it knows whether this square here is occupied by Blackstone or Whitestone in Othello.
David Bau: Right?
David Bau: And there are 64 squares.
David Bau: And so there's 64 probes.
David Bau: like, I wonder if we can get, like, high-accuracy approach for all 64 squares. Like, this collection of 64 approaches is like a world model of Othello.
David Bau: And if you can get high accuracy on all of them, then
David Bau: Definitely, this model has, like, not only information about Othello, but, like, the whole board mapped out.
David Bau: Does that make sense? Is that cool?
David Bau: So there's, like, another example, like, what you might look for. The mapping people might look for things like this, right? Just like this, right?
David Bau: And I'll tell you, this is what we found.
David Bau: We train a whole bunch of probes, and here's, like, the 64 probes, and you get the little vectors for the probe, and they're arranged in these cool geometries all over the place. But the cool thing about the geometry is not just the geometry, these probes are, like, 91% accurate, 98% accurate. They're pretty good, right? If you train the probes.
David Bau: But we train the probes two different ways. When we trained the probes, when we train nonlinear probes, we get 98% accuracy. It tells the story that we wanted to tell. We wanted to tell the story. These transformers have a world model, right? It's like, it's like looking inside the crow, and you see a little othello board in there. It's amazing, right?
David Bau: But the sad reality is that when we… when we actually did this experiment, I kept on hounding, kind of, to train linear probes and get linear probes to work.
David Bau: And the highest accuracy he could squeeze out was, like, 79%, which is, like, rounding up. Yeah, I think that, like, he's probably more at 75%, right? And so, but, like, he, like, the most he'd get out of Linear Pro is 79%. So, like, 20% of the time.
David Bau: 25% of the time, they're wrong.
David Bau: Right? Which… which is not that convincing for such a simple…
David Bau: problem where it's just classifier between white and black, right? It's like, it's just only, like, a bit better than random, right? You want to be in the 90%s.
David Bau: And so…
David Bau: So, he had to write this text, and then… so we did this, we made an MOP and said, so what's wrong with a multi-layer perceptron?
David Bau: Who doesn't like multi-layer posts?
David Bau: Oof.
David Bau: You don't like multi-layered probes. Who that you read doesn't like multi-layered probes? I'll give you a hint. We already read Bean Kim's paper, we talked about that, so who was the other paper?
David Bau: Do it!
David Bau: Okay, Hewitt doesn't like multi-layer probes, and I'll tell you why in a minute, but I'll tell you why through this story first.
David Bau: Right, okay, so this is, like, in the appendix or something. Here's, like, the error rate for Odella on linear process. It's sitting in the 25% range. Like, it's pretty bad, right, for such a simple problem, right? Yes. Two minutes. Okay, so the upshot is…
David Bau: Their shot is… After we published this paper, Like, a month later.
David Bau: Neil Nanda posted this blog post.
David Bau: And it says… Actually, there's a linear probe that works.
David Bau: And he says, if you use a linear probe.
David Bau: Here are the accuracies you get.
David Bau: And where white is, like, zero, right? Or, like, you know, near zero.
David Bau: And blue is, like, 50%. So, near the edge, you can guess some potential. The reason that we're up to 25% is because, oh, maybe at the edge, we're better at guessing, but, like, in the middle, because, like, zero, like, you can't get any accuracy, or 50%, right? So it's, like, random, right?
David Bau: But then says, hey, but look, if I take this classifier, and I don't train it on whether it's white or black on every move.
David Bau: But if I turn it, whether it's white or black, on every other move, Maybe I don't move.
David Bau: It was accuracy, 100%.
David Bau: Right?
David Bau: In the other move.
David Bau: if I… if I train the classifier, but I flip its parity, so it's not answering whether it's white or black.
David Bau: And it's answering whether that square is occupied by the person to move, or the person to play, the person opposite.
David Bau: Right? That is also 100%. So we were… we had…
David Bau: trained the wrong classifier. We were trained the classifier on white or black, when the right classifier was, is this square occupied by the person to move or the opponent?
David Bau: Right? Split-putting. And so, if you think… if you… I mean, so neural network people will tell you.
David Bau: to convert from a flip thing to a not-flip thing is an XOR function, and XOR is the thing that you can't implement in a single-layer neural network, so you need a second layer to do it. So the whole reason you need a second layer is to implement this XOR function.
David Bau: So that you can convert from one to the other.
David Bau: It's like a total waste. Anyway, so… Hewitt says, don't use MLPs.
David Bau: MLPs have this overfitting problem, Oh, we have a lot of other questions to get through.
David Bau: You'll talk about it next time.
David Bau: But the way to really do a paper is that
David Bau: Like, this is the bottom line.
David Bau: The green line is linear probes.
David Bau: And your way of saying, what you really want is you want things that are selective, that they're surprisingly good. You want something that's surprisingly good.
David Bau: You don't want something that was really hard to extract the information out of. And here, this green line is always surprisingly good. It's, like, floating above these other classifiers that you had to train so hard to get something out of. And so… so just… so Pua's takeaway is that
David Bau: Accuracy is not the point of a probe. You're not trying to get the most accuracy.
David Bau: Right.
David Bau: like, it's the surprise. It's like, how much unexpected accuracy you have. Like, better than… accuracy, better than some baseline.
David Bau: Okay, so Hugh would suggest some baseline, and he says the gap between those accuracies is what has chronic health care, but that's… so it's an important point to get. I can talk about it a little more next time.
David Bau: But, but yeah, that's the story of probes.
David Bau: Right?
David Bau: Okay.
David Bau: Maybe Orlando's in front of me now.
David Bau: That's why the train doesn't trust the money.
David Bau: They have some clear is the lack of opinion.