David Bau: What?
David Bau: Let me just…
David Bau: The last couple slides that I didn't get to go over yesterday while we want major people to arrive.
David Bau: And instead of describe what's going on here, and sort of beyond… it'll be on the Zoom recording.
David Bau: We're trans.
David Bau: Yes.
David Bau: Not here at Chuck Rhode Island.
David Bau: How do we translate their ambulance system?
David Bau: Why is it by a mother of nothing?
David Bau: Yay.
David Bau: Great.
David Bau: So…
David Bau: Yeah, so I finished, you know, last week I was telling you the story. It was just sort of my personal lesson of how to do probing wrong.
David Bau: And, and so… You know, the basic idea of our bank, buzz.
David Bau: then, you know.
David Bau: Once you have a probe that compels.
David Bau: the difference between A and G, you know, between one classroom and opposites, then… then you can use the accuracy of the approach to kind of
David Bau: guide you.
David Bau: As an estimate for… How much?
David Bau: How much your representation knows the concept, right?
David Bau: So, so we did that here for… A fellow.
David Bau: But there were two different types of classifiers that we contained.
David Bau: financially trained, linear class times, But the linear classifiers were…
David Bau: And we, we trained multi-layer, classifiers, like, two-layer…
David Bau: And they work, right, like, 98% of the time.
David Bau: And so, so we thought, well, if you contain it and classified.
David Bau: on the page, then that'll… that'll simply show us the information that's in there.
David Bau: So, so we published it.
David Bau: And, and then I sort of told you the story that
David Bau: Atlanta came out a few weeks later and said, no.
David Bau: This LLP, you don't need to do it, you can get the information out with a linear classifier.
David Bau: And, and so this is… I really like this chart. So this is the chart where he says, here's, like, you have white, which is a really bad guess, is, like, 25% long.
David Bau: In the past, I've only been up, and then…
David Bau: And then, you know, black, or really dark blue here, is, like, 0% wrong.
David Bau: And…
David Bau: And so, you know, our MLPs were 0% wrong, but he trained this linear classifier to do it. He trained the same linear classifier that we put in the GitHub.
David Bau: And it was… it was bad. It was, like, 25% wrong, most of the time.
David Bau: But then, when he switched it so that the class buyer predicted,
David Bau: The things not on every turn, but on every other turn. It'd be like, instead of looking at the representation on every word, look at every other word, or instead of every sentence, looking at every other sentence. Or when you're… if you're studying dialogue between two people, like, oh, look at… don't look at, you know, every… every sentence, but every other sentence, right?
David Bau: you know, he got the accuracy up to, you know, 99%. And,
David Bau: And so, what's the heck? You know, the reason was for this?
David Bau: What would kind of be rotated in Python models?
David Bau: It had a representation of the Othello Ford, but it was encoded in a way that we didn't expect.
David Bau: Instead of encoding, a white square is, you know, a white pebble is on this square.
David Bau: And the black pebbles on another square. It was encoding a different concept.
David Bau: Which was not the concept that we had anticipated.
David Bau: with our training staff, it was encoding
David Bau: Is the pebble on this square owned by the person to move?
David Bau: Or the opponent.
David Bau: Which flips. Every day of the time.
David Bau: in settlement.
David Bau: So I think that there's a couple lessons from this. So… One lesson is… Look at your data.
David Bau: So this is, like, one of the reasons that…
David Bau: Before I went into causal mediation analysis and all these other things, I was like.
David Bau: Hey, let's just make some scatter plots to just look at some vectors, and then just get into the practice of doing that.
David Bau: I think that,
David Bau: There's probably other ways of looking at the data, too, but that's the most obvious, obvious thing to try. But if you have a way of just, like, looking at the raw data somehow, so Neil has, like, a bunch of different visualizations he was doing to try to look at the raw data.
David Bau: And just live with that for a while. Like, get a little sense for it. You might find out that you're asking the wrong question.
David Bau: Of your model, that you're not… that you have your own human bias?
David Bau: For how you think the model might represent something, but the model might represent something in a slightly different way.
David Bau: That would… would change what you find a lot. And so this is… I thought this was really interesting.
David Bau: Because it's a clear case.
David Bau: Where the model headed it.
David Bau: And the equivalent concept, But one that… the classifier definition of a concept that we've had
David Bau: Was totally wrong, so wrong, that
David Bau: You know, it gave us this really bad accuracy.
David Bau: And so… So, but then there's a second takeaway designer, which is that
David Bau: We were able to get the accuracy up to 98% with an MLP.
David Bau: Which, which, which leads to this question.
David Bau: Like, were we fooling ourselves somehow? Like, what's wrong with an MLP?
David Bau: Thank you.
David Bau: And that's what… that's what the Hewitt paper is basically about. And so I just wanted to go over this.
David Bau: And, and the basic idea of the Hewitt paper
David Bau: is… yes, before we get into this, did he mention anything about his…
David Bau: intuition for why did he look into BioCDR?
David Bau: I think this was the first finger, where he showed it, and I think that, you know.
David Bau: he was looking at… I think that he was visualizing the data in different direct ways. I don't know exactly which visualization he used. Like, in the end, he had a lot of these kind of color diagrams of what the predictions were and how accurate they were.
David Bau: And, so he was doing… like, this is… this… you might think of this as sort of a logic lens kind of visualization or something, a fellow board.
David Bau: And so I think he was just building a lot of these, sort of, Logic Land-style visualizations, and…
David Bau: And then it… and then I don't know what gave him the idea, That maybe, instead of…
David Bau: maybe the classifier could have been made better by making the problem easier. So this is always the pattern in research, right? In research, you have something, you're trying to make it work.
David Bau: And, you know, you really believe in your hypothesis. You gotta chase your hypothesis for a while. You think that there's gotta be something here, but everything that we do to measure it is like coming up noise.
David Bau: And then the right reaction to that, if you… until you stop believing your hypothesis, right, the right reaction is.
David Bau: you know, is there a different way for me to set up an experiment? Maybe I have some confounder in there. Maybe I've got something that's making it unnecessarily hard. Maybe I've got something in there that's all noise, that isn't a necessary part of the experiment that I can remove.
David Bau: And… and so I think that's, you know, that's what this experiment here is. It's like, O'Neill says.
David Bau: Well, maybe, you know, for whatever reason, maybe doing this classifier every single turn
David Bau: is somehow confounded, right? Maybe it'd be easier if we picked the easy turns.
David Bau: to do. What would the easy turns be? Well, maybe the easy turns would be all the turns where the same person is moving or something, you know, every other turn, right?
David Bau: And, you know, you can see how that would be, like, an easier problem.
David Bau: Right? You only have to classify half of the banks.
David Bau: And, and so… so I, I often will recommend this.
David Bau: when people are doing different pieces of insert things, it's like, oh, your results are really noisy? Maybe some sort of subset analysis.
David Bau: would be useful. Maybe you don't have to, you know, get your result to be clean on the whole data, maybe on a subset, maybe on half the data, maybe there's a certain third of the data
David Bau: where your result is really clean. What is that one-third? If it's systematic, then you still learn something, right? And that's basically what Neil did here. He says, okay, let's… let's do it on half the data.
David Bau: And, and… and boom!
David Bau: It worked.
David Bau: I was not a very experienced advisor.
David Bau: when I was working with Kenneth on this problem otherwise.
David Bau: I'm very embarrassed that we didn't do it.
David Bau: You know,
David Bau: You know, all I did was, how Kenneth really should be a linear classifier, right? And he tried. He tried really hard.
David Bau: But we… we didn't do something like, let's do a sub-head analysis.
David Bau: Okay.
David Bau: So,
David Bau: The second lesson, though, that comes out of it is NLPs can kind of fool you, and that's what the John Hewitt paper is about. And so I just give you a little bit
David Bau: of… Overview of what's going on here.
David Bau: And so, the idea here is…
David Bau: So this diagram here is the diagram of, like.
David Bau: Actually, I don't really even know how to understand this diagram, but it's a diagram of what a control task might be for part of speech tagging, and…
David Bau: And the basic idea is that normally, to do part of speech tagging, you have to do a lot of interesting things. You might have to disambiguate ambiguous words, like I was telling you, play can sometimes be a noun, sometimes it can be a verb, it can sometimes be transitive, it can sometimes not be transitive, and whatever, right? And so… so to figure out, like.
David Bau: what kind of verb it is, or what kind of noun it is, you sort of have to understand what's going on in the sentence, and so part of speech tagging is kind of tricky. You gotta do this thing.
David Bau: And and so, what John says is, instead.
David Bau: what he's gonna do… so he says, so… so if you train a classifier to classify part of speech tags out of one of these networks, you'll find that it has a certain accuracy. It can, like, get the part of speech tags out
David Bau: With a linear classifier at, you know, maybe 85% or something like that, which, you know, is okay. And then he says, but he's frustrated because a lot of people publish papers saying, oh, I can do better.
David Bau: Then the last person who wrote a probe paper saying that I can get this out at 85%. I have a new paper, and I can get it out at 99%.
David Bau: And I've done it using a multi-layer perceptron probe, like a fancier probe, like a, you know, more complicated machine learning thing. And so now I've really proved that the information's there. It's like 99%.
David Bau: And… and John's point is… That that's not really… The point of it.
David Bau: I, you know, we had, we had a math class in here just, A few minutes ago.
David Bau: And I feel like it's kind of like asking people, can you solve this math problem? And you give them a pencil?
David Bau: And then, you know, it gives you a measure of why they can solve the math problem. And they say, yeah, they're learning something from the class, like, 80% of the people can solve the math problem. My probe has succeeded in telling me how much they know math.
David Bau: And then… and then somebody else says, well, I'll really tell you whether they know math. And they distribute, like, pocket calculators to everybody. And they say, with pocket calculators, can you solve this math problem? Like, it's like 100%, like, everybody can solve it. It's like, so now, does that give you, like, you know, a better sense that everybody knows what's going on in the math? I'm not sure it does!
David Bau: Right?
David Bau: And so I think that this is more or less what… You know, John Hewitt's…
David Bau: Point of view is on the difference between a linear probe
David Bau: and then LLP. And the way that, He… he proved it.
David Bau: Was to come up with a control task.
David Bau: baseline suggestion. And so what he says is that what MLPs are good at is they're really good at learning how to solve
David Bau: Due to problems.
David Bau: That your neural network didn't already know how to solve.
David Bau: Does that make sense? Like, oh, it's like… the analogy would be, like, not, like, giving calculators to everybody in class. What if you gave everybody ChatGPT in class, and then you gave them the test?
David Bau: Right?
David Bau: you know, ChatGPT is not only, like.
David Bau: good at solving the math problem. It's good at, like, solving new problems!
David Bau: That, that, you know, that you haven't seen before.
David Bau: And so, you know, even if you come up with a really creative test with all sorts of new things that are getting the students to think they're using ChatGPT, it's a, you know…
David Bau: That test instrument that you're using.
David Bau: might hide… might hide the signal. And so, the idea here is that the control task is
David Bau: the new task. So you've got… you've got a task that is your regular hypothesis, like, part of speech tagging, and the control task.
David Bau: It's something that would be hard to learn.
David Bau: But… That would be kind of meaningless.
David Bau: And so, the thing that you did here is random tagging.
David Bau: So he went to every single word, and he gave it a random tag instead of part of speech tag.
David Bau: Like, so what is this? It's like, cat is normally a noun.
David Bau: But instead, Kat is part of speech 37, or something like that. I don't know, like, he didn't…
David Bau: you could have shuffled it and said, oh, part of speech is a verb or something like that, right? And then dog is normally a noun, but, like, dog and I always, like, part of speech 15 or something, like, so cat and dog are, like, in different classes, just totally…
David Bau: Totally messes it all up, right?
David Bau: And so, if you want to… So, the control task.
David Bau: You'd have to sort of memorize this nonsense Garbage. And… Make sense?
David Bau: And so, and so the idea is.
David Bau: If you give this special AI to the system.
David Bau: and you ask it, can you use this AI together with your own brain?
David Bau: To solve the part of a speech task.
David Bau: And can you use the AI together with your own brain to solve control tasks?
David Bau: And it gets, like, 100% on both of them.
David Bau: And that's sort of an indication
David Bau: that the AI is doing all the work. Because we don't think that your own brain is actually doing the control task.
David Bau: Does that make sense?
David Bau: We don't think that your own brain actually represents cat totally different from dog, like… Do you think that…
David Bau: Like, it thinks both of them are nouns, kind of similar nouns, if that makes sense?
David Bau: But the control task has some other garbage thing, which we don't think is in your head.
David Bau: And then, so if we give you a probate tool that measures 100% on that.
David Bau: It's probably not measuring what's in your head.
David Bau: Make sense?
David Bau: So that's the basic idea.
David Bau: Of the Hewitt thing.
David Bau: And so, here's the… Here's the regular probe.
David Bau: And then, here's the control task probe.
David Bau: And here's what happens when you have this MLP, and you make the MLP really weak, so you make it so it's barely an extra layer.
David Bau: You make the extra layer, like, not too many neurons?
David Bau: Right? And then here's where you make it really fat extra layer, right? You've got, like, a lot of extra, you know, neurons in this extra layer.
David Bau: Right.
David Bau: And… and when the MLP has a lot of extra neurons in layer, you can see that it can take this garbage task.
David Bau: And get it to near 100%.
David Bau: Which is… which is meaning that the MLP is learning the new thing by itself.
David Bau: It's not that it was already there, and that…
David Bau: In the brand that you're testing.
David Bau: Make sense?
David Bau: You get it? But here, when the MLP is really weak.
David Bau: You know, you can get really mad.
David Bau: Bye.
David Bau: Any questions about this?
David Bau: Shuyi. Is she here?
David Bau: Yes.
David Bau: Yeah, the adjusting is very strong, I could go back then.
David Bau: We will say that if it's not a blueprove, then it's not fair.
David Bau: That's right.
David Bau: So, you would say, if it was here, you would say, You're at 99%.
David Bau: But… Like, for the red one, you'd say it's 99%?
David Bau: But… You know, it's not a very good program.
David Bau: And so, actually, what, what these, what, so it, the…
David Bau: the logic is actually… what John would say is, it's this blue one, which is at 99%, which is not a good probe. So even though the blue one is measuring the thing that we really believe
David Bau: Isn't it?
David Bau: John would say, don't believe this blue one.
David Bau: Because the gap to the red thing is tiny, the selectivity's…
David Bau: low. So even though the accuracy is 99%, the selectivity is, you know, 1 or 2%.
David Bau: So is that… so do you think that that's fair?
David Bau: What do you think is unfair? I don't think it's unfair, because that means the probe… if the probe is very strong, then you always think it's not a good probe.
David Bau: You think if the probe is very strong, then… You always think it's not a blush probe.
David Bau: Both are not sprint.
David Bau: Well, no, but this probe… You know, let's say this… Blue Point was up here.
David Bau: Let's just say it was up there.
David Bau: You wouldn't say it was bad, you'd say it was good, because the gap between it and the red thing is pretty large.
David Bau: Yes, ma'am?
David Bau: That makes sense, or do you think they… Yeah, I think one way to describe it is, like, just because you give somebody chat GPT and they can solve
David Bau: doesn't mean they don't know how to solve it without chat. Like, it could be… Or maybe it's unfair. It could very be the case that they still know how to solve it, but I think the question… I see, I see. …that what John is saying is… I see.
David Bau: how big of an update is it? That's right. Even though… So we can't fail you… We can't fail you, but we can't say that you've learned it. That's right. That's right.
David Bau: That makes sense. So, so if we, if we, like…
David Bau: If… if you get, like, 100% on the test.
David Bau: And by using ChatGPT. And then you can also get, like, 99% on a garbage test using ChatGPT.
David Bau: Can we fail you?
David Bau: And the answer you're saying is, no, that's not fair. We can't fail you.
David Bau: Right? But what we can say is it's not a very good test. It's not a game bunch. Does that make sense?
David Bau: That's right. Yeah, but it's not enough to fail you. It's not enough to say that you're dumb.
David Bau: Right. Okay, maybe that answers your question. Okay. But… but it does mean… so what I'd say is, is selectivity fair? It's… it's… it's fair against the test, is what it is. So we're backing up one level and not asking how good the model is, we're asking how good the test is.
David Bau: And it's a good way of saying it's a bad test.
David Bau: I'll say. Okay, so… Going on, had a… had a comment.
David Bau: I'm not here.
David Bau: No.
David Bau: Not even online.
David Bau: That's… Did pop up.
Guangyuan Weng: Oh, can you hear me?
David Bau: Yes, what was your question?
Guangyuan Weng: Oh, oh, yeah, I was asking, like, why do they argue regulation to reduce generalization gap is not sufficient if your goal is selectivity?
David Bau: Right.
David Bau: So… Is regularization enough?
David Bau: I think that, in a sense, this plot
David Bau: Is one of ex… an example where it says, well, regularization is not nothing.
David Bau: So, one form of regularization is to just reduce the parameterization.
David Bau: Right? And… and and you can see that it improves sleep selectivity.
David Bau: Here's more different types of regularization here. So here… They're doing weight decay.
David Bau: Right?
David Bau: And some other things.
David Bau: And… And I guess what they're… So… So, I'd say that…
David Bau: The conclusion from here is that you know, regularization, I… that's how.
David Bau: Somewhat.
David Bau: Right, when you regularize more, When you do… this is sample count, reducing the steps of early stopping.
David Bau: You reduce the rank, you know, reduce the number of parameters, you have higher dropout.
David Bau: You have higher weight decay, you know, so all the higher regularization things are on the left.
David Bau: You can see selectivity generally rises.
David Bau: when you go to the left. So… so the paper isn't…
David Bau: if you looked at… I don't know about the words of the paper, but if you look at the actual results, the paper isn't totally down on regularization.
David Bau: It's like, okay.
David Bau: But then there's this other trend that… he notices.
David Bau: That's even stronger than the regularization, which is the difference between the green lines
David Bau: The red and the blue lines.
David Bau: Does that make sense?
David Bau: And so the green line sphere, Are all the linear models.
David Bau: And the red and the blue lines are all the MLPs, like the… Of the nominator models.
David Bau: And…
David Bau: And for the same amount of weight decay, or dropout, or early stopping, or whatever, the green lines are pretty consistently
David Bau: Like, a pretty big gap.
David Bau: It's selectivity over?
David Bau: the red and the blue models, and so that's why John Hewitt is like.
David Bau: I don't know about… I don't… I'm not going to recommend that people use regularization, I just recommend people use linear models, like a simpler solution.
David Bau: Right.
David Bau: You'll get… you'll get better selectivity.
David Bau: Does that make sense?
Guangyuan Weng: Thank you.
David Bau: Sure, sure. And so, now,
David Bau: So I think this is a little bit of the background why I kept on harping on our student, Kenneth.
David Bau: To try to do a linear model.
David Bau: Because of… because of this kind of thing. Now, some people will still probe with nonlinear models, because it does tell you something, but you have to watch out for this type of issue.
David Bau: Related to overfitting, and…
David Bau: And you know, it's a classic machine learning, problem to be aware of. So… and selectivity is… and, you know, designing a randomized control task
David Bau: If you have a complicated probe, even if you have a very high-dimensional linear probe.
David Bau: When you write your papers.
David Bau: If you want to get them through peer review, this is not a bad way of convincing a reviewer
David Bau: That your probe is a good test.
David Bau: Right, come up with a random… Control task. Say, John Hewitt.
David Bau: Say we're gonna, like, show the gap, we're gonna measure the selectivity against this thing, you might have to put the chart in the appendix for room.
David Bau: But, you know, if you can show a gap like this.
David Bau: And then you'll convince a reviewer. Even if it's not linear, even if it's MLP, you could defend it as well. Yeah. If the idea is that, like, performance is only meaningful if it, like, disappears when, like, the meaningful structure is, like, destroyed.
David Bau: How do you, like, what kind of intuition do you use to identify that, like, meaningful structure, like, that we need to control of?
David Bau: Oh, like, what a meaning, like, what a good control task is? Yeah. Yeah, you asked that question.
David Bau: What a good control… what a good control task is. Yeah, because I feel like…
David Bau: the control task that they… they're like, control tasks are a function of word identity. Yeah. And I'm sort of like… I want to just give every word a random part of speech. Yes, and I think that, like, that seems to tell you, like, how well can you compute
David Bau: random lookup tables as a function… like, I feel like how hard it is to compute a random lookup table depends on how hard it is to derive word identity, and… or how hard it is to determine the function that you're interested in. And so I don't really know if there's an obvious…
David Bau: Because, like, when you saw his… so he had another control task for it.
David Bau: I don't know if I have the thing, like, part of speech tagging, but then also, like, you know, dependency relationships, you know, so, which he said, oh, we'll have a weird control task, which is we'll randomly send an arc to the left, or to the right in the middle, or something like that, right? So,
David Bau: And and then you look at that and you think, huh.
David Bau: I don't know if I would have invented that control task. It's just, like, some random… some random thing, right? So, like, a random part of speech for every word, yeah, sure, that, like, maybe. Like, you know, so there's a classic…
David Bau: experiment that machine learning people like to do, which is, they like to say, you know, how powerful is my…
David Bau: Network, by neural network.
David Bau: By taking…
David Bau: is shuffling all the labels. So you have, like, you know, a million images, and some of them are dogs, and some of them are cats, and some of them are sailboats, and some of them are trucks, and whatever, right?
David Bau: And, and then what you do is you just take exactly the same dataset, and you just completely scramble all the labels, and then you train your model on it to see what accuracy you get on that. It's exactly the same.
David Bau: as this experiment here. But it's… it's like the… but what they call that…
David Bau: Is they call that the memorization baseline.
David Bau: Right, so there's these two things that you can have a model do.
David Bau: Machine learning, people like to say that you can have a model generalize, or memorize.
David Bau: And so when a model generalizes, it's supposed to be finding regular patterns in the data, and after it sees a few of the same regular patterns, then even if you show it a new example of a new sailboat that it's never seen before, a new cat that it's never seen before, because it…
David Bau: recognize the patterns for a sailboat, or a cab, right? It can guess.
David Bau: The new one.
David Bau: Right? But the other thing that a network can do is just memorize. It can remember that I saw exactly this cat before, and this cat.
David Bau: is A. And I saw this other cat, and this other cat is Class B, and this sailboat here is Class A.
David Bau: And this different sailboat is, like, Class B, and it's just like, you know, they're just all scrambled. Some sailboats are A, and some sailboats are B, and some cats are A, and some cats are B, and they don't look anything like each other.
David Bau: But… If you have a powerful neural network, it can…
David Bau: Remember that anyway, after you train it for a while.
David Bau: But it won't do what we call generalization. If you show it a new cat, then it will have no idea, give it a random label.
David Bau: Because it has no idea what… Random hits.
David Bau: Does that make sense?
David Bau: So, so that, so, so, so that a classic control task is to shuffle the labels.
David Bau: So, like, if you need to come up with a control task.
David Bau: Then, the natural thing to do… would be…
David Bau: to just say, well, just like Hewitt, we're gonna shuffle the labels.
David Bau: you know, we have stripey things and not stripey things, and we'll have a control task, and we'll just, like, randomly assign stripey things to not stripey or something like that. And we'll see how well that program does that.
David Bau: But… but he would say, well, sometimes that's… Too hard.
David Bau: And, like, if you really shuffle every single thing, it might be too hard, so you might want to arrange things in groups, and then shuffle the groups, or something like that, to make it a little easier.
David Bau: And the reason is because he's looking for this gap.
David Bau: control tasks too hard, then the control task can, you know, sit down there at low accuracy, even though
David Bau: you know, he's arguing that what you want to do is you want to find that there's a meaningless task that it could have learned instead, so…
David Bau: You know, so this is a whole… it's kind of a hole in his analysis.
David Bau: If you need to come up with a good control task that reveals the weakness of your… your test.
David Bau: Right.
David Bau: Make sense?
David Bau: It's like… I think that after…
David Bau: After spring break, I'm supposed to have
David Bau: A professor coming… come here and watch me teach.
David Bau: And,
David Bau: And so, you know, they're gonna see if there's a hole in the way that I teach. Something that I'm doing that doesn't convey to you guys the actual knowledge, but just makes you feel like you have the illusion of getting knowledge, or something like that. And,
David Bau: And so, but… but what's the test, right? Well, you know, you could devise a test.
David Bau: And, and, you know, it passes it with flying colors, but maybe with a more stringent test.
David Bau: you know, You could say, oh, there's a gap here, there's a selectivity problem.
David Bau: And, but there's an art to Camper.
David Bau: So then the last thing I wanted to talk about probing was other things that you can do for probing. There were a bunch of questions about… from Ayush and Eunice about, probing ideology and personality and biases and things like this. And so I just want to point out that there's been a bunch of work
David Bau: Doing probing for interesting things. There's this nice work from Rita Chen over at Harvard, that builds probes that look into LLMs that tell them… that tell you, like.
David Bau: you know, does the LLM think bed.
David Bau: You are.
David Bau: a certain age, a certain gender, a certain socioeconomic status, a certain education. And what these classifiers are, is they're basically probes.
David Bau: You know, I think they just trained a bunch of linear probes on a bunch of text.
David Bau: that came from young people, that came from men and women, that came from whatever, right? And they said, you know, how accurate is the model at
David Bau: You know, classifying
David Bau: this information, and they took their high-accuracy probes, and they hooked them up to a user interface. So there's… so yes, so you can probe for things like beauty, you can probe for things like socioeconomic status.
David Bau: And, you know, and… and it often works pretty well.
David Bau: Okay.
David Bau: That's all I wanted to talk about for pros. Now let's go on to the presentations.
David Bau: Any other questions about probes before we move on?
David Bau: Any probing questions?
David Bau: Meanwhile, we have Team QK up here. Like, really any questions I have to answer any questions about pros.
David Bau: Yes. Could you probe a probe? You probe probes? Yeah. Like, if you had an LLP probe, could you apply the linear probe on that to see if it…
David Bau: Yeah. In a sense.
David Bau: you know, in a sense, that's what we're doing. We're doing this weird thing.
David Bau: Right?
David Bau: Because our neural networks are already machine learning models, they're already fake ones.
David Bau: And then, it's like, why are we taking these fake models and putting other fake models on top of them?
David Bau: And it's this weird exercise. So you can probe a probe. That's actually all that we're doing. Like, the LLM itself is a probe.
David Bau: You know, for text.
David Bau: The probe for text is, like, tells you what the next word is.
David Bau: And, and we're probing little pieces of it.
David Bau: I think the way to think of a probe is, like.
David Bau: So is there… have… have people probed probes?
David Bau: Oh, no.
David Bau: I don't know if people have done that, yes? I think there was, like, some stuff where they tried to decompose
David Bau: probes into SAE features, so they did some kind of weird… Really? And then, yeah, I think they did some stuff, but… Alright. Ask Grace.
David Bau: Grace will tell you on Discord if people are pro-probes.
David Bau: Grace is… Grace is our labs programming expert, so I'm a little… I'm a little embarrassed.
David Bau: Great last name for that. You know, lecturing about probes right around grayscale.
David Bau: So, Yes, good question now.
David Bau: Alright, Team 2K.
David Bau: Yes, ready?
David Bau: Good. I have some slacked food. Why not?
David Bau: I like high nuts. Yeah.
David Bau: It's not just on probes. Oh, yeah, sure. Okay. Yeah, it's okay. You guys are now into the… I think that you have enough methods that you can just do whatever. Okay, cool. Yeah. Because I haven't… I, like, kind of restructured.
David Bau: The task a little bit, and,
David Bau: So, okay, we'll just start from here.
David Bau: You want to push the next time? Okay.
David Bau: Okay, cool. So, I basically… we're the ones that are trying to better understand if there's some sort of, like, speaker role represented? So that I have time. Sorry.
David Bau: How much time do I get?
David Bau: They're restricting really fast.
David Bau: Okay, yep, so basically we want to know if there's role representations in guidelines when you give an LLM a transcript that has not already been differentiated.
David Bau: And so, this kind of comes into world entity a little bit, but the question is, do LLMs encode with speaking as some sort of structured signal? There's a theory linked to this back from 1990 in Smolensky's work, where he looks at tensor product representations as, like, rolls and fillers.
David Bau: And so, in a transcript, the role would be speaker 1, Speaker 2, or something, and then the filler would be, like, Alice Font, so…
David Bau: The LLM needs to, like, bind Speaker 1 to Alice, and Speaker 2 to Bob, and then whatever Alice says later on has to get bound to Rule 1. Whatever Bob says later on needs to get bound to Rule 2.
David Bau: That's, like…
David Bau: a very theoretical framework that Smolensky created, and his whole take on that is that there should… it should be decomposable through linear operations.
David Bau: Right, and so then there was this, like, empirical link from more recent times about how do language models find entities and context to show models can track entity role assignments, so that's basically what we're trying to explore.
David Bau: So, I got Claude to give me 20 different dialogues between two speakers, Alice and Bob, and they have 80 turns each turn, meaning, like, the text that Alice said is one turn, the text that Bob says is another turn.
David Bau: Then I did some length control so it didn't get too crazy, and I like the speaker swap stuff, ignore it for now, because I thought it would be a good, control, but it's probably… now that I think more about it, I don't think it is, but… yeah, so basically, this is just a screenshot of what I have, like, there's 6 different topics Alice and Bob like.
David Bau: Have some sort of, like, conversation or dialogue or a debate on these things.
David Bau: Okay.
David Bau: So I'm looking at residual stream activations. I'm only extracting from layer 20 right now. I will eventually scale up to other layers.
David Bau: So basically, for each turn of the dialogue… oh, I should've…
David Bau: This is a typo here. It's not the full dialogue but turn, it's, like, the dialogue up until the current turn gets fed into the model, which is, like, as a text string, so it's been concatenated. And then the context is,
David Bau: The only tokens that gets tokenized, after the forward pass and the hidden states are captured, we're only looking at the tokens of that current turn text here.
David Bau: We're not looking at anything else. And then, so these are averaged into a single vector.
David Bau: per turn.
David Bau: Now, because of the results I'm going to show, I'm actually going to change this a little bit, too. I don't… I don't want to average it anymore, because I think it might diffuse a signal. I'm going to look at the last token of each turn moving forward, but I just wanted to say, like, for this one, it's averaged.
David Bau: Okay, so the first thing we did is, for all the dialogues together.
David Bau: do a PCA, and try to see if you can tell if there are separable clusters. Not really, as you can see. And so that's the big takeaway. It's like, and here you're just seeing the swapped label condition just overlap with what it should overlap with. So Bob swapped, Alice base, Bob base, Alice swapped.
David Bau: Okay, so then I was like, okay, well, let me see what one dialog looks like in this global PCA space to see if that's, like, a little bit more separable, and it seems like maybe, maybe not. Like, I could probably draw a line right here.
David Bau: Something else that's really interesting about this is that
David Bau: But, so each of these dots are turns, right?
David Bau: Down here, these… this is, like, turn 0, and then this is turn 1, and this is turn 2. And so when we get up here, this is the later turns. I thought that was cool. I don't know what to make of it, but… I don't know.
David Bau: And so, now that I see this, I looked at a couple different, like, dialogues like this and saw
David Bau: By eye, I feel like I could probably draw a line. That was, like, pretty good.
David Bau: So, I was like, why don't I look at this per transcript then?
David Bau: So…
David Bau: Within transcript, we're looking at the role axis over time, so is there a stable role axis within this dialog? So, what we're doing is we basically have these two clusters, right? And so I want to find the centroid of the Bob cluster and the Alice cluster, and then I want to find the midpoint between those two clusters.
David Bau: And so the idea is that
David Bau: we're going to give it a score, like, for each turn gets a score, where we take the turn, which is X sub T, minus the midpoint, and then take that dot product of that with the roll direction vector, which is just, the centroid of Alice minus Bob.
David Bau: In this case. And so that's what I'm kind of showing, like, plotting here, and you can see now, clearly, there is some separability here, right? And so everything that's blue is Alice, everything that's orange is Bob. So I'm basically just projecting each turn onto this, like, vector rolled, roll direction vector thing.
David Bau: And here, I just highlighted where this model messed this up.
David Bau: like, I pulled out the actual, like, text to see why. They're… they are not, like, they don't have great personalities, it's all very neutral, so it's like nothing obvious jumped out to me, the human, about why the model miscategorized these.
David Bau: Can you make the screen big? What is the text? How do I? I just have a slideshow.
David Bau: The little white button in the upper, yeah.
David Bau: And so, which, which line of text was the one that was… So, this is Alice, the blue dot, which is crossed over into the wrong region. This is Bob, Alice again, and then Bob.
David Bau: And they're talking about… How cities should be structured.
David Bau: So, I actually pulled out a couple different projections like this for the dialogues, and some of them were messier than others, but generally speaking, the majority did separate them.
David Bau: So, I wanted to, like, figure out a better way to kind of,
David Bau: like, define that. And so now what we're doing is taking this, like, role direction vector per transcript.
David Bau: And then we're looking at the cosine similarity. And so obviously on the diagonal, they're going to be perfectly similar. That's good, so it worked. And then off-diagonal, you see a lot of red, some gray, some blue. So basically, the red is gray, because it's sort of pointing in the same direction, or at least has the same sign.
David Bau: Gray means that they're completely orthogonal.
David Bau: And then blue means it's pointing in the opposite direction, basically, so it's, like, not super great. So there's something, I think… this is where you're… I'm like, okay, I messed this up somehow because I think there's a confound in the transcripts themselves, which is what's maybe actually getting picked up, not the role direction vectors themselves.
David Bau: So it could be, like, the stances that each Bob and Alice take. It could be, like, one of… maybe one of them is arguing pro something, the other is saying, no, con. That might be actually what we're looking at, as opposed to speakers.
David Bau: Okay, and then… so this is, just to summarize it all together, we're looking at distance over dispersion versus silhouette.
David Bau: And so, distance versus dispersion is kind of going back to what I talked about earlier. We're just looking at the between-centroid distance between the two clusters, and then…
David Bau: clusters can be different, because they might be dense and, like, pulled in together, they might be really, like, scattered. That plays a role in separability, and so we, like, created the separation score using… between-centered and then cluster dispersion.
David Bau: And so what this says is how far apart are the centroids relative to the spread of each of those clusters? And silhouette, I just used this from scikit-learn, it's how close are the points to its own cluster versus the other.
David Bau: And so, on the y-axis, you're seeing silhouette x-axis, you're seeing distance over dispersion, and so, basically, like, the more separable they are, the more within one cluster.
David Bau: Okay, and then I kind of, like, tried to take some inspiration from this, like, concept of function vectors. So, I wanted to see if, across dialogues, if we could learn a single role detection vector from training data, and then classify held-out turns.
David Bau: And use the sign of the projection on that direction, and so I compared it against some baselines, like label shuffling, and then I also tried just random directions.
David Bau: So, it's barely, like, above chance, in terms of roll direction, and label shuffle and random directions are at chance, as they should be.
David Bau: And again, this is a cross-dialogue, so this is where I was like, okay, I need to try to look at it per transcript. So then I had to split the transcripts' turns themselves into train tests, so it learns transcript-specific direction, and then on the training turns, and then we evaluate it on the held-out turns. Here, again, you see this, like.
David Bau: Distribution, where it is… it does get pretty good for many of the transcripts, in fact, but there are some that are below chance or around chance, so…
David Bau: This might also have to just do with the dialogue, the topic, how exactly Alex and Bob are speaking to each other, things like that.
David Bau: Okay, the linear probe experiment, I don't know if I did it perfectly great, but, I, again, I trained this, directly on the average directors from the residual stream activations, then…
David Bau: So I used it to try to predict role, which is what we care about, speaker A versus B, variant, which is the original dialogue or speaker swapped.
David Bau: And then topic, because I had this, like, nagging feeling that topic was the issue here. And so, as you can see, the linear probe seems to have better accuracy when it comes to topic, based on basically for variant and role, so… Nice. Yeah.
David Bau: And then I tried it, per transcript for role decodability, and you can see it's just above chance, not, not that much higher.
David Bau: Nice. Nailed it, that was the last slide. Great. Yeah, nice experiment, nice negative result. So you can see the value probes, it's like, oh, yeah, my broken mistake, I didn't want to learn, it's perfect. Yeah. So I, I'm gonna try this again on the last, token.
David Bau: Just to see if it, like, kind of gives a clearer signal or something.
David Bau: I… it's one question, I maybe missed this part, but in the part where you calculated the vector, are you explicitly, like, have you given Alice and that speech to the vector, or is it just…
David Bau: the speech. Which vector? The one before this? When you're doing the… the one before this.
David Bau: Yeah, this one, low direction vector, so when you're calculating U of A, I'm assuming that's Alice? Yeah. So, is the, model… does the model have access to the fact that Alice did speak this, or does the model, like, just have access to the text?
David Bau: This is not… this is not the model, it's just, like, I'm just taking all the vectors, and taking the mean of the vector, and then taking the mean of… this is, like, this is just kind of geometry, it's like, there's no model here doing anything. Yeah. How do you get the vector?
David Bau: The vector is, like, so I have the… all of the, turn vectors, right? And so, so, like, there's a bunch of turn vectors for Alice, there's a bunch of turn vectors for Bob, and I just take the average.
David Bau: For Alice, does the turn vector have added certain… No, no, no. Yeah, oh, so you're saying the label is completely separate. It's like, the model doesn't see it, the… like, yeah, yeah, yeah, yeah.
David Bau: I'm sorry, I thought you meant, like, the models I used to… No, no, no. Yeah.
David Bau: Do you think that might be too much of an implicit signal? I don't know, because…
David Bau: the text that Alice is speaking may not, when you're deriving the vector, may not be, like, related to what Al, like…
David Bau: This is Barcelona. Yeah, most likely. I mean…
David Bau: the model, when you give an LLM a transcript, it is able to be like, oh, there's two spiders here, right? Like, if you use, like, as Yasmin's shown, so many examples, like.
David Bau: So how, like, the whole point here is, like, how does this figure it out? Is it… is it really contextual? Just, like, finds people with, like, two different opinions, and it's like, that can't be the same person, or… or it's one person with a split personality? Like, you know what I mean? Like, how does it determine this exactly?
David Bau: It's kind of, like, how we're trying to hack it.
David Bau: It was nice that you saw the separation for a single transcript. Yeah. And I guess you're… I was surprised at how many dots you got. Actually, it looks like your transcripts are pretty long, like a single one. Yeah, 80 times. And so, are they long enough that you can have it hold out, so you could, like, drop out?
David Bau: a bunch of things, and then… Yeah. …test on the holdout. Yeah, yeah, I think… I mean, I think I can make them longer, too, but I think 80 is quite a lot, and plus, like, I have 20 different dialogues, I can just… I've been able to repeat it and look at the standard deviation. I see, I see. I wonder if we could prop them in, like, give them personas, so they're, like, more distinct, too.
David Bau: Yeah, like, no, like, the personalities we're talking about, making them, like, maybe goofy or fun, like, or… this kind of goes to the power group, like, having one be an advisor versus student role, like, that kind of thing would probably… but then it's, like, we're getting away from, like, how… what is… how do you disambiguate role from, like.
David Bau: character that's played by the role. The other thing I wanted to try for an experiment here is that both Alice and Bob have two different stances in the conversations. I want them to agree by the end. So, like, I'm gonna tell the Alan, like, starting turn 50, start making them see each other's side and come to an agreement.
David Bau: Because then I want to see this plot again, and then see if it completely collapses at that point.
David Bau: Is, like, the next, like, thing to kind of test that.
David Bau: There's also… a good setting for that control task idea from John Hewer, right? Yeah, yeah.
David Bau: Like, if you took the same transcripts and you shuffled Alice and Bob somehow.
David Bau: Would it make it harder for you, bro, to figure out this plan? Yeah, that's true. …be able to do this anyway?
David Bau: So instead of a speaker swap, just, like, random. If you just randomly said half of these things were Alice, half of these group out, just shuffled.
David Bau: But, like, linear props can be pretty powerful, too, so, like, I wonder if we can use, I appreciate it.
David Bau: Yeah, well, then it would be like, oh, this is not so cool after all. Depends on everything else, also. Yeah, but yeah, it's just gonna be…
David Bau: Okay, yeah, click on the next team so I don't know who it is.
David Bau: Doesn't have a great job.
David Bau: Lose, lose representatives feedback today.
David Bau: I like the choice that you guys made of picking a presenter.
David Bau: That's not a bad way to do it. But you can just, like, choose a different victim every week.
David Bau: Super stop. So, last week, what did you do?
David Bau: Oh, yeah, absolutely.
David Bau: So basically, last week, we are trying to separate the, label from trinary and binary. Binary is, like, without, without uncertainty. The trend rate is, like, low, medium, and, no, no, intermediate, and high uncertainty.
David Bau: And this is the result we get. As you can see here, first of all, the binary, AISR accuracy is much higher than the trinary.
David Bau: result here.
David Bau: It's here. So… but surprisingly, even if we give the definition and the label meaning, it doesn't really mean that the model is
David Bau: can make the agreement. Actually, you can see here, given the definition and label, actually makes the, accuracy lowest. Oh, really? Yeah, among all the… for the trailer risk settings.
David Bau: So, the top one is the without definition and without labeling.
David Bau: This is the, only give definition of the uncertainty. This is only give the, definition of different labels. It's just without, given everything.
David Bau: while this is lowest among all the three models. And that's accuracy with respect to human labels? Yes, right? Yeah. This is not too much. We only have, like, 70 labels. Yeah, we only consider these. But there's a very interesting…
David Bau: like,
David Bau: decision boundary shift, if you're given the, definition between no and intermediate, uncertainty. But there is no… basically no from high to no uncertainty confidence. They're not very common.
David Bau: Because our data is too complicated, so we are creating synthetic data to, to see whether the learning model can do uncertainty. And we have three kinds of data. One is numerical data, which means there is numbers in the statement. For example, we have low uncertainty, to high uncertainty.
David Bau: Yeah, and here is the result, because, the poor label is too complicated, so we do two labels. So can you… so… so before you show the data, can you describe the experiment in a little bit more detail? So, what did you do with the text, and what are you reading? What did you repeat it to? Yeah. With inputs and outputs?
David Bau: inputs is a statement, and so, literally, these four statements, no, you must have more than just these four statements, is that right? Yeah, yeah, of course, we have, hundreds of statements. And how did you generate them?
David Bau: It's okay, that's cool, that's not a lot, I'm just… you're just trying to understand what it is that you did. Yeah, we made a lot of effort, because, learning models always generate with a template, and it makes many seem
David Bau: syntax symbols on the different, labels. So, we've generated a lot of them and chose some.
David Bau: I try to choose ones with different syntax. Yeah. So right now, it's a little hard-coded, but, like, we might try API later. Basically, Shui did… what Shui did was she sort of, like, created different levels of synthetic data, where, like, this one is the, like, probably the easiest, where you have, like, mean and variance.
David Bau: Mathematically, and we also have, like, economic synthetic data that are, like.
David Bau: A little more complicated, less complicated than the earnings calls. Nice, and yeah.
David Bau: That's very cool. Wow. Yeah, yeah, so you don't have to… so actually, one of the… one of the things I feel like
David Bau: As you get into this, you know, Guy.
David Bau: getting towards making the peer-reviewed paper, you realize that, oh, actually, data preparation, like, the whole story that you have about, like, how you synthesize your data sets, that's, like, an interesting story in itself, right? That's, like, an interesting part of the research itself.
David Bau: feel free to, you know, talk about, you know, all the dilemmas you face with that in the future. So, okay, what do you get?
David Bau: here is the PC from layer 0 to 7, and we can see that in the first layer, it has some eyelet, and then it,
David Bau: became clusters.
David Bau: And in, in the middle layers, it became very separate.
David Bau: It's… And oh, it's a…
David Bau: And, it's a… it's a memo.
David Bau: The previous one was the last token, and in the import, the layer 0 is already separate, but not island.
David Bau: And then this… the, second…
David Bau: delivery of the data. It's equal data. There's no numbers in the statement.
David Bau: No numbers. No numbers. So basically, you have, like, those are also a little bit templated, so, like, for no uncertainty, there are, like, things like immutable schedules, like, a bunch of it, but,
David Bau: And for high uncertainty, you have, like,
David Bau: a lot of different directions, moving pieces, a lot of things going on. I see. And,
David Bau: Yeah, I guess this data is, like, more realistic, so, we, before, like, doing PCAs and stuff, like, we tried model performance on this, and, like, we basically gave, like, two-shot prompts to it, and then see, like, whether it gives…
David Bau: The correct label for the,
David Bau: So, like, in this test, we only extract the no uncertainty and high uncertainty ones to, like, just be binary.
David Bau: And, like, we also tried, like, feeding in the definitions, just, like, what? And,
David Bau: It seems like the model is leaning towards, like, giving uncertain answers, even on the, samples without uncertainty, so… and also, like, a similar trend observed is that when you give definitions, it's not necessarily improving the performance, so, like, the definition one is exactly the same as the one who's used. But, like, the second one.
David Bau: because the definition one didn't work, so I tried, like, looking into the dataset and, like, make a much precise, a little bit dataset-specific definition, which definitely makes it, like, better. Oh, no. But,
David Bau: I don't know, like, this definition might not be realistic, given that we have, like, we're looking at our income.
David Bau: But, given those, even though, like, it's bad at, those without uncertainty, we still feel like at least it is, like.
David Bau: Separating things, like, for example, on the… on those certain samples, it's, like, half-half, but, like, on the uncertain samples, it always… it predicts, like, 100% of the time.
David Bau: So, we try, like, a little bit of activation patching, where, our question is, like, which layer encodes information regarding a statement's uncertainty level, and for this, we,
David Bau: We did not do, like, token-level, patching, because, like, at the end of the day, uncertainty concepts would be dispersed across the entire earnings calls, so we do not really care about, like, oh, which token encodes that information, but rather, we sort of, like, ask which… well, like, in this
David Bau: experiment, for example, we basically patch at the end of sentence period, for this statement, and so we construct, like, contrasted pairs, so, like, for each
David Bau: For each sample, we have, like, a corresponding contrast sample that talks about the same thing, for example, a GPT growth, a GTP growth, sorry. But, like, with the uncertainty label flipped.
David Bau: And, we ran this across, like, 100 contrasted pairs, and it seems like the…
David Bau: So, you could see, like, Clean baseline is the ones with uncertainty, so, like,
David Bau: this is, like, basically the logic difference between yes and no, and it's, like, very high. But, like, for the corrupted baseline, it's, like, the logic difference between yes and no on the certain samples. Similar to, like, our, model performance results, it is, like, near zero, because it is, like.
David Bau: half percent of the time, it also outputs, like, yes when the sample has no uncertainty. So, like, it sort of corresponds to our empirical testing, but basically, the takeaway is that at layer 12-ish,
David Bau: Hatching in, the clean samples token activation at the period token, helps recover, like.
David Bau: about half of the legit difference. So, we do think that this is, like.
David Bau: A strong evidence that, uncertainties
David Bau: concept is, like, present here, I guess.
David Bau: And here is the PCA result, and then we can see that in the layer 0, it's a… okay, separate, but not that separate. But in the next layer, it's becoming very separate. Next, please.
David Bau: And in the middle, they are, they are all very separate and next to mixed.
David Bau: And in the last, it became mixed again.
David Bau: Yes, it's very expensive. And it's the four, four-label PCA. And, in the… Okay.
David Bau: Yeah, that would be pretty, Cindy.
David Bau: We'll skip to… Oh, oh, we also want to,
David Bau: see that weather learning model can tell value from risk, so we do, also… also… what is that experiment? Okay, oh, we did an experiment to see… to decouple this. And it's a result can see that in layer 20 and layer 24, it's almost, 90…
David Bau: So for this one, basically, we were talking about, like, the first and second moment separation of concepts. The second moment is, like, the uncertainty, which is the variance. The first moment is, like, how bad things are, which is, like, sentiment, basically. And we want to see if the model
David Bau: sort of, like, understands, oh, the, market is bad doesn't mean the market is uncertain. Like…
David Bau: So, like, we tried sort of, like, getting the direction of uncertainty versus the direction of sentiments, and see, like, calculating the correlation, cosine similarity between those.
David Bau: And this is on the toy dataset. So I guess…
David Bau: like, one direction for our research project would be, like, going further, like, I think this direction of separating the two,
David Bau: like, uncertainty direction and the, like, first moment, second moment directions is really interesting. And, so we had, like, localized, sort of, like, localized the uncertainty concept.
David Bau: using this synthetic data, we might want to make it better, make it less templated, and then actually recover this uncertainty direction, and do the same thing on the sentiment direction, or, like, the first moment direction, and then see, like.
David Bau: If we could do some causal intervention.
David Bau: And see, like, are they… are the model using them separately, independently?
David Bau: Yeah, a lot of good stuff.
David Bau: Any suggestions for this team?
David Bau: Can we see the activation of how she feels like reports.
David Bau: The contrasted players.
David Bau: You want to see the data centers? Yeah.
David Bau: That's the way you're looking at that.
David Bau: One of the questions that I have about this is.
David Bau: You know, when you have all this synthetic…
David Bau: Data, you, you wonder, with these small data sets, actually, even with the natural data.
David Bau: You know, are you… are you getting… are you basically getting your grav detectors? Are you just, like, you know, detecting…
David Bau: A vocabulary chart.
David Bau: And I wonder if it's possible
David Bau: To do something, to generate text that had the same vocabulary with words in different order.
David Bau: That… that conveyed it, and…
David Bau: a little bit of… it's a little bit of one of these crossword puzzle problems, right? Is there some way of constructing different sentences that have
David Bau: Great different uncertainties using the same words.
David Bau: C. Okay. So the answer would be…
David Bau: If the model, like, performs differently, then…
David Bau: Yeah, like, it's a little suspicious, like, when you have the PCA separation, and even at layer 0, right? It's, like, pretty separated.
David Bau: It's a problem of data generating, is that if we create it with, same words, that means that there's only a little word that will tell the difference of uncertainty. Yeah. And, maybe model will easily separate them. You could easily separate it. Yeah. Yes.
David Bau: Yes, and you just, you just, UFCA, you're just getting…
David Bau: Yeah. The separation between active sports and doesn't have sports. Yeah, yeah, yeah, exactly.
David Bau: Oops.
David Bau: But what we were, when we were thinking about this.
David Bau: This, let's contrasted pair.
David Bau: Cool.
David Bau: Yeah, our contractors are… I guess I've been thinking about more similar pairs.
David Bau: But then, my problem with that was Harvey… Just picking up the…
David Bau: the difference between the words that we have changed, or the concepts. Right.
David Bau: Right.
David Bau: Yeah, this tree, yes. You like this better? Yeah. You like this better?
David Bau: How do we lose the, precision on the token position level?
David Bau: Because this is, like, more abstract than, like.
David Bau: It's like, they looked different, but…
David Bau: because I, like, I guess our belief is that you couldn't expect uncertainty to be located, like.
David Bau: on, like, a word level, which is exactly why we're looking at it. Like, we wanted to…
David Bau: Okay, maybe a dumb question, but, like, I think one of you works with Baker from Baker, Bloom, and Davis. Isn't EPU, like, literally, they just look for words, though? Like, they're looking for words like tariffs, or, like… Yeah, you mean, like, the model is…
David Bau: I don't even know what I think their model is, they just have, like, a list of words, so, like, I was just… because you said that, like, it's not…
David Bau: like, uncertainty isn't being expressed at the word level, but… like, when I read through transcripts, like, for earnings calls, it's usually just, like.
David Bau: They're talking about, like, tariffs or something.
David Bau: And there's, like, a… like, a pretty major benchmark, like the Policy Uncertainty Index, and I think they also just look at words.
David Bau: I guess the motivation Veronica originally had was that, like, yeah, people have been looking at, like, words, back-of-words, style models, but, they're also, like, trying to use LMs to gen… like, and see, like, if they are better, and I think they are. There are some, like.
David Bau: for example, one paper used LMs to, like, generate the, like, the EPU measure, and then, like.
David Bau: that actually… that, like, on a scale of 0 to 10, something like that, and, like, that correlates with economic outcomes. And, like, sort of, like, the point is to see, like, oh, why LLMs can't accept those.
David Bau: If the conclusion is it is on the word level.
David Bau: That's, like, a failure, but, like…
David Bau: If you could have just come up with a backwards model, Boolean.
David Bau: That's kind of disappointing? That's what you're saying? Yeah. Yeah.
David Bau: You have these different types of… I still, like, This question of…
David Bau: Are different types of uncertainty the same?
David Bau: And you have different data sets. You've got… Your number experiments.
David Bau: Where I can… I can totally imagine number experiments having balanced tokens.
David Bau: Where, you know.
David Bau: You switch the number… the rolls of the numbers around, or something like that, in some way in a sentence, and it changes.
David Bau: you know, the probability is a lot, while it's just the same numbers. And, and…
David Bau: And I can totally imagine that the models are pretty good at understanding number sentences.
David Bau: And then for these other sentences, you know, I do buy it that there's some words that just are inherently about uncertainty, like the word uncertain
David Bau: You know, maybe you just say the word, and it's uncertain. And so the thing I wonder is…
David Bau: when you have, like, unigram uncertainty, when you have, like, an utterance or something like HubSpot uncertainty is…
David Bau: It's just an abstract concept, and you have uncertainty that shows up because of a situation, maybe because you have certain bets or something like that.
David Bau: is that the same thing to the model? Right? Like, I'm kind of interested in… as a reader, I'm like.
David Bau: Interested in whether… these two worlds.
David Bau: are the same inside the battle line, right? Like, the old backwards thing that the economists were doing.
David Bau: just looking for words. When the LM sees a word, does it…
David Bau: Does it have an internal reaction that's the same as when it sees, like.
David Bau: You know, a certain set of cards on the table.
David Bau: Something like that, it's like, oh, you know, now, now I have no idea.
David Bau: whether I should bet one way or the other, because there's a lot of uncertainty for this poker hand.
David Bau: You know?
David Bau: Without, without having to say that word uncertain.
David Bau: You know?
David Bau: Yeah, so, like, is that… or are those totally different concepts?
David Bau: We do need to poke around with the data, I guess, a little more.
David Bau: Like you would, for example, show…
David Bau: Yeah, like, what would… what would it do? It would, like, if you… Like, if you patched…
David Bau: If you found ways of patching costs.
David Bau: Very different domains. If you fetch from the number domain to the word domain, or from the word domain to the number domain, or something like that.
David Bau: And, and, you know… You can have the same effects?
David Bau: That might make sense. So, you catch a little bit, you have some effects, but sort of in domain.
David Bau: But, maybe, maybe approximate batches worth.
David Bau: Playing with… I don't know.
David Bau: I don't know, it's just a crazy idea.
David Bau: Anything else?
David Bau: We would suggest for this, this project.
David Bau: I was going to suggest to our group, but, like, one of the easiest things we can do is, like, switch numbers around.
David Bau: Which is, like, if… Like, have one number that's, like.
David Bau: Like, we expect, a 0.1% variance and, like, 20% growth, and it's, like, 20% variance and 0.1%.
David Bau: Yeah, of course.
David Bau: Yeah, it's the same numbers, but, you know, play different roles.
David Bau: Yeah.
David Bau: That's cool.
David Bau: So, yeah, you're seeing separation.
David Bau: You've seen sort of causal effects.
David Bau: But you're, you're all looking, you're all looking like cold, so do we have something yet?
David Bau: And do you… it doesn't feel like you have something yet, right?
David Bau: Yeah, right. So you have, like, you have these questions about Layer 0 responding, so there's all these confounders.
David Bau: Right? And… and then… and then even if you do get peer separation, you're not sure that it's surprising that you know that the models
David Bau: And classify, so you look inside, you can classify what's the big deal.
David Bau: Right.
David Bau: So I think that it's… but maybe the questions are…
David Bau: That are interesting are these kinds of things, like.
David Bau: I guess, like, what you asked, like, what is the model using to classify it? Does it need the words?
David Bau: To be able to tell what's going on with that sort of do if you take away the words.
David Bau: does it… does it still work? Like, if you… is it the same concept, just…
David Bau: It had very balanced numbers and things like that.
David Bau: That's another way of saying…
David Bau: You know, it doesn't need the words. If the words are present, they'll use them.
David Bau: But I think about it some other ways.
David Bau: Yeah, I'm not sure. What would you have to do to nail down that story?
David Bau: If we could have, like, a dataset that is,
David Bau: Not having the bag of words, but…
David Bau: like, our uncertain, like, we know of that. That would be great. Oh, you just, like, blacklist the whole world, so you have the standard economic model.
David Bau: you turn on your LLM, make a lot of unsearched texts, The violations model.
David Bau: Hello, Elle, your mission is…
David Bau: To… to come up with that score badly on this.
David Bau: But actually, the other way.
David Bau: And then you can look inside those.
David Bau: Yeah, maybe something like that, that's cool. That would be a cool dice.
David Bau: Do you know what, like, some of the transcripts look like, where, like, a model labels it as uncertain, but it doesn't, like, contain any of the words?
David Bau: Like, like, like, did you, like, I was wondering if you guys had looked at, like, basically transcripts that get labeled as uncertain by, like, an LLM, but don't contain, like, any of the, like, bags of, like, the words in the, their, like, EPS bag of words.
David Bau: It's just… that's really interesting that it can be better. Actually, there are a few, because in reality.
David Bau: doing a planning call, you, like, the one who speak, don't want others to know they're uncertain about their financial
David Bau: like, conditions.
David Bau: So they're… when they're talking, they're actually hiding.
David Bau: I would say. So… In many cases.
David Bau: the, like, I'm not quite sure of all other, like, uncertain… these rules were not directly showing in the, conversation. But the LLM knows.
David Bau: Yeah,
David Bau: I would say there are directly nos. There are, like, as you can see, the result, there are, like, nearly 50% of the disagreement on different level of uncertainty based on different models. So I would say models are quite…
David Bau: Not very… quite sure about the, uncertainty.
David Bau: Oh, and the words aren't present. Yes. Oh, interesting.
David Bau: But in aggregate, like, it outperforms EPU on predicting, like, different economic…
David Bau: But… but that question, like, one…
David Bau: We haven't tried, like, larger models, so we are not quite sure whether the model's parameters will affect, like, given a…
David Bau: More frontier models will… Gave better result, but… As far as we have.
David Bau: it's not… works good. What's that policy, isn't?
David Bau: Gemma to 9B, Lama 8B, and Q1 7B. It's, like, rather small. 7? 7, yeah. All less than 10 billion. Yeah. So I've been very happy
David Bau: with the capabilities… so the next size up, like, 30… like, 1030B, you know, Llama 70B, like, I've been pretty happy that they…
David Bau: you know, can do pretty good linguistic understanding, they have pretty good theory of mind, they have good other things. Whereas, when you get down to 70 bottles, like, they're pretty grammatical, they understand a lot of things about the world, but they're very simplistic, right? It's like talking to a little bit, and
David Bau: They don't have good theory of mind.
David Bau: Yeah, so, like, if you think about that as a milestone, right, like, the difference between a 4-year-old and a 5-year-old is…
David Bau: Like a 5-year-old.
David Bau: We'll be able to second-guess you.
David Bau: You also, you know, play a theory-mind game with you, and a 4-year-old will just believe everything that you say, everything's just surface level, there's nothing…
David Bau: Nothing under the surface.
David Bau: And maybe if you look at these small models, you can't do clear your mind, that's, like, the mindset you're kind of look at.
David Bau: You're sort of… you're sort of asking… asking a 4-year-old to read the earnings columns.
David Bau: It makes sense. Yeah, yeah, yeah. Yeah, yeah. If you use Open Router, it's good for testing them out before you go, like, all in, and you're like, I'm gonna collect activations. Oh, I forgot to mention to everybody, oh, we have, there's a budget.
David Bau: for, or some, For some compute.
David Bau: that, that Aruna has raised.
David Bau: Like, thousands of dollars for you guys to just spend.
David Bau: So, and so she has… she's gotten a credit card that we can reimburse, and got an account.
David Bau: And all you have to do
David Bau: is go on the Discord, or, you know, find, find, find the email, or whatever.
David Bau: It's… so there's a RunPod, thing for running, you know, detailed experiments. So if you want to, like, set up some training runs or something like that, it's inconvenient to do other ways. You can use this RunPod thing. And…
David Bau: Apparently, the invitation expires this week. Today, actually. It expires today? Expires today.
David Bau: Not even tomorrow? Okay.
David Bau: So don't forget, before you go eat your dinner, right, go get, like, this free account thing. And, yes, if you miss it, of course, we can go back to revision.
David Bau: You know, say, oh, sorry, I missed it. But, yes, go get the free thing. There's $1,500 right now.
David Bau: Yes. I was wondering if it's, like, the Teams 1500, or… I think it's the class. It's just the whole class. So, yeah, so use it fast before it's all gone.
David Bau: No, it'll be fine. I think that we can refill it, stuff like that. It'll be fine.
David Bau: Thank you.
David Bau: Yeah, I think… Okay, thanks a lot, you guys. Yeah, who's next?
David Bau: Is it Team Spatial the team that wants to do more next week, or is it this… both.
David Bau: Can you stay tuned.
David Bau: He came into the washroom. Oh! Team Special is in the washroom.
David Bau: I don't know if people are doing it on the Zoom. Anybody on Zoom?
David Bau: It's next. We can, we can click next for that.
David Bau: Anybody on the team that wants to present today?
David Bau: Oh, that's a bad thing.
David Bau: Oh.
David Bau: Pretty good.
David Bau: Okay, we'll come back to the spatial next. Thanks, Ross, but then… Okay, but you wanna, but you wanna, you wanna skip? Yeah. Okay, so we'll go to the next.
David Bau: We'll come back to 15 special when 5 different times, so… So, here, next one here.
David Bau: to empower.
David Bau: Really cool the name.
David Bau: Cool. Okay, we have some, patching experiments to report on, and some data set construction, kind of next-step stuff stuff to report on. So we'll try to go quick. The first is, so this is, like, the setup that we've been playing around with. This is, like, the prompt in the following sentence, does entity exercise power over someone or something else, true or false?
David Bau: Then insert the sentence, what's the answer? We've been trying to shorten this a little bit, and then playing around with true or false, yes, no, but this is what we're going to be doing this setup.
David Bau: So, like, this is the example. Sometimes Tony reassigned them to the basement office, or they reassigned Tony to the basement office.
David Bau: And, we've been, playing around with hatching. So this is what that sentence looks like, and we get a lot of, like, clear signal down here at the period, and then at the colon, in terms of, filling the outcome from true to false.
David Bau: And then, it's a little bit, kind of.
David Bau: a little bit more indecipherable in here. This is the they token, so going from they to Tony, and then Tony to they, and that actually bumps it in the opposite direction, so it makes it more likely to be false when you kind of patch there. So,
David Bau: I think we're… oh, it's weird, so it goes the opposite. Yeah, but I think because we're, like, switching the position of the people, right, so the pony…
David Bau: When Tony gets a patched down, now there's two Tonys in the sentence? So you think this is a pointer effect when you're, like, mixing up the pointers, is the thing.
David Bau: As opposed to mixing up the power structure.
David Bau: Yeah, that might… yeah, there's something going on about… yeah.
David Bau: Yeah, we don't have, like, a very clear articulation of that. And then if we flip the prompt around, and so it's something or someone exercising power over Tony, we get somewhat similar structure. It's less pronounced on the first token here, but as a Tony token, there is a big…
David Bau: shift, and then,
David Bau: Yeah, so within this setup, we've been trying to figure out if there's other ways to kind of get a power beyond just kind of flipping the order of the entities, so Armuta's been thinking very carefully about this, and yeah. Over to you.
David Bau: So it's the… This is another, set of data that we switch over, see how it impacts the output, and we can see that,
David Bau: And fire is changed to greeted.
David Bau: Then there is some… Something moved on here, where… Also these two positions.
David Bau: But, I'm still thinking that it's the, if we change the, like, prompt to the, to other things, like, not, exercising power, but something that,
David Bau: Asks about a characteristic which, true and false, means something for it.
David Bau: then I still expect to see this here, see the same, like, pattern results.
David Bau: Even if it has nothing to do with color.
David Bau: Right. Yeah, and still thinking, how we can, right?
David Bau: Picks the power from this.
David Bau: Right.
David Bau: Do you have an interpretation of what this means? Like, it's switching from true-false at that word. Is that it?
David Bau: Also, some, the dynamic of the power is, very related to the verb.
David Bau: Like, firing, or doing something, forceful.
David Bau: Life.
David Bau: There's a lot of semantic meaning in the verb, that's kind of what we're trying. One interesting experiment would be if you're…
David Bau: make it a third-person, sentence. Like, you click the sentence out and switch the position of the form and see if that… like, instead of just switching the verb, also switching the position of the verb and see if this moves to that position. Like, the employee was greeted by Sean.
David Bau: The combination of this one and the previous one.
David Bau: Yeah, just to confirm that it is the farm,
David Bau: And it stores the relationship between
David Bau: I'm just thinking, like, if it's a position thing or a warm thing.
David Bau: Yeah, because we're asking about Sean, and then this is the word that comes after it, and then it goes to, like, switching this into the passive voice. Yeah, yeah. Yeah.
David Bau: Also, there are other, prompts that can make difference in the outcome, like, to exercise power.
David Bau: or, you hold power, does this person hold power over someone or somebody? So for the sentences that the power is not much clearer for them, it can make a difference.
David Bau: Yeah, and then we're, like, I think where we kind of started with this was kind of, like, how much do we impose a definition of power? And, like, I think one thing that we're kind of struggling with that is that maybe we're shading into all these different kind of conceptions of power, that we're kind of bumping into different concepts within the model. And, like, so, yeah, I think we're in this place where it was, like, a little bit of a give and take between, like.
David Bau: Having a kind of clear conception of power, versus kind of trying to figure out what is the breadth of this within
David Bau: the model. Yeah, you… I don't know, I put a bunch of other stuff in here, but this is you, yeah. Okay, so, sorry. So, coming on into there, so we… so looking at these examples, we can see… we can see that in some cases as well, from pre… From experience in the previous weeks.
David Bau: that, that, that there are… that there are limitations to even the instruct models versions.
David Bau: conditions of power. For instance, there are bad false negatives. There are certain contexts where it's less confident if you have multiple modifiers, right? So as we get into different definitions of power, one… one thought was that one, one thing we advanced on was, trying to systematically expand our synthetic data set.
David Bau: So front… so that's one thing here. So now we're cross-referencing,
David Bau: about 200 granular power mechanisms and different ending-based conditions, so we want… we want to try to…
David Bau: So, for instance, like, we start with, like, I think 11 or 12 hard categories, economic power, military power, ex… you know, digital power, etc, right? But, and then from there, we articulate different contexts, so this might… in the case of military power, it might be embargoes, str…
David Bau: Define legitimacy, strategic weapons, etc. So, and that results in about 220 different contexts.
David Bau: around here with different attributes, because we want to be able to try and isolate for different things. So, for instance, there might be contexts where power is loaded to, using the military example, you might have an instance where it's like, oh, the commanding officer decided something, right? As opposed to the subordinate decides that, right? There are modifiers, it might be based on gender, rank.
David Bau: and other descriptors that might encode notions of power. So to really kind of stress test that spectrum, like the landscape of it, we're trying to expand the synthetic data sets to account for these things.
David Bau: as we said, we're doing a full maintenance cross-referencing, so it's about 3,000 records, for positive, and then we're also constructing one for negative. We'll get into that next.
David Bau: One other thing is that, as we said.
David Bau: we want to be able to try and scan for if it's verbs, entities, there are certain things that capture meaning, right? The issue with a lot… the issue with a lot of things, as a lot of the valuable stuff Armita and Isaac were doing, was, like, certain verbs
David Bau: can contain a lot of information about power. So one thing is we want to have… we want to also try and create contexts that… that clearly demonstrate a power imbalance, or a lack of, but… but use… but use realistic, but less…
David Bau: a singularly overloaded term. So, a national army surrounds a separatist militia. Surrounds is… is not necessarily the same thing as bordered, dismissed. You know, water can surround water.
David Bau: An island. Same thing, intentionally relay evacuation procedure. Hear the power is not a single verb.
David Bau: Right? So we're trying to capture a lot of linguistic variance, too. So the whole, the whole point with this expansion is that, for next week or later on, we're trying to create something we can use to create a systematic profile. How many, how many different types of power again did you say you guys?
David Bau: So, I think we're at… I removed… Yeah, it's about 220, yeah, different contexts, and I… I removed a couple, supposed to be attribute conditions, but we're gonna have, like, 11 bias conditions.
David Bau: So it's gonna be… and that's for positive power, there being no power. We want to actually open this up a bit, because while we're talking about isolating for verbs.
David Bau: The… there's a rough… I mean, absolutely do some quality control things, but the rough draft of this data set, at least in the first instance, is close to ready. But we… while we're talking about different conditions to isolate for, one thing that Armita brought up was.
David Bau: Are there different formulations of power by which we can ask the prompt, right? Because if we're saying, is power being exercised? I mean, I believe you had a couple of other formulations we can ask the question, right? Because if we're asking the question the same way maybe we're trying to generate the data, that's not correct. There might be different…
David Bau: she identified several different ways we can rephrase the question, so we're trying… I think the next step
David Bau: is going to try and be do a systematic scan, and then we're probably going to try and hone in a couple things, but as we expand the data set, we're actually hoping to ask some questions about what might be some interesting dimensions to
David Bau: interact with.
David Bau: data sets.
David Bau: Yeah.
David Bau: Yeah, so my feedback, so… I… I,
David Bau: So, for this and some of the other projects, right, I do think that
David Bau: You know, people ask, what, you know, one of the questions was, you know, is probating… Second best.
David Bau: Whatever, right?
David Bau: I… You know, once you develop a dataset.
David Bau: That has a pretty clear idea.
David Bau: Of what the concept is, that data sets, you know… Trying to compose.
David Bau: then probate is just a really natural way to do. I mean, so it's a classification data set.
David Bau: And you can go around all the hidden states of your…
David Bau: On your model, or ones that you think are… you could train folks on them.
David Bau: And so I think that the causal… so, like, you've got these two different types of experiments.
David Bau: And you can… you can go back and you can think about, like, what Sam Marks did.
David Bau: in using these two things together. Well, so he didn't do prob… he did math, mean, things, but he had a very simple binary class, is it true or not? And now you have, like, a…
David Bau: I don't know, maybe it's a 3000 weight classifier or something, I'm not really sure, right? So, but,
David Bau: But… But he did it in two stages. First, he says, well, I could probe anything.
David Bau: But really, I'd rather probe
David Bau: the… the places in the model where I think is where it's really paying attention.
David Bau: this concept.
David Bau: Like, what the previous…
David Bau: You know, work has found is often at the end of the sentence or something like this, but you could do a causal
David Bau: Our analysis, just to highlight the
David Bau: these are the promising places to probe. And then… and then you could… you could train probes, right, to see what accuracies they get at different locations. And then… and then once you have those probes, then there's other things you can do. You can do…
David Bau: Or, you know, causal, so, the probes will tell you what… you know.
David Bau: What one of these classes… At least a probe thinks the model is thinking about.
David Bau: You can see if the probe agrees with your prompting strategy. You put a thing after.
David Bau: That sentence, and you say, what kind of power was it? And see if they agree or whatnot.
David Bau: And, and, you know, a natural thing to do would be
David Bau: Is that really what it's looking at to make that assessment? Let's say the probes and the prompt degree.
David Bau: You can go and force in a different representation into the probe and see
David Bau: Is that this prompt to behave a little differently.
David Bau: We… You know, that would be… I feel like that…
David Bau: That's sort of a way of bringing these things together and connecting it to, like, an internal thought.
David Bau: I think eventually you're gonna have this question of, like, so what do we care?
David Bau: About what the model's thinking inside.
David Bau: As opposed to what it's saying.
David Bau: Right?
David Bau: Are there situations, like, so, you know.
David Bau: If you could just ask a model.
David Bau: What is thinking, and why do you need things?
David Bau: You know, why do you need to probe around on the inside?
David Bau: And and so I think that, you know, so these are my suggestions on, like, things that you can do. These are just, like, mechanics, things you can do to try to pin down the thing, but I think that you also have a larger question of, it's like, okay, so let's say this all works, and we can probe it out, we have positive effects, and we know
David Bau: Where the concept of power is, you know, You know, what do we…
David Bau: you know, what do we get out of that? Like, why do we care? I can imagine there's a couple sorts of things you could get out.
David Bau: of…
David Bau: But do you guys have a sense for why… why you want to look inside the bottle at power?
David Bau: So, our, like, going back to our original research info we started about, one thing was we wanted to… we wanted to try… try to identify certain… at least certain triggers in the model, or at the very least, if it was possible then.
David Bau: you know, if there were certain noise or signals you could… like, look at the words that caused this inference. Yeah, or even in particular, in certain models, for instance, if it was possible to affect the signal in some predictable way, to have it redu- to have it, you know, change its decision-making over.
David Bau: Yeah. Oh, that… I really like that, because, like, I'm in this, like, prompting group, and they're like, oh, like, if you don't capitalize things, or you have a typo, like, it gives you much worse results, and, like, stuff like that, or if you slang, like, and it feels like what your question is, like, asking, like, why does that happen? Like, was there some…
David Bau: some circuitry in there that's, like, making that… I feel like that'd be really cool to know, like, if the model is treating me differently because of some inference like that, like, maybe it only typed in lowercase or something.
David Bau: Interesting. I think one thing I should do, like, for some of these is, I think what could be interesting is, for instance, maybe I'm wrong, say, for some of these examples, right, we've extracted, like, entities and verbs. If we, if we do that…
David Bau: And then use those as labels. It might be interesting if there's an experiment we can do, for instance, where we can reliably have it reverse decisions it makes based on patching things from different prompts in a controlled way. I don't know, though. That'd be…
David Bau: Yeah, just to build up that a little bit, like, I think,
David Bau: Another way of thinking about this is, like, it's not just about power, but it's about, like, how models think about social relations more generally, and this is a very, kind of, particular
David Bau: conception of social relations, but maybe there's this, like, this kind of template for thinking about other types of social relations as well, so there might be a little bit more of an interesting interest there. I like that a lot. So, like, there's this… so I like the… so, like, so you're taking a very broad view right now, which is, like.
David Bau: Which, it leads to this thing, like, okay.
David Bau: We're gonna catalog all the parts, we're gonna get 224 things, and that would just lead to this natural thing of a 224-way classifier.
David Bau: That you just go and you say, and then, like, the natural experiment would be, like, oh, if we have some piece of text that has to do with making ice cream.
David Bau: I wonder if you…
David Bau: if you zap that classifier at it, if it says, oh, actually, this ice cream paragraph is secretly about, like, nuclear weapons, we're talking about that, right? You're like, that's really what the model is thinking about. So I think that that's kind of the direction that this data set would lead to, but there is this opposite direction.
David Bau: Right, which is really narrowing it, right, to say, hey.
David Bau: If you just have sentences with two people, like your other data set.
David Bau: And… and there's a relationship between them, Like, how is that…
David Bau: How is that social relationship represented by the people are like, this is a binary classifier. We're just changing the different conditions, and it's just… It's always binary, but you have, like, 224 ways it's, like, you know, like, situations, where… so you could actually do the other thing. You could also make a 224-way
David Bau: classifier on… on this large data, if you wanted to. It's just the same data that you could just label it another way.
David Bau: And it's actually a natural question, like, when you ask.
David Bau: How do models think about power, and you've asked this several times when you present, Like, wow.
David Bau: You're preparing this data that you could use this approach to see if any of these, you know, works better, or…
David Bau: You know, is misclassified, or… does it have… it has a similar accuracy over all these classes, or…
David Bau: Is it higher accuracy on some of them, or not, or yeah. I think in a class, I think you mentioned as well, even… let's say there's an interesting interaction, but it's reliably in one sector. That's still what you said, that's still something, as you said, right? What do you mean? I believe you said, so for instance, let's say the models are not interesting results for…
David Bau: all but one category of the data, but for… but it's… but it's reliably interesting in one or two areas. Oh yeah, then that could still be interesting. Yeah, absolutely. Right.
David Bau: If you're like, oh, the model is, like.
David Bau: It doesn't have a very clear representation of most of these things, at least when we're probing this way, but the representation is clear this way.
David Bau: At least given the state that we are today, where we don't know anything about how the models work inside, just in a part of it, so it's pretty interesting.
David Bau: Huh.
David Bau: I think the sentences that are unrelated, random says there is a power relation, which,
David Bau: I've always been fascinated by that. I think this is so… when I showed you the,
David Bau: the paper from Yudi Chen Wandberg's group, where they're like, hey, here's…
David Bau: Here's, like, a bunch of probes for, are you, rich or poor, are you educated, are you not? Are you, you know, are you, are you male or female?
David Bau: The fun thing about that
David Bau: is that you can just put in other random sentences, you can say, you know.
David Bau: You know, I was reading the Wall Street Journal the other day, it says, oh, you're definitely male, or something like that, right? Well, that's a bias, right? You know what I mean?
David Bau: And something that's kind of interesting, you know, if you have a classified like this.
David Bau: Now it gives you another tool.
David Bau: You can do. Now, but now, of course, the other… what… it's… it's not a full paper by itself. It's just a tool.
David Bau: Because it leads… because if you tried to make a full paper out of it, people would just say.
David Bau: Well, why don't you just… Go to the language model and just ask it.
David Bau: Do you think I'm a boy or a girl, or do you think there's part of that you're just in the way that you're doing it?
David Bau: And, and then it tells you then, why do you trust it.
David Bau: Right, so there's a couple situations. You could be in a situation where, well, for some reason, you don't want to trust the model because you think they're lying to you. Or it could be a situation where, oh, the point is not
David Bau: to just read the signal out, but to see where the signal's coming from, or something like that, right? Or the point could be, you have 200 different types of signals, and you're trying to tell which ones are more faithfully represented in the model, and which ones are not, as sort of, as your proposal for the models.
David Bau: Know about this, and they don't, and you can see evidence that they know about it.
David Bau: From the way they represent, or something like that. Like, there's a few different…
David Bau: You know, takes that you could…
David Bau: They've got, like, wide one with consent?
David Bau: But… but so… so you're getting there, right? You're… you're, like, you're looking around at enough things now that maybe you should have a stronger opinion.
David Bau: Great.
David Bau: About which kind of equations to that.
David Bau: Thank you.
David Bau: Anyway, that's my comment, but I think that it looks good.
David Bau: Nope.
David Bau: Any other suggestions?
David Bau: Alright.
David Bau: Oh, it's like…
David Bau: suggestions. I think it's also, like, having… so currently, it's kind of a static situation, where there's one… one, like, it's basically the subject, verb, and object, so you have
David Bau: the subject is asserting power over the object, probably not. You mean… you mean in that form? Yeah, in that form, right? I think most of them are like that, yeah. So, like, can you see how… can you make this into a dynamic situation, where, say, over a number of turns, it's,
David Bau: moderating. So, does the model assign power just to, say.
David Bau: Alice, or in the case of Alice and Bob, does it always assign power to Alice irrespective of the situation, or irrespective of the job? We did mess around with that, didn't we, where we were switching the subject and the object, and then in terms of we were asking who had power, we were doing the patching?
David Bau: Did we do an experiment like that?
David Bau: Yeah, I don't think we had it, like, evolve over time, though. They turned the tables kind of thing? Yeah, it's a good movie plot.
David Bau: Okay. Interesting. Yeah. Yeah. So yeah, it would be like having a sequence, then ask, at this point, who has power in this situation, sequence runs a little bit longer, who has power at this point? Yeah.
David Bau: Yeah, because I don't know if this one's helpful, there's a paper… I don't even know if it's a paper, but I saw a presentation by this person, and they basically gave two country names, and they asked.
David Bau: like, if you had to save someone from this country, or would you save someone from this country? And they swapped their country, like, they didn't, like, keep it always. For example, would you save an Indian, or would you save an American?
David Bau: And surprisingly, it always saved, like, the Global South than the Global North.
David Bau: But they also, like, switched up stuff, so then I could be,
David Bau: The one always stuck through to the identity.
David Bau: How would you invent that?
David Bau: We're done? Okay, we're done. So how… so… so, like, so we… we can go through about half of the teams. Should we do that pace? Do you want to do half the teams one week and half the teams the other week, or do you want to just do every team, every week? Like, how do you… how do you guys feel about that?
David Bau: I think it would be better for one team who can do it twice a month, because if you're doing it per week… Do you want to… do you want to scale back to that?
David Bau: Like, I… maybe… Sean, what do you guys think? I don't know.
David Bau: Yeah, it often feels like there is that. I think they feel… I think it might be good, since… especially since we're getting… I think a lot of us are moving towards… You're getting… you're getting more into it. It's not… you're not just, like, proposing, oh, I don't know what we're doing, like, you're doing stuff, right?
David Bau: So, why don't we do that? So, there's a few teams that…
David Bau: So I, I didn't write down…
David Bau: Nikhil, can I ask you to write down which teams presented this week? So, which team to present? The first three. The first three, which was… TK. TK. Economics. Economics. EK. Economics. Power. Empower. And we had finance yesterday.
David Bau: So, like, by essence, like, half, okay? That's fine. Yeah, and… and… and there, and so why don't we do this? Instead of…
David Bau: have you guys present on Tuesday. The next three groups just present on Thursday.
David Bau: Next week. Does that make sense? And then we'll go into this…
David Bau: And we can… we can be a little flexible, obviously not to do an extra presentation.
David Bau: Okay. Okay.
David Bau: Thanks. Do remember to sign up for that, complete resource. If you get any issues, please let me know.
David Bau: It leads me to talk about the news.
David Bau: Oh, listen… Let me double-check it with David. If you're working with an E for C…
David Bau: I think so. I think so. And so… My Instagram has one, but…
David Bau: I don't have a strategy, if you don't mind that, too.
David Bau: You're gonna take action, so we're going to have…
David Bau: Yeah, that's alright, like, I just followed that on me. Yeah, yeah.
David Bau: So basically, I'm working on that part more right now. Maybe I can, like, tried to parse out some of it? Asking for, like, site paramise.
David Bau: Because, like, I think one of the things that say you're supposed to that, I think is… Seriously, I see. We can avoid the predicated prompts. So it's like, let's say a prompt is, like, one of our sentences, right? What?
David Bau: Okay, okay, okay. Yeah, yeah. Wait, wait, wait, wait, but, like, yeah, let me get that taken care of. Also, do you need keys or anything? I have, like, open router… Oh, yeah, I need to think about that a little bit more.
David Bau: Oh my gosh, yeah, I'm so sorry, I think it… I think I did actually, but… Your budget is just really high for some of these, so we can do that particularly. I use that for a Q&A.
David Bau: What we can do, I think…
David Bau: It's power, it's classification accuracy. Thanks for, thanks for doing all the, to, like, set up for the meetings and everything.
David Bau: Yeah, I think I had seen that, and I thought that it expired.
David Bau: I think what could be interesting is if we do do, if we do the road, but it's just… I think this is not, like, it's just…
David Bau: Again, if we just kind of take a shot. Or not hunger! Not sleep, I see if there are different conditions.
David Bau: Because at that point, because at that point… I don't know what it is.