David Bau: It's a great idea.
David Bau: Yeah, I saw that. Oh, this is your… is that this is your CVE work? I don't know. This is your question.
David Bau: Father's Day, just a good amount of bread.
David Bau: Look at that. Yeah.
David Bau: But this doesn't mean that they are, you know, kind of…
David Bau: How come the video's not showing? There's supposed to be a video camera in this room. Here. Nothing. Nothing, nothing.
David Bau: Yeah, that's not about that. That better hope you should be.
David Bau: It is. Right. All right.
David Bau: So, Team S. Team S. Right here, but my teammate hasn't arrived yet. Okay, should we put it in a different order? Yeah, sure. Okay, I'm gonna switch the first two.
David Bau: Which means of Team K.
David Bau: Is TJ okay? Can I have, like, 30 seconds? Yes, of course. It's gonna take me 30 seconds. Oh, actually, can we go third? What? Should we go first.
David Bau: Is M here? I'm just out of breath, I think I was on the stairs. So don't worry.
David Bau: Jasmine, I'm gonna put you first, but you can pick up any equipment, okay? Because, because… I can think we're finished, okay? Okay.
David Bau: Just take a minute, and what you can do while you're taking a minute is, like, zoom into the thing?
David Bau: And actually, all… everybody on all the teams, you can attach the Zoom, turn your audio… attack without audio, but then so that you're ready to…
David Bau: switch over so that we lose all your time. Thank you, Nick, no, but…
David Bau: Does that make sense? We've done this a couple times, lately, and can we just see if she's…
David Bau: paying bills Oh, look, this timer's already coming! It's just sitting…
David Bau: Fine, how long has it been going? For 35 hours. Who knows? I thought you had a contact. No, no.
David Bau: Everyone wants to have one that I don't.
David Bau: I'm joining… Don't even show.
David Bau: As the Jag, which is fine.
David Bau: And, and then, after you, after you go, you can help yourself, too. Oh, absolutely.
David Bau: A prize? Oh, officer. Some of us had a prize. Oh, you got it beforehand.
David Bau: It looked pretty good. So we're gonna… okay, well, we'll start with Jay. Jay ready?
David Bau: They're so good! Weekend good. What? We're 8? They're so good. But they're number 4 right now. Oh, goodness. I'm not…
David Bau: We can go. We can go. You're welcome here.
David Bau: Wanna go first? If no one else is able to go, we can go.
David Bau: We love you, Claire, so much. All right, all right. Yeah, ready to present. Look, I'm switching, you guys. Sorry. Will it or not, because we have to go. Sorry, can we get out of here?
David Bau: Okay, so 7th is E, J…
David Bau: Thank you. You moved to number 4. Is that okay? Yeah, that's… thank you guys. Okay, guys. Now…
David Bau: Okay, oh.
David Bau: Do we need that?
David Bau: Welcome to the last day.
David Bau: So what I'm gonna do is I'm gonna keep everybody to a very firm
David Bau: Well, so you're… you should present for 8 minutes, but I'll put… put up a 10-minute timer here. You know when it hits 2, you're… you're out of time.
David Bau: But, but, but, you know, but, you know, you can go over a bit. But then, at 10, I'm just gonna…
David Bau: We'll stand up and clap you off the stage. Okay. That's how ML algorithms work, too. That makes sense.
David Bau: And so, Oh, I have to turn this on.
David Bau: So this is, so the idea is…
David Bau: I'll make a crisp presentation, you don't have to talk about everything that's going to be in your paper. The paper is due at a due date I put on the website, like, in a couple days.
David Bau: Because my grades for your class are due a few hours after that, so if you have the ability to hand in a paper early, that makes it easier. I'll actually
David Bau: you know, have a little bit more time to read them. Some of you might already be close to done.
David Bau: So, so hand it in when you can't, but there's a deadline.
David Bau: on the website.
David Bau: And, and so welcome, everybody.
David Bau: Let me see… is it… are we recording on the Zoom? I'd love… I'd love to record it. Yes. You can.
David Bau: Okay, great. So welcome.
David Bau: Indeed. Yeah, yeah, I got it. You gonna hide that? Yeah. Or screen it off to the side? Yeah. As best we can. Oh, sorry, it started. So, we're a DE. We're working on localizing and steering economic uncertainty in dark flash language models.
David Bau: So, economic uncertainty affects all investment and market behavior. And now, people are using LLMs to analyze earnings calls, and there is also financial GPT, financial clock, there are so many financial… different financial LLM statements.
David Bau: And there's also
David Bau: trying to measure uncertainty using LLMs. So, what we're going to ask is how uncertainty is represented inside the model. So, do LLMs actually have an internal representation of uncertainty, or are the models just counting the word risk and thinking everything that says risk is actually uncertainty? That's not a question here.
David Bau: Yes. So, as you can see here, we… the pipeline basically contains two stages. Firstly, see the left part? We got the…
David Bau: two different words in the, which one… the first difference is high intensity, and the, the second one is lower intensity. We do activation patching here, and after that, we calculate the uncertain direction by, using the high intensity minus the low intensity.
David Bau: After that, we apply this into intervention stage, so that we can, test it on the real synthetic earning costs.
David Bau: Clear.
David Bau: And how do we construct our data sets? So we got two different parts. First one is, the real earning cost. This one is, you know, from real-world examples. It contains 200 examples.
David Bau: After that, we also use cloud to generate it based on these, like, real-world examples to get the similar, but a little bit easier economic statements. So.
David Bau: So, 400 samples… pairs of samples in total. So, each pair's not statements differ only in the uncertainty level, which is high and low, with online economics topics are exactly the same. I think it's important to clarify that both examples include, this synthetic data set.
David Bau: was created using Cloud, using real NSCO's data, which is important. Yeah.
David Bau: So,
David Bau: Prior, works on measuring uncertainty in earnings calls use bag of words model, specifically text frequency inverse document frequency.
David Bau: Where they took the frequency of risk-related words over the total amount of documents they appeared in them, and used them to assign a score for uncertainty.
David Bau: So, we used that same bag of words model on our datasets to see how well they would perform at separating the high and low uncertainty statements. And we found that they actually performed quite poorly.
David Bau: Another question we had was, what are the words that the language model is paying attention to compared to the bagged words? So we're in,
David Bau: a UDIF input attribution on the last period token, and looked at the words that they would share.
David Bau: Of the words that Llama 3.370B and Bagward shared were pending and unpredictable, which are two very…
David Bau: Keywords related to risk.
David Bau: But I looked at the top 100 words that, LMs were looking at, and over 98% of the other words were not shared.
David Bau: Llms also have the ability to focus on bigrams and trigrams, which the bag of words model we used did not.
David Bau: Next we talk in detail how we set up the, pipeline.
David Bau: we introduced earlier. So, first, we talk about how we localize the uncertainty direction. So, this is our, activation patching setup. We basically, set the high uncertainty statement as the source and the low uncertainty statement as the target. We patch from the high uncertainty statement to the low uncertainty statement. So, basically,
David Bau: They only differ in this test statement because they're in a two-shot setting. We put high uncertainty statements here and low there, and we patch from here to there.
David Bau: And, this is our pattern results. So, basically, as we can see, for both template and synthetic datasets, LM seems to summarize information at the end of the economic statement, so this is exactly the period token of that statement.
David Bau: And, so we're, it happens at, like, around layer 12, so we are sort of, we are pretty,
David Bau: Confident that they do this summarization, and also from our patching experiments, you can see that for the templated datasets, individual words also have meaningful patching effects at the early layers, which align with our expectation that in the templated datasets, there are a bunch of words that already carry individual uncertainty signals.
David Bau: But in the synthetic datasets, our patching results show that things are much, much quieter, with,
David Bau: almost, like, zero individual words patching effect. So, with this in mind, we…
David Bau: Extract direction exactly at this spot.
David Bau: What we did was, we had a set of, 100…
David Bau: train pairs from the dataset, so it's, like, because we want to test our direction on a held-out test pair. So on those train pairs, we extract the activation from the high uncertainty statements and low uncertainty statements, we take the mean difference, across the dataset, and we set that as our uncertainty direction.
David Bau: to test how good it is, we basically tried to project held-out test pairs activation onto that direction and see if we could add, if, like, positive… positive results correspond to high uncertainty and negative corresponds to low uncertainty. So basically, if we can use that direction to classify held-out statements.
David Bau: And we found that for, the within dataset setting, where we extract and task on the same dataset, we got perfect accuracy.
David Bau: And for the crop data set setting, where we extract the uncertainty from one dataset, and,
David Bau: apply that to another. We got pretty good, accuracy.
David Bau: when we extract from synthetic, but when we extract from templated, we got, like, bad accuracy. That's why we proceed with, the maximum probes, introduced in the truesomeness paper, accounting for the variance of the, activations, and it seems to be that accuracy a lot.
David Bau: Finally, we tried to see if this direction has causal effects. What we did is do a model inference, where we asked, the model to classify a statement as high or low uncertainty.
David Bau: We add that direction in with a scaling factor alpha, so the, so the idea here is that with a positive alpha, it makes the model giving
David Bau: High uncertainty more, and vice versa.
David Bau: And, our results correspond to this expectation, where basically, as we increase the alpha from negative to positive.
David Bau: we observe a monotonic increase in the proportion of high predictions. So, we could see that full override occurs at alpha equals 7, where basically, regardless of the input, whether it's high or low uncertainty, with that scaling vector… with that scaling factor, the model output's high all the time.
David Bau: Oh, the last thing we did was a downstream economic experiment.
David Bau: And the idea here is that we're gonna use, actually, 40 real statements from earnings calls, so we have company information, and we also have financial information for, like, 3 months, right before an earnings call. So we're gonna strike these excerpts, and we're gonna have these sentences that are actually statements from executives, and we also have this financial information, like, right before the earnings call, okay?
David Bau: It's important. And then we're gonna ask the model to allocate $1,000 between a risky asset, which are stocks in that company, and a safe asset, which are U.S. Treasury bonds.
David Bau: And in economic theory, and the empirics also, this is, like, very proven, is that more risk is gonna move investment into the safe assets, which is why, when there's a lot of uncertainty, the market tanks, and everyone's gonna buy U.S. Treasury bonds. That's sort of the idea.
David Bau: And what we're gonna find is that
David Bau: It's gonna find evidence of this in our model, so when we increase the uncertainty, when we steer the model towards more uncertainty, we're gonna have lower investment in the stock allocation.
David Bau: She's pretty interested, and as you can see, when we steer towards less uncertainty, the model is going to invest more and more in the company's stocks, which is super interesting, it was really nice for us to see. So what we're gonna say… we're gonna conclude here is that uncertainty is localized, and we can control it, and it actually has an economically meaningful downstream behavior in our model.
David Bau: And that's it.
David Bau: Minute for questions?
David Bau: Is Team S ready?
David Bau: I liked the experiment at the end, that was very smart. Yeah. And cool, and hot, and fresh. Yes, it's great. This experiment didn't exist two days ago. Oh.
David Bau: It's… but it's perfect, because, like, it's genius. It's exactly the experiment. It was even though it works. It wasn't like… It didn't just come out of a… It was a process, it wasn't processed. It's great.
David Bau: It's great.
David Bau: So every… so you guys were impressed with the last experiment, too? Yeah, nice work. Nice work.
David Bau: Yes. I'm looking forward to reading that.
David Bau: Alright, to the left.
David Bau: Sorry. You're here next. Okay, fine.
David Bau: Okay, great.
David Bau: You shall live.
David Bau: I can tell you in Chinese, but no.
David Bau: So, in the middle side? Yeah, I think.
David Bau: Alright, I'll present it there. Yeah, something like that, so…
David Bau: Alright, nicer. Okay.
David Bau: Great. What is iPad?
David Bau: Okay, welcome!
David Bau: Team S, which are… is… they are brave…
David Bau: vision language model teams, which is more of a challenge because of the architectural differences than the other LLM.
David Bau: projects, but I'm looking forward to seeing how the project, turned out. All right, welcome, PMATS!
David Bau: Hello, everyone, we're Imaz, and our topic is more focused on visual language, although just, like, what David said. And my name is Vichy.
David Bau: I'll be sad.
David Bau: Okay, so, today, vision language model can do a lot of parts that humans do. Like, if you show vision language model an image and ask, does the image look soothing? And most likely, the vision language model will give you the same answer as human.
David Bau: That means at a behavior level, a visual machine language model has the same effective perception as humans, but the problem is.
David Bau: How… how the visual language model computes
David Bau: this concept inside of the model, is it also, like, a similar pipeline as human, too? And, this is our, research passions, like, how the visual language model process this kind of information from the image.
David Bau: And, we need to… we cannot just look at the behavior, we need to, look into the model to see the, activation.
David Bau: Yeah, so, by doing this, we look at the different applications, so we honor the context and processes that occur in the language model that we can, take out of the intermediate
David Bau: And, we set up, same prompt for, different images, and…
David Bau: Just to describe this image, and we broke on both encoder and decoder for encoder. So we're doing, meaning for over all the image tokens.
David Bau: And we can see, the probing accuracy across different layers in our visualized models. And, the, the red line represents the non-effective green, and the red line
David Bau: represents the vector main. And, especially in encoder, we can see a clear difference to see that, on later layers, I mean, for effective main, the accuracy goes up, slower than the, not effective main.
David Bau: So, from the probing, we know that some information exists in this, heat vector, but how, how does the model use this kind of information? So, by doing so, we,
David Bau: proposed some of the experiments using catching. And, again, we did the same experiment on both encoder layers and the encoder layers. We see that, for, for the
David Bau: Attributes for non-exactive and effective. Broken on both.
David Bau: I mean, touch on both Attention and MRP, they show a similar trend, which shows that even if they contain,
David Bau: The different concepts, but they still work the same… on the same mechanism, in the both encoder and decoder layers.
David Bau: And move on to next, we want to see, like, how are… all of these are interpreted in the decoder. So we do separate experience on non-effective attributes.
David Bau: Things like, if the scene is, a rock scene, or, we… if this scene contains natural light.
David Bau: So, we have the non-effective graph alert here, and we also do effective,
David Bau: attributes, comparison right here. It mostly, we analyzed the, scary, Susan, the stressful attribute.
David Bau: And you can see, from these two, pictures, first we have a difference in the, image tokens and attribute tokens. When we do, encoder patching on the,
David Bau: for, for a meme poll on the all-image token activations, you can see, like, other, both non-effective and, effective attributes are, they, jiggle around,
David Bau: 15 to 20 layers, so that's… that's where we suspect that the, when we patch the image tokens, the VLAN makes these decisions around
David Bau: These two layers. But we, when we, move on to the, when we patch the attribute tokens right here, you see that,
David Bau: It, struggles to… the main drop is happening on the 22…
David Bau: 25 decoder. So, this is our first finding in this, graph. And also, you can see that across, from non-affective to effective,
David Bau: Attributes, we can see that
David Bau: For effective attributes, the in, The slope is, pretty…
David Bau: I would say it's not that steep compared to the, non-effective, attributes, so we think that the concept is maybe,
David Bau: Distributed, stored distributed in the decoder layer, compared to what we have in the attribution context, where, they share pretty similar, results over here.
David Bau: And this is for activation patching, and we also do… next slide. We also do a indirect effect analyst right here for, so we want to see how much,
David Bau: We want to see how much attention, layers and MLP layers place in each run, for, inside the decoder. And, you can see that for image tokens, we try to, run the analyst, but the recovery rate is, barely, effective.
David Bau: So the scale is actually different from the attribution… The attribute tokens part.
David Bau: And, but, so…
David Bau: But right here, if we try to, inspect the attribute tokens in the decoder, we see that, attention and NLP all have a pretty,
David Bau: Good recovery rate at, they are, 15 to 20.
David Bau: In not effective,
David Bau: Attributes, but for, effective attributes, like, scary, sufficient and stressful, which we, mainly focus on researching,
David Bau: the…
David Bau: The MLP takes, just a little bit of the influence. It mostly uses attention to, decide their, to do the decision of whether the attribute, whether the image contains the certain attribute.
David Bau: And yes.
David Bau: Okay, so let's put the picture together now, and, in Ecoder,
David Bau: the effective and non-effective attributes share the same MLP, attention MLP pipeline, but, effective attributes just need more, layers to,
David Bau: to spread the concept, and in Decoder, there is, like, where we see the different evaluation for the concept, effective and non-affective shows different patterns that,
David Bau: For a non-effective concept, it needs strong attention and strong MLP, but for effective, it just needs a weaker attention than the
David Bau: anymore, MLP.
David Bau: And, this beast, This piece reminds us of,
David Bau: Some finding in, effective science.
David Bau: Which is, appraisal theory. Emotion is not triggered by events, but, by how events are evaluated. So for visual language model, we also find, similar patterns, that in encoder, which is perception, procedure, it doesn't,
David Bau: they share the same pipeline. For the same image, they have the same representation. But in Decoder, the way we evaluate the concept is different, so that's how different, precising pipeline for effective and ineffective concepts.
David Bau: That's all. Thank you.
David Bau: Any questions for the team?
David Bau: What's one interesting thing that's not in your presentation?
David Bau: It wasn't think She, like, said to me, like, pow, what's up?
David Bau: But you were like, it's not here.
David Bau: That's a good answer.
David Bau: So, you found that the patching
David Bau: Didn't work as well when patching over
David Bau: image… image data rather than cache data, for positives. Do you, do you have a theory for why that is?
David Bau: Sorry.
David Bau: Because we did, not only on Quinn Bales, but also to other models.
David Bau: Because we sneak at the way they, are trained.
David Bau: are different, but it turns out the result is… Similar across all of them. It's interesting, yeah.
David Bau: Yeah, I think, like, the image tokens are well precise in the encoder already, so for a decoder, I just need to read the information rather than write down some new information in the image tokens. Right. Have you seen,
David Bau: Some of the… Papers that suggest that when… to have success patching image tokens.
David Bau: you have to patch a lot of the tokens, like, you know, huge regions of the image. I'm not sure, like, so when you patch… did you patch, small regions or big regions? We patch all the image tokens. All of them, and you still… you still didn't see as any strong effect, at least for these words.
David Bau: Interesting. Okay. Well, great.
David Bau: Thanks very much.
David Bau: So the next team, Team M.
David Bau: We're going on at a clip.
David Bau: So that we… we can get to everybody.
David Bau: So, go ahead and reject.
David Bau: Perfect.
David Bau: Hey.
David Bau: So…
David Bau: partly populated it? I mean… Let me show you… Quote. Okay, cool.
David Bau: So, hi everyone, we are Team M, and we are trying to, to analyze the representations of geography in larger network models, and this is our team members, and some of them actually traveling across the different countries, and, yeah, which is very interesting at this stage.
David Bau: Yeah, so, yeah, I'm open to this. So, this is how we interpret our world. So, there are different countries, they have different continents, and they have oceans.
David Bau: And also, we may interpret… we may interpret some, like, locations, for example, Canada is at the… Canada is at the north of the United States, so we have different… a lot of ways to interpret our world in many ways.
David Bau: So, we wonder here if LM do the same as, we do in our mind.
David Bau: And in 2024, Gurney and Thermat, they showed that LM actually encodes some knowledge of the world as a function of latitude and longitude. And here, they asked the model using the activation to predict the landlord of the specific, cities and the project on a, on, on, on a, on a, on a map like this.
David Bau: And we are trying to replicate the same thing, and we build our own data set where we include 24 hundreds of cities across 24 countries, and 8 cultural regions. And we asked the LM, specifically Gwen and Alama, for this specific question, where is the city? And with the question mark.
David Bau: And we extract the activation at a different layer, and using a linear probe to
David Bau: predict the lead and alone. And tested on 20% of the test set, and we used the square to evaluate how good they are actually predicting this lead and a loan, and find the best, best layer.
David Bau: And this is an example specifically for Gwen, for Gwen, 12.57TB.
David Bau: And the larger R-square, meaning that they have better representations of the information, so as we can see, as B goes through deeper to this model, we find there are better spatial representations.
David Bau: However, if you think about it, LLM at its core is a next token predictor, so the question shouldn't just be, like, what is this model representing, but what is this next token representer, how does that represent spatiality?
David Bau: Look at this prompt, when LLM is tasked to be a navigator and travel from Mumbai to London, when we ask Metalama 38B to do this, this is the path it chooses. It goes from Mumbai to Dubai to Frankfurt, London.
David Bau: We did this across many paths, and we noticed there was this tendency for the model to always choose Europe, irrespective of where the model is coming from. Even if it had to travel from America to, say, China, it would still go through Europe.
David Bau: So why does it do that? Why does it always choose Waypoint via the Bay in Frankfurt? Is it just more preferred? Is it more convenient? Or are these models biased to choose certain locations?
David Bau: We claim that question, we ask, are there certain cultural regions that the models within the LLMs are biased towards? The way we look at culture. So this is… we use the Ignat Wellsville World Cultural Map, which is collected from the WVUS survey, which is done over
David Bau: Hundreds and hundreds of participants, where they kind of map our countries into these 8 cultural regions.
David Bau: And we then conduct a representational similarity analysis. So what we… from these 8 cultural maps, we get 2,400 countries across 2,400 cities across 24 countries.
David Bau: And we map, first, the city locations, that is based on latitude and longitude, that is what the… using the habit side distance, and we map the city embeddings, that is, the model's internal representations.
David Bau: And then we see, we compare these distant matrix, matrices.
David Bau: Let's say we then compare that as a measure of geographic encoding quality, where, for example, if you want to see how well Tokyo is represented, you'll find a correlation between this geo-distance matrix and this activation distance matrix.
David Bau: A low correlation would mean that the model's understanding of how Tokyo is represented is not exactly the way how we represent Tokyo. Also, the higher rep correlation would mean that Tokyo is well represented.
David Bau: Oh, yeah, so here is specifically comparing the performance of two models, Lama and Guang, because they are originally from different countries, and one is, you know, basically in the US, and one is developed by a Chinese company.
David Bau: And we saw, like, a significant difference for different regions. For example, let's focus on the East Asia.
David Bau: Sorry, I forgot to mention that the… so the dots actually placed in their, branches, let alone, but the size of the marker stands for the RSA. The larger means that they have better, geospatial knowledge, relative distance knowledge.
David Bau: And so, let's look at this East Asia apart. That's it. We can find that, actually, Guam did better than Lama, because they had that, larger,
David Bau: larger markers. And for English-speaking countries, for example, Australia, and all, those, some, from the UK here, actually, Lama, did better than Gwen.
David Bau: And we go to, deeper under trying to understanding, like, how well each region is encoded in each llama, in different… in different layers.
David Bau: And,
David Bau: And this is an example, just during the Lama, and you find that, actually, the protested Europe and, and also the Catholic Europe, has the best representation. And while the, West and South Asia has the worst performers in the stats across all layers.
David Bau: It's Scotland.
David Bau: Okay, okay, yeah, and we're trying to comparing, like, how, how these different models,
David Bau: doing this, how, how, how does different models doing, performance differently in, in this, in these tasks? So we, at each layer, we rank all the regions by their main preceding RSA, and do this, like,
David Bau: stress boosting to testing, like, actually, which model performs better in different regions. For example, on the right-hand side, where the blue bar shows that actually the, grant performs worse, while the Llama works better, specifically meaning that the,
David Bau: LAMA, encodes better presentations of protested Europe, includes speaking and, orthodontics, Europe, relative distance, knowledges, while the Guam has better knowledge about the West and South Asia, Confusion area, and Latin America.
David Bau: Okay, sure.
David Bau: Yeah, so we then look at, like, precision and recall, in terms that we… for, say, Tokyo, we look at, like, the k-nearest neighbors of a city, and we penalize… for recall, we look at… we penalize the model if each missed geographic… if it misses any neighbor, while for precision, we penalize if it adds a false neighbor.
David Bau: We see that across models, it usually performs better when they have higher precision and low recall. English-speaking cities fall more in the robust performance zone, while more West Asia, African Islamic cities, and Latin American cities fall… usually fall in the poor performance zone.
David Bau: Now coming back to this example.
David Bau: We've asked the model to be, again, a navigator of traveling from Mumbai to London.
David Bau: But… and we know that the model does really well on… or Manorama does really well on Protested Europe. What if we were made to forget the model… we were made to forget that Protested Europe even exists? We…
David Bau: every direction that exists in, for protests in Europe, what do you think will happen then? Do you think the model will still, like, force itself to going towards Europe?
David Bau: We see no. If the model decides to take the turnaround way, it goes to Mumbai to Delhi, Bangkok takes a really long way, but it does not… But, yeah, that was basically our presentation. In conclusion, we show that… we validate that LLM shows strong representations when encoded with latitude and longitude.
David Bau: We also show that a bias in representation, there exists some bias between cultural regions. The model doesn't bias… the model doesn't represent every continent equally. There are certain biases that exist.
David Bau: And through causal intervention, we've provided early evidence that the model kind of anchors its representation in the logic of viewing this as cultural regions.
David Bau: Yes, thank you. Great.
David Bau: Any questions?
David Bau: The last experiment is new. Yeah, we did that. It's cool. so I have two requests. So one is… so I'm looking forward to reading more about the experiment.
David Bau: One is the West Gurney reproduction, where you have this beautiful graph going up.
David Bau: I'd love to have in your appendix some Wes Gurney-style maps to see what your reproduction looks like in terms of his visualization.
David Bau: Like, this kind of thing. If it's messy or whatever, it's all very interesting. Like, you know, if it came out worse than what's his thing, it would be nice to see.
David Bau: And… or even from layer to layer, and how it changes.
David Bau: And then your last experiment. Is that… is that on a multi-token rollout, when you ask it to give you
David Bau: the whole… the full itinerary? It's a single, term. Like, we just ask it… we say, okay, Mumbai dash, and then we ask it to roll out the entire. We don't intervene at multiple stages.
David Bau: Oh, so, but it's… but you, you do do one generation of, and then you have it?
David Bau: say as much as it wants. Yes, yes. So we have kept a limit of, like, 150 tokens. And is that a steering where you're making an intervention during the generation, or is it a different type of intervention when you do it? So we, basically use a classifier to identify
David Bau: the most linearly predictive direction for, say, Protestant Europe in this case.
David Bau: And we find, through the classifier where we null out the orthogonal direction.
David Bau: And that's the direction of protection Europe, and then we subtract that from the null activation, which is then projected upon before the generation. So you do it at, like, one token before the generation? Yes.
David Bau: And then you don't do it anymore after that? No. Wow. And then it has an effect on the entire generation? That's what we see. So, we did see that this was, like, false… the particular example was for layer 20. As it goes towards layer 30, it kind of starts fighting back. It's interesting that happens anyway. I'm looking forward to reading it. Yes. That's great, that's my question.
David Bau: Yep. All right, next team, next team. We don't have much time, so… Sorry to…
David Bau: Make it go so fast.
David Bau: It's very interesting. It's a very nice, very nice choices, the last experiment. It's really cool.
David Bau: Nice.
David Bau: Nice background.
David Bau: Hands are yours.
David Bau: Excuse me.
David Bau: Hey everyone, so, we're Team TK, or Team J, whatever it was, originally. I'm Jasmine.
David Bau: I feel like you guys know who you are, but…
David Bau: You guys know how to deal with this songs?
David Bau: Bye. Okay, that's a no.
David Bau: Okay, so just a quick roadmap. Basically, we're asking, you know, do models robustly, model and represent speakers? We're going to have one section on evaluation, so which models have this capability?
David Bau: One, an interpretability, so how are speakers represented in activation space? And then finally, the mechanism. So we'll talk about what specific kind of mechanism they… Oh, hey!
David Bau: So one big question. Do language models build, maintain, and use speaker representations to reason about dialogue?
David Bau: Why does this matter? Well, language models are increasingly being deployed in social settings that demand an exact, yet flexible knowledge of who's speaking. If you look at the App Store, the top 3, you know, like, things you put on your phone right now are all LMs.
David Bau: And McKinsey shows that right now, around, like, 79% of businesses are actually integrating Gen AI into their different business applications. So basically, the settings they're working in are becoming increasingly complex, and the stakes are, you know, growing ever higher.
David Bau: So, how do we assess, like, how do we figure this out? Like, right now, people do a lot of vibe checks, but we want to do something more principled. Well, we take naturalistic text.
David Bau: that has, clear roles and active interpersonal engagement, so basically these are groups trying to figure out, like, this survival tasks, like your plane crashed, let's figure out what items we wanna, we wanna, like, have, like chocolate, a gun, I don't know.
David Bau: And there's also, most importantly, limited extraneous identity giveaways, so it's not like Obama speaking, or, you know, Dario Amade, so in those cases, maybe the models memorize a lot of external, kind of, like, structure, or idioms, or…
David Bau: etc. So we really tried to keep it, you know, like, naturalistic, but also very controlled. And so here, the conversation dynamics are what create the structure.
David Bau: Importantly, in our setting, we give, you know, a model some transcript, and we ask it to reason about how many people are speaking at the time. We only use models which are either instruct or have reasoning toggled off.
David Bau: And also, we share this conversation before we ask the question. So, this is really important, because what we want to understand is.
David Bau: did the model have these representations, and was it building them up to understand the conversation? We don't want to ask the question first and have it be like, hey, like, this is a thing that I should be tracking. And we also don't want it to use reasoning to just brute force, be like, okay, like, one guy's there, another there, etc. We want it to specifically, like, build this up across the conversation and use it as a part of the reasoning strategy.
David Bau: So these are examples of our transcripts.
David Bau: So we see on evaluations, models pretty handily complete the label task. We want to make sure that they're, you know, smart enough to do the very simplest case, where it has the speaker's names, and there's very clear structure. And on Unlabeled, we do see a performance gradient, but it is very clearly above chance. We also see some pretty interesting scaling story, I'm not going to go too much into it, you know, where we have frontier models as pace setters to make sure that the questions are not impossible.
David Bau: Interestingly, the older GPTs actually tend to do better than 5.4, and this is something I also saw anecdotally.
David Bau: Didn't have time to look into it, but thought it was cool. So, now we move on to the interpretability section.
David Bau: How do language models figure out who is speaking?
David Bau: And then I thought Giuseppo's thing was very cute. It's like, I'm a magician, said Michael. I'm a skeptic… oh wait, this… I put this in, sorry. I'm a skeptic, Michael replied. Obviously, these are two different Michaels. Like, how does a model know that? That's crazy.
David Bau: Yeah, so, adjustment said to be this kind of behavior and this model, so, I was really…
David Bau: more thinking about how it gets represented in the models, this idea of speaker identity. And so I created, like, a bunch of different transcripts I'm going to focus on, too, today. One is called Distinct, it's the Baseline, where two people are just talking, and then there's another one called Quote Intrusion.
David Bau: Where, basically, if something Bob says, Alice will, like, verbatim in this, like, argument sense, like, well, you said this, and that makes…
David Bau: So that ended up being the most interesting experiment, so we're focusing there. The pipeline is, we just take the transcript's tokens from 1 to T, meaning, like, for every turn of the dialogue, we feed every, every bit of the transcript up until that current turn of the dialogue, then we extract
David Bau: just that turns, tokens. We average them at layer 20, by the way. We average them, and so you end up getting one vector per turn, so you have, like, all the Alice vectors, you have all the Bob vectors.
David Bau: And then we trained, using L2 logistic regression.
David Bau: And we test it on held-out transcripts. Now, this is, like, going to be important to keep in mind when I go into some of the results later. And I use shuffle, label, good quote, meaning, so we have 15 transcripts out of the 20 that are for training.
David Bau: We have 5 for testing, and so while those two classes are separated, we just iteratively shuffle the labels and test the probe on that.
David Bau: Nice use of the Hewitt & Young.
David Bau: And then your training data, does it say Alice and Bob in it? Like, so those are all stripped? Those are just labels, yeah. I see.
David Bau: Yeah, so then, so now this is… these are the probe results, so you get…
David Bau: I… what I'm showing is Llama, but I also tested this on Olmo, and it has pretty similar results, but slightly worse in terms of accuracy, but…
David Bau: The point here isn't like, oh, look, I found a great probe. The point is, like, I found a direction, and it's above shuffle, and I think that's cool.
David Bau: So that's the left-hand side that you're seeing right there. On the right side, what I did was, again, with the whole train and test set separated, I trained the probe on the first half of transcripts, and then I tested on the second half.
David Bau: a completely different transcript, so there's no leakage or anything. But this is to just say, like, what if, Bob and Alice said hi to each other in the beginning? Is that what's really happening? And this says, no.
David Bau: Or at least for me.
David Bau: Okay, and then, so, the next result is… this one's a cool one. We trained a probe on non-quote turns, so we're only looking at the quote intrusion dialogue at this point.
David Bau: And we tested on quote turns. And so non-quote turns, I say, where a producer, with whoever is speaking during that trans… Like, the chat.
David Bau: is the one who is just saying it. That content is, like, for the…
David Bau: When there's a quote, the producer, the person speaking, that's not where the content is actually coming from. It came from the previous speaker, correct? So there's a little bit of a dissociation going on here.
David Bau: So basically, what we're seeing is that, the ProBasic just inverts. Like, it's not even just doing, like, randomly bad, it's doing spectacularly bad, which is great for us.
David Bau: Because what it kind of says is that, probe might be, like, Consistently looking at the content.
David Bau: Of what's being said, as opposed to trying to understand who is actually saying it in a dialogue sense.
David Bau: So I'm look… on… so that's the results on the left side. On the right side, what you're seeing is, like, an example transcript.
David Bau: The green squares and dots are all from, like, the standard probe, whereas the gold is from the, probe that was only trained on non-quote turds. So you're seeing, like, how it very much, like, consistently, except for a few up here,
David Bau: It has low confidence, and it's ascribing it to the other speaker.
David Bau: Something else that's interesting is that the squares are actually the quote terms for the standard probe, and the standard probe does pretty well with the quote terms, as it turns out.
David Bau: Yeah, so…
David Bau: It's crazy, it's almost like it's strategically getting it wrong. Yeah. Well, that's, like, that's the main point here, is, like, again, I'm not trying to say we found a great rope, I just want to show, like, the sign flip is, like, the most important point here.
David Bau: Yeah, and so these are just, like, some numbers on accuracy. Like, accuracy on quote terms by the transfer probe is .376, which is well below chance, well below the shuffle controls.
David Bau: And then the accuracy of the standard probe on quote versus non-quote turns is about 06.93 to 0603.
David Bau: So I'd probably have to scale this up to actually see if those numbers are meaningful at this point, but… yeah, so basically both signals are there. Tracking content origin seems to be, like, the default for a probe that's never learned.
David Bau: the concept of people quoting each other, but then a probe that's been exposed to quotes, seems to abandon content and go more towards some sort of attribution framing. So now, again, that's, like, our leading into that.
David Bau: It's like layers of knowledge.
David Bau: So basically what we do is we set up a controlled, like, two-speaker setting to study how Lottoms link speakers with their contacts when the names and the attributes span across different conversation lines.
David Bau: So, this is the causal activation patching. We find that the model does use a binding ID, and the strip at the Alice carrier token kind of proofs that, because when you patch from source to the corrupted one, the model is there between authority and some friends, despite never seeing Alice and only seeing Claire as his context.
David Bau: So from the activation patching and a few other circuit analysis things, we identified the following binding IDE mechanisms. The model first assigned binding IDs to the speakers as they see them, and then the binding IDs are kind of propagated to the attributes.
David Bau: And when the model sees the question, a lookup balance in the conversation, it resolves the binding ID to triangle as the lookup key, and then later at the final layer, it uses the key to look up front as the final answer.
David Bau: Note that in the original settings, Atlas Entity always goes first, and the first country always gets the triangle binding ID. Our dialog setup allows us to naturally break these coincidence between binding ID order and the appearance orderings, so we… what we do is we add this, like, grading line, like, the third line, hi, Bob.
David Bau: Which allows us to see that the Bob's country actually comes first now.
David Bau: We also have the reset setting, where we, prompt the model to be like, hi Alice, which Alice cannot be grading
David Bau: herself, so the fourth line now belongs to Alice. We see from the actual… What are you busting?
David Bau: trying to tell Okay, keep going! We're not done on the previous slide, yeah.
David Bau: Anyway…
David Bau: It shows pretty similar activation patching patterns, including the strips and the Curialis tokens. So we take this, and then we train the leader prop from both the entity and the attribute tokens, labeled by, this triangle and rectangles here.
David Bau: Slide.
David Bau: We test the train leader probe to, like, new, more complex transcripts, and we see that the model assigned by the IDs based off of summarization term structures and disparate views. So, like, for example, and the Alice turn on row 3,
David Bau: Bob and friends actually received Bob's finding ID because I was born to fall off, and then, and, grow poor.
David Bau: Thailand is receiving Bob's binding ID.
David Bau: first… using the first-person queue of, like, I live in despite Alice appeared earlier in the same turn, so this kind of tells us that, the speaker attribute binding is context-aware.
David Bau: Okay, so basic… I don't know, our central findings, models create structured representations of speakers. They're linearly decodable from residual streams. There's also this interesting quote behavior, where probes without context have a richer kind of behavior than we'd expect. We also identify a binding ID mechanism which follows discourse structure. And, for future directions, we're going to apply these in monitoring and defense.
David Bau: to diagnose role confusion and prompt injection vulnerability. We're going to try and do some training to improve role representation and inference. We're going to unify the settings.
David Bau: Just, you know, and try to connect the interp results with actual role inference ability, and then also do some multi-speaker work. Thank you, guys.
David Bau: Alright, next team up is Team B.
David Bau: Alright.
David Bau: Am I wrong?
David Bau: So, we will talk about scoffancy today.
David Bau: our, kind of story beginning from the searching for political bias, but we end up with a search for discrepancy, and then ask every other shift their express beliefs and leaves to match users.
David Bau: As a political scientist, I'm well aware of this fact. He has man-eyed dynamic in the political bureaucratic inefficiencies, but this is also a kind of… this is a long history and evolutionary… evolutionary anthropologists say that this is a kind of a survival strategy in hermit for the social cohesion.
David Bau: And there's a Venom TV series I recommend to watch in 1980s.
David Bau: But information system algorithms put us in, kind of, curated, kind of, bubbles and chambers.
David Bau: And then kind of shape our behaviors, and then harms democracy and deliberation.
David Bau: LLMs raise the stake, and then tailorize the responses, and shape our judgment. And there are a bunch of studies, I'll just screenshot the titles. You can see, LLMs shape our perceptional judgment, and scope authentic AI decreases both pro-social intentions, and it persuades our, kind of.
David Bau: It shifts our, kind of, political beliefs, etc, etc.
David Bau: So, so that we understand this is kind of a, this matters to study, and then we kind of…
David Bau: Do a couple of, experiments, and then find the pattern.
David Bau: Which is, let me agree with commonly held post-liberal beliefs.
David Bau: But, disagree when you say you are a conservative.
David Bau: And we asked, alright, why do LLMS systems change their opinions? I agree with you, sir.
David Bau: Leave it.
David Bau: Alright, so moving into interpretability land, past works have shown that we know where succancy happens in LLM, so we can localize it, we can kind of toggle it on and off.
David Bau: But we don't know why or how it happens, like, what's going on in those layers? Like, what is the model thinking when it switches its response based on what you say your views are?
David Bau: So…
David Bau: To try to investigate this why question, we look at two existing extreme hypotheses for why LLMs switch their views.
David Bau: One is the stochastic… what we're calling the stochastic parrot hypothesis, which is that LMs agree just because it's most likely. So, as Emery pointed out, you know.
David Bau: There's a lot of echo chambers online, a lot of text is people saying, that's such a good point to each other. And then the other extreme we're looking at is the sycophancy hypothesis.
David Bau: Which, we're going off the word sickmancy, which in English kind of implies an insincere flattery, so we're imagining that LLMs kind of know what they really think, but say something else instead.
David Bau: And we stress test these hypotheses in Quen 2.57b Instruct and LAMA 3.370b Instruct on a very small dataset, so we're gonna make some strong claims, but add a lot of salt, grains of salt to them.
David Bau: So first, we start out with stochastic parent. Do LLMs agree just because they're a lookup table, they're just memorizing what's likely online?
David Bau: And we claim that no, LLMs agree because they have linear representations of user beliefs that are causally implicated in model outputs.
David Bau: So, to extract these representations of user beliefs, we use the same methods from the persona vectors and assistant access papers from Anthropic, but apply it to the user instead of to the assistant.
David Bau: And we prompt models with a description of the user, so, like, I'm a devout, so the user says, I'm a devout evangelical Christian, and then asks the model what they think about some question, and we extract the activations for the assisted responses to those questions, and mean them for the conservative user and the liberal user.
David Bau: And then using these contrastive vector, we can steer along the model's representation of the user's politics, and find that steering on the user's politics will shift the assistant's stated belief from up high as more liberal response is scored by LLM, and below is more conservative, and 50 is neutral.
David Bau: Okay, so…
David Bau: We've eliminated one extreme. It's not just memorizing something, there's a causal mechanism. But what about the sycophancy hypothesis? Do LLMs, like, think one thing, and then they say something else entirely different?
David Bau: And we claim that, no, at least for the models we were looking at, we can't recover what… how the LLM would have responded if they didn't know anything about the user in contexts where they're flipping their response to respond differently in response to user opinions. And specifically, the bar to look at here is… so…
David Bau: We're training probes to see whether we can predict that the model would have responded conservatively in a neutral context, but respond liberally when prompted by a liberal user, and we find we can do no better… we do worse than chance, in this context, so we appear to be a coherent, linear representation of what the model really thinks.
David Bau: So, okay, so now what do we do? We know nothing, we're…
David Bau: We have no idea what's going on. So now that we've eliminated the two extreme hypotheses that LLMs are essentially a lookup table, or that LLMs are, like, intentionally being sycophantic, we can say that these mechanisms don't cause LLM movement, but what can we say about what does?
David Bau: Well, something we're interested in is that in our, first two hypotheses, or we're eliminating them, and in a lot of mechanistic interpretability work, and even social simulation work as a whole with sycophancy, we're looking at synthetic persona descriptions, not actual real human users. So we wondered whether LLMs have different representations of user politics dependent on whether users are synthetic or real human, and if that might be why we see these causal
David Bau: So to do this, we set up two match datasets, one with real human descriptions about beliefs, values, and principles from the PRISM alignment dataset, and then we have the match dataset, where we ask Claude's son at 4.6, to create personas responding to the same prompt.
David Bau: We also want to, look at the difference between the assistant and the user framing. So the user framing is what Grace has already done, where we give the persona to the user, and then we collect activations as the model reasons about the user's, politics.
David Bau: And then in the assistant framing, which goes back to the assistant vectors paper, we are actually giving the persona to the LLM and saying, you are somebody who thinks X, Y, and Z. Then we have a user message asking the model to reason about its own political beliefs.
David Bau: What we find is that the mechanism behind LLM agreement actually differs depending on whether users are synthetic or whether they are human. We find a high cosine similarity between synthetic users and synthetic assistants.
David Bau: right here. Essentially, what this means is that when a user is liberal, synthetic user is liberal, it's represented in the same space as, whether the assistant
David Bau: visible itself. On the other hand, human user politics are represented distinctly to assistant politics. We have a comparatively lower cosine similarity here between synthetic assistants and real user politics, so the assistant and the user when it's liberal are represented in different spaces.
David Bau: What we hypothesize here is that the, user, or the LLM actually recognizes that it's talking to another LLM when it's talking to a synthetic persona, and it's saying, or representing its own politics and the user's belief politics as the same. Thus, we have this little Spider-Man meme.
David Bau: We do some steering, experiments, and we find that the results do differ between synthetic and human user politics vectors. This is a little bit of reading tea leaves with our small sample size, and of course, our, our large,
David Bau: bounce here. But we do find that, when we are steering with the synthetic user politics.
David Bau: The model will continue to steer liberal, but it has sort of a bound in how conservative that it will steer, versus the real human politics sector allows us to steer more towards the conservative direction. We hypothesize that this is because the LLM is representing the synthetic user as itself, and we already know that the, assistant character of an LLM has a natural left liberal lean and is not more conservative.
David Bau: And so we're actually steering within the bounds of this assistant character when we're steering for the synthetic user, because they are entangled in the spy.
David Bau: So, this comes to our, SSS contribution, our early evidence that sycophancy is systematically shaped by self-recognition of synthetic users.
David Bau: Yeah, so, if you forget everything in this presentation except for one slide, please make it this slide, our hot and fresh takeaways. So, you know, in existing, mechanistic explanations of LLM succfancy, we find, you know.
David Bau: Both the kind of stochastic parent hypothesis of them just being dumb token repeaters, is
David Bau: probably wrong, and the second hypothesis of them being insincere and thinking one thing and actually saying the other is also probably wrong. Instead, it's probably a third secret option, where, LLM assistants have recognized synthetic users as themselves, and…
David Bau: That changes steering effects and behaviors, within the elephants.
David Bau: some impression.
David Bau: Rules of hypothesis, that's great. Any questions?
David Bau: It's really cool, it sounds like there's this pretty big, unexplored space of just, like, synthetic users as, like, assistant personas, and… I don't know, maybe that's not a question, but… Do they…
David Bau: I don't know if you have any thoughts? Yeah, I mean, just if you had thoughts on that space down there, and your third secret option. Yeah, I think we want to explore it further, and not just in the realm of, like, political sycophantasy as well, but in terms of how similar that, like… Like, if you see here, we have this, like.
David Bau: Real, user and, like, transform, so we need to look at, like, linguistic variability and kind of some other factors in kind of pulling apart a real user and a synthetic user, a realistic.
David Bau: synthetic assistant. So I think there's lots to do in that area, but specifically with the political sycophancy, we find that the steering results are pretty promising, that we can kind of push liberal, with the synthetic user, but it's not, like, pushing conservative in the same way that a real user is.
David Bau: Cool, yeah. Yeah, with Evan Perez out there telling everybody to do this, I think there's, like, there's so much work.
David Bau: All based on synthetic data. Yeah. And if you guys were to come out and say.
David Bau: Thank you, Evan, it'd be amazing. And so it was very exciting, although you guys have fairly thin evidence right now, so, but you… Yeah, yeah, 100%.
David Bau: All right, thank you very much. Thanks. Is TP here? Do we have TMP? TP had a timing issue. Are we okay with it? Yes. Oh! Chris showed up!
David Bau: Okay, great. Yeah, Chris!
David Bau: All right, project?
David Bau: We got the notes.
David Bau: That's fine, though. Okay, we're too empowered. So, to begin, we wanna…
David Bau: throw some questions out there, and figure out how this works. Great. Okay, so, do models discriminate against people with certain identities?
David Bau: Do models side with people in positions of authority? Do models understand actions that harm others? So these are kind of central questions and debates about AI fairness and safety, and we tend to think of them as discrete.
David Bau: But there's a long tradition within the social sciences that views these questions as related and about power.
David Bau: That is a kind of thread through all of these. Empower being the ability to get another person to do what you want them to do.
David Bau: So today, we're not gonna be answering those questions.
David Bau: a much more modest step towards a mechanistic understanding of power. So trying to understand, at a basic level, do models have internal conceptions of power? And if so, what do they do with those internal conceptions
David Bau: So more specifically, we're going to be addressing these two questions. First, does the model encode information about power relationships between two people? If so, where? And the second is, is this information causally, relevant for model behavior? So,
David Bau: Before we get into the actual experiments we've, ran, I'm going to talk a little bit about our data that, spans the experiments. So we put together a series of synthetic, data sets with our friend Claude.
David Bau: And, that cover, two sets of contrasts. And, they're each around this question, does someone exercise power over another person?
David Bau: So we have sentences that look like this. Ravi disciplined Leia. So in this sort of setting, we would want to say, yes, right, Ravi is in a kind of asymmetric position of power, exercising influence over Leia, that Leia's not exercising, over Ravi.
David Bau: And we contrast that with sentences that look like this. Ravi helped Leia.
David Bau: So, is power being exercised in this sentence? It's a little bit more ambiguous, maybe there's a case for yes, but it's a symmetrical sentence, and it's not implied as such a hierarchical relationship, right? So, definitely in relationship to the first sentence, this is less power being expressed. So that's our first set of contrasts. Our second set of contrasts
David Bau: have to do not with the relations embedded within sentences, but within social roles. So you think about the doctor helped the patient. Here is the roles that the individuals are occupying, express a relationship of hierarchy, power, authority between them.
David Bau: So with that, I'm going to turn it over to Kai to talk a little bit about what we've done with these sentences.
David Bau: Yeah, so using this setup, where we have different entities that have different power relationships.
David Bau: We collect activations from the model, asking them to identify which entity has power, which one is powerless.
David Bau: The top…
David Bau: chart here is PCA of activations from one of these layers, where we find that it has the most separation between entities that have, power.
David Bau: Which are these kind of purple dots at the top, and between prompts where the entity is powerless, which is kind of these yellow dots at the bottom.
David Bau: And we also notice that there's this kind of separation between, situations where an entity has power or not, and activations, across different kinds of tasks. So, in this bottom PCA plot, also from layer 15,
David Bau: We're just asking the model to write an email.
David Bau: Using these kind of identities. So, in some cases, it's like, you're a doctor, write an email to a patient, and in other cases, it's reversed.
David Bau: And the top blue dots, are activations from when a model is writing from a position of powerlessness, like it's a patient talking to its surgeon.
David Bau: And yellow dots is the opposite, and we're still seeing this kind of separation in both settings.
David Bau: We train a probe using the activations from the setting where we're asking it to identify whether entities have power, and we find that it has high accuracy at identifying whether the model is writing from a position of power in this email writing task.
David Bau: So we are hypothesizing that this means the model has some kind of shared representation of power, even across different settings and tasks.
David Bau: And we find that the accuracy is fairly high, around 85-90% at most.
David Bau: And that it's localized mostly around the middle layers, which, kind of fits into past research, showing that, this is kind of where the model is building its representations of abstract concepts.
David Bau: We also find that power directions, like the ones we just identified in the previous slide, can have causal effects on how the model thinks about power. So here, we're steering the model using power direction.
David Bau: Where we're doing mass mean steering on the same kind of activations that we saw on the previous slide, and here we show that increasing the steering strength towards the direction of power gets the model to say that an entity has power, even when it doesn't.
David Bau: Our results are a bit more complicated and mixed in a downstream setting.
David Bau: This is another experiment we set up, where we're asking, should a powerful entity receive priority support over a less powerful entity?
David Bau: And in most cases, the model just says no. You shouldn't have any preference to giving support to one entity over another.
David Bau: But when we patch in activations from a sentence where the subject is powerless, we find that there's kind of a modest increase in preference for the powerful entity.
David Bau: So, overall, from these results,
David Bau: We basically think that the model is encoding information about power, so yes, it is doing that.
David Bau: But the interpretation, doesn't…
David Bau: translate directly to the downstream setting. We find that there's a lot of, kind of, complicating factors, such as, you know, if the model should be making decisions based on morality, what kind of formatting we're using in the prompts.
David Bau: If the model is refusing to answer the prompt entirely because it…
David Bau: Is trained not to make these kinds of decisions.
David Bau: But we believe that this kind of power direction is a useful, future direction for, studying
David Bau: How models encode inequality and justice more broadly.
David Bau: Any questions for Team Power?
David Bau: I like the focus on the transfer to the email task. It's great. Although, again, a very fresh experiment, right? So, relatively little data. I think it's good. I saw the steering plot, and it wasn't… it was working quite well on the true…
David Bau: Label, but not on the box labels. You have, like, a… Dary backup?
David Bau: Yeah, so this was kind of a sticky confounding factor that we tried to get rid of. This is our best attempt at getting rid of it, and we do that by kind of
David Bau: Taking into account whether the entity is first or second in the sentence, whether the true response should be true or false, and whether we're asking the prompt in a way that is,
David Bau: Saying, like, true means power, or false means power.
David Bau: So we're accounting for all of that, and this is the best we could get, but in general, we find that the model is kind of more biased towards saying true, in general.
David Bau: So just piggybacking off that, other experiments we did to…
David Bau: The false positive case, so when there is no power present, the model accuracy was down, so…
David Bau: Even on the instruct model, it seems to be that it's good at detecting power which just when they're present, but it might… but not when they're not present.
David Bau: Very nice.
David Bau: Thank you very much.
David Bau: It's a fantastic… Good books.
David Bau: Beautiful.
David Bau: You know, asking questions that we don't normally ask as computer scientists, with the models are thinking about.
David Bau: Social science concepts.
David Bau: Things, and and so I think, you know, some of the projects you guys are hoping to bring forward.
David Bau: I'm hoping that…
David Bau: That the techniques that you've learned, if not the very specific questions you have, are things that you can bring forward. And also.
David Bau: The people that you met, you know, I encourage you to strike up
David Bau: collaborations, as you go through your PhD, try to find…
David Bau: other… other cool ways of, trying to cross this divide. I think it's… I think it's pretty important to,
David Bau: You know, for our community to go in this direction.
David Bau: So… but thank you very much for the class. You guys can hang around and help me finish this food, and and then, yes, any… So the… everything is due midnight tomorrow, or… Yeah, so let me see. So I… I put it… I put it on the website, a few weeks ago when I looked up
David Bau: When my grades are due, to give me enough time to grade it?
David Bau: And so did I… did I put… did I put a time on it? No, no, just a time.
David Bau: Just a day.
David Bau: Yeah, let's just… so it's due tomorrow, is that right?
David Bau: Yes, I think that my grades are due the day after. So yes, so you can have up to midnight.
David Bau: Tomorrow. And then I'll get the grade then after that.
David Bau: Great.
David Bau: Yes. Where should we submit it?
David Bau: Yeah, so, do two things. So, put it in the…
David Bau: G-Drive for your team, like a PDF,
David Bau: And then email me and Kiel a link to how we can open it.
David Bau: Okay.
David Bau: Great comment.
David Bau: Thanks a lot.
David Bau: Hey, you guys.
David Bau: So…
David Bau: How are we away from tomorrow? From tomorrow, yeah. I'm gonna be on that flight. No, no, no, I'm a person with Philly to visit my girlfriend, and I will just directly.