d.bau: Lovely Yeah, but it's a lot. This one's gonna talk about power, like, Deep down in the history.
d.bau: Common.
d.bau: That's a weird way to phrase it, but, like, events that you didn't really expect, like, are not examples.
d.bau: And then, secondly, like, first movement versus second movement. There's, like, how I derived moment.
d.bau: So, meaningful economic activities, for example. So basically, just like how people have done this kind of screening, using our output, we use… We did a whole technical neural networks plan.
d.bau: In one day.
d.bau: Covered all neural networks from… What's in her on?
d.bau: to modern transformer allies.
d.bau: a day, right? So… Today, I think we're gonna talk about Entire… Debate.
d.bau: about… Computational models of…
d.bau: concepts, all in one day. Also, it could be a whole class, or it could be a whole, you know, semester course, a whole year course.
d.bau: So we'll try to cover it all in one day.
d.bau: So, we'll see.
d.bau: I think it's relevant to a bunch of your research.
d.bau: But it's gonna be faster than one day, because… Other things to cover.
d.bau: Besides that.
d.bau: There. So, we'll see.
d.bau: Good Lord.
d.bau: Fair enough.
d.bau: So,
d.bau: Now, you haven't modified the notebook, right? I modified it, like, for the puns, case. Okay, did you push the modification?
d.bau: I gave it to Nick Hill to let it go. Oh, you push it. And, the thing is that the pun stuff doesn't work programmatically, like, Jake and I were trying to figure it out. There's some, there's some,
d.bau: discrepancy between what PR does and what's happening programmatically, even on the default model, without steering, so… Yeah. Like, these discrepancies. I think, you know, we… you know, so there's actually… I'm serious, so,
d.bau: Adam, on the end of April?
d.bau: Is interested in putting together some sort of frequency paper.
d.bau: Are these discrepancies?
d.bau: Yeah. Like, we use transformer lens, and there's some potential, like, yeah. Yeah, what are the differences?
d.bau: Nobody's doing anything fancy. It's just like, Yeah.
d.bau: Yeah, so who was here for Genghast?
d.bau: Is that chief thing? Maybe nobody.
d.bau: Yeah, yeah, but it's investing kind of stuff.
d.bau: They're,
d.bau: But, you know, with them. They're doing anything different to these models, but they… they behave differently inside.
d.bau: What the heck's going on?
d.bau: None.
d.bau: So when do we start? Do we start now? Okay, let's start now. We'll start… we'll start a little… a minute early instead of 5 minutes late, because we have too much to cover.
d.bau: Alright, so, who is, whose team? S?
d.bau: Facial theme?
d.bau: Okay, are you guys ready to present? We'll see, who's Team E? Oh, team E's, like, ready to go. Okay, okay, TeamS.
d.bau: Okay, so let me, let me, let me, let me, cover for everybody what we're gonna do today. As you guys come on, you can come on up.
d.bau: This, RV, should I announce?
d.bau: Yeah, you can announce whatever you want to announce. So, okay. So, first of all,
d.bau: You know, Nikhil is our TA? Yeah. Hello, everybody. Okay.
d.bau: And so, that media. And what's the announcement? I forgot the Inaskan. Yeah, I think there's three logical things.
d.bau: If anyone has any enrollment issue, please, like, you know, as soon as possible. Oh yeah, because it's like, at the end, like, the registrar won't listen to us in a couple days, so…
d.bau: So, yes, any involvement issues.
d.bau: I haven't got it for the course. Unless you want it in your transcript, you don't need to do anything. Okay.
d.bau: And what was the second thing?
d.bau: But one thing was, we have updated the link to our NDI server.
d.bau: On the website? Oh, the Discord server. Yeah, the Discord server. Oh yeah, that invite link wasn't working. We should use that link to join, join the Discord server.
d.bau: Yeah, if you want to talk to the people as well as…
d.bau: Anything in real time, you can use that. Yeah. It's updated for the class, so it's optional. Okay.
d.bau: Okay, before we do this, so today's plan, there's lots to do.
d.bau: We're gonna have Team S in E first, receptive Project Pitches.
d.bau: You know, 5-10 minutes each, and then we'll… and then I'll give a very brief… because we're doing this at the same time as Project Rich, we'll do… I'll do a very brief introduction.
d.bau: I'm going to try to pack an entire year course into this one half-hour lecture, so, so we'll… but we'll do it interactively, so that, you know, we can kind of see where everybody else.
d.bau: And we'll do this discussion, and I'll closely point you to the more reading this section.
d.bau: And,
d.bau: And then, and then we have a notebook on steering at the Aruna, and so Arena will step us through that. So there's a few things, about half an hour each. Okay, go ahead. So go ahead and plug in your HDMI, and then you can, you can, click over that, that button way in the corner.
d.bau: If you click on the… and then… okay, there you are! And then when we're done…
d.bau: The other team, Team S, can plug in, so go ahead.
d.bau: Hello, everyone. We're working E for Excellence in Economics. And economics, and we're working on… we're thinking of working on economic uncertainty and LMs using earnings calls. I'm Veronica, I'm a third-year PhD student in economics at Boston University.
d.bau: Northeastern, events involved.
d.bau: open to.
d.bau: So, expectations are key to economics. If a firm has
d.bau: Expectations that the economy is going to do well, they're going to hire more people, which improves the amount of, spendable income in the economy, which in turn improves the economy and improves expectations that things are going to do well.
d.bau: So, there's this virtuous cycle that, as long as everyone is believing everything is good, everything will stay good, but uncertainty kind of breaks that cycle.
d.bau: So, uncertainty, like tariffs, Brexit, Trade policy disputes, all this stuff,
d.bau: And break up the economy, as you can see in these headlines here. And in the market today, it's really interesting.
d.bau: Yeah,
d.bau: And do economists care about uncertainty? Yes, they write about it in, like, 45% of their papers, so it's most obvious that they do care about uncertainty.
d.bau: But, since we're economists, and we like to measure everything with numbers, we're gonna use a definition of uncertainty that's regarding the moments of a, like, a statistical distribution.
d.bau: Usually, you think about it in, like, the terms of returns to stocks, for example. So, usually we think about uncertainty in the terms of moments, so the first moment that we're going to study is the sentiment. For example, you can see if firms are happy or sad, and then a second moment that you're going to study, it's whether firms are thinking that things are going to be uncertain. This is like a volatility measure.
d.bau: And these examples from earnings calls, for example, we can see changes in sentiments and changes in… oh, this is supposed to say uncertainty.
d.bau: For example, this is Nucor Corporation, and they're… they have this expected decrease because of…
d.bau: their seasonal effects, and this is something that they probably face every year, so it's not really uncertain it's gonna happen. And then this is Apple in their second quarter of last year. They…
d.bau: They said that they didn't really have a lot of tariff impact in the first quarter, in the March quarter, but for June, they really weren't sure on how to… how tariffs were going to impact their outcome. So this is clearly some uncertainty regarding tariffs, and this is just…
d.bau: some feeling of, like, things are not going that well. So these are two different things that we're trying to distinguish.
d.bau: And how do we measure this?
d.bau: So there is a lot of literature trying to measure uncertainty using terms, using text. They use, like, newspapers, Twitter.
d.bau: Like, reports, earnings calls, and we have a slides on why earnings calls, but I'll… I don't have time.
d.bau: And the idea is that you can make a dictionary of words that you consider are related to risk, so certain risk…
d.bau: unsure, things like that, and then you can measure how many words, regarding uncertainty show up in a sentence. So that's usually how people have measured uncertainty so far.
d.bau: And there's this, Brexit paper, which is really good for us, because they…
d.bau: they separate the idea of the Brexit sentiment and the Brexit uncertainty, and they really use that to measure uncertainty in, like, all of Europe, basically.
d.bau: And economists have really quickly taken on LLMs to do different things.
d.bau: And, there's this big paper now that is measuring geopolitical risk.
d.bau: And in this, they used… use the LM, they provide all the entire earnings call, which are very long, and they provide definitions and concepts about, like, this is what you consider to be
d.bau: a tariff. This is a tariff, this is a boycott, this is a sanction, and then you identify where those things show up in the earnings calls, and firms are talking about it. That's the idea.
d.bau: And they just, like, measure what your political risk is.
d.bau: And there's this other paper that uses a specific definitions of
d.bau: Uncertainty, and then something that we do in economics is that we need numbers.
d.bau: So what they use is, like, they fit in this idea, what… and they ask.
d.bau: is this economic policy uncertainty? And then they ask GPT or, like, any other LLM to provide a measure. So from 0 to 1, how much uncertain is this? Like, that's sort of the idea.
d.bau: Behind this paper.
d.bau: So, to summarize a little bit about LLM's work in economics, people usually have this concept that they want to define. So, think about it like industrial policy uncertainty.
d.bau: The meaning of life, someone's telling the meaning of life. I don't know, that's econ, but that's the paper.
d.bau: And then we want to use the LLM to classify a certain dataset using that definition.
d.bau: sometimes people manually have a golden dataset, but recently people have just been, like, using, like, bigger LLMs to annotate one dataset, and then expanding.
d.bau: And there is also some cases in which people just ask, is this uncertainty without providing a definition?
d.bau: And then…
d.bau: Optionally, we want to create measures from, like, 0 to 1, or, like, from 0 to 100, of how much uncertainty, or whatever that is.
d.bau: And then we want to use that to predict. So, for example, to predict returns to stocks and stuff like that.
d.bau: And there is some literature on that.
d.bau: You wanna talk about this one?
d.bau: Terrific.
d.bau: In terms of, like, LA…
d.bau: of the use of LLMs in economic, where they kind of examine the internals. There's, like, this one paper where they, study how LLM process financial information by scanning internal states, so they basically use, SAEs to, like.
d.bau: Decompose, like, the latent activations, and then, they sort of extract this one particular feature that represents risk.
d.bau: And, so they also tried to, like, steer this risk feature such that, like, it sort of, like, positively,
d.bau: it correlates to… with the model's output, of, like, risk tolerance. So, for example, they might prompt the model, help me, like.
d.bau: allocated asset between S&P 500 and, some more risky, bonds, but when we steer this, like, risk feature, the models tend to be, like, more and more or less risky.
d.bau: So,
d.bau: I guess, building upon all of those, we proposed three research questions. They sort of, like, build on top of each other. The first one, we do want to understand, like.
d.bau: how LLMs interpret uncertainty internally. So, because a lot of previous methods, in economics use, like, bag-of-words, those traditional techniques, we sort of want to know, like.
d.bau: if they… if LLMs go beyond those simple techniques. So for example, those techniques might use, like, direct mentions of uncertainty, then
d.bau: can we, like, sort of, for example, examine if LLMs also activate on those non-direct mentions, but those that imply uncertainty? For example.
d.bau: the investigation on the head of Federal Reserve, which we know that it seems uncertain, but, like, to our lamps,
d.bau: Well, that words definitely won't work on that instance, but, like, dual lands work on that instance? So, some techniques we are thinking of are… include, like, activation catching or SEEs, for example, you could, extract an SE feature and examine, whether it activates on
d.bau: Both direct and indirect mentions of uncertainty.
d.bau: Going beyond that, we mentioned how, people, distinguish between the first and second moment. So, we want to know if all labs also learned this kind of, distinction. If, like.
d.bau: So previously, the bag-of-words model, they, extract mentions of
d.bau: positive sentiments to represent the first moment. They extract the mentions of risk-related synonyms to represent the second moment, but we want to see if, LLMs
d.bau: Internally represent, or, like, distinguish those two concepts.
d.bau: And lastly.
d.bau: Previous studies, just like we talked about, have, used LLMs to derive, the economic policy indices, so, like, oh, yeah, economic policy uncertainty. So, as a measure of,
d.bau: how uncertain this sort of text is, but can we, like, develop reliable measures of this uncertainty from its internal representation? And, for example, those studies have shown that these indices correlate with
d.bau: Some interesting economic activities, like,
d.bau: on how people invest, etc. So, can we, like.
d.bau: For example, directly prompt the model to predict, the economic activities, and then examine its internals, and extract that uncertainty,
d.bau: measure, so that… and to see, like, if those uncertainty measures correlate with the economic, or, like, financial market activities. And also, we could even…
d.bau: Manipulate or, like, steer the… those representations to see if those uncertainty measures causally affect the,
d.bau: Predictions on economic.
Natalie Shapira: Hey, this is Natalia Shapira. Can I interrupt with a question?
d.bau: Yeah, I would've done.
d.bau: I don't know where to go.
Natalie Shapira: Okay, can you give concrete examples of, uncertainty in text?
Natalie Shapira: As… as opposed to texts that are… Clear, without uncertainty.
d.bau: Can you go to the second or third slide? The one that has accepted one?
Natalie Shapira: Yes, sorry, I was 5 minutes late, so I might be…
d.bau: It's okay, yeah, I don't know where to look, but we have this example, and this should say uncertainty in the… in the second example.
Natalie Shapira: Okay, thanks.
d.bau: Where's the camera?
d.bau: My camera's right here, so she can see you guys.
Natalie Shapira: panic steaks, I can see.
d.bau: more questions?
d.bau: I just have a couple days, a letter of paper…
d.bau: Let's try to understand 15 neurons.
d.bau: I think it was last year. Uncertain 18 neurons.
d.bau: In what? In what? Oh, what? Yeah, it wasn't really specific to economics per se, in any sense.
d.bau: They were just trying to understand how does certain things replicate inside the world.
d.bau: find the relevant pieces. Can you share?
d.bau: So, related to that, I would be, you know, very interested, because the topic of setting…
d.bau: an AI's uncertainty about its own internal reasoning is sort of a very machine learning concept. A lot of machine learning people are interested in how uncertain AIs are about their own internal reasoning. Probably the paper that Nikhil is going to share with you is about that. But you're studying how
d.bau: It understands uncertainty in, like, a third person, Piece of text, and…
d.bau: I'm kind of… you might consider.
d.bau: Asking the question whether those are the same thing.
d.bau: Right? Whether its own internal… if it knows when somebody else is uncertain, it's sort of the same thing as when it's uncertain itself, or if those are completely different.
d.bau: Thoughts.
d.bau: Well, something that we thought was, if, like, internally, what does it mean for it to understand uncertainty? If we don't feel the definition, and we ask it, is this firm uncertain about the future of trade policy, what does it mean?
d.bau: Yeah. And how does it change when we provide a different definition? If it changed at all, because it kind of did not change.
d.bau: So something we thought, maybe related to that. Yeah, it's interesting.
d.bau: You might be able to… it might be interesting to see if…
d.bau: You can parse apart different types of uncertainty that might have different representation.
d.bau: You know, another, another question is…
d.bau: It might know that the speaker on the earnings call feels a certain level of uncertainty, but it might have its own perception of uncertainty, which is different from what they say.
d.bau: You know, doesn't that happen when somebody on an earnings call says, we're absolutely certain?
d.bau: But then there's some other cues.
d.bau: That you might, as a listener, pick up and say, you know what?
d.bau: My pick is very certain.
d.bau: And, and so it'd be interesting also, you know, so… so these are… I think these are beyond your first-order things, maybe? The thing to look at first, right? But I think these are…
d.bau: Reasonable things to keep at that point.
d.bau: Like, I remember there's some… I don't remember what government agency did it, but they tried to map, like, when people say probably or certainly, like, what
d.bau: actual…
d.bau: likelihoods do people mean when I say words like that, and seeing, like, how calibrated is the LLMs? Like, you could try to map it to… is it encoding, like, what humans tend to… the distribution of
d.bau: meetings that people… like, when people say it's, like, certainly gonna happen, some people mean, like, 99.999% chance, but other people mean…
d.bau: 80% chance, or something, so… That could be interesting if…
d.bau: Okay. One more question. So, I'm thinking down the finer thing. So, how peaceful is this? So, do you have data sets and so on that, that, you know, you use to power this?
d.bau: My boss wrote the… so my PI is the guy from the, the other one, the Brexit paper, so he wrote, like, a bunch of papers on using earnings calls and using these dictionary methods, so I do have the earnings calls, and the measures that he builds, we can use.
d.bau: Oh, that's great. Yeah, because it's, like, it's a very simple formula, it's like an index. Yeah, very… that's fantastic.
d.bau: Nice baseline. Yeah, I thought we chose baseline.
d.bau: Okay, Team S. Team S.
d.bau: Gosh.
d.bau: Oh, thanks.
d.bau: Yeah, somebody needs HTML and plugging into that thing.
d.bau: Oh, that's weird.
d.bau: You guys have reached your mic?
d.bau: Oh, God.
d.bau: Yes, I mean, that's… okay, great. There's not sharing here.
d.bau: Oh, there's another chair. Send me, hey, who's next? Olivia, Arnold.
d.bau: Oh, yeah, yeah, oh, here is.
d.bau: Thank you for the coffee.
d.bau: Definitely.
d.bau: You ain't a fool human.
d.bau: Is it invisible? Yeah, yeah. Okay, cool.
d.bau: Elevator pitch. I don't know, too many people for an elevator. Okay.
d.bau: So, hello everyone, we're group ads, and our topic is, spatial and perceptual reasoning. My name is Yu Qi, and I'm currently a CS Master's student.
d.bau: And, this is my last semester here. How you have, PhD in civil engineering? It's, oh, I'm a master's here, too.
d.bau: Okay, so, when we talk about spatial reasoning, we always think about, object-level relationship. So, for example, in the first image, we may ask, where is the pizza? Is it, on the adage of the dining table?
d.bau: And for this kind of question, we always talk about the, objective relationship in the image.
d.bau: However, there's another kind of question that we may ask in our real life, is, for example, in this image, is it a pla… is this place look safe?
d.bau: This question, is the same level perception that,
d.bau: And we still don't know that the vision language model understand
d.bau: the concept of save from the image and how it understands it. So, our research is mainly motivated by this kind of question.
d.bau: And, we, we want to explore, does vision language model understand the concept of SAFE, and how, it makes the judgment based on the image we give.
d.bau: And, there are still some, potential sub-questions we want to explore. which region or which queue in the image is more important for the visual language model to make the judgment.
d.bau: And, how the judgments emerge and, differentiate in different layers, and how the reasoning process aligns with humans' reasoning process.
d.bau: Okay, so our motivation is, we want to understand how the VOM works. We want to, like, look into how the visual concepts are encoded in the VOM.
d.bau: And, the second motivation would be, like, what the model's output for scene-level perception, is similar to humans' perception, or we want to, like, make it better to match more with humans' perception.
d.bau: And, the third reason would be, like, we can't actually use that for embodied, household agents.
d.bau: Or, like, auto, novice driving, and things like that in the real world. And, this is also beyond the object-centric realms where we currently have.
d.bau: Yeah, there's some, well-established, literature review on the, how, previous work
d.bau: is that, there's some existing work on object-level, special relation, which is, described before, such as VLASR, and it, evaluates the pairwise objective relationship, and some other, VLMs, and,
d.bau: There are previous studies revealed that,
d.bau: there's some monosomatic features in the language models, and we, have some object-level, visual interpretation from, Contrastiver, like, clip, and there are some, GAPIs that, how does, VLM
d.bau: Encode holistic on scene-level concepts, and,
d.bau: That's the, integration across the entire environments, and, does it, show the mechanistic coordination beyond the object-level regions.
d.bau: Yeah, so, and, there's some, scene-level spatial condition theory, that is outside of, computer science, such as, where, Kaplan, defined a peripher risk matrix about the urban environment, in four dimensions, such as coherence, complexity.
d.bau: legibility, mysterious, and, some other,
d.bau: others like Lynch, they described, image theory, how to,
d.bau: the imaginability of the scene in five different special elements. And also, in the urban…
d.bau: planning. There's a, yuri about, how does the people, feel about the environment.
d.bau: Okay, so move on to this. Where do y'all…
d.bau: have the MIT placepass, 12, which have the 6 special dimensions for, human data on each scene.
d.bau: And they, for each picture, they have, the score…
d.bau: Wealthy, wealth, foreigners, and, the presidents.
d.bau: And, move on to the next. This is, oh…
d.bau: Yes, we have seascapes and, monthly wisons, and these two, datasets are, for segmentations, and we,
d.bau: We also want to use these for simulations, to, see, like, whether we remove some parts of the picture, would that affect, the model somewhat?
d.bau: And that would be you all.
d.bau: for our… Any, questions and queries?
d.bau: Yes, how do you refine safety of workers? Like,
d.bau: It is interesting, so we want to, like, see how actually VLMs understand it, so before we actually define it, we, want to first look into the VLMs first.
d.bau: In some data sets, they have a human-rated score of how safe this place is promoted.
d.bau: how did they, like, what are the, like, factors within safety? Like, like, I could imagine that it being, like.
d.bau: A place, and then… is it, like, crime statistics? Or is it, like, a classroom that's, like, a safe space?
d.bau: Okay. That's just, baseline image. I'm gonna show you.
d.bau: Any other groups.
d.bau: Yes. Yeah, I think this is super fascinating. There's, like, a long-standing
d.bau: debate within the world of policing around this thing called broken windows theory. Oh, right. …in the 80s, and was… and so, like, when you're talking about this, I'm thinking very much about that, and, like, the basic idea there was that, like, things like broken windows signaled the place is unsafe, people don't care about it, if you fix the windows, you'll fix the serious stuff.
d.bau: There's been a lot of research since then that suggests that that kind of perception of safety is completely divorced from the actual safety of a place.
d.bau: And that people's perceptions of safety tend to map on more things, like, racial composition of the neighborhood.
d.bau: How good are the schools? How nice are the cars? And these sorts of things, that, end up being pretty orthogonal to the question of, kind of.
d.bau: Safety in a physical sense.
d.bau: So I just wanted to, like, flag that for you all. I think it'd be super interesting if you actually found that the LLM was encoding that perception of human safety, but that also might have some within the kind of, like, finer framework implications on the ethics sides of things.
d.bau: If you have any… you know, articles or something about Ferg Windows theory or any of the things.
d.bau: throw it for us to the team, or to the Discord, that'd be really helpful, I think. Yeah.
d.bau: Go ahead. Kind of going off of what you just said.
d.bau: I was thinking it's more from, like, also a perspective of who the individual is, because one individual might find a scene safe, while another might not, depending on their background and experiences, right? And so…
d.bau: I do wonder, like, how can I run a vibes towards one serving type of food?
d.bau: For example, on how it chooses to portray something, or label something someday.
d.bau: Right. Yeah. So you guys are getting a sense for the I and finer.
d.bau: Right? Which is, like, what people would be interested in.
d.bau: Which is, you know, so I would, like, sort of take a note of this, right? I think a lot of traditional computer vision research in the last few years would have taken something like this and said.
d.bau: Is my model capable?
d.bau: Of matching my labels in my data set.
d.bau: Of whether something's safe or not.
d.bau: But I think that we're beyond that. I think that models are capable of matching labels in the data set, and now the question is.
d.bau: where does this inference come from? How accurate is it? How many different personas can you take on, and where did it learn it from? So, you have this interesting VLM,
d.bau: It's… it learned from vision.
d.bau: It learned from language.
d.bau: I don't know if it's gonna be possible, but it seems like an interesting question.
d.bau: If it has some idea that this is safe or unsafe.
d.bau: Did it learn that from reading some text somewhere?
d.bau: Or did it learn it from, you know, its visual…
d.bau: you know, pathway. Is that, is that, like, a piece of textual knowledge?
d.bau: If it's… if it's textual.
d.bau: Does it have different concepts of safety, like, you know, people are talking about here?
d.bau: You know, maybe stored in different ways, or if you give it a different persona.
d.bau: you know, it might… it might perceive something differently, and can you find how that's represented or processed and said, I think…
d.bau: These are just sort of the… B.
d.bau: The questions that would differentiate you from.
d.bau: If you had done this project 5 years ago.
d.bau: That makes sense.
d.bau: Any other suggestions, any questions?
d.bau: I need to add on that, because there's an anthropic paper that was recently published, which was studying personas and language models, and so it sounded what you were saying, David, was quite similar to what they were doing on a strictly language basis, and more focused on the actual, like, persona rather than language.
d.bau: Quality of the scene, so… Yeah, I could maybe share that paper.
d.bau: the Discord for context, if I talked to me.
d.bau: Doesn't seem like a place to start from with that specific topic or something.
d.bau: It's often easier to do research where you compare two things.
d.bau: Rather than just look at one thing. So if you look at one thing.
d.bau: It's like, oh, does my… so a bunch of your research questions were…
d.bau: phrased as, does my model do this? Can I do this? Whatever, right?
d.bau: But it can be hard to analyze, because if you sort of publish a thing saying, oh yeah, we think it did it, it could do it 27%, that means it did it. Some other people will look at it and say, well, 27% means it can't do it.
d.bau: So, you know, what's your conclusion? But if you compare two things, it's a lot easier to say, it does more of this in this situation, less of this in this other situation. If we… if we train it as a reasoning model, it does more of this. If we don't train it as a reasoning model, it does more of this. If we tell it, it has this persona.
d.bau: that does more of this if we tell them how to exercise. So it, like, that could be…
d.bau: Easier to… To come up with an interesting conclusion.
d.bau: I mean, one point on that, so I think somebody mentioned from behind that
d.bau: Once you have different perception of safety, it might be the case that models also have different
d.bau: perceptions of reality. I mean, if you use your dataset on different types of models, let's say models from American companies versus Chinese companies, they might have different behavior.
d.bau: On that dataset. Maybe you can use that
d.bau: I mean, two… these two models for comparison and 19 Australian.
d.bau: are some technical suggestions and advice. So, with DLMs.
d.bau: At least last… till last year, before QAM VLMs came out.
d.bau: the long-running, consensus is that when it comes to preference, VLMs heavily
d.bau: prefer their language side than the visual side of things. For instance, even if you show the picture of
d.bau: Northeastern University, but you caption it by, say.
d.bau: It has broken windows, that's it.
d.bau: Most of the time, the language, the VLM would say, yeah, it's unsafe.
d.bau: Right, so I think…
d.bau: This was not done on the latest models, and it's seen that these new models are much more capable.
d.bau: I think it would be interesting if you can somehow…
d.bau: Try to figure out if it comes from the tech side of things or not.
d.bau: the image side of things. And there are, like, some good baselines you can think of, right? One is you don't capture the image, and just ask the question, is it safe?
d.bau: And then you don't provide an image, but just caption it somehow, like, provide a black image and caption it.
d.bau: You see how we've become…
d.bau: For pop-ups. It would be a good baseline, I think, for your kidney, where this is.
d.bau: And second one is, again, when it comes to preference.
d.bau: people found it easier if you gave a VLM two images and asked them to compare them, so that it's easier to look at the attention patterns and see what it is they're looking at when they're comparing two images.
d.bau: I think it would also be an interesting setup, if you…
d.bau: If you end up not finding the single image set up.
d.bau: Pretty throat.
d.bau: Thank you, please.
d.bau: You might want to ask Rohit for more advice as you go through the project. He's sort of our most vision expert person in the lab.
d.bau: Yeah, upper 20s.
d.bau: Oh, any other questions, or…
d.bau: Suggestions for the project?
d.bau: If not, then, thank you very much. Fantastic.
d.bau: All right, guys, so… alright, how we doing on time?
d.bau: Are we… are we half an hour in now? What are we in? 20 minutes? Oh, not too bad. We're going through so fast.
d.bau: Alright, lots to do. Good!
d.bau: Okay.
d.bau: the research details. Nice, nice work. So, I think that…
d.bau: So, remember, on the recent project, you know.
d.bau: I think that we started off with the eye. Just, like, polling our classmates, giving a pitch, trying to get a sense
d.bau: For what?
d.bau: Other researchers would be interested in it?
d.bau: We have no idea if any of these ideas are feasible. They're all… I'll tell you, like, every single one of the proposals right now is too broad. There's no way you can answer all the questions that any of you
d.bau: then you can end up with something much more narrow, so…
d.bau: So the, the, you know, the next step is for you to start getting an intuition about what's feasible to F in Pine R.
d.bau: So, the assignment for Thursday is…
d.bau: to try using some technical methods, try using… I mean, so just… we have these exploratory methods that we're gonna look at. We have LogitLens, we've got… you know, Aruna will show you Neuronpedia.
d.bau: You can… if you know other methods or other things, you can talk about other… other things, but you've got at least two exploratory research methods for
d.bau: You know, putting some text or putting some input into a model and trying to see what insights or what intuition you can kind of get out.
d.bau: None of them are rock, you know, Rock solid.
d.bau: Complete experimental methods, but they're… they're experimental things, you know, exploratory things you can use to get a little intuition.
d.bau: And so on Thursday.
d.bau: Your team's job will be to put together your 5 or 10 minutes of, experimental results from your explorations to try to suggest what
d.bau: What part of your research question you think is the feasible one?
d.bau: Or the feasible set that you're gonna pursue. Or… or not. Or if you have a negative result, you say, well, we looked at.
d.bau: Everything's a mess, and just share that with us as well.
d.bau: Does that make sense?
d.bau: Okay, so that's for Thursday.
d.bau: you guys, so probably some of you have already been doing that, because you know it's coming, but I just want to remind you.
d.bau: Alright, so, the lecture part.
d.bau: So, there were a few readings, you know, I don't know if you all read all three of the things. You know, some people read some of them, at least, enough to, you know, submit a question, that's nice. So we're gonna try and do the same thing, and I have…
d.bau: I have, so… The challenge here, so what this meeting was about this week,
d.bau: was about two things. One is, I… I want to catch you… catch everybody up with this notion
d.bau: That concepts or vectors, that we are trying to understand the power of the representations in this neural vector space.
d.bau: Which is how a certain school of… a certain community of neural network people have been thinking about, concept representations.
d.bau: You know, certainly for the last few years, but actually for decades. But that comes in
d.bau: a larger context, a larger historical context, that I wanted you to get a little flavor of.
d.bau: And I, and I, but I… But
d.bau: Covering the larger historical context is sort of a full semester course.
d.bau: So I didn't… Give you guys, like, 10 papers to read.
d.bau: But I'm gonna try to give you a little sense for it now. So, there's a central question. What is that concept? And actually.
d.bau: Who's Isaac?
d.bau: Isaac.
d.bau: You, you, you asked this in your question, like, right? What was your question? Yeah. You can kick us off. Sure. So, all of the papers are talking about concepts. Yeah. Distinguishing concepts from language. Yeah. But they're not…
d.bau: necessarily formally defining… Without defining it is. And so, it seemed like each of the three readings may have a slightly different idea of what a concept is. People just flinging around this idea, this work, concept. Seems pretty loaded.
d.bau: Yeah, yeah. And what do you… what do you think that is going on there? Why… why do you think people are not sitting down there? It should be a sloppy here.
d.bau: I'm not totally sure. Yeah, yeah, yeah. I think partially it's they're being sloppy. I think partially that they also know that there is this historical baggage around it.
d.bau: And it's like, well, this is not the place to get into it. You know, choose your own definition, right? Because it can be quite involved. And so,
d.bau: And so, okay, so Christopher asks a question that I like. So, Christopher, go ahead, yeah, what was your question? Well, so my question here is, so…
d.bau: what I felt dark in the paper was, like, you know, here's a lot of theories and a lot of grounds that's like, you know, here's circumstantial reasons for why this is plausible, why it's inconvenient, why it's beneficial to adopt this representation, but they don't offer a falsifiable statement saying, well, if we're wrong, this is how we'll know. Like, there's something missing in this Piantadosi reading that I gave you.
d.bau: And so… and so what's missing is, he's not sticking up for his intellectual opponents, right? He's like, this is my view.
d.bau: And, you know, I think I'm right.
d.bau: And when you read it, because I just gave you this paper in isolation, you're like, okay, well, that seems fine.
d.bau: But what would it mean if he was wrong? Like, how's that even falsely? So let me give you that context here. So let's do that. So this is him.
d.bau: So here, here's, here's, like, the world…
d.bau: that he lives in is… so the classical name for this is connectionism. Have you… who's heard of the term connectionism? Okay, so connectionism is a term that the neural networks people made up for themselves, that's what they called themselves in the 80s.
d.bau: They're connectionists.
d.bau: And, and so there's, there's this name… Hinton won the Nobel Prize, he was a student of Romahart, and, you know, there's McLeeland, you know, other, you know, sort of cognitive neuroscientists.
d.bau: There's a guy, Smolensky, who I recommend reading. I think that… did you… did you say that you met Smolensky? Is there still active research today at MSR? That really interesting work by Smolensky. You read the Piantadosi paper, but basically, these are folks
d.bau: Who think that concepts are vectors.
d.bau: that basically what a concept is, is, of course, there's different perspectives on what a concept is. I'll come to the next slide, but then the most useful way of thinking about a concept is how it's represented in a neural network as, like, a vector representation, and they ask.
d.bau: You know, how do you understand how concepts relate to each other in terms of the geometry of vectors, the linear algebra of vectors, other things that you do with vectors, distances, angles.
d.bau: Right.
d.bau: And, okay, but they're not alone.
d.bau: So you read the Piantadosi paper, and he says.
d.bau: Yeah, concepts are vectors. So, there it is. So…
d.bau: There's a couple other intellectual schools of thought here, like the right way of thinking about computational… the computation of concepts.
d.bau: And so, so there's the symbolic AI people, the symbolic
d.bau: It's, you know, the symbolists.
d.bau: Right, so who's heard of Noam Chomsky?
d.bau: Yeah.
d.bau: is the man, right? And so, and then, you know, so there's Minsky and Chomsky, and there's a bunch of other people. Fodor is actually weighted into this debate and has some really interesting readings that I'll put in the supplemental readings that
d.bau: That, that are there. And so… so…
d.bau: They… what they would say a concept is.
d.bau: is more of a classical view. Like, a concept is, like.
d.bau: something that you could write a sentence about, like a dictionary definition. A concept is like a cassette of rules. It's a cassette of…
d.bau: You know, a concept comes from, like, what you define it to be.
d.bau: Like, if you asked a lawyer, what is the concept? They might, like, say, okay, well, it's this, but there's some caveats, and the more you write, the more you can pin down the concept, and…
d.bau: you know, a lawyer might be able to give you, like, a long paragraph about a concept, and really, really pin it down, and they're like, yeah, that's… that's… that's the exercise we go through when we're really trying to understand a concept, is we… we are coming up… and so… and if you want to understand it computationally, they… they think a lot about language, that's why the Chomsky thing is so important, like, how do you express…
d.bau: Like, John Loves Mary. A nice concept.
d.bau: Right?
d.bau: And you can express this concept as a relationship between a couple things as a sentence, and you could… you could add more adjectives or more sentences to that to refine the concept. So that's… that's sort of the idea, and it's very, very compositional. This… this… the way that symbolic
d.bau: people think about concepts, and it also relates to the way that we program all of our computers today. Whenever you write an if statement, and you have an if this and that, or that, and you have all these conjunctions and your predicates, and you string together all this logic, it's all from the symbolic tradition. Like, all of the…
d.bau: by hand, You know, computing things that we do.
d.bau: Our models of symbolic concepts.
d.bau: Right? Make sense?
d.bau: And so… and then there's a third school that's a little bit more modern, and so it comes from…
d.bau: 90s, people started to say, these guys are too abstract, right? Like, we're just, like, what makes us…
d.bau: you know, focus on these computational things. We're just biological creatures. We live in the world, we perceive things.
d.bau: you know, the things that, like, all of the things that we think and perceive and understand as concepts are just side effects of, like, our life in the world, and the way that, you know, when I touch a thing, and I feel it, and see it, and I interact with it, this eraser thing, then, then I…
d.bau: My concept of the thing derives from this
d.bau: invited experiment… experience that I have.
d.bau: And so there's this embodied…
d.bau: school of thought that says, that has this other perspective on what a concept is, yes. I feel a little bit naive, maybe you've also experienced before, but…
d.bau: I don't see why connectionism and the grounded view are… Are different. Yeah, couldn't go together. Yeah, they, they, they, right, they kind of go together a little bit. They all have…
d.bau: Relationships to each other.
d.bau: Like, nowadays, You know, we have these huge neural networks.
d.bau: And what are the… what are our favorite neural networks? They're processing language!
d.bau: Right? You know? They know all that when people avoided language for a long time.
d.bau: But… but now, like, language is the way to do them. So… so there's a lot of…
d.bau: I think there's a lot that has been gained over the decades from these three
d.bau: Warring factions, battling against each other intellectually, saying, you're missing this, and you can't do this, and your model of thinking doesn't work this way, and they all try to patch up.
d.bau: what they're missing, and I think that it's actually advanced the field.
d.bau: And so a lot of the stuff that we do, large-scale AI today.
d.bau: Is actually motivated by these old debates.
d.bau: you know, like, the huge investment in language, I think.
d.bau: Is motivated by a lot of intuition.
d.bau: But yeah, there's a lot of stuff.
d.bau: In language, you can't get out of a regular neural network unless you take it seriously. And also, the huge investment in multimodal
d.bau: AI.
d.bau: With vision and other modes, comes from this debate with embodied folks.
d.bau: That, where they say, there's no way you can understand the world unless, if you've never seen it, you don't even have eyes, right?
d.bau: Does that make sense?
d.bau: And so… so here's, like, the debate semi-diagram. So this is… I think this is… I really like this slide. This is an AI-assisted slide, but…
d.bau: you know, curated by hand, so you go into the PowerPoint, you can see what I curated.
d.bau: And, And so, between each of these schools, you've got
d.bau: Different things. Okay, so yeah, so actually, so Jasmine.
d.bau: What was your question? What was Jasmine? Oh, I forgot. What was your… oh, yeah, it's something about versus symbols, or something like that.
d.bau: What was your question? You have too many questions!
d.bau: Sorry. Yeah, it's alright.
d.bau: Ask any question you have.
d.bau: Your turn.
d.bau: You don't remember your question? Okay, it's alright. I probably have a record if I didn't bring it.
d.bau: And so… So… right.
d.bau: And so… These are the debates between the neural network connectionist people And the symbolic people.
d.bau: And so these are, like, I would say two primary debates.
d.bau: So… This… the neural network people, they go to the symbolic people, And they say, yeah.
d.bau: I was just gonna ask, like, given that now, you know, we have these language models and everything's a bit… are the connectionists willing?
d.bau: What did it be?
d.bau: I think that's what Piantadosi is saying. He's like… he's like, victory?
d.bau: Let's, let's publish a paper in.
d.bau: P&AS, and… Hello, everybody.
d.bau: Forget all this stuff.
d.bau: Concepts or vectors? Probably. This is probably, right? But, like…
d.bau: You know, we're done here, is basically what he's saying.
d.bau: And that's, that's what his paper is trying to say.
d.bau: But… but, like, I'm trying to put it in a little bit broader context so that you can understand, because there's some… some of this stuff will actually impact the way you might decide to choose some of your research topics. So let me give you, this debate, and then you'll see.
d.bau: So… so on one side, it's… we know, we're familiar with the debate. You say, hey, you symbolic people.
d.bau: You can't even make complex systems that…
d.bau: deal with ambiguity and gradations of truth and things like that, and you can't even learn, right? I mean, every… everything you put into a symbolic system, you have to, like, program by hand and, you know, make long lists and big dictionaries and databases and stuff like that, and while you look at the whole world of software today, you actually can build a lot of
d.bau: complexity with this symbolic approach, but there's a lot of stuff you can't do. All the cool new neural network stuff.
d.bau: you can't do with symbolic systems. It has no answer to learning, it has no answer to this sort of approximate reasoning.
d.bau: Pattern matching doesn't do very well.
d.bau: And so… so that's… we're familiar with that.
d.bau: That's what's driving the stock market higher. Maybe not today.
d.bau: But yeah, that's what's going on.
d.bau: And so…
d.bau: What about the other side? So, what do… the symbolic people are still around, you can see them on Twitter. What's the name of,
d.bau: Gary Marcus! Right! Gary Marcus. He's always… he's always the reliable defendant.
d.bau: for the symbolic view, if you go follow Gary Marks on Twitter. And this is, like, so the strongest arguments actually came from Fodor, and I put a link to Fodor's old paper, where he introduced this argument back in the 1980s.
d.bau: As… as… I just added it just this morning as extra optional reading to Week 2. And,
d.bau: But… If you think about how you represent a concept, as a vector, then you…
d.bau: And then… but then you think about what kind of concepts
d.bau: Humans really need to deal with.
d.bau: You end up with this puzzle about compositionality.
d.bau: and binding.
d.bau: Right? Like, oh, John loves Mary.
d.bau: Right? So, okay, we might have a factor for John.
d.bau: We might have a vector for Mary.
d.bau: So what's the vector for, like, John loves Mary?
d.bau: It is a vector for loves, and you do, like, add them together? Maybe you add them together, or something like that.
d.bau: Well, if you add them together, then how do you… what… is that the same as Mary Loves John?
d.bau: I don't know, I think it might be different. I think John might love Mary without Mary loving John. Maybe Mary really doesn't like John, right? You know, and like, how do you represent that?
d.bau: Does that make sense?
d.bau: And so… so this compositionality, like, how did… you just add the vectors together? Like, how the heck does this work?
d.bau: is a puzzle.
d.bau: If, if you believe the concepts.
d.bau: directives.
d.bau: Right.
d.bau: And so…
d.bau: So that's what… so the… so the symbolic people wrote this long paper in, you know, 1988 or something.
d.bau: How you're never gonna do it, you're never gonna figure out composition in the spectrum view of the world.
d.bau: Mic drop.
d.bau: Don't do neural networks.
d.bau: Right? Stick with our expert systems and databases.
d.bau: Right? And, but then, in the years since, the neural network people have been trying to answer that.
d.bau: And…
d.bau: you know, some of our research, like the binding research done by any field, it starts to ask, you know, how are these neural labs actually solving this problem? But what happened is, you know, even before Pantadoski, before the LLM era.
d.bau: People were asking this question theoretically.
d.bau: They said, is it even possible for vector space?
d.bau: to represent… binding and relationships with a certain amount of power, and I'll talk about that in a minute.
d.bau: Right, does that make sense? And then… and then… and so they have these, like, abstract theoretical arguments of, like, how you could do it, and how the math would look, and how the neurons might hook things together.
d.bau: And now, like, our empirical world, since 2017, has caught up and exceeded what the theoreticians have been reasoning about.
d.bau: Clearly our models can do binding, at least to some level. Clearly our models can do…
d.bau: You know, a bunch of these compositional things.
d.bau: Yet we actually don't know if the theoreticians were right. If all these, like, mathematical structures that they proposed, like, if you were a neural network, you could do it this way.
d.bau: We don't know if they're actually… Doing these things.
d.bau: Or if they're doing something different.
d.bau: And so… so it's… it's worth, like.
d.bau: You know, if you're kind of curious about rooting your research and some of the old stuff, it's worth reading some of the old things and seeing.
d.bau: when people are thinking. Okay, so, the debate between the…
d.bau: So, what's the difference between these guys? Let me ask that. So, of… Right.
d.bau: You know, between the neural network people and the body people, seems like the same.
d.bau: Right?
d.bau: So, it's a little bit of a funny intellectual idea.
d.bau: But there's this question… That psychologists even ask.
d.bau: They said, what if you had been born But…
d.bau: And maybe even born without a sense of smell or touch or anything.
d.bau: All you could do is process language.
d.bau: And you only knew what people told you about her.
d.bau: Ben, would you be able to understand the world.
d.bau: Like, a fully, like, you know, if you couldn't move around the world or anything like that, you just heard stories about it, did you actually understand what it meant?
d.bau: To have a sunny day.
d.bau: Right, to walk to class, You know, to have to go get a Dunkin' Donuts.
d.bau: Whatever, right? Like, could you understand these things?
d.bau: Or…
d.bau: Or is your understanding, like, meaningless in some algebraic ways? These are just symbols that you're just manipulating, and it's inevitably gonna be missing something. And so, and so the neural network people say, yes, all of knowledge is all relationships. It's all relational.
d.bau: Like, grounding is not so important.
d.bau: Right? Because it's all about how all the different concepts are related to each other. So there's this wonderful
d.bau: Paper by…
d.bau: But what's fulfilled by soil is left, the platonic hypothesis would be the first author on the… HQH on.
d.bau: Right, Jacob Hutt and Philip Isolated just came out a year ago. They're firmly in this camp. They said, hey, if you train a model just on vision.
d.bau: And you're training a model just on language.
d.bau: They, they, they, like, form these internal graphs of, like, relationships between all their internal concepts, and if you look at the graphs.
d.bau: If you look at this in the kernel, right, it's all the dot product related chips, right?
d.bau: They're the same!
d.bau: Even though the vision one has never seen language and the language one has never seen vision, like, they have the same model of the world if you train them big enough, anyway.
d.bau: And so… so, like, the point here is maybe you don't need ground aid.
d.bau: Right? You can build a model of the world without ever having seen it.
d.bau: Right, but the embodied people are like, no, you gotta, you gotta see it. There's things you'll always miss.
d.bau: And so, and then, so now the neural network people are like, well, okay, sure, that's fine. But…
d.bau: But, like, we can just solve this by putting a video camera on our LLM, right? And now we can see
d.bau: Attaching it to a robot, now it can touch.
d.bau: you know, giving it some problems to solve, now it can plan, now it can do all the other things that you experience in real life. And so that's sort of the path that we're going down now, is trying to fill in these gaps. But there are some people who argue that, you know, maybe it's not even so essential, like, without, like, you know, it's, like, maybe easier than we thought, because you could learn a lot just from language.
d.bau: Okay, so those… that's that debate. And then there's also this interesting thing here, where people, like, if you say everything's about grounding.
d.bau: You know, it's the same argument here, the symbolic people have no grounding, but then the objection from the symbolic people to the grounding people, like, okay, people… grounding people, you think everything is about what you can see and feel and touch and see, right?
d.bau: What about… the concept… of 7.
d.bau: Where does 7 come from? Have you ever touched 7?
d.bau: And what is that? What about the concept of justice?
d.bau: Right, how does that…
d.bau: How does that play into what you feel and touch and see, right? So the symbolic people are… they love these language
d.bau: Puzzles.
d.bau: Right? And they're like, you're missing this. So that's the three-way debate.
d.bau: Between these communities. Just a little… kind of a… that make sense? And so we're… so we are sitting here. This whole class, you can think of as continuing this tradition, and… and one way of thinking of it is.
d.bau: For 50 years.
d.bau: People have theorized in order to resolve these debates about what can be done here, but now there's this engineering solution of just making these systems, and they can do all these things. They can do them.
d.bau: But we actually don't know whether the way they're solving them matches up with what the theorist Post.
d.bau: And… and there's, like, a 50-year-old…
d.bau: history here that's kind of interesting to retrace is, like, I have to answer. Like, is it working the way we thought, or should we be surprised? Maybe it's, like, it's working, but it's working in a totally different way.
d.bau: And so by peering inside the frames and inside the neural networks, you know, you can resolve some of these questions, if that interests you.
d.bau: Okay.
d.bau: So, okay, so… so for the neural network view, I think there's… so there's,
d.bau: There's a couple other questions, so let me let Hayu and Hai
d.bau: Ask… ask questions as I… and maybe I can use some of the materials here to…
d.bau: How about? So, hi, you…
d.bau: Hi, are you right? Yeah, what was your question? I was reading Paul's paper, and I think, things like that the features and the number of neurons has a superposition.
d.bau: Yeah, I don't understand how to supers position you.
d.bau: Great. So we'll talk about supervision in a minute, but you… so one of your questions is, like, what is even a feature? Yeah. Right? And so, a feature is another rough synonym for concept.
d.bau: Right? It's just like, it's what machine learning people say when they've got a represented concept.
d.bau: And they don't want to wade into the whole debate about what is the concept. So then they say, well, what we have as a feature?
d.bau: It's kind of like that. It's a concept, but why don't we even want to debate about it?
d.bau: And so, so…
d.bau: So there's really 3 perspectives as a neural network person that you have on any feature or concept that you're looking at, right? One is the primary thing we're studying.
d.bau: Which is, oh, there's some neural configuration. You know, you get your neural network in some state, you can put a number on every neuron, right? That's sort of like…
d.bau: That's, like, all the information that's in your neural network, right? Or some subset of the information or something, right? Some sub-vector. Right, so this is, like… so one way of thinking of, like, what is a feature or a concept is some part of this thing, this big vector.
d.bau: You've got.
d.bau: Right?
d.bau: But then there's… there's these other perspectives, right? So, oh yeah, and if you're… if you're a neuroscientist, then take the fMRI. That's…
d.bau: That's… that's what you're saying, right?
d.bau: But… but then, like, Informally, or maybe philosophically.
d.bau: Right? That, you know, the brain, or maybe the neural network, is sort of thinking a thought, is thinking something.
d.bau: Alright, so you might have, like, like, a concept, like, the food that Alice loves.
d.bau: You know? You get your neural network to think about all the food that Alice loves. We're gonna get ready to cook some food or order some food for Alice. Gotta know what food Alice left, right? So, that corresponds to some state the neural network is in.
d.bau: But then it might also correspond to, like, just informally, like, oh yeah, like, all this different food!
d.bau: Alice, I was like, the actual… the actual concept?
d.bau: Right?
d.bau: And then… and then the… and then the third thing it corresponds to, and this is the interesting thing, because, like, this is easy for people to understand. You can say, oh, she loves vegetarian food, except nothing with pineapples, please.
d.bau: Right? And, you know, if you could imagine all the different types of food, Right? But…
d.bau: But it's kind of, like, how do you map between this thing and that thing? How do you, like, do science on that? It's really kind of intractable, it's like…
d.bau: This is like a fuzzy, informal thing.
d.bau: Right? And so… There's this other perspective.
d.bau: that was proposed by, like, David Maher and other people to say, oh, the other way of understanding a concept is as a formal computational role.
d.bau: Right, and so the computational role here might be for, like, the food that Alice loves, It's like…
d.bau: In principle, That concept is a function. It's a function that maps from any food.
d.bau: to how much Alice loves it.
d.bau: That make sense?
d.bau: And, and, and, like, and so there's some abstract… mathematical formalism, Of that concept somewhere.
d.bau: And then… and then you can kind of, once you've written it down as, like, a mathematical object, you can ask, oh.
d.bau: Can that mathematical object… Can that be stuck in the specter's face? Can it live in here somehow?
d.bau: Right? And if we have a space that, oh, the food that Alice loves, the food that Bob loves, the food that Carl loves, all these different foods that people love, right? You've got this whole space of different foods that people love. Can they all be, like, you know, represented in here, and how, and how does it work? Right? Now.
d.bau: now it's sort of more mathematical, and you can prove theorems about it, or you can set up a quantitative experiment or something like this, right? And then you could express your data set this way. So this is… so, like, okay, so yeah, now… now you can ask your question. Oh, I guess I had a different one, like, like, I guess, like, what makes some concepts feel very, like.
d.bau: You're like, food, you're like, you see it, you know it. Like, I was having this conversation with a friend, he was like, if I'm a shareholder, like.
d.bau: at McDonald's, am I complicit in, like, the obesity crisis? And we're, like, arguing over the word complicit, like, that feels like a very leaky abstraction. But it was, like, you said pretty… with great confidence that you were a shareholder, like, that seems to be, like, a pretty firm one, like…
d.bau: Is there a frame here that helps explain that discrepancy?
d.bau: So, like, what's the difference between, like.
d.bau: you know, the real meaning of something, and just, like, saying something in a word. Yeah. You know, I… so my philosophy on this is after, like, appearing inside these neural networks for, you know, a couple years.
d.bau: is that they are all concepts at different levels, right? In other words, you can have purely linguistic concepts, or just… they just sort of say, well, somebody uttered implicit, that's just like the word they used, you know, who knows what that means.
d.bau: But then, like, deeper in the neural network, there may be other more abstract concepts that correspond to certain levels of, like, taking responsibility, or really having responsibility, or perceiving responsibility, or something like that. And then, it's a very interesting question. When you get away from just the literal words.
d.bau: like, whether the LLM or whether the AI is actually trying to parse apart these different action concepts, you might imagine they have to
d.bau: So, here's the basic dilemma.
d.bau: You could… you could write down… so you could write… you could write a book on one of these concepts, like.
d.bau: like, you know, what does real responsibility mean, what does real complicity mean? And stuff like this is what it would entail. And then once you really understand that deeply.
d.bau: You might be able to say, okay, well, there's different levels of taking responsibility, there's different…
d.bau: There's different agents and parties that could be responsible for different objects and events and things and duties.
d.bau: And what, really, what understanding, taking, you know, the levels of taking responsibility is, is you could say that, like, a full accounting for that would be some mathematical object that would say, okay, you know.
d.bau: your belief in taking some responsibility would be some kind of thing, some kind of information structure, right? Now, if you write down the information structure really abstractly like this.
d.bau: pretty quickly, you get to, like, an infinite-dimensional thing, which is, like, too big. It's like, even if you write it down as, like, a mathematical object, right? This is, like, this mathematical object is too big to fit in any computer. Certainly not, like, a regular, like, a neural network.
d.bau: Right.
d.bau: And so… and then the question is, well, like, my neural network doesn't seem to be phased by that. This, like, this object is too big, it's like, it's reasoning about all these things anyway.
d.bau: Right? So basically, one way of looking at this exercise that we're doing
d.bau: Is we're asking, how do you jam this thing?
d.bau: Into that thing.
d.bau: Right? Like, so you got, you got some… abstract…
d.bau: Representation of, like, the concept that you want, Does this thing have it?
d.bau: like, there's a few ways you could jam something in. One way of jamming it in would be, like.
d.bau: round it off. You say, you know what? There's, like, a hundred, a thousand million different ways of taking responsibility for something.
d.bau: I have room for all that. I just, like, have one way.
d.bau: In my model.
d.bau: And that'd be, like, a way of saving bits.
d.bau: Right? Or, there might be some way of decomposing it. So I think that this gap
d.bau: It's the thing that, historically, people were struggling with.
d.bau: And, you know, all this old classical work on connectionism.
d.bau: And I think that SCAP will permanently be with us as we work with LLMs and we understand their capabilities and limitations, and so this is, like, this is part of…
d.bau: You know, what we're… what we're after by keeping this field going.
d.bau: It's understanding this guy.
d.bau: Okay.
d.bau: So, alright.
d.bau: Let's see, who else had a question? There's another Kai.
d.bau: I had a question that I thought was appropriate to talk about at this point.
d.bau: That was Kynron? Yes.
d.bau: I was going on kind of a tangent, because the why concepts are probably vectors paper. Yes. Talked mainly about concepts in, like, word vectors. Yes. And they talked about the relationship between those. Yes. Dog is to dogs is cat is to cat, that kind of thing. Yes. And there's other work showing that models also kind of have those relationships, like…
d.bau: We train a model on summarization and, like, news, and a model on
d.bau: classification in news, then the difference between those two models is
d.bau: like, analogous to that difference in other domains, too. Yes, yes, yes. And so…
d.bau: kind of a broader question that I had was.
d.bau: If we treat models as, like, billion dimensional vectors, are they representing things like, domains or tasks? Like, what level of knowledge can we represent with vectors?
d.bau: Yes, so you're asking a couple meta questions.
d.bau: And… And, so you're sort of… you sort of skipped over to this…
d.bau: thing here. They said, there's other… you know, so there are some really simple abstractions. Like, is there a dog in the room? It's a true or false, like, concept.
d.bau: Right? And a lot of, like, the literature is dealing with, like, oh, if you have a whole bunch of true and false concepts, then how do you represent them, and so on. We'll come back to that. But then you're like, oh, but there are complex concepts, like models of things, and relationships between things, and so on. So you're sort of down here.
d.bau: And so, I think this is where, you know, the way to think of it is to say, yes, there are a lot of really, really interesting, like, meta-concepts. In fact, like, a whole neural network, a whole model, a whole agent.
d.bau: You can kind of think of as one of these things. And then you can ask.
d.bau: oh, does that situate inside, like, inside a general network somehow? Is it struggling to try to represent this and understand it, and how or not? Now.
d.bau: In a neural network, there's a couple levels of abstraction, a couple levels of reasoning, and some of these are part of the debate in this discourse, so I think that there's this really interesting distinction
d.bau: Between… Like, long-term memory?
d.bau: and short-term, Context, or short-term memory.
d.bau: And most of what we are playing with in this class has to do with What can you represent?
d.bau: In the activations, in the short-term memory, in, like, the neural states.
d.bau: Right.
d.bau: But most of what the rest of… Machine learning is working on.
d.bau: is how do you get things loaded into long-term memory? How do you adjust the parameters
d.bau: Of neural networks so that those parameters serve as an imprint of all the long-term knowledge that you want that
d.bau: neural network to remember when you boot it up and you wake it up and it has no context, right? Like, what does it already know? It already knows that Einstein was a brilliant physicist, it already knows that Madame Curie, you know, discovered radiation or whatever, you know, like, it already knows all these things, you didn't have to tell it.
d.bau: Right?
d.bau: And so, But… so… so there's these two different things, and… and then… and then…
d.bau: And traditionally, they're sort of separated, but one of the interesting things is, oh, can you kind of simulate one?
d.bau: Any of the other, right? Like, in the short term, knowledge, can you, like.
d.bau: Do things that behave as if
d.bau: you're learning something in the long-term style, and so on, right? So there's… so there's…
d.bau: So, so there's this whole world of, like, can you take a traditional learning algorithm down here and then also jam it up?
d.bau: care and things like that, because I think in your question, you asked something about, you know, models. I have a question about just…
d.bau: you know, in interpretability as a field right now, like, we look a lot at, like, single token, you know, concepts more than, like, single token. Yeah, yeah. So, when you're talking about this sort of functional view of, you know, representing, like, complex, concepts, like, do you feel like…
d.bau: the answer to, say, like, multi-token representations or long-form rep… like, a concept that's diffused over, like, a long set of tokens, like, do you think the answer to that also lies in finding a function that somehow maps? So, I think that there's definitely things where, like, you know, Nikhail is looking at some of the binding questions.
d.bau: And the structures that he's finding definitely span many tokens.
d.bau: Now, these probing into these things, because of the tools that we have, looking at things one token at a time.
d.bau: But when… after he looks one token at 5, he sees these structures and these look-backs and things that go back
d.bau: many tokens, and so I think that, yes, Ultimately, the strategies that the…
d.bau: Networks probably use when they're representing complex things are exploiting the long sequence rollouts.
d.bau: But how that structure all works, I think, is still very obscure.
d.bau: We're just, like, scratching the surface, and so this is just a little view of, of, like, you know, the problems that need to be solved, not, not really a view of, like, oh, somehow we know what things are.
d.bau: That make sense?
d.bau: Does that answer your question? Okay, cool. Okay, Connor. Okay, so some quiz. Okay, so, oh, did I get everybody? Wait, did I get everybody? I got everybody, right? Kai? Kai? Let me talk about Kai's questions a little bit. Okay. So, okay, quiz.
d.bau: So, there's these three different views, and even within the
d.bau: the connectionists view, right? They're sort of…
d.bau: three different ways that you could think of a bag of neurons, and so… so I'll introduce, sort of, three… I'll make up three names for these communities, they probably don't like to be called this way.
d.bau: So let's say we have this problem of knowing what animals are in the room.
d.bau: Right?
d.bau: Any animals in the room? No animals in the room.
d.bau: Oh, just a whole bunch of humans. That's great. So, but, like, let's say there were some, like, zoo animals that might be in the room. Like, oh, is there a giraffe in the room? Like, how big is the giraffe?
d.bau: How's the young? Is your dog in the room?
d.bau: And… and so the question is… Mike…
d.bau: If we had 5 neurons in our brain.
d.bau: And how many different animals could we represent?
d.bau: Oh, I already clicked forward one!
d.bau: Okay, so here's a hint.
d.bau: So if you were a localist.
d.bau: If you were a localist, then you would say.
d.bau: Every neuron is a concept. Who's heard of the idea of a grandmother neuron?
d.bau: Right, so this is just an old joke that came from Professor Lefin from 50 years ago. He's teaching people about neural networks. He says, imagine that you might have a neuron that represents every concept. Well, think about all the thoughts that you can have. You clearly have a concept in your head for your grandmother.
d.bau: who makes you apple pie every time you go home for Christmas and wears a nice red dress that you remember, right? And so, what if you had a neuron for that concept? What if it had a physical manifestation of this neuron? Then you would have this concept of your grandmother.
d.bau: Isn't that interesting? So every time you thought about your grandmother would light up.
d.bau: Every time you are thinking about your grandmother, you would be quiet.
d.bau: That's kind of cool. And so that would be the model for the localist view of…
d.bau: animals. You might have one for a cat, one for a dog.
d.bau: One for the giraffe, one for the lion, one for whatever, right?
d.bau: But if we only had 5 neurons, then we could represent 5 animals, and that's…
d.bau: That's it, but we could do it in any combination.
d.bau: Should be nice.
d.bau: That makes sense?
d.bau: So… Now…
d.bau: There's this other view of what you would do with 5 neurons. Oh, yeah, I had another question. Like, so, you know, like, the whole, the monosomatic view of… well, if you had, like, as many neurons as the number of features you're trying to represent, like, they all, like.
d.bau: each neuron would represent a creature. Is that, like, verified? Like, in the sense that if you had 5 neurons and 5 concepts, like, does polysynaticity or, like, does it still emerge? Like, is there still…
d.bau: comes far, sitting with the model… If the world that you live in only has 5 animals, and the brain that you have to use has 5 neurons.
d.bau: Then, then the toy experiments that were done, actually, in Hajj.
d.bau: suggests that you don't get polyasomicity. It says, oh, look, we have the perfect number of neurons for the perfect number of concepts that we have in our data. I have one neuron per concept.
d.bau: Done.
d.bau: And you get this. So, this actually shows up in Berkeley.
d.bau: We'll talk about that in a minute.
d.bau: Right. Okay, but now, you're clever computer scientists. How many things can you distinguish if you have 5 bits?
d.bau: 5 factorial again? Huh? 5 factorial?
d.bau: I don't know, that's a little bit, like…
d.bau: Yeah, yeah, what would you say? 2 to 5. 2 to the 5, 2 to 5, yeah, there's more traditional computer science.
d.bau: Right, so 5 bits is, yeah, 5 bits, not 5 quarter bits. Neurons can be on, they can be off, that's what we'll think about. They don't get shuffled.
d.bau: Right? Make sense?
d.bau: And so… so, like, in a combinatorial encoding, which I have here, there's, like.
d.bau: If you count very carefully, I got Gemini to draw me 32 little red dots there, right? This is a hypercube.
d.bau: you know, projecting into 3D space with 32 corners, right? So, I don't know if you know why I'm saying it's a hypercube, but basically, yes.
d.bau: So basically, there's this combinatorial view, which is.
d.bau: we could represent 32 things. So what if you had, like, 20 neurons? Then how many things could you represent with this combinatorial view? Well, you could… well, look at this view, you could, like, represent 20 animals.
d.bau: But with a common frontal view, what is 2 to the 20?
d.bau: Any computer scientist, quick, quick with 2 to the 20. What was 2 to the 10?
d.bau: I think 226, about a million? About a million! About a million, yes, because 2 to the 10 is about a thousand.
d.bau: Did you notice? Okay, right, so…
d.bau: And 2 to 20 is about a million. And so, yeah, it goes… so it explodes very quickly. So you could, like, distinguish between, like, a million different situations if you had 20 neurons. Just 20 neurons! 20 neurons, not that many.
d.bau: There's a lot of situations to distinguish.
d.bau: But then it's very limited.
d.bau: Because… If we had our catalog, our zoo of a million animals.
d.bau: And every neural configuration was, like, one of the animals.
d.bau: Then, if some other animal walked in, it would have a very different configuration.
d.bau: I'd have trouble with computationality. Like, well, not both animals are in the room. Like, I don't know what state.
d.bau: To put my system in. Now, in the localist thing, no problem.
d.bau: I can just light up both of those interactions, have both of them.
d.bau: That make sense?
d.bau: But… but in the… but in this combinatorial view, it's very fragile. I can't, like, have… I can't even have two animals in the same room.
d.bau: Like, if I… if I added those two bit strings together, then what would happen is I would, like, hallucinate some third animal, right? You see how that would work? That makes sense? Because I did some other bit pattern.
d.bau: Yeah. And so… Great number.
d.bau: El Hajj actually argues this is where a lot of hallucinations come from. A lot of, right, is from, like, these kind of things.
d.bau: Okay, so… so there's this geometric encoding.
d.bau: Which is… which is the third thing.
d.bau: Which is, like, oh, don't think of every neuron as a separate thing, think of it as, like, a vector. If you had to dosi says, it's a vector.
d.bau: Well, I don't know, what does that bias? I don't know if it buys us anything.
d.bau: Right, like, how many independent things can you represent if you have a 5-dimensional vector?
d.bau: Anybody remember the linear algebra? Like, how many independent things can you…
d.bau: Represent in, like, a five-dimensional vector space.
d.bau: life.
d.bau: It's no better than the other things!
d.bau: Right?
d.bau: So, like, what's the big deal about vectors?
d.bau: I could just stick with neurons.
d.bau: That's how I started my PhD. I was like, well, I'm just gonna look at what the neurons are doing.
d.bau: Right, and in some domains, that works pretty well.
d.bau: But the vectors have an advantage, because what happens is.
d.bau: This limitation, that you can only represent 5 independent things in vector space, is only true
d.bau: If you constrain your… vectors, To all be linearly independent.
d.bau: Or, or, like, if, you know, or even orthogonal to each other, but, you know, like, literally independent from each other. If you, if you…
d.bau: If you actually relax that a bit, And allow them to be…
d.bau: Usually orthogonal to each other, usually almost orthogonal, usually independent from each other if you sample a subset of them.
d.bau: But if you took all of them, then they're not independent anymore, but if you took any random subset, then they're…
d.bau: Usually independent.
d.bau: Right? So if you relax that a bit.
d.bau: Then… then you can get these, like, good enough, almost bases.
d.bau: That have more.
d.bau: than 5 things. You could… you could very reasonably represent, like, 20 things.
d.bau: Get in.
d.bau: And so, what I've shown here is, like, a beach ball.
d.bau: kind of thing, like the Borenoi diagram of…
d.bau: Of, like, you know, how different regions of belief might… might be organized.
d.bau: It's not… things are not orthogonal to each other.
d.bau: But instead, you get things where you have similar beliefs, because you're sort of nearby.
d.bau: But, but,
d.bau: Okay, so I'll talk a bit more about this here in detail for a second, so… Oh, oh, out, oh!
d.bau: oh, I had that great figure, and I didn't paste it in. I was like, I worked on this figure for, like, an hour, and I didn't paste it into my slide.
d.bau: Oh, where is it? Where is it? Let me find it here.
d.bau: Oh, it's too late. I can put it on the whiteboard. Okay.
d.bau: So… Okay, so I'm gonna review, like, the Hodges, experiment here.
d.bau: Right? So this is… so this is a magic…
d.bau: So sad. How I worked on this nice figure? Okay, I'm fighting for another one.
d.bau: So, Ahodge did this. He says, what if we just have 5 neurons, and how many animals can we… he doesn't talk about animals.
d.bau: very abstract. He's like, X sub i with importance 0.7 to the eye, and with sparsity 0.3, right? So what is this?
d.bau: So, what he's saying is, Let's say we have, like.
d.bau: 20 animals in the zoo that might come into our classroom.
d.bau: Right?
d.bau: And… and I have to build a little neural network to remember, to sense and remember and represent, like, which animals I have here. And not only that, but, like.
d.bau: How big they are, or something. Some real number about each animal.
d.bau: Right?
d.bau: And, and so… So the… so, I have to remember a random size of 20 animals.
d.bau: Right?
d.bau: But I only have 5 neurons to use.
d.bau: Some animals are more important. I'll tell you, if the lion is here, I don't want to forget that. Super important, right? But, like, if I forget the squirrel, that's okay. Like, less important. Make sense?
d.bau: And so, I have, I have different levels of importance of every animal.
d.bau: And then the nice thing is that with some probability S,
d.bau: Like, any given animal will be absent. Like, I could set S to, like, 50, 50%, and that means that, oh, actually, 50-50, like, any given animal's just not here.
d.bau: Animals just know the line's not here, so it's sparse.
d.bau: So this is, like, the data situation that we're facing.
d.bau: And so we have 20 different animals, and only 5 neurons deal with it. What happens? And so… so, yeah, the diagram that I was gonna draw was this, like, photocoder type of diagram, where we have, like.
d.bau: These 20 inputs, Right? And then we're gonna, we're gonna, like, make a little neural network to, like, assign…
d.bau: Of five-dimensional vector to every animal.
d.bau: Right, just a table?
d.bau: Every animal can find a measured vector for our, for our neuron, for our neurons.
d.bau: And then if multiple animals are in the same room, that we just apply the normal
d.bau: Connectionists roll. Like, oh, two things are true at once, you just add them.
d.bau: Right? 10 things are true at once, you just add them all. And so you get some vector, representation of, like, the set of animals.
d.bau: And the reconstruction rule is.
d.bau: The way that we read things out of neural networks is we do dot products.
d.bau: So we say, hey, we know what the vector is for elephant.
d.bau: And I've got some weird 5-dimensional vector.
d.bau: I wonder if there's an elephant in the room.
d.bau: How big it is?
d.bau: And so I will… so to do that, I just take this 5-dimensional vector, take the elephant vector and dot product in.
d.bau: And the dot product should recover the component
d.bau: Of my five-dimensional bandwidth is distributed by elephants.
d.bau: Alright, so we read that spec out.
d.bau: And then, now, any linear algebra person will tell you.
d.bau: Since 5 dimensions is a lot smaller than the 20, it's gonna be lossy, there's gonna be some error.
d.bau: So they just put this thing in an optimizer, and they say, okay, well, you can come up with any dictionary you want.
d.bau: Give any five-dimensional vector to any one of the 20 animals. Just minimize your error.
d.bau: Right, and then how do you minimize… and you just have to step into an optimizer for 5 minutes.
d.bau: And they said, come up with a nice dictionary.
d.bau: for all of the, you know, five-dimensional vectors for every animal, they minimized this error, weighted by this importance, like, we don't want to forget about flying, but yeah, it's compromising on the squirrels okay.
d.bau: Make sense?
d.bau: So that's what… so that's what the situation is, isn't it?
d.bau: Sorry if you're not having the diagram. And so this is what… this is what they… this is what they found.
d.bau: Right. Which was… that… If you, if you do this with a regular linear network.
d.bau: Or, if you do this with sparsity, Zero.
d.bau: In other words, every animal is always there.
d.bau: The whole zoo is always here.
d.bau: Right?
d.bau: But sometimes, like, the lion is really big, sometimes the lion's really small.
d.bau: Right? Sometimes the squirrel's big, sometimes the squirrel's small.
d.bau: Right, but the zoo is always here. Or, if the model is, you know, perfectly linear.
d.bau: Then you get this picture. This is… this is the… this picture is equivalent to PCA. Who's ever used PCA in their research?
d.bau: Or SVD, yeah. So this is…
d.bau: This is… this is saying that what the, what the brain learns
d.bau: Is that it's got 5 dimensions available.
d.bau: And it's gonna allocate those 5 dimensions to the 5 most important animals.
d.bau: It needs to keep track of all the carnivores.
d.bau: And none of the other animals matter.
d.bau: Right? It's gonna allocate zero resources to any of the other animals.
d.bau: Because… The carnivores are always in the room, needs to know how big they are.
d.bau: Thank you.
d.bau: And all the other animals are in the room too, but they don't matter.
d.bau: So… so that's… so that's what the network does. But as soon as you introduce nonlinearity in the network, so you make it a neural network, so, you know, you have a relative
d.bau: You know, it's gotta be just technicalities that it can… You can do nonlinear functions.
d.bau: And… and you, and then you, add sparsity.
d.bau: to the assistant. You say, oh, what do they say, previous person? Can I read this off?
d.bau: Sparsity 0.3. So with 30% chance, any given animal's not here.
d.bau: Right? So maybe the carnivores aren't here, like, a 30% chance that the lion's not here.
d.bau: 30% chance the cheetah's not here.
d.bau: Great.
d.bau: then,
d.bau: Then what it will do is it'll start to say, well, you know, I only have 5 dimensions, but…
d.bau: I can come up with a codebook that has codes for 7 animals.
d.bau: Is that in 7? There's a chance it's fine.
d.bau: Oh, maybe you're right.
d.bau: 70% should not know you're right, because it's 1 minus S.
d.bau: Thanks.
d.bau: Gotta get the numbers right. Yeah. 70% chance. So, so I guess you have to… it has to be pretty sparse for this to show up. If it's only a 30% chance, you probably don't see this. Probably sticks here for a lot of time.
d.bau: But it needs to be, like, it has to be… it has to get pretty rare, We're annually around.
d.bau: And so, if 70% chance the animals aren't around, then it starts saying, oh, you know what? I'll write down codes for 7 animals instead of 5.
d.bau: And now, what is going on with these yellow things?
d.bau: Is that they're not linearly independent of each other.
d.bau: They're like, there are these other vectors that…
d.bau: can be confused with one another, right? So, like, if we have a few of these animals come in.
d.bau: And it's like, do I have… A really big squirrel.
d.bau: And a medium-sized ostrich, and a bat.
d.bau: Or, is that what I get from a giant toad?
d.bau: And some other things, because those two situations would give me the same vector.
d.bau: Like, I don't know, I can't tell.
d.bau: Which, like, it's ambiguous. It's truly ambiguous. But, when you do the readout.
d.bau: It says, oh, if you just do a dot product, I'll give you an estimate.
d.bau: Of, like, how big the toad was, and how big the ostrich was. And you get these numbers, and those numbers are close.
d.bau: They're as close as possible. They actually optimize the loss compared to… Like, throwing away the data.
d.bau: That make sense?
d.bau: And so, so yeah, so that's, that's the solution. So you see this, like, slightly larger clipbook, and then as sparsity increases, so now 97% chance
d.bau: You know, the animal's not here.
d.bau: Right? You know, 99% chance the animal's not here. You get a fully sparse codebook.
d.bau: Right? If it's rare for any of the carnivores to ever show up.
d.bau: then don't give them the whole dimension. They don't get their whole own neuron.
d.bau: Right? They only get the ball of neuron if they're, like, pretty frequently there.
d.bau: Right? But if 99% of the time they're not there, they don't get a neuron, they just get something in an overlapping codebook.
d.bau: Right.
d.bau: That make sense?
d.bau: Yeah, so that's… that's a superpositioned…
d.bau: story, and this is something that Neural networks can do.
d.bau: that… the bi-neuron localists, one neuron at a time thing cannot do. It can pack more concepts in.
d.bau: Right? But, with some error, with some loss, with some ambiguity. But, inevitably.
d.bau: Because it's… it's going beyond what linear algebra says you can do on ambiguously.
d.bau: Now, do you guys understand the El Hodge experiment?
d.bau: Pardon?
d.bau: Yeah, hi, my question was about, how stable superposition is across different models, and, like.
d.bau: Yeah, are they consistently grouping the same features together, or assigning the same sparsity? Yeah. And if they are so different, then how do we know that the superposition examples we see are actually reliable?
d.bau: Right. And Avery, you've had a similar question, is that right? Is Abrid here yet? Yeah, I asked that. Exactly the same question. Exactly the same question. Okay, another project.
d.bau: So, so, so yeah, the answer is,
d.bau: the jury's out. When people try to do this, on purpose, like, you know, so there's all these people, like, training SAEs, which are
d.bau: that are, like, a form of doing this on purpose, like, trying to find server sparse codes. One of the big problems is SAEs are not stable every time, like, you go and train them again, they, like.
d.bau: Like, divide up the space very differently.
d.bau: On the other hand…
d.bau: It says that, for really important concepts that have relationships to each other, they tend to
d.bau: Learn the same structure, right?
d.bau: I think that… I also think that the… the research so far has been relatively…
d.bau: oversimplified? Like, this toy model story and SAEs themselves, they have this assumption of independence between all the features.
d.bau: If your features are all independent, then there isn't any particular reason if there's some of them to group together.
d.bau: Or not. But in real data, you have a lot of groupings, a lot of correlations, and a lot of relationships, and so it may be that in real data.
d.bau: You tend to get more, reproducibility?
d.bau: Superposition, but it hasn't, like, It hasn't been studied very well.
d.bau: And so, it's an open space for me.
d.bau: Everybody wants to do that for their PhD.
d.bau: Another question from Yuki.
d.bau: Oh, God, that's funny.
d.bau: I'm curious about what I see it used for. In my eyes, it's just, like, a map from a low dimensional to a higher dimensional. Yeah, yeah, what's this buff? So, in this thing, if I remember to draw the diagram.
d.bau: You can see this toy.
d.bau: Example is a map from high dimension to low dimension.
d.bau: So you have, like, 20 companies that deal with it?
d.bau: What the SAAs are, claiming or attempting to do.
d.bau: is you… is, like, you have this five-dimensional thing. Can we guess what the true…
d.bau: underlying 20 features were… that were pressed into this thing. So essays are an attempt
d.bau: And doing an unsupervised decompression technique to reverse what they hypothesize as this kind of process.
d.bau: And if you feed this kind of process in, a toy process, the essays are pretty good.
d.bau: at decompressing it, but of course, they apply SAEs to real data to try to decompress it, and I'm actually kind of skeptical. Like, they have all sorts of issues, like, with instability and things like this. I'm not sure they're really getting a good decompression of all the concepts. They're probably finding a few of them, right? But,
d.bau: But that's what they're trying to do. Does that make sense? Yeah. Okay, yeah.
d.bau: And then, so that's sort of how SAEs fit into it.
d.bau: But a lot of people work on SEs, and they're getting better over time.
d.bau: So, Luz I had this question about safety.
d.bau: How do you say your name? Lucy should be online. Oh, Lucy, are you online?
d.bau: I'll ask Liz's question.
d.bau: Lizzie?
d.bau: Yeah, sorry.
d.bau: So, what does this mean for safety?
d.bau: Like, if these things are, like, so approximate, like, how can we ever assure any of the behavior ever?
d.bau: And I think the answer is, offer the traditional
d.bau: mathematical sense of, like, the symbolic reasoning people would say, you know, we're going to make a proof that this thing is never going to misbehaved or something like that. It's pretty dire for that. I don't think you're ever going to get a proof out of a system that uses more dimensions than you have capacity, or you're going to get aliasing.
d.bau: And problems like this, you're gonna get, like, hallucinations are just occurring.
d.bau: And so, the old-fashioned type of safety.
d.bau: Right? I don't… I'm not… I don't think it's that… I'm not optimistic.
d.bau: Right? You know, after this kind of analysis. Now.
d.bau: you know, I could be wrong. Like, there are definitely people, there's a whole community of people out there making assurances.
d.bau: Around provable guarantees on neural networks, doing really impressive work, getting, like, you know, propagation of bounds and provable guarantees through, like, GPU hardware, trying to scale this up for larger neural networks, but…
d.bau: But I… but, like, I think…
d.bau: Even if you can figure out what the error bounds are and everything like that, you're still stuck with this situation that in order for the networks to perform well, they end up playing troops like this.
d.bau: which end up, you know, giving you aliensing, which I don't think is about some error bound. I think it's just…
d.bau: how they work?
d.bau: So, on the other hand, there's other ways of achieving safety, like measuring them, and also, recognizing what they're representing and understanding whether that actually aligns with what you're trying to do. So, the previous, paper.
d.bau: a story where I'm like, hey, you come up with this thing, and you see how it jams into that, does it fit, does it not? What's, like, thrown overboard? I think that if we understand this, if we understand what compromises the network is making, and, like, how it's representing things, then that's sort of the path to safety, right? So then at least we understand
d.bau: like, what the network is trying to do, what it's not trying to do, what it's good at, what it's bad at, because of, like, what compromises it's making. And so I think that's… that's sort of the path. But in terms of, like, oh, can we guarantee that the network will never utter a thing? I'm not really sure.
d.bau: Make sense?
d.bau: And then, okay, question about Claire Spark City.
d.bau: It's clear here.
d.bau: Yes.
d.bau: I think my question is related to, like, sparse neural networks and com… Yes, yes, yes, and KANs and stuff like this.
d.bau: So, K-ans I'll set aside for a little bit, just because… so K-ans are a different way of doing neural networks. Instead of having, weights between things, you can learn functions between layers. There's an old system called a generalized additive
d.bau: models.
d.bau: Which KNs, generalize by stacking them layer to layer. So, I think the reality of KNs, and so some people will disagree with me, is that they're a theoretical construct. The people who published a big KN paper a couple years ago said they're gonna outperform neural networks.
d.bau: you know, people start showing up with, like, venture capital money and saying, oh yeah, you're gonna beat Allen on the horse, here you go. I think in reality, nobody uses it because
d.bau: They're computationally really efficient, they're hard to train, they're hard to run.
d.bau: So they're not that practical. It's a nice conceptual idea. So I'm going to set KNs aside, because it's a little bit of a niche idea. For the sparsity.
d.bau: the thing I wanted to make a point about is that
d.bau: This situation doesn't come up because your neural network is sparse.
d.bau: This situation is coming up because the world is sparse, because the lion is not always in the room. 97% of the time, the lion is not here. So most of the time, you could ignore the fact that there might be a lion, and, like, it would be wasteful to, like, allocate a whole neuron to a lion because
d.bau: Because 99% of the time, it's like, that neuron's, like, would be zero, it'd be off. It's like, what a waste.
d.bau: So instead, the neural network learns to have, like, a more efficient encoding.
d.bau: But it becomes sort of inherently,
d.bau: ambiguous. But this thing comes from sparsity in the world.
d.bau: As far as at the end of our own.
d.bau: Right, so that's a, that's a sort of a distinction that one.
d.bau: To clarify.
d.bau: Okay.
d.bau: So, dot products… so, like, what, like, what this geometric view is all around, you know, constant to a vector. So, what does it mean when we say constant or vector, right? Like, it's just… a vector is just a sequence of numbers.
d.bau: Like, how's that… how's that differ from just saying concept is, like, a message? You know, it could be, like, written in ASCII, and then there's ASCII vector, and so on? So, like, what you normally don't do with, like, an ASCII vector, or any explicit numbers you see somewhere in text is you normally don't take dot products of that.
d.bau: Right? And so, when Priam Tadosi says, oh, concepts are vectors, and you're talking about connectionist literature and everything, this is the heart of it.
d.bau: Like, how do you know if a sequence of numbers you have is a vector? It's… as soon as somebody starts doing dot products between vectors, then you're like, that's a vector, you're in a vector space, here's some dimensionality, there's some geometry here. And there's three ways
d.bau: of thinking about dot products. One is, it's just a way of telling you how well two vectors match.
d.bau: So there's this old trick people would do with, like, punch cards, is they'd say, hey, are these punch cards similar? And you could line up the two punch cards, and you could see, like, how many holes
d.bau: They have in common?
d.bau: That's a dot product between punch cards.
d.bau: Tells you how similar… the more light shines through, the more similar they are. That's literally what a dot product is. So I like this kind of picture. Right, so dot products are just, like, the simplest possible thing you could do to, like, look at similarity.
d.bau: Between, like, two vectors.
d.bau: Like, holding up two punch cards. But there's also this geometric view, which is this, like, angles between things, right? Maybe Brand learned in middle school. And then there's also this Fourier view of, like, oh, you've got some really complicated thing.
d.bau: And you're gonna decompose it into components, like Fourier components or something like this. And this is another, you know, view of, like, what .
d.bau: And so, and so, yeah, so all these views are all true, it's all the same math, right? They all exist at the same time.
d.bau: And… and this is what the basics for… so… so when we… when we play with things, okay, so a couple more come down before we get done here. So…
d.bau: So, more intuition building about vendors.
d.bau: So, in middle school, we all learned about vectors.
d.bau: In, like, two dimensions. And when we're back to high school, they're like, oh, there's a third dimension. Oh, three-dimensional vectors. Okay, so, but, in neural networks, like…
d.bau: price for entry is, like, a thousand dimensions, so now we're in a thousand-dimensional vector space, and I'm sad to say a thousand dimensional vector space works very differently from a three-dimensional vector space, even though all the math
d.bau: you know, sort of the same formulas, but just pumping up that number to 1,000. So if you take two random vectors in three dimensions.
d.bau: Two random unit vectors, and you took a dot product between them.
d.bau: What would be a typical dot product between two random vectors in three dimensions? What do you think the mean is? Who wants to make a guess for what the mean dot product is between two vectors and data?
d.bau: Zero, no, it's higher. It, like, if you take two random vectors in three dimensions, in three dimensions, in three dimensions, yeah. Like, how, how, how, like…
d.bau: So a dot product will tell you, like, how close to parallel things are. So if you just took a vector in 3-space and another vector in 3 space, they're not going to be parallel, but they've got some closeness to parallel, and so I need to figure out the cosine of the angle.
d.bau: So, like, what's the average cosine of the angle between these two vectors? What would you… what would you say? Well, if it were perfectly parallel, the cosine would be 1, so it's definitely going to be less than 1.
d.bau: And they're perfectly orthogonal, it's gonna be 0. That seems rare also, so it's going to be more than zero. So in three dimensions, like, what do you think, like, the average cosine is? Something between 0 and 1.
d.bau: Hey, just throw the number. 45. 45, good guess. 0.46, very good. Okay, so it's about, about, about half.
d.bau: Right now, we're in 1,000 dimensions, we do the same thing.
d.bau: We just pick two random vectors, and we get, like, a dot product. What's the average?
d.bau: Say again? Louder, louder? What's it?
d.bau: Closer to zero?
d.bau: How close to zero?
d.bau: Zero? Zero exactly zero? Yes, ma'am. No, no, no, no, no. It's never exactly zero. You never get, like…
d.bau: There's some curve!
d.bau: But it's closer to zero, so I'll tell you how close… 0.02 to 5.
d.bau: 025 per minute.
d.bau: So, it's pretty close to zero.
d.bau: Right? So, in, like, in three dimensions, that would be indistinguishable from being right to right angle. So, that's why in thousand dimensionals, you just take random vectors.
d.bau: the assumption is, like, if they were not at right angles, it would be, like, weird. It's like, wow! Like, does that practice between these two things is, like, 0.1.
d.bau: That's, like, 5 times bigger than the dot product should be. Like, I… that's a, like, high dot product.
d.bau: Oh, no, not bypassed like that, it's 50 times higher.
d.bau: Or whatever, it just… like, you could see… And so, Okay.
d.bau: If you have, like, buy them.
d.bau: in high dimensions. So if you have a bar, It has what?
d.bau: IRQ, whatever, you know, that, that, that, that, that volume?
d.bau: Right? If you shrink the ball by 1% radius, now you have another ball inside.
d.bau: How much did you shrink the volume of the ball?
d.bau: In… in three dimensions. It… so, well, now the… now the new ball has a new volume.
d.bau: So the old ball had volume 1,
d.bau: And so the new ball has volume…
d.bau: Let's say the new volume is proportional to the old volume.
d.bau: Shrinking it by 99… Drinking it by 1%, eating it to 99%.
d.bau: Yeah, yeah. Oh.97.97. Yes, physics percent, exactly. Linear approximation of cubing a number.
d.bau: 97.
d.bau: Alright, does that make sense? So, like, the volume, like, the linear dimension goes down by 1,
d.bau: That means that the volume goes down by 3, because you're in 3 dimensions, which should give you a hint.
d.bau: What happens if you're a thousand dimensions?
d.bau: So you just shrink the ball by 1%, what happens to the volume?
d.bau: Am I out of time.
d.bau: I'm out of time. What happened to our lab?
d.bau: I, I, I said, I'm gonna go through the stress, I'm gonna only have 6 slides. And I, like, used an hour for 6 slides.
d.bau: Sorry.
d.bau: I don't know how to go through this fast, there's too much material.
d.bau: You want to do it Thursday? Sure, I don't know if I'll be able to come in person on Thursday, because I have a…
d.bau: Okay, but you guys have to do this for homework anyway, so there's a notebook. Will you guys…
d.bau: Try it tonight.
d.bau: And…
d.bau: log into the Discord, and share if you have any trouble, and, and then if, and then Karina can also…
d.bau: You do that, it's actually supposed to be helpful for what you're talking about for the Thursday at the very, very Tuesday.
d.bau: Does that make sense? I'm gonna just show you the rest of these numbers here. So the volume, so .004%,
d.bau: How much of the cube is filled by a unit sphere? In three dimensions, it's 52%.
d.bau: In 1,000 dimensions, it's this crazy astronomical number, 10 to the negative 2700.
d.bau: Right? But basically, a cube… a hypercube and 1000 metric was all corner. It's no… it's no interior. Like, basically all corner.
d.bau: If you have a normal distribution, right, so you take a normal vector, people are like, hey, take a Gaussian distribution, it's like, well, I know what that's like, it's like fussy, right? So actually, in three dimensions, like, it's pretty rare to have, like, a Gaussian normal distribution, even though it's, like, the mode.
d.bau: It's pretty rare for it to have zero length, even though it's, like, the most powerful vector.
d.bau: Right, but the length is, you know, around 1.6.
d.bau: Right? With this kind of distribution. When you get to 1,000 dimensions, this is the length of an adaption, vector.
d.bau: It's, 31.6.
d.bau: And, and the length and time set is anything to do that.
d.bau: Okay, I'm out.
d.bau: Okay, here you go. Here you go.
d.bau: Oh, no, I have so many things to talk about.
d.bau: Oh, what did you choose?
d.bau: Okay, yes.
d.bau: I'll put it online. Thank you very much.
d.bau: Hey guys, buddy, okay.