David Bau: Okay.
David Bau: And… What do I have? What is my slide decks?
David Bau: Nope.
David Bau: Lost my slides.
David Bau: move.
David Bau: The four slides.
David Bau: I can't… I can't reach any of my… my menu bars because the Zoom is covering everything up.
David Bau: Okay, I've lost my slides.
David Bau: We'll work for them again here.
David Bau: By the time.
David Bau: All right, session 2, reverse chronological, we love that. So I just want to… so I don't have anything technical this week, but I wanted to congratulate everybody on getting
David Bau: Their submissions in.
David Bau: And oh yeah, we should do this. So, how do I…
David Bau: I would put this in full screen mode.
David Bau: There's, like, a mode thing, right? Full screen?
David Bau: We're gradually learning how to use this version of Zoom.
David Bau: And as soon as we, like, are comfortable with it, I'm sure they'll change the UI again.
David Bau: Thanks.
David Bau: Yeah.
David Bau: So, on the left, Congratulations, everybody!
David Bau: You actually, like, wrote all the words in all the papers.
David Bau: Which is… on some of them, it was a little tight.
David Bau: But it's good. I'm really proud of all the papers. I really like all of them. I think they all tell an interesting story. I was telling people that, you know, I like the… I like the flavor of the topics that you've all chosen.
David Bau: I think that they all provide a bit of insight into something new, which is, which is, you know, what we're all trying to do.
David Bau: And a bunch of interesting topics. So, so congratulations on all that. I, I, I also am enlisting things that showed up in my, open review for NURIPS, which, went through review.
David Bau: Just a few days earlier than the ICLR deadline. And to my surprise, all the NRIPS reviewers, or the ACs, or…
David Bau: program chairs or something, like, had pity on us, because pretty much every one of these papers had split reviews, where some of the reviewers, you know, had recommended, reject, and so on. But…
David Bau: But the… the Nurips, you know, gods looked down kindly upon us, and, and actually accepted all these papers, and so congratulations on everybody there as well.
David Bau: You know, so we have… we have a lot of, things to tell the world. Now, just getting the paper through this peer review process is sort of the first step of the process, so there's a lot of…
David Bau: like, you know, we should create websites, and clean code repositories for people to use, and, you know, explanatory materials, make sure the archive is all up to date, and all that type of stuff. But, but congratulations on getting through, you know, these stages of the process. It's a sign that
David Bau: People, appreciate.
David Bau: kind of, questions that you guys are asking, and the research that you're doing. So, congratulations, everybody.
David Bau: Okay, let's go on to the plot-a-thon.
David Bau: All right, who's… who's this? Who's this? I can't see on my screen. Alright, David.
David Bau: Okay, so…
David Bau: Last time I gave an introduction to this task, and just as a reminder, what I'm doing is I'm training a model on, to mimic choice behavior according to randomly generated preference weights.
David Bau: And so, what I'm doing here is I'm like.
David Bau: Well, so what is actually being trained? So, I take two checkpoints, and the first checkpoint, checkpoint 1000, which is on the top there, can perform the choice task, but when you ask it, like, if you were to do this task, what weights would you use? It doesn't tell you anything with any, like, any real relationship to the choice weights.
David Bau: So you can see, like, the, the green line here is the introspection behavior, and it's quite low at the start. And so then I…
David Bau: take this adapter that I've trained, and I… or, like, set of adapters, and I… Wait, so hold on, so, sorry. Choice behavior only. Yeah. So… What does choice behavior only mean? So, like,
David Bau: I guess, so basically, if you're, like, you are, Alexander the Great, you're shopping for a toaster.
David Bau: Toaster A has, like, these 5 attributes with these values, toaster B has the same 5 attributes with different values. Which toaster would you choose?
David Bau: And then you train it just on that, and then later you ask it, if you were choosing between, like, two other toasters.
David Bau: what, what weights would you use to decide? And so the… what weights you would use is the introspection task. Exactly, yeah. But then, which toaster would you choose is the choice task. Right. And so you're training only on the choice task? Yep.
David Bau: Only on choice tasks. And, what you see is, like, initially, the choice task performance goes up, but the introspection behavior does not. So that's where these two checkpoints come from. The checkpoint 1000
David Bau: on the left, or on the top, can perform the choice task, but cannot perform the introspection task. So we see that the blue line, which represents,
David Bau: Choice task performance is quite high, before you ablate anything, and the green line is quite low.
David Bau: On the bottom, the… this is, like, the fully trained model. Both are quite high. And so, on the top left, what I'm doing is I'm ablating every…
David Bau: every layer of the trained adapter, previous to the, to the layer index. So…
David Bau: I guess what I found interesting here is, like, the… you know, I've drawn this red line on layer 40,
David Bau: And so what we see is, like, at layer 40, if you remove everything at layer 40 and below, on Checkpoint 1000,
David Bau: it really doesn't change much. So, like, most of the interesting training is happening at the top of the model, or the end of the model. But on Checkpoint 3000, which is the fully trained one, which can introspect, at checkpoint 40, if you remove everything previously, like, it does quite affect things. And so, I guess this… this is…
David Bau: I guess the source of my, like, temptation to speculate wildly that what happens is…
David Bau: At the beginning, it's kind of like training in hard heuristics, like, cheap heuristics.
David Bau: And that's why you can't introspect, and… So, like at layer 50 or something? At layer 50, yeah. At early checkpoints. And then by later checkpoints, what happens is this, like, the thing that's changed kind of moves down towards layers. It finds the place where it's kind of like the natural slot for this… for this information, and at that point.
David Bau: the pre-existing introspection pathway can activate and access that information and, faithfully represent what the model is doing. If you can put the information at layer 40, then it can introspect. Yeah, right. What a nice story. That's… that's…
David Bau: Right, that's a story. So I'm trying to figure out if we can confirm that. Yeah, what are the ways of verifying that?
David Bau: That's a nice story.
David Bau: Yeah.
David Bau: I don't think I've seen this data before. Anyway, if anybody has any good ideas on how to verify that story in a different way.
David Bau: That's gonna be your challenge. Yeah, definitely let me know. Okay. Thanks, David.
David Bau: Very cool.
David Bau: Oh, what's this? Andy. Yeah.
David Bau: So I'm, like, trying to understand this thing called singular learning theory, which is a bunch of, like, math people who are trying to understand how models generalize, basically.
David Bau: And trying to see if they can come up with some theory of deep learning that actually tells us something.
David Bau: So, okay, the main, like, thing I would like to maybe explain here is…
David Bau: That, imagine, you know, on the…
David Bau: On the XY plane, you have, like, your weights, like W1 and W2, and then, on the Z, you have, like, your log likelihood or something.
David Bau: And so, the, the really nice case is represented at the top, where you have some, like, really nice-looking bowl shape that you imagine when you imagine ideal things.
David Bau: And you have, like, a nice… you can… you can understand this nicely by looking at, like, you know, the second-order Taylor approximation and Hessians and stuff.
David Bau: But, like, seems like in practice, like, things are a lot messier.
David Bau: And, you know… Me too.
David Bau: You have, lost landscapes that look, like, more degenerate.
David Bau: So some people talk about flatness, which is, like.
David Bau: Your… your kind of, like, curviness is… is less, but,
David Bau: what I'm trying to get across here is some notion of, volume rather than flatness.
David Bau: So, what I mean by volume is…
David Bau: You can… let's say you look at the minimum point of your… of your loss,
David Bau: And then there's a question of, like.
David Bau: How much volume around it, or area around it, in parameter space can you go?
David Bau: Where you, like, kind of keep the loss roughly the same. You keep it within, like, some epsilon.
David Bau: And you can see that between these different, geometries, this quantity is… is very different. Like, at the bottom, there's, like, a really wide volume of, like.
David Bau: basically parameter space that behaves similarly, like the losses within some epsilon, that's like the z-axis thing.
David Bau: Versus, you know, some, like, very sharp thing.
David Bau: And then there's some argument that's, like, an Occam's Razor-like argument, where you could say that the, like, bottom
David Bau: Ones with, like, larger volume like this.
David Bau: would, you know, generalize better, because they're simpler solutions. They're simpler solutions in the sense that you need less bits of information to encode them, because, like, you can specify anything within that big volume on that bottom right, if you're looking at that bottom right.
David Bau: And you get, like, a similar solution compared to some very, like, sharp thing.
David Bau: Anyway, so this is, like, what I'm reading about, and I don't really understand it all very well yet, but, like,
David Bau: So is this volume thing, sort of,
David Bau: Is it, like, related to higher order than second order? Yes, exactly. So if you look at, like, for example, like, X to the fourth.
David Bau: If you do, like, a two-second order, like, Taylor approximation at zero, like, this will not tell you anything.
David Bau: But X to the 4th.
David Bau: should be differentiated from, like, X to the 10th, or, like, X to the 20th. These are very different, right? And so you can't actually distinguish these from, like, second order. So do this… the singular learning theory people try to get at these higher order things? Well, they try and…
David Bau: They think that, like, estimating this volume
David Bau: I see. Good thing to do. Okay, it's just, they just go up to the volume instead of after derivatives. Right. Okay, got it. Right. Nice. Thanks for the explanation. Yeah. I had a question. Yeah.
David Bau: What do you mean by that blue region, blue volume? Is that in that region, if you move, so you're not… there's some people waiting downstairs. Oh, yeah. It actually could someone hit back, just for this question.
David Bau: Yeah, so if you, like, so what the volume… if you just look at the, like, right-hand column, what that volume is showing is, like, if you start at the, like, zero point, like, how far outwards can we move such that the loss only increases by, like, an epsilon?
David Bau: Oh. It's like an epsilon ball from the bottom, and then all the things which are… that… all the loss curve that's contained in the epsilon ball?
David Bau: No, it's not an epsilon ball from the point. You're just saying, like.
David Bau: Like, if we're here, how far out can we go, such that the loss
David Bau: for all of those corresponding points is, like, only epsilon above… Right, I mean, it's the intersection of the curve and an epsilon ball, so the curve would be inside the epsilon ball.
David Bau: I'm not sure. I think it's the same thing, but… Okay. So… so, like, if you have a very narrow code, moving… moving a little bit will, like, increase your loss a lot, right? But if you're very flat.
David Bau: And you can move a lot without changing the loss very much.
David Bau: And so, in some sense, like, this is a simpler solution, because you could be sort of, like, anywhere in this space. You, like, need less bits of information to specify where exactly you need to be.
David Bau: or some very precise thing that you need to get exactly on that point to… to obtain that loss. That's kind of an intuition. Is it… is it more like, let's say we're training a model to reach to the minimum?
David Bau: Yeah. Yeah, yeah. And so if… it's more kind of like a flat region, so we can, like, reach at any point in that flat region, and we'll, like, achieve a greater…
David Bau: Or better convergence report. Yeah, similar loss, but the argument here is that, like, maybe these, regions give you better generalization because they're simpler. So…
David Bau: Alright, yeah, that's… yeah. Awesome. Well, actually, since people are coming, I think I'm gonna try and, like, make space. Like, I can just, like…
David Bau: Bitcoin, I got you.
David Bau: Yeah, we can just shift… we'll just shift by one. Alright, so the volume thing is kind of, like, almost, like, intuitively, like, a probability that you'll land in the optimal loss region with respect to the task at hand. Kind of like saying, oh, it's like a generalized… it can understand and learn the thing without overfitting to something very obscure solution. There's a lot of ways you can get there as soon as you can possibly…
David Bau: Yeah.
David Bau: Should we go on to the next one? Yeah, yeah, yeah. Okay, cool.
David Bau: So last week I was presenting the, like, essentially whether a chain of thoughts can generalize in terms of, like, sharing these chain of thoughts across different models.
David Bau: So it turns out, like, like, it is actually better, like, the perceived model quality of the chance that's better with generalization. When I was checking for data where,
David Bau: like, the answers match but are wrong. I noticed that it included answers that also did not reach a conclusion, so I, like, after dropping those, I realized, yeah, essentially, the updated figure is that, like, if you have better
David Bau: Generalizability, you can actually assign the users that are focused more, as opposed to, like, the ones being done, right?
David Bau: local accurate or not. And in terms of, like, another project that I, like, started, like, discussing, working on, is essentially, like, hallucination interp. So, okay, that's… it looks pretty small from here, so I'll just kind of, like, leave…
David Bau: what each of those questions are. So, like, one, can you tell which instructions or questions are high impact, low impact? And so usually, like, maybe refusal, like, the work from, like, refusal,
David Bau: vectors and those may be, like, relevant to be able to find what are considered, like, high impact and low impact.
David Bau: But the reason why I care about this is, for the next question, whether we can, like, offset, like, representations of, like, subtle to severe hallucinations, because instead of, like, stopping a hallucination and saying, like, okay, like,
David Bau: like, you could say, for example, I don't know, but sometimes
David Bau: We, like, some of the hallucinations are kind of, like, reasonable inferences, and so maybe instead of, like, completely stopping hallucination, we can go from, like, subtle to, like, getting, like, egregious ones for those that are, like, low impact in terms of, like, creative writing, for example.
David Bau: And so, in order to be able to go back and forth with that, it's kind of, like, the third being, like, a steering sort of thing, whether you can move across different quadrants of…
David Bau: hallucination and problems. So, there are some related works that I'm, like, trying to, like, now get into more, but I thought I'm just sharing it in this meeting.
David Bau: Oh, no, so they work that high. There we go.
David Bau: It's your truck.
David Bau: Yep.
David Bau: Like, Sanjima's paper? Yeah, yeah, that's what I'm working with her that much, yeah.
David Bau: I like, I like Adam Kalai's way of looking at it, where there's different… it's not about the fact that's being hallucinated, but it's about the context.
David Bau: Like, if you were just, joking around with your buddies, then they don't really care. You got some fact wrong.
David Bau: But then, you know, if you're in a court of law and you're testifying, you know, after having sworn an oath or something like that.
David Bau: Deciding whether to send somebody to pay out.
David Bau: Then… then it's, like, a little higher space situation.
David Bau: And it could be exactly the same fact, it could just be, like.
David Bau: You know, who is Justin Bieber's mother?
David Bau: Right.
David Bau: But in those two situations, that very same fact.
David Bau: you know, you might want to have different calibration. Alright, so I'm kind of interested in whether models are aware of these
David Bau: different contexts. I don't know if that's a different spin on it. If there's a comment on that.
David Bau: I really liked… you didn't put this thing on, but you mentioned you were looking at the,
David Bau: the false chains? No, you don't have to put it up. I just wanted to comment on this piece of data, and it's worth taking a look at putting a paper in the end, so,
David Bau: So Koena has this measurement of how generalizable chains of plot are to other
David Bau: agents, when you use a chain from one agent to the other one. And, you know, last time I'd asked this question, oh, so if it's generalizable, if the first agent made a mistake, if their chain led them to the wrong answer.
David Bau: Does a generalizable chain lead a different agent to the same wrong answer?
David Bau: Or does it somehow, you know, ignore it? And I thought it was pretty interesting, because Karina has this measure of generalization, and her measure of generalization is totally
David Bau: you know, correlated. It's basically the same.
David Bau: you know, it's a proportional number to… if you only look at the incorrect following. So if… if a model
David Bau: is making what Karina calls a generalizable chain.
David Bau: then, then it's just a convincing model. It'll convince you. It can convince you of the wrong answer.
David Bau: And and which I think is kind of, kind of interesting. It's this night's…
David Bau: Way of showing that the metric is 4,000 or two.
David Bau: accuracy, correctness, whatever, right? It's just the convincingness of a chain.
David Bau: So anyway, I thought that was cool. It was a cool measurement that you made in, like, the last week of the…
David Bau: week of the paper, and it was a nice addition, I think.
David Bau: Okay.
David Bau: To add another comment. Yes. Like, in fantasy stories… Yes. There is always hallucinations. Yes, it's all hallucinated. None of it is true.
David Bau: In that… in that context, if you're… if you stick too close to the truth, then you're not doing your job, it's like negative calibration.
David Bau: Yeah, yeah.
David Bau: Yeah, so this was a fun thing I got to work on as part of the paper with Arnib.
David Bau: So, this was applying the filter heads to something that could be useful in practice.
David Bau: So, here on the left, we have a lie detector, so if you have some sort of context, and you set up a predicate at the end and say, is anything in here false?
David Bau: Then you can just use the attention pattern to discern which sentences are false, based off of which period is being attended to most strongly.
David Bau: They just said some…
David Bau: Upper quantile, of which you want to underline, and then anything that is false according to
David Bau: You know, these filter heads gets marked as false. And then, on the right, we're doing the same thing, but with sentiment.
David Bau: So this is just a cool way that you can have, like, sentiment analysis or lie detection without having to actually do any training. You just have the filter heads of the model, and you have some predicate, and you can match to it.
David Bau: Nice visualization.
David Bau: Okay, it's really cool. I'm curious if you train a probe, whether the probe has similar, like, whether the heat map looks similar, or whether it would, like.
David Bau: Different. Yeah, we actually did train a probe, and curiously, it was a lot more noisy.
David Bau: So this is much more clean than the probe, which is a cool thing, too. Oh, did you report that? No, we didn't report that. Yeah, it's always hard to report a negative result, because, you know, you're like, oh, we didn't spend that much time on it, and so do we really trust it, right? But, yeah, but it's…
David Bau: But it's a nice one if you… if you build some confidence, you could add it to a future version.
David Bau: It's cool that it started attending to the god delusion once it sees the god…
David Bau: Like, once… like, as soon as it starts seeing that, that book.
David Bau: being mentioned, it's like, wait, this is… Starts lighting up. Yeah.
David Bau: Thanks.
David Bau: Yeah.
David Bau: Very nice.
David Bau: Carl Sagel is an American football player. I like how it, like, for boring movie, it's like, oh wait, no, sorry, I thought that was the lie detection one. It was negative sentiment. Okay, yeah. Yeah, so this is boring, it's okay, this is… It would have been funny if they thought it was a lie. Yeah, like, Joss isn't boring, what are you talking about?
David Bau: Cool. Yeah, it's very nice. It's very nice.
David Bau: Yeah. So, in some previous experiments, we were able to, probe what the letter identity is out of the diffusion models, like, internal states.
David Bau: But the way that we went about doing it, it was dependent on the size of the actual letter patch.
David Bau: But we were interested in trying to identify some kind of way of also doing this probing as invariant to, like, how big a letter is in the image.
David Bau: So one thing that… this is an experiment where we basically just took…
David Bau: Each token that is in the letter patch that has some kind of, ink
David Bau: and just tried to probe it from that token instead of the entire letter containing it together. So you can see that there's, like, a heat map of…
David Bau: The class probability of… classifying
David Bau: the letter K is the letter K. And you can see that it's… it… that letter information is also just, like, within the token level, not just at, like, the patch level.
David Bau: And so the tokens are the tiny little…
David Bau: Like, pixelated things, is that right? Yeah, so the way that the, like, image is patchified within, like, the transformer diffusion model, it's like, it comes out to, like, 16 by 16 patch, something like that. Yeah.
David Bau: Wait, each, each letter is 16 byte? Oh, each, each token.
David Bau: Oh, each token is 16 by 16 pixel? Yeah, 16x16 pixel.
David Bau: So, okay, so what are you showing in the middle and versus the right? Yeah, so one thing that… so the next thing that we thought was…
David Bau: We were trying… we were interested there in some kind of attention pattern, or trying to find some kind of attention heads that
David Bau: does this sort of copying task, like, we were… we're fairly certain that there's some kind of attention pattern that is able to say there's some capitals here, this is this particular font, or it's this letter. So the idea here is that we just have some kind of target mask, so we have some mask that's, like, the letter P,
David Bau: Here, and then we're just gonna optimize… we're gonna…
David Bau: produce attention patterns for each head, and find the ones that maximize the F1 score that max… that maximize the target mass, so it's like, we're basically finding the heads that attend the same letters.
David Bau: Within an image. So, you can see on the right that, like, the blue square, it's kind of hard to see, but on the bottom left, the P,
David Bau: that token is attending to, like, the… on the top row, there's another P.
David Bau: on, like, that specific token. So…
David Bau: That was one thing that we were looking at, and… If you looked at the same head on one of the other letters, then it would find the other instances of the other letters, is that the idea? That's the idea, yeah. But we… it's kind of noisy. I haven't been able to, like, visualize it fully, and we… we also have, like, there's a lot of… there's, like, a large combinator of space, I guess, of, like, oh, like, is it the same…
David Bau: like, varying the font, or, like, it should have 10, maybe? Like, could you find the one that tends to take both capitals and lowercase? Like…
David Bau: Could you, like, along those lines, I haven't, like, gotten to, but this was just, like, an initial, like.
David Bau: Concept that we did.
David Bau: It's really cool.
David Bau: It's nice that one little hat… one little token does have the letter in it, otherwise it would have been… I don't know how we would have figured out how to, like, concatenate all the different sides of the letter, so you can just look at one token, and it actually is, like…
David Bau: Can you tell which token ahead of time?
David Bau: Can you tell which token? Yeah. Well, I guess what, what this was showing is, like, is, like, any, any of the little tokens in the letter… Oh, any of them would work. Yeah, well, will… you can probe that letter, which letter it's in. So they all… like, this disjuncture of all the tokens were. Yeah, that's what the left… Cool. That makes sense.
David Bau: Nice. Very cool.
David Bau: Got it, that's mine.
David Bau: This is… so, some context for people who are new. I… we trained a…
David Bau: Small transformer to learn, like, abstract algebra problems?
David Bau: And…
David Bau: So one of the things that it learned how to do was, like, solve the identity, cases. So the identity is, like, something at times…
David Bau: Like, A times X, if A is the identity, like, you get back out X.
David Bau: And so the way that the model does this is…
David Bau: You can see these… there's these two attention heads up here. So one head, the yellow one, promotes, like, both of the answers that are in the question. So, like, A and E.
David Bau: And then there's another head that demotes the… identity answer, so…
David Bau: instead of saying, like, A and E, now we just say, like, A times E, we get rid of E, so now it's A.
David Bau: So I thought that was a pretty elegant solution, for, like, how this would work. And then, on this, this little,
David Bau: 3D plot here is kind of showing the model has, like, a clean separation between how it, like, how it represents identity facts versus non-identity facts.
David Bau: So, like, it clearly is, like, dealing with them in a different way.
David Bau: when… this is when you do PCA on the, hidden states.
David Bau: And then, on the bottom is just kind of an example of, like, how we're verifying that this is what's happening.
David Bau: So the top sentence is what the model normally answers for, like, a query DF, and so this one turns out it's not an identity question, so it spits out E.
David Bau: But if you steer in the direction, like, that moves stuff towards the identity facts, then the model starts putting out both of the answers in the question, so D and F.
David Bau: And if you insert, like, a fake fact that happens to demote one of the symbols.
David Bau: Because it thinks… the model thinks one of those is the identity.
David Bau: Then it'll drop out the logit for the identity case and promote the other one.
David Bau: So, like, in this third case, it's… we're sticking an identity fact that makes the model think D is the identity, and so the logit of D drops down, and then it predicts F, and then if you do the opposite, where you promote… you make it think that F is the identity, then the logit of F drops, and the model predicts D.
David Bau: So yeah, I thought it was a pretty elegant, way that the model solves the identity.
David Bau: That was pretty fun. So we threw that in the paper. Thanks, Rohit, for…
David Bau: Figuring that out for us.
David Bau: That's very cool.
David Bau: Have you tried steering the other direction? Yeah, so if you go in the opposite direction, so, like, if the model's going to predict an identity, and you go in the opposite direction, then it'll just predict something else that's not. Kind of random. Yeah, interesting.
David Bau: You can also control it. So, for instance, you have a very good fact where the
David Bau: Final answer is not an identity.
David Bau: and you introduce a very out-of-distribution false fact, saying one of them is identity, the clean model would say, oh, I know this, this is an identity.
David Bau: But then if you steer it in the negative direction, it forgets this false fact, and starts predicting the right answer again. So it's, like, more controlled. That's awesome.
David Bau: I like this appearance of this switch. You know, you see these switches, you know, conceptually all over the place, but this is, like.
David Bau: There's a single MLP, Because this model is so small, that when you steer it with this one
David Bau: global vector, so it's a fixed vector, right? The screen vector you've got.
David Bau: It has this effect.
David Bau: On this one specific attention head's output, where that attention head could have a lot of different outputs.
David Bau: Right? And so, if you think of it like this funny transistor, you know, you've got this signal.
David Bau: It's, like, transmitting lots of interesting, different types of information down this very specific wire.
David Bau: Right? And then you've got this one-dimensional little switch that seems to be able to turn this signal on and off, right? They can either let it pass through or not.
David Bau: I think it's really interesting, I think, that
David Bau: Yeah, at some point, if anybody's interested in, like, how the heck an MLP, like, a single MLP layer.
David Bau: Does that acts like a transistor, like this? Might be…
David Bau: worth looking at it in detail, maybe making more plots or something. I think that it's cool. I don't think we've seen
David Bau: You know, an instance of a neural switch, like a neurotransistor, that's as clean as this one.
David Bau: You kind of need to understand it.
David Bau: Okay, next one.
David Bau: Oh, this is mine.
David Bau: So, this stuff is from a paper from…
David Bau: neuroscience, and I speak about the confusion in the… in the literature about the concept of theory of mind.
David Bau: that there are a lot of papers that uses different terms. Sometimes it's called theory of mind, sometimes it's called mind reading, sometimes it's called empathy, etc, etc. And also, they try to measure different things.
David Bau: With different measures, and each
David Bau: Each component, each paper treats different components, and each component is measured by different, ways.
David Bau: And they tried to organize and to, make a concise,
David Bau: minimal criterions for what is theory of mind, or empathy, for mind reading. And, they mentioned two minimal, criterions, the non-merging and mentalization.
David Bau: Non-merging means that the ability to separate the self and others, to understand that
David Bau: I have a different perspective from others.
David Bau: And the mentalization is the ability to mentalize, To have a different perspective.
David Bau: to take… To… to…
David Bau: In Hebrew, there is a term to get into the shoes of a different person, to… to be able to look at the world from a different perspective.
David Bau: So… Thanks, thanks.
David Bau: So…
David Bau: So this came from which, which, which literature? It was psychology, you said, or… Yeah, it, it published in the… the researchers, Neil Scientists.
David Bau: I'm not mistaken. And, it was published in a psychology literature.
David Bau: Excellent.
David Bau: So are they kind of viewing this as, like.
David Bau: this… the criteria… the minimal criteria that they come up with are, like, the floor, and then, you know, it's okay to add in other things in addition. Or are they viewing it as, like, no, all it is to have theory of mind is just this minimal criterion, and when people add in other things, that's, you know, superfluous.
David Bau: So they mostly concentrate on if someone is, having
David Bau: suggesting a new measure that do not include those criteria, so it doesn't, mean tier of flight. Or maybe someone is measuring something that, is sub-levels, before those,
David Bau: Before even reaching those criterias. So, just pay attention to this. Okay, great. So they're saying these criteria are necessary, but they're not making a sufficiency claim. Yes. Okay.
David Bau: Thanks.
David Bau: Okay, so this is,
David Bau: also the same thing that Adrian was talking about, except from the other angle. So, we're, so we're working on this project of trying to understand how… how flux represents, like, letters, or, like, text and images. So, what Adrian was looking at was in the image space, but
David Bau: This week I was also trying to look at
David Bau: just, like, in the text embeddings. And so, yeah, like, and so when… when you're conditioning
David Bau: flux, you could, you know, ask it to generate an image that's just, like, banner with text that says… and then give it, like, a random token. And at least for the examples I've tried, I've tried… I've been trying to do… stress test this a little more. It's, like, usually really good at, like, actually
David Bau: Putting all the right letters in there, even if the token is kind of, like, random and seems really unlikely.
David Bau: And so I was wondering if that means that it's actually storing
David Bau: the letter information in, in, like, the text embeddings, because I feel like it would be a lot easier if the, you know, the text embeddings already have that, and so…
David Bau: I was trying to probe to see if that's the case, so I was just training, like, a little, like, logistic regression thing,
David Bau: for each letter, just to say output 1 if the letter is in that token. But actually, it's, like, not very good
David Bau: at this, I might have not had enough training examples, so I'm rerunning it again with, like, like,
David Bau: triple the training examples. But at least, like, when I ran it this time, the test accuracy, or the test F1 scores were, like.
David Bau: You know, the max is, like.
David Bau: Maybe .6 or so, and then, like, on average, it was, like, 0.2 across all the letters.
David Bau: And so, I'm not sure if this just means, like, I have to do a better job
David Bau: probing, sometimes with more examples, or if it means that the model doesn't actually have… or, like, the text encodings don't actually have letters in them, and maybe the model
David Bau: Either in the image side breaks it down into letters, or, like, it doesn't even need letters, and it's just, like, memorizes what every single token looks like.
David Bau: without understanding the compositional aspect of, like, oh, all these tokens are actually made of the same units, basically. And so, yeah, that's the…
David Bau: The challenge is to try and figure that out.
David Bau: Because we know, at least on the image side, from what Adrian was showing, is that
David Bau: It does actually… it does represent these letters, but yeah, I don't know.
David Bau: Have you ever thought about the letter forms of the FedEx logo? Have you ever looked at the FedEx logo?
David Bau: Oh, you mean with the arrow? Yeah, you notice the arrow, right? It's in the white space between the X and the…
David Bau: Right, whatever the letter it is, right?
David Bau: And so…
David Bau: Yeah, so that's… that's what I wonder, when I look at your, your, sort of, your, your probe that doesn't have anything, I wonder.
David Bau: Oh, does this mean that the model…
David Bau: isn't thinking of KTE in terms of the black part of the letters.
David Bau: Maybe it's thinking of it in terms of that funny white space between the K and the T?
David Bau: Maybe it knows the shape of that, right? Kind of like the way you notice the arrow and the FedEx logo? Yeah. Which, it'd be really funny, but then it would have to do that for all 30K tokens, but maybe it did do that. Or maybe it's just, maybe instead of having a big table of what
David Bau: letters look like. It has a table of what bigrams look like, and it says, oh, this… this token has these two bigrams, and it has a KTE bigram, it has a TE bigram.
David Bau: Oh.
David Bau: Yeah.
David Bau: Yeah, I don't know, it's just…
David Bau: There are clinical tests in the psychology for a Gestalt. Gestalt? Yeah, and they are concentrating looking at the… the shape, or the… what's around. Yeah, yeah. You have the same, like, text and, like.
David Bau: like, in different ways of presenting, like, you know, a book with a tin, like, that says KTE, and then you take the representation of that, whether it, like, focuses on the KTE, let's say capitals and gas, and to give KTE capitals was the most…
David Bau: focuses more in the background, again, be able to upload.
David Bau: I'll duplicate it.
David Bau: You're saying that… You're looking for some, information on the embedding, so that if the embedding is actually going to the profession of the learners.
David Bau: So, is it… is it more like, let's say I have two tokens, one is butter, one is shorter, so if, they are predicted into the embedding space, so they're… the… if they contain information about the letters in the embedding space, so they might be very similar in that space. Is it something like that?
David Bau: So is that, yeah, I guess, I guess what we're… we're wondering is that it's in some… on some level, yeah, if, like, you have two tokens… Because the only difference between butter and shutter is B, it has a B letter in the front, and it's an SH at the front, and there's all of the letters are the same in the token. So, there's a chance if there is…
David Bau: if, like, tokens are represented in the embedding space, so those two embeddings will be very similar. Yeah, at least in that subspace. Yeah, yeah. So, yeah, that's kind of what we're… we're… we're…
David Bau: Would this be just to a single character, like, a banner with the text that says K, and then tramp over K.
David Bau: Yep.
David Bau: Well, I guess if you did that, though, then it would just be…
David Bau: There's only one token for you.
David Bau: Like, if you just have one letter, Then it's like…
David Bau: I'm not sure. For that simple case, would the regression work?
David Bau: Like, if you just have a single token in the banner.
David Bau: Yeah, I mean, I could try… you mean that just as, like, a sanity check, kind of? Yeah, so… That's what you did. The way that Flux would look at KTE,
David Bau: is if, however, the T5 tokenizes KTE, right? So, it's possible that it's not KATNA. Yeah, right, so this is one token. KTE is one token. And then you probe for KTE? So, yeah, KTE, this is, like, I only use single token, single token words, and then probe that single token.
David Bau: Oh, you're just probing for KTE here. It's not K and P and E. So, KTE, but then have, like, a probe that is, like.
David Bau: A probe for K, and then a probe for T, and a probe for E, yeah, yeah.
David Bau: So, like… like, so far, I was only restricting it to token… one… a single token, and then just, like, the probe should output 1 if that letter is in that token.
David Bau: But to the model, it's just, like, one token. So from Adrian's experiments, or your previous experiments, it's highly likely that if you just have a single token, like K, and probe for K, it should work, right? Yeah. Like, you have some attention layers that look at it. Yeah, I think so. Then do you think it's just the problem of tokenization?
David Bau: Yeah, I guess it's a… it seems like…
David Bau: Yeah, I don't know, but I don't… yeah, like, because, like, you know, with strawberry, like, models, the strawberry thing, models don't know how many R's are in strawberry, so, like, they don't know all the letters, and… So, when you're looking at the token embeddings, you're… you're… are you looking at the T5 token embeddings, or are you… so this is… you're looking at the very first layer, right?
David Bau: Yeah, so, actually, so I was trying it at every layer. This was the best layer, but most of the… Oh, which layer was this? This… I wrote layer 15… Oh, 15, pretty far in.
David Bau: Right, and, like, the non-single block ones, but honestly, it was kind of around this for most of the layers, it was like. And so…
David Bau: Yeah, so I was assuming, like, I was thinking maybe T5 doesn't have it, but then once it goes through enough layers of flux, it has it, but even then it doesn't really work. Right. Makes sense. Makes sense. Yeah. It's disappointing that…
David Bau: I mean, you'd think that the T5…
David Bau: Embedding should know what letters are in each token, like, it should be in there.
David Bau: Guess it doesn't…
David Bau: Yeah. Makes the transformer learn it. Yeah, makes Flux learn. Yeah. But then it… yeah, but then it doesn't seem like… but maybe also I just… But then Flux also doesn't learn it. Yeah, but maybe I just need more data, so I'm rerunning, rerunning it. Oh, I see, just make it bigger. Yeah.
David Bau: training with more data, like, are you, like, are the tokens that you are choosing, like, especially towards the end, like, what if you had the same thing, saying, like, PATE, text written on banner, and, like, in that position, because that one gets more focused on the…
David Bau: That might be a good way of getting more… more examples out of it, because right now I basically…
David Bau: I'm limited by how many tokens there are and stuff.
David Bau: That would be good, just to bury it like that.
David Bau: That might also be open.
David Bau: One last idea. It might, like…
David Bau: The model has to know that it starts with K, and, like, the second letter is T, and that the third letter is E, right? Like, not just that it's a bag of words containing K, T, and E. Right, right, the order. So I wonder if, like, probing for, like, starts with K,
David Bau: Or ends with E, or the second letter is T. Yeah, it's a positional… I'm thinking of, like, if I'm a language model, I might, like, I don't know, like, have the first letter in this space, and the second letter in this space, but if you pro for, like, contains K, these might be represented, kind of.
David Bau: differently from the perspective of, like, a linear probe. Sure. Yeah, nice. I am actually running for the first, first letter right now, so hopefully… I hope it works. If it doesn't work, I don't know.
David Bau: Sheriff also said that PTE is, like, one token itself, so maybe it's how they don't consider it.
David Bau: Guaranteed.
David Bau: Yeah. The information has to be in there somehow, right?
David Bau: Yeah, it has to somehow know I have to take this one token, and now I have to write K and T, then…
David Bau: That's great.
David Bau: I love it.
David Bau: the information has to be in there somehow. Where is it? Not there. There's no… it's not… Whoa, what happened? Oh, Nikhil! Nikhil, Nikhil's drawing… Nikhil's drawing a blank. I just had that thing.
David Bau: Oh, oh, you put the same screenshot I did. Is that right? Yeah, it's just my website. Congratulations. I did not add any thoughts. That's okay. That's totally okay. It's great.
David Bau: All right. Oh, Rohit had something. Oh, yeah, so this was, this is something that I very briefly spoke about in my… in the previous slot on…
David Bau: So that…
David Bau: The current work was, like, you can train a meta network, like a feed-for-row network in the blue up there.
David Bau: Which takes a reference image and a text description of your reference image.
David Bau: And it gives you, like, a bunch of vectors, which you can pick, and then use it in your diffusion model to customize.
David Bau: this, particular reference image that you have. So, for instance, if you see on the top row, those are, like, some of the reference images I used, like, a dog, basically.
David Bau: And I take the delta for that dog token and plug it into my inference prompt when I use the prompt photo of a dog swimming in the ocean.
David Bau: So what I was hoping for is that I would get the dog in the reference image.
David Bau: to, start generating in the, the diffusion models inference pipeline.
David Bau: And it works pretty good, like, so once I trained this for, like, some 3 days compute.
David Bau: It starts learning what dog it has to draw, and it has a pretty good generalization of putting it in different instances.
David Bau: But then, my training data did not have, like, the right… I didn't really check it… really good, and it had a woman in it.
David Bau: And even though the prompt was photo of a dog, and I give a reference image of a woman.
David Bau: It somehow just plugs in a woman, and then… put certain…
David Bau: in an ocean, swimming. So the… the left column are the actual images generated by flux.
David Bau: And everything afterwards are just these customized vectors plugged in.
David Bau: So, even though the model has a heavy bias to draw a dog, because, well, I asked for a dog.
David Bau: And I give this, this strong feed-forward networks vector to the diffusion model.
David Bau: It's very happy to just change it into a woman and then keep, woman in different…
David Bau: poses, so it's… I think this was very interesting.
David Bau: Yeah, I don't know how much of it, like, what to take out of this, even though, like, I would call this, like, some heavy de-bysing of concepts, so it's…
David Bau: the model should have a strong opinion on what a dog should look like, but this one single vector, if it can turn this bias away, I think it's pretty interesting.
David Bau: but yeah, that's… That was a funny…
David Bau: observation… Does it work for anything? It works for anything.
David Bau: Yeah. So do you try… so you feed in a different picture? Yeah, I feed in a different picture of a woman or a man, and it just works. And it can just make that the dog? It can just make that the dog. I see. And, if you put in something that's not, like a living thing? Oh, this hot dog, yeah, it's for a hot dog itself, right?
David Bau: Ouch.
David Bau: I'm just curious, like, if you…
David Bau: Like, larger dogs? Smaller dogs? Color dogs? Nope.
David Bau: Yeah, I had more from, like, previous…
David Bau: a lot of fun, like, from a previous Friday meeting, there were, like, other large dogs, different breeds, and it still works.
David Bau: Yeah, yeah.
David Bau: But it only works for dogs, because I trained it only on dogs. Yeah. And the backgrounds don't make a difference. Yeah, backgrounds don't make a difference. So the new set of experiments we are running is…
David Bau: So, for the reference here, I'm just using image of a dog.
David Bau: But I can also use a reference prompt called image of a dog posing in front of a background.
David Bau: And now I can take the background token and plug it into any different generation, like a person walking in the background. It could bring the background from the reference image.
David Bau: And drop it into the… yeah.
David Bau: Have you tried using, like, multiple dogs in the referencing?
David Bau: Yes, yeah, it usually draws the… what… how many other dogs are there. Oh, so if you have two in the reference image, it'll draw… Yeah, but you can… so the way we build this feed-forward network is, basically, it has a cross-attention. Okay.
David Bau: You can literally block the cross attention for one of the dogs, and then it just draws the other dogs in it.
David Bau: Oh, I see, so you could…
David Bau: Yeah, that makes sense. So you could supplement it with a pointing user interface or something like that. Yeah, yeah. That's interesting.
David Bau: Nough.
David Bau: That's interesting.
David Bau: Yeah, it's just ready to be a product, I think. I think so, maybe close to a dog product. Why do you say it only works for dogs if it clearly doesn't only work for dogs?
David Bau: I mean, I… this should have some bias towards dogs. It's only trained on dogs. The network has only seen dogs.
David Bau: So I'm guessing whatever attentions it has learned, or whatever…
David Bau: MLPs it has learned is…
David Bau: probably fine-tune towards dogs. Yeah, like, dogs have so much variation that, like, I would expect an abstract…
David Bau: Enough away that it would just work.
David Bau: Maybe. Yeah, I should, like, take a totally different reference image and reference prompt, and see how it works. I haven't done it yet. I was just…
David Bau: Now you guys are thinking kind of like machine learning people. So you train it only on dogs, and maybe you're done. Maybe you don't need to train on anything else, right? Because, like, maybe now it knows everything. Like, you could put a spaceship there, whatever, you know, right? Because, like, you know, it's like profound generalization, right? It's like construction training.
David Bau: You just train it on, you know, 10 different types of instructions, and then it can…
David Bau: You can do anything, right? Because it's learned the general idea of an instruction. So now it's just learned the general idea of just substituting the main subject.
David Bau: Right, maybe.
David Bau: Maybe, maybe you're done. So, I should say, the feed for our network was initialized from some of the diffusion model's weights.
David Bau: So the attention and the MLPs in it are initialized from diffusion model. So it's possible that it knows all of this. Did the previous already knew? Yeah. Yeah. Well, you know, you have a base model that you're… I mean, you're… that you're modifying that knows everything.
David Bau: Right, yeah, even different models are things, right? So…
David Bau: But the vector has to come from feed-for-row network.
David Bau: Like, for the customization. Right.
David Bau: Right. That's interesting.
David Bau: Okay.
David Bau: That's… that's… the one is more kind of recognizing what's there as the main object in the image, and then it's using the tool
David Bau: project on different backgrounds. That's why, like, when you even put a woman in the image, that's the main object. The woman is the, like, the main object in the image, and when you ask it to put it in the ocean, it just often extracts the main object and put it in the ocean.
David Bau: Yeah, it could, yeah, it could potentially have thought dog as a main object, right? Like a foreground. It can be very bad, like, if you put a cat or another person, or maybe something which, like, kind of is in the foreground, in the background, and there's a clear contrast between them, and if
David Bau: Then it puts that also in the ocean, that can, like, kind of inference that it's generalizing on the foreground object and putting it in the background. Yeah, I think so, Tonya. So, yeah, it works for all living things.
David Bau: Animal… Yeah, we're not sure about non-living things. Yeah, if the non-living things work too, then potentially I could say it's a foreground background.
David Bau: I feel like another axis of generalization.
David Bau: Right now, you always put the backlers in the dock.
David Bau: tax tokens. Yeah. So how about if you generate under…
David Bau: Latmost colony during a photo of a horse, a photo of a jazz.
David Bau: And apply the vectors… Of the dog into the hat. Use the dog model, but still, but apply the vectors to the cat. See if it still works. That's, yeah, that's interesting. I can try that.
David Bau: There's… maybe it's just a sense of, like, you just need to put the vectors in some token where it represents subject. Yeah.
David Bau: That way we can just swap the subject along.
David Bau: That would be interesting. I think so too, yeah. Or, like, you get a blend of a cat and the dog in the picture, maybe. Yeah. Which would be nice, too.
David Bau: Are you testing the limit and see where it fails. Yeah.
David Bau: I thought you were suggesting take this dog token and put it in…
David Bau: like, the photo or, like, a swimming token and see how it works. Oh, yeah. That doesn't work. Yeah, I… Yeah, like, dog has to go into a dog, but I haven't tested dog going into, like, a totally different token. Yeah. I can try that.
David Bau: Oh, that's… oh, is Rohit the last, or anybody else with a last-minute update? Okay.
David Bau: Thank you very much, everybody. Do you want to quickly talk about the project's website? Oh, the project's website, yeah, okay, so yes. Thank you.
David Bau: So… Cool. So the, So…
David Bau: Now that, you know, ICLR is finished.
David Bau: we should… we should do some housekeeping on our… on our lab website. And so, there's a new subdomain that's not really linked anywhere, but if you go to…
David Bau: Projects, spaoulab.info.
David Bau: then Avery has put together this really nice
David Bau: Website that has a card for every, page.
David Bau: That has a,
David Bau: you know, has a webpage. Most of our project pages are not linked anywhere from the homepage on BioLab, and so I'm going to add a link to the projects
David Bau: website prominently so that people can find all the projects. But right now, this is not in a great state, because some of these things are just…
David Bau: You know, just a citation with no text.
David Bau: You know, other ones, you know, we have this nice text that you can kind of see with the thumbnail. And so what I want to ask everybody to do is.
David Bau: You know, make sure your thing is linked here.
David Bau: you know, copy from your abstract or whatever, right? Just some nice piece of text, find a nice thumbnail that you think is beautiful, you know, lay this out. There's instructions on the, on the GitHub, so actually, what is it? So it's… is it under my…
David Bau: username? Yeah.
David Bau: So, I can add you all to this GitHub that you can push to. What's it called? Projects? Projects Web? Projects Web?
David Bau: Something like that, so…
David Bau: Yeah, so actually, you know, after we wrap up here, you can just give me your GitHub user IDs, and we'll just… we'll just add everybody to this GitHub.
David Bau: And you can push… there's, like, you can push your modifications to this thing. It's a… yeah, do you want to… really quickly… so it's a… it's a YAML file, YAML…
David Bau: Oh yeah, it was gonna be too scary.
David Bau: It's not scary. Oh, wow, look, it's really nice, look at this! There's this? Wow.
David Bau: So nice! And so let's look at a fully populated one.
David Bau: Oh, there's this description, there's this extra… wow, what, like, this is really nice, like, formatting here.
David Bau: Concept sliders. You put the thing in… Yeah, so, there's, like, 2 or 3 things you need to be careful that are all, I think, in the README. If you have
David Bau: Colons in your titles, which is very common.
David Bau: That is a reserved. Oh, you can do that. Don't put colons in your title. No, no, it's fine, it's fine. It's fine. Oh, you shouldn't, as a writing, as a writing? Oh, okay, okay, sorry. Okay, well, sorry. Many, many titles of papers have colons in them, I have found, and so that ruins the YAML formatting, unless you put it in quotations. That's easy to do.
David Bau: Just, just, just be aware.
David Bau: I see. And then how do you… and then after you do this YAML, then you just, then you have to run Python generate, that's it. I see. So if you… if you don't generate it, then the actual HTML won't…
David Bau: won't publish yet. Isn't this just because of your complicated box that sucks up those. I'm that complicated.
David Bau: Not that complicated.
David Bau: That's it. And the URL just has to be, you know, access… the image URL has to be accessible, and the projects
David Bau: URL, just… Link to that, and you're good to go.
David Bau: So should we put, like, a… I don't know if we should put full abstracts here, but maybe a few sentences or something? Yeah, I would… so I would make it snappy. You can… so it actually can be… so that's what the extra thing is for?
David Bau: So there's, like.
David Bau: keep it really snappy for the main description, but then if you want to just keep on saying stuff, you can put in an extra, and… and I… actually, I've done that all over the homepage, and if people, like, click on… Yeah, it clicks on the more, and then… right, so you see what I mean? So, like, yeah, that's the more thing. I don't know if you know this.
David Bau: Notice this funny more UI that we have.
David Bau: It was coded very strangely, I have to say. It is, it is. It is very strange. No, it was fine, it was fine. The selector was very strange. But it's… yes, it's designed to work very well.
David Bau: like, if, if people browse around with JavaScript disabled and stuff like that, then it, like, continues to work, right? It's very old-fashioned.
David Bau: Question about the projects. I'm assuming that if we add our own information, we'll just push it into a new branch, and then
David Bau: What'd you say? Like, if we all decide to update, like, the projects page, then, like, I assume each of us have to just push into a new branch, so… No, no, no, just merge. So, like, you know, it'll say, oh, you have merge conflict, because somebody else was, like, just pushed 2 seconds ago, so then you gotta merge, and then it'll be okay. It's, like, since the almost pretty clean text.
David Bau: It'll, it'll merge cleanly.
David Bau: And you know, and if you merge wrong, then we can pick through the…
David Bau: the good history. You know, find it, find it again. Yeah.
David Bau: It'll be okay.
David Bau: Yep.
David Bau: So, yeah, so update all that stuff, you know, sometime this week, and then we'll… I'll update the homepage and link it in.
David Bau: Stuff like that. We gotta get this website up.
David Bau: I'll… I'll rationalize. Thank you very much. It's roughly in… it's roughly in, reverse, or, like, newest first, by… by year, but that's… that was a complete guess, so I… Yeah, and we put links to everything, so if your project is secret, you can… then you can, like, delete your thing, it's fine. If you want to keep it secret, that's alright, but… but, you know, you can…
David Bau: Put all this stuff in.
David Bau: Okay?
David Bau: Alright.
David Bau: Okay, cool. So that's it. Thank you, everybody.
David Bau: Great.
David Bau: When I steal them.
David Bau: Yeah. I think… yeah, you're right about that. The loss.
David Bau: The ammo's beautiful. Oh, wow. That is so nice. Euclidean Lost Distance is less than ever. Exactly. So it's like… They're all going out.
David Bau: Absolute value. Yeah, the ones with… Yeah, that's right. Oh, when you said, like, epsilon ball, I was thinking of, like, epsilon or parameter space. Yeah. I guess just put it under…
David Bau: I was curious about this. I fully understood it. Yeah, I'm looking into it. Like, yeah, if you're curious, it would probably be helpful for me to try to explain it. I love it. Yeah, yeah, yeah, of course. So, yeah, if you want to take me up on that, like, I'd love to in the coming weeks, yeah. Especially the more mathematical part. Okay.
David Bau: I like math. Yeah, it goes pretty deep, so I don't know how deep I will get, but… I could serve as a reasonable limits test for, like, some of the stuff. I don't know what the actual theory is for it, so I don't know, like, what branches of math or whatever, but… It gets into, like, algebraic geometry, which I'm sort of avoiding.
David Bau: That's kinda cool.
David Bau: Like, yeah, I don't know, it, like, spans a lot of stuff like classical statistics. Not really.
David Bau: But yeah, but… I don't really… Algebra's the part that I, like, don't have a lot of… If it's, like… If you try to learn it… I haven't done algebraic geometry, but I've done a fair amount of algebra, and real analysis, and whatnot, so I could probably at least, like… Like, I certainly would be able to, like, probe you in a way that you would have to think and answer, and maybe even just, like, help understand it.
David Bau: Yeah, yeah, that could be helpful.
David Bau: Cool, yeah, maybe in a couple weeks when I'm, like, ready to be, tested. Yeah, go do, like, a whiteboard session. Yeah.
David Bau: Right.
David Bau: Perhaps.
David Bau: Nope.
David Bau: That is it.
David Bau: That was a local Thursday.
David Bau: Oh, internet, so they have San Jose.
David Bau: Oh, okay. Do you want to pay street, David? No, I'm good, thank you.
David Bau: profit.
David Bau: Love it.
David Bau: I'm happy. Yeah.
David Bau: I mean, I don't… Yeah, I don't try to embarrassing.
David Bau: Also, what do you mean earlier?
David Bau: Oh, okay, great. Well, I, yeah.
David Bau: Always…
David Bau: Nice results, I like it. Thank you. Yeah, that's interesting. I think that's a good setup to be in. Is this somebody? I don't think so. I think they were just plating them. I don't learn what they've got.
David Bau: Yeah, so I was… yeah, I was thinking, if you need any collaborators within the lab, I'd be happy to join. Yeah, I would love it. Though I… I will… not really sure how much I can…
David Bau: I mean, if you just want to come to the meetings, like, that'd be great. Yeah, so actually, the reason for me to… or the motivation for me is, I think I started thinking for Auto Interp.
David Bau: Oh, yeah, yeah. And I think our project seems… is probably the most auto-interp we are doing in the lab. So, auto-interp…
David Bau: I guess I could see that as, like, agents, or I could see that as training the audience? I'm broad with the definition. It's not really agent, I get, I get it, but I'm… Yeah, yeah. I'm not any… I'm not…
David Bau: excited with any of the attractions. Yeah, I think it's… it's…
David Bau: Did you see that paper that I put in last night? The, the linear direct… or the language models and code?
David Bau: Training data recency in, linear direction? No, I just saw the title, that's it, yeah. I didn't really read anything as real. What's that?
David Bau: I feel like it's probably, like, kind of trivial. Do you feel like it's real? What's the result? Like, the training data which comes later? You can pro… you can train a probe to find out, like.
David Bau: when this current data came in, when it's great.
David Bau: Yeah. And so they do… I haven't read the paper, but they do, like, a synthetic setup, where they have, like, 6 different runs of, like, you know, they vary the order.
David Bau: But… and then they also find that they can train those models to self-report.
David Bau: Alright. What? Now the ICLR is over. But, I guess, like, that seems like an interesting… Like, if you train the model to, like, self-report enough interesting things… The one that has the most, like, a general frequency. But there's never an hour block. Yeah, how did you train the model?
David Bau: So they… I… the impression I got is they were just, like, they took a model that had been trained in some order, and then they just, like, took a bunch of training data, and they were like, okay, entity…
David Bau: you know, 75… Internet's so bad, right? What percentage of the way through training did it come? And that's just a fine-tuning data set. So it's like Michael Jordan, you know, 75. Oh, okay.
David Bau: Yeah, see? If we pick any day at Wednesday, I don't think David is available on Wednesday, honestly.
David Bau: I imagine… I imagine, like, what actually happened to something, or, like…
David Bau: Like, something, like, trivial, but they say they tested for a bunch of stuff. Okay, so the fine-tuning one. We're after that.
David Bau: Basically, they sample some examples from their training data, and they have the index of that training data, their entire data set. I don't know whether to index or persona. Something like that, some quantization of where exactly it is in the training data, and then they are fine-tuning another model. I think they're fine-tuning the existing model, but that's… so the idea is, like… Oh, they're fine-tuning existing
David Bau: Because it's more interesting if it's self-report, right? Okay, yeah, okay, so it is able to self-report itself.
David Bau: Okay, I have a meeting. Yeah, if it was another mode, it would just be random.
David Bau: Cheer.
David Bau: And, yeah, like…
David Bau: Yeah, okay. It might be related to what you were saying, the layers thing as well. I don't think there's really any good… Oh, right, right. Where exactly that thing is included in the layers, like, at which layers the… I don't know, like, these three, like, in between… And sorry, and what about the probes? I did not get the probes part.
David Bau: the… just that they can, like… Oh, and they can bring the folks as well.
David Bau: Oh, okay.
David Bau: Yeah, then if that layer thing is… This is the next good alternative. Yeah, so then you can see the layers where the probes are actually effective. Yeah, that would be cool, yeah.
David Bau: That seems like a proven thing. But so the auto-interpreter that you're interested in, is this, like…
David Bau: Like, training a model to be faithful. Self-report.
David Bau: I've not thought through it, but my, my, my general… the place where I'm coming from, I think from…
David Bau: the projects that I've done so far in my PhD, I think those have been very labor-intensive. I have to do a lot of work to get it done. And now I feel that maybe I can still do it.
David Bau: But… But I can do it now, I think, so I feel confident doing that. Yeah. So it's probably try to do something new. I feel automation is important there. See, but that's, like… I don't know, Chris.
David Bau: I think it's a little bit odd for… because, like I said, Atticus is joining the fire engine. It seems a lot easier outside of academic labs, because it's so software engineering-y, you know?
David Bau: This is true. Not to say there's not interesting stuff. Yeah, but I feel that the space is open enough that we can contribute ideas even from academic lab. Yeah, definitely ideas. Yeah, Ravik is, I think, putting in… Yeah, I think they hired… I think Adidas was leaning towards that. His motivation is probably similar
David Bau: to my, I think, when I was working with him in my previous project, we were talking about this thing as well. Yeah. Because these things have become, like… Yeah, you could automate the scale, yeah, yeah. I think he, he, he…
David Bau: He's betting on that now, basically.
David Bau: Yeah. Yeah, how would you do that? Wait, who's for the Thursday morning? I mean, the nice thing is, like, a lot of these techniques… Thursday morning?
David Bau: You just, like, the patching setups themselves would be obvious, but… Okay. And are they available… it's not available anytime in the… I mean, the negative thing, if you have a ton of RL, it's just, like…
David Bau: Jamie, not at all, aren't there. Like, one thing you do is something like Maya, but, like, I was thinking tomorrow she should try this, like, do Maya while… But then Tuesday, maybe? Just, like, do… or L on this process. Tuesday, did I say I'm unavailable at any time? Or… I don't know. Yeah, if you, like, propose a patching setup, if it, like…
David Bau: do RL on the book. Yeah, I was kind of thinking about that as well. Yeah. Maybe…
David Bau: We can train an RL agent, instead of just prompting a frontier model. Yeah, yeah, and probably the RL would be good enough.
David Bau: that, yeah, after certain training, we might just be at a good level that it might actually… Should we do on Tuesdays? Yeah, for one? Yeah.
David Bau: For auto, like, we were talking about auto… automated interpretability. Like, given examples, come up with an explanation. Sorry, what? Like, given examples, come up with an explanation.
David Bau: Or… Yeah. Like, propose an experiment, interpret it, propose the next experiment. Yeah, so basically automate my job.
David Bau: automated interpretability.
David Bau: Yeah.
David Bau: Exactly. But doesn't Maya already do some level of it? That is very neural level, very, very microscopic. What's Maya? Yeah. Like, John, like John only interface for, like, Monday to Wednesday.
David Bau: But it is iterative in nature. 15 people. It's not just…
David Bau: You just take the example, and you get a description? SAE is, like, I feel like you just do, like, get your max activating examples, and get, like, the auto-interp on those. So then, in the…
David Bau: Maya 1, I think, like, you start off, like, maybe by seeing which, like, what, I guess, property, I think the model might be looking at, and then I think they have, like, a description of all these different neurons that they try to then…
David Bau: go back and forth and experimenting. Yeah, so the main difference is basically the iterative nature of the… iterative nature… The task is the same, you have some essay feature that you want to explain. Okay, but they look at the neuron-level things. But yeah, idea is the same. So given a neuron.
David Bau: for which you can figure out its activations or different examples. Can you describe its functionality? Yeah. I think it's multimodal, right? Because you can give an image, right? Yeah, I think they were looking into it. They're not like, how does the model do, like…
David Bau: theory of mind task. No, no, no, that's the aim, that's the… that's the… that's the objective here, but… You're working on this? Like, trying to automate interpretability?
David Bau: I haven't started thinking about this, what we were talking about, at core. Yeah, I've started to think about it, started to think about it. Yeah. How do you create good training data?
David Bau: seem to be. So that's, what you're trying to achieve? You know, that's, like, what… when I was doing the Anthropic Fellows program, they had, like, an opt-in thing that was, like, record your screen so that we can train our automated interpret… er…
David Bau: automated alignment researcher on your screen computer usage. Yeah, no, it makes sense. I opted out. I don't know, basically everyone opted out. When I was interning with Apple, the first day, my manager said, everything that you do is Apple's property. Everything… Did you have to record your screen?
David Bau: No, I don't have to record my screen, but… They probably had, like, software on your computer. Who knows that? It was their laptop. Oh, yeah, yeah. They installed some bunch of software, which I barely understand. I might have been doing it, but… yeah, but I think that was the first thing my manager said. Everything that you do is basically logs for the apples.
David Bau: Crazy. I mean, it seems like the way you get good training data is, like, you… you come up with a nice… Yeah, maybe you ask, and then you can just, like, throw a computer. So… Well, I mean, like, you could do something like Maya, you could do something like…
David Bau: Even if… Christian Merrill, I think… Right, right, I know, but if you're, like…
David Bau: I don't know, like, let's take IOI. You're trying to do IOI or something. You're just like, here's some distribution.
David Bau: And then you're trying to come up with, like, a minimal circuit, or a minimal set of units, and you can just kind of search over that space, and, like, you have a continuous menu. Yeah, a little bit.
David Bau: He, yeah, he was recently mostly at Industry Lab, right? And now he is here, is that right? It'd be nice to, like, design some, like, toy models. Which course is it? Or something. Oh, wonderful! I didn't know he was teaching that. That's great! It's really, it's like… I was thinking about that.
David Bau: Like, last year, like, RAS, but… And so you could be, like, the closest I've seen is… And I talked to him about… What's called Interpunch? A little bit more natural. But the problem is, like, that's also using this, like… what's the… is it RASP? It's not RASP, but what they're doing is they're doing, like, interchange
David Bau: Where they generate a false model, generate, like, a circuit, and then, like, train this model to respect the interchange interventions, behaviors according to how you'd expect.
David Bau: And then also, like, behave according to, you know, if you throw on random stuff, it doesn't change. So it has sort of wrong traits. It's a little bit like… you're kind of restricted time. I thought it was tracer models, but okay, maybe I'm wrong. I don't think it's a tracing models.
David Bau: surplus training, so it looks more real or something? Yeah. Yeah, semi-synthetic. But these are still small ones, right? I don't know.
David Bau: I think the danger would be that, you know, we don't really know what the units or, like, the granularity of the causal model should be.
David Bau: Or maybe you're just, like…
David Bau: Yes, and much larger image models. Oh, no, so it's not true. They compare it to everybody.
David Bau: Like, these are the weight magazines. So it's like, this is one spot that was, like, distribution, right? Right, that's what I'm showing right now.
David Bau: Pretty sure the patient uses. It's metal.
David Bau: The other thing is just, like, training, like, small models. Like, how I learned Interp was, like, doing interp on toy models, which are trained on, like, easy problems. Or, like… Well, you know what?
David Bau: I don't know, things like this. You could train, like, I don't know, two-layer small ones that can do these with these, and then you can then actually, like, inverse sense, explore a lot of men.
David Bau: Yeah, monthly thing, right? A monthly thing, yeah. And so, is there, is, is it public?
David Bau: Go to his website. Okay. Yeah, so for each one, he, like, posts solutions, basically. Oh, well, that's nice. And he releases the model, etc. It has collab posts. And when you say solutions, it's basically supplements.
David Bau: to our plat-a-thon. No. The solutions are, like, not in the same, like, format or something. It's, like, every other week, we do… Yeah, in the same way that, like, not all papers written about interpretability are, like, the same format. The interface is not, like…
David Bau: a circuit of, like, these heads… Okay, then I have to look, try to address my product and project, and… Interminability seems messier than just…
David Bau: Okay. Now, I was actually thinking about you, if… if they actually had circuits… We might be able to use that. I'm planning that now. …evaluate a few things.
David Bau: few, few methods. You could turn some of them into circuit stifling.
David Bau: Yes.
David Bau: Deciding with the journal club timing? Yeah. Tuesday or Wednesday looks like the best time.
David Bau: Would those work well for me? Starting next week? Yeah, I think so. How does, like, paper selection work? So what we've done in the past is…
David Bau: We'll, like, pick someone to be in charge of the… that day, and they'll…
David Bau: either, like, pick a few papers that people can vote on, or just, like, pick a paper that they want to read.
David Bau: I don't think there's… you have a better suggestion for a setup, but… That seems pretty good.
David Bau: How do you assign people to days, like, round robin? Usually.
David Bau: I should probably do that early. That would be good.
David Bau: Look at all.
David Bau: the easiest way, I would say, to describe subjects is that, you know, like, the emergent misalignment figure, where they're like, we asked…
David Bau: researchers to predict ahead of time. Oh, yeah, yeah. Yeah, that was great. I would love, like.
David Bau: every week, I get a newsletter, and they're like, here's the setup, what do you think the results are gonna be? And then you predict the results, and then they tell you what the actual results are. Yeah. It's very hard to do that, to, like, yourself, because you're gonna be, like, contaminated by the abstract.
David Bau: Does that make sense? Shall, say that again? What kinds of things you're trying to predict? So, like, the… the thing we just talked about in the training.
David Bau: I would love to, like, get a newsletter every week, and they're like, here's the setup, we trained a bunch of models, we, like, trained creatives on this, we go, do you think this would be successful or not? Oh, I see. And then they give you the ground truth. Yeah. But it's always going to be successful, because it's, like, a paper, right? Well, right, so you'd have to choose stuff that's, like, it could go either way.
David Bau: Which I guess is harder, but, like, those papers do… you need some, like, false… you need some false results also. Yeah, I guess you could do it that way.
David Bau: Yeah.
David Bau: I bet you could set up, like, a… if you're not always reading the latest archive dump, then you could set up something to scrape
David Bau: Yeah, he had Claude to do it. And, like, yeah, just, like, set up a Claude instance that, like, scrapes, summarizes an experiment that somebody did, and then come up with, like, some fake ones, throw them in there, there's your… and then send it out. Yeah.
David Bau: Like, I think it's really interesting to see, like, how good people are at actually guessing, like.
David Bau: Like, what their intuition is. That'd be kind of interesting. But the fake ones have to be genuine what? Yeah, yeah, yeah.
David Bau: Yeah, like, I… yeah, I feel like Claude could be pretty good at coming up with…
David Bau: It'd be cool to see if, like, if, like, David was better than all of us, like… Oh, that'd be interesting. What was the,
David Bau: Yeah, what was the results with this, like, training data time thing?
David Bau: What I took is, like, you can train a linear frame to recover training order.
David Bau: And then, two, you can train models to self-report. Yeah, the,
David Bau: At the NUSE, yeah. So what… tell me where you are in your program.
David Bau: I'm a second-year undergraduate. Second year undergrad? Yeah. Oh, that's great. And so you're… you saw some data… Yeah, what kind of… you saw some data about… You said first. So remind me, remind me of your name again? I don't know, I'm… I started, like, teaching myself machine learning, and I'll end up kind of the end of my first year. Oakland is a kind of quiet place, there are only 900 kids left.
David Bau: And so you get a lot of time to, like, learn stuff you want to learn. So, and you're second year in Oakland? No, I'm second year in Boston. Oh, you came to Boston? Oh, okay. So everybody from Oakland was a first-year student. We keep on changing the program. People like that were later in training. So they didn't have the professors in the classes.
David Bau: I started learning machine learning and learning LLMs, and kind of run into mechanistic interceptibility because of Near, and then I saw you talk, and I was kind of sold on it. Yeah, it's very fun. I've loved it. I've spent a lot of the summer trying to teach myself everything, so I don't know, I'm really excited to come to meetings and stuff like that.
David Bau: Like, you're reinforcing it, like, the models, like, getting it earlier, or something. But in that case…
David Bau: You see, like, data X, data Y, data C. You don't keep doing… Well, that's the Harvard route, right? Right, they dropped to 8. Oh, okay. It would be sort of different, right? Yeah, I don't know… It's only my fourth week in Boston, so I haven't been really anywhere. There's no H in it, so you can just go by. Alright, well…
David Bau: you know, I don't know, I… it's possible that… Yeah, so… Yeah, that's what you would be getting. You know, they have reading groups, but they have, yeah. They have different, like, kind of… They don't explain in the paper? They just say a prose can distinguish? Yeah, so one of the things I noticed was, so one of our… We should read it. I know, I should read it.
David Bau: You know, spent a lot of time… I didn't bring my slider today. It's, like, so… feels very special. That's right. That's right. I spent a lot of time over there, so… No, I actually followed up, and I ran some, like, cloud jobs overnight. Mentorship applications for big programs, like… I didn't expect it to be that long.
David Bau: Yeah, they're on server. So yeah, I have my day, but then I woke up at an awesome time. I can't shut off my avatar. No, it's just, like, still running. In my hotspot, yeah.
David Bau: then you get your… you can get your minute of fame. Just… just get, you know, ask David Atkinson for the link, and you can throw it aside. I think it'd be more like a minute of feedback for me. Anything else. That's great.
David Bau: It was really nice to meet you. Nice to meet you, too. I'll be back. Why is this table so dirty? There is, like, a seminar. Wow.
David Bau: That's right, yes, so we've… so there's more one person talking through the… Yeah, I think I'll be back in the next couple months. I have a,
David Bau: Question. Yep. Should we be recording presentations?
David Bau: I think we are. We are. I know, but ought we too? You mean, you think we should not? I think we should consider, like, or have we not? I don't know.
David Bau: Well, so we don't publish all the publications. Okay. It's like an option on publishing. So which ones are published?
David Bau: So, so we have… so how do we decide what to publish? So, so the person to know…
David Bau: is Emma Bortz, who is our outreach Manager for NDIF. She is the one who runs our NDIF YouTube channel.
David Bau: Okay. And so far, the YouTube channel just has 5 talks on it, out of the many, many dozens of talks we've recorded. Okay. So far, they're all external speakers, except for me, and
David Bau: But they have access, the end of folks have access to all our recordings, and they're sort of curating through them to see which ones would be good for YouTube.
David Bau: channel presentation every week, they… they want to sort of edit one more. And, and so… so if we ever record something that you're like, you know, I'd rather not have this show up on YouTube, you just send Emma an email, and it won't, you know, they'll know not to put it up.
David Bau: Okay.
David Bau: Does that make sense? I feel like that should be the default. Yeah, I think… Yeah, at least Emma has asked if… she's, she's always asking… so I don't know if that's… Yeah, so that, that is the default. Only reason I bring this up is, like, I wonder if…
David Bau: I don't know, if I were giving a presentation, I feel like I would… I would potentially present, or make certain decisions, or say certain things, depending on whether I know I'm recorded or not, or something like that. Oh, because you might not comment on your opinion about somebody else, or something like that. That, or, like, maybe there's something I'm not sure about, but, like, you know, I might want to pontificate, or something, but I wouldn't
David Bau: I wouldn't feel comfortable, let's say, like, having it in a recording on YouTube or something like this. Right.
David Bau: Potentially, like… That's right. And for example, I wonder if the speaker from last week, like, might have told us things that he didn't tell us because he was being recorded, or something like this. Oh, I see. It's a possibility. I see, it's a possibility. I,
David Bau: Yeah, it's a good question. It's a good question. So, I guess… I guess the feeling that I have is…
David Bau: I… I… bias towards putting on the recording, just because…
David Bau: Because we've been in a lot of situations where it's like, oh, that… you know, in retrospect, that was a really…
David Bau: you know, good talk that we'd like to share with the world. And at the end of the hour, you can't go back and hit the record button. That's true. The,
David Bau: Whereas on the other side, like, you know, not publishing is easy to do after the fact. I do, you know, I do make a point to ask everybody, you know, is it okay to record before we have a guest speaker recorded? And so,
David Bau: So, certainly, we should make sure that people
David Bau: Like, are, you know, know that they can…
David Bau: say no, that's totally fine. You know, including all of you. I wonder if it might be nice to ask before the… because I feel like if I was in front of an audience, and I was like, no, like, then it would feel a little… Yeah, so it's okay, should we record you or not? We do make a… I do screenshot a tweet where somebody is bullying somebody else. Would you rather not be recorded?
David Bau: Yes. Yeah, yeah. Yeah, we don't have to record. Yeah. Yes. yeah, so I, I think that… so that's a good idea. So I'll ask people, you know, privately whether it's okay to record.
David Bau: And, and then, yeah, so Emma is pretty good about…
David Bau: Asking everybody, whether it's okay to post their stuff on YouTube before they get posted, so there's another double check.
David Bau: And, and then in addition to that, we edit the videos.
David Bau: So…
David Bau: you know, if you… if anybody is ever in a recording, and Emma calls you and says, want to put this on YouTube, it's, like, the best… it's the best talk ever.
David Bau: And you're like, you know, I'm not comfortable with this 5-minute discussion that we had, or whatever, right? They will edit it out. No problem.
David Bau: And so… So yeah, we can do all of the above.
David Bau: Correct.
David Bau: And then, and then I think that for this kind of thing where people are uncomfortable saying something.
David Bau: Yeah, I… the nice thing is, that's one of the nice things about having people come here every person. I feel like, oh yeah, we should totally take advantage of, like, talking to people in the hallway, talking to people privately, you know, outside. Does that make sense?
David Bau: Yeah, tell me how you really feel. Yeah, tell me how you really feel, right, exactly. Like, what is, like, what was behind this, you know, opinion that you had?
David Bau: you know.
David Bau: I think, I feel like… I feel like hallway discussions… Are always more honest.
David Bau: Even recorded or not.
David Bau: Stand better for the crowd. Yep. Yeah.
David Bau: But good feedback.
David Bau: We should give people a more low-pressure way of saying, no, it's unrecorded.
David Bau: Updating.
David Bau: Fair.
David Bau: Book.
David Bau: Excellent.
David Bau: Excellent, excellent.
David Bau: So, yeah, it's interesting, because, be, there are various things that…
David Bau: Are plenty about this world that we live in.
David Bau: But, you know, how online the world has become is one of the funny things.
David Bau: You know, if, like, we don't really have a YouTube footprint for the labs, but, like, if we had a YouTube footprint, too. A YouTube footprint.
David Bau: Like, we don't have, like, a YouTube audience. Right, yes. But if we were to have a YouTube audience, then that lab would have more impact. What? Should we do that? I'm glad you didn't think about it. Like, sometimes papers are, like, have a 5-minute explainer.
David Bau: Exactly. Like, I remember when the, like, alpha geometry paper came out, like, I wasn't gonna, like, go through the whole paper, but then the author, like, put a 10-minute nice explanation up on YouTube, and it made it very easy for me to, like, eat my dinner and, like, learn about alpha geometry. Yes, yes. So I think that the general feeling is that the answer is… yeah, so, like, I guess… so here's my…
David Bau: here's my weird professor perspective on this, right? Our job is to be educators, and if the way that people are learning is on YouTube, then why the heck aren't we on there? Posted that one about the…
David Bau: Owls, but they changed it to eagles. Yeah, Alex, Alex, I think, most of the time. Yeah, yeah, yeah. That was a really great video.
David Bau: No, no, no, it was Eagles originally. The original paper was on eagles, and the, like, explanation of the phenomena was… But it's funny that you think the owls, they changed it to eagles. Oh, I don't know, the video I saw had eagles, that's all I know. Yeah, yeah. But I know the paper slide on owls. The paper that the video was on was the original.
David Bau: The original paper. But so my video included a segment of recent album. Got it.
David Bau: There's a five numbers right here dump section on my desk, and I wanna, like… Yeah, you should go ahead.
David Bau: bookshelves for them.