pal.k: Oh, and we have a lot of online people, too.
pal.k: you know that's great.
pal.k: Can you just pull that out?
pal.k: So Lisa is in her final year of her. Phd, she's on a lot of really interesting
pal.k: work in in machine learning, but especially in interpretability. She has looked in. She's she's done some interesting work, having to do with
pal.k: alleatoric and epistemic uncertainty. And she's done other work looking at other aspects of interpretability. But the the work that I asked her to come and talk about today, which I'm anticipating you'll talk about. But you're of course you're free to talk about anything, Lisa is she? Did this really really interesting work?
pal.k: Trying to extract
pal.k: new human on new knowledge unknown, previously unknown to humans
pal.k: about chess from the alpha 0,
pal.k: a chess system.
pal.k: And so so in doing that work, she had to wrestle with
pal.k: a bunch of new types of problems. And that was just so fascinating. I want to want to hear you know, talk a little bit about it.
pal.k: So
pal.k: so yeah, so welcome, Lisa, the the floor is yours, do you wanna see if your your screen sharing works.
Lisa: Yeah, that sounds great. Thank you so much for having me.
Lisa: okay.
Lisa: I think.
Lisa: Can you see?
Lisa: Just sort of my slides.
pal.k: Yeah, yes, let me where something's wrong with our our projector on our side. So let me take a second here. Oh, if for some reason it's small.
pal.k: Maybe your screen.
Lisa: Oh,
pal.k: Oh, maybe you're sharing a window. That's not the same aspect ratio as as ours, or something like that.
Lisa: So, if you make it.
pal.k: If you make it wider it'll fill up more of our screen.
Lisa: Me.
Lisa: See? Okay.
pal.k: Where's the keyboard?
pal.k: Try?
pal.k: And so
Lisa: sorry. I don't know what's happening on my end.
pal.k: It's okay. You can try try again. So I think that if you what you did before was fine, but maybe I'll ask you to make the window wider.
Lisa: Yeah, it is looking better now.
pal.k: Give me a second for it to sync up with. Zoom!
pal.k: Let's see
pal.k: it, says Lisa started screen sharing. Oh, look! It's beautiful! It's totally full screen.
Lisa: Awesome. Okay?
Lisa: let me. Just okay.
Lisa: Great
Lisa: I just want to see if I can get my notes up, too, but otherwise I will do it without my notes.
Lisa: We're a friend.
pal.k: The audience, don't worry.
pal.k: Oh, yeah.
pal.k: Oh, oh, sure. Oh, yeah, yeah. Just a single. Yeah.
pal.k: Okay. Oh, she'll be showing up there as well. Yeah.
pal.k: Okay.
pal.k: Oh, that was bad choice. Can you minimize
pal.k: masters? Okay, okay. I think we've got everything set up on our side, too, Lisa. So whenever, whenever you're ready.
Lisa: Okay, great. Well, thank you so much for having me. I'm really excited to talk to you. Oh, sorry this is not. This is the 1st slide about my research.
Lisa: So maybe
Lisa: as a very brief introduction, I'm a final year. Phd, student at the University of Oxford. I'm currently visiting the Coco Psi lab here in Princeton with Tom Griffith.
Lisa: and yeah, I'm excited to talk to you about my work.
Lisa: If you have any questions. Please feel free to just jump in and interrupt me.
Lisa: I probably won't be able to see the chat, or I can't at the moment. So
Lisa: just shadow any questions you have.
Lisa: all right.
Lisa: So maybe before I dive right into the project, I want to talk a little bit about the long term vision that we had when we were doing this project.
Lisa: And so
Lisa: basically our idea is that we want to use AI to create a positive impact. And more specifically, we wanted to push push. Sorry the frontier of human knowledge by leveraging interpretability tools for knowledge, discovery.
Lisa: and to unlock new model capabilities in a safe and responsible way.
Lisa: And
Lisa: that was sort of the underlying idea. In this project we want to find new human knowledge or new knowledge. Sorry that we could teach to humans.
Lisa: so the project that I'm going to specifically be talking to you about is this paper that we wrote called bridging the human AI knowledge gap.
Lisa: concept, discovery, and transfer in Alphazero
Lisa: and.
pal.k: Hopefully that works.
pal.k: Sorry, just.
Lisa: Oh no!
pal.k: Frozen on our end. But I don't think it's a user. I think it'll be okay. There was just a hiccup for 2 seconds. But you're back online.
Lisa: Okay.
Lisa: so this is a project that I was really lucky to do with a lot of incredible collaborators and mentors, namely, Nanad, Tom.
pal.k: Oh, no! Oh, no! Is this gonna be chronic? Maybe just everyone get off your computers.
Chris Wendler: On my end. The connection is fine. By the way.
pal.k: Oh, so it's on us.
Jannik Brinkmann: It's it's ours.
pal.k: Everyone turn off the Internet. Oh, no! What to do.
pal.k: We could probably just use one of our laptops to just hear the audio. And
pal.k: why don't you continue on, Lisa? And we'll see if this connection works.
Lisa: Actually do you mind if I just fix one thing on my screen?
pal.k: Go ahead!
Lisa: Okay.
Lisa: none.
Lisa: Is it okay? From your end?
pal.k: Looks good. It's you're so we're you're in a window now, but so it's a little smaller, but it's fine.
Lisa: alright.
Lisa: So maybe to sort of
Lisa: describe sort of the way that we think about this project so if you look at this circle.
Lisa: This is sort of circle represents what we as humans know.
Lisa: And then you can think of a second circle
Lisa: as representing what machines know.
Lisa: And so
Lisa: there may be some knowledge that humans know. But machines don't know which is represented on the left, which is H minus M.
Lisa: Then there's knowledge that humans and machines have in common. Which is this middle. So M intersect. H,
Lisa: and then there's some knowledge that's unique to machines which is on the right side which is shaded in.
Lisa: And so previous work has focused on the intersection between the 2.
Lisa: So this is when we ask things like, you know, which known concepts does model encode. So this happens very often in probing.
Lisa: and is what the previous paper focused on.
Lisa: And in this work, what we try to do is we want to target n, minus h. So what does the system know that we, as humans might not know yet?
Lisa: so in our work. We want to find concepts in M, minus H and teach them to humans.
Lisa: And we decided to do this in the domain of chess. So we think that the domain of chess is the ideal sort of domain to pursue this goal for a couple of reasons.
Lisa: The 1st one is that humans have extensively analyzed chess for a long time, which means that we have a well-defined sort of human knowledge space. So this we have a lot of insight into the circle. H.
Lisa: The second thing is that AI systems have performed at a superhuman level for a very long time.
Lisa: which would suggest that they may have knowledge that we don't have.
Lisa: And the 3rd thing is that the ground truth, evaluation is much easier in the domain of chess. Right? So if you compare it, for example, to the domain of like medicine or science.
Lisa: where, if we find something, we may need to perform wet lab experiments, it's a lot easier here.
Lisa: for example, to
Lisa: simulate games to find the ground truth outcome.
Lisa: So maybe before I go into this as a quick question.
Lisa: who here knows how also 0 works, or has read the paper, or
pal.k: Maybe 10%.
Lisa: 10%. Okay?
Lisa: Good. Then I'll give you a brief overview.
Lisa: so
Lisa: I'll start with a bit of a technical overview, a very, very short one, and then I'll go into
Lisa: the way that I kind of think that it plays chess. And so, Alpha 0, as sort of like a method. It has 2 main components. It has a policy value network.
Lisa: And essentially, what that does is it takes us input a chest position and then it'll output 2 different things. The 1st thing is the probability distribution over the next action.
Lisa: So this is kind of similar to like an Llm. Right like you have like a probability distribution over the tokens. It's the same here. It's like a probability distribution over the moves.
Lisa: And then the other thing that it outputs is a value which is an estimate of whether it thinks that
Lisa: white is better, whether it thinks that black is better or whether it's equal.
Lisa: So that's the 1st component, the policy value network. And the second thing is that
Lisa: it also uses a search algorithm, multicolored tree search which it uses to select the move which it wants to play.
Lisa: Now, when I think about Alphazero.
Lisa: I think about it a little bit as follows, so I used to play chess professionally, and
Lisa: so I don't know. I think it kind of. I like to think of it like in how it relates to how humans play chess?
Lisa: And so when I look at a chess position like the one that's shown here, and maybe actually, before I continue, how many people play chess.
pal.k: Be 50%.
Lisa: Okay, that's quite some. Right? So when I look at this position,
Lisa: I first, st you know you can't calculate all possible moves. That's impossible. And for those of you who play chess black is to move.
Lisa: So instead, what I do is, I say, like, Okay, you know, what are the most important aspects of the position.
Lisa: And here I might be like, okay, it's a very early stage position. So I want to develop as many pieces as I can. I might think a little bit about like, you know. What are the structural weaknesses? Where do I have more space? What is sort of my long term plan.
Lisa: But there are sort of these high, level features that you extract, and that's the 1st step that humans do.
Lisa: And you can think of Alphazero doing roughly the same because it takes a position and it passes it through the policy value network, which is also essentially extracting these features.
Lisa: So next as humans. What we'll do is we'll based off of these features. Try to come up with a couple of moves.
Lisa: So here is black to move
Lisa: for the chess players. We might do something like Bishop d. 7. Try to develop the pieces. We might castle, bring our king into safety.
Lisa: or we might think of like slightly more creative moves like rook. G, 8.
Lisa: Which happens to be the best move here
Lisa: and
Lisa: all for 0 essentially does the same right? So it's like extracting these features in the policy value network. And then what it does is it basically creates sort of.
Lisa: you know, this probability distribution over moves, but there will be usually a couple of moves which have a lot of mass on it. So basically, there are a couple of moves that it thinks are the most interesting.
Lisa: And then, after it's done that, what it does is it essentially starts calculating? And we do the same as humans. So when we think about a couple of moves and which one we want to play, we'll kind of go through this loop of saying like, Okay, if I play this move?
Lisa: What is my opponent's best response? What am I going to respond to that? And you'll do it sort of in the same
Lisa: way that's described here, and Alpha 0 does the same. So once it's sort of selected, these moves, it thinks that it's most interesting to calculate.
Lisa: It'll use Mcts to actually find which move it wants to play and calculate the next best continuation.
Lisa: Are there any questions so far
Lisa: gonna move on.
Lisa: So now, moving on to our work. There's sort of 4 main components that I'd like to talk about. So I'm going to start with talking about what is a concept. Then I'll go into, how do we find concepts?
Lisa: And then how do we find concepts in the space M minus H, so concepts that are unique to Alpha 0 that humans may not know yet.
Lisa: And then how do we, or can we teach these concepts to humans?
Lisa: We'll start off with what is a concept?
Lisa: So there are several possible definitions of a concept like in the literature. You'll see things like.
Lisa: you know, they're human, understandable, high, level features. Or, for example, it's an abstract idea
Lisa: in our work. We define a concept as a unit of knowledge.
Lisa: and there's sort of 2 main aspects to this definition.
Lisa: So the 1st one is that it is useful. So it's something, for example, that can be used to solve a task.
Lisa: So if you think about this more generally like, if you have the concept of a beak, we can use this idea of a beak to identify birds.
Lisa: Or, for example, in the context of chess
Lisa: we have the concept of material.
Lisa: rare material is the aggregate value of the pieces.
Lisa: And we can use this, for example, to determine whether or not. We want to exchange certain pieces.
Lisa: And the second aspect of this definition is that
Lisa: there's sort of this notion of minimality, so that a concept should be concise, and that irrelevant information should be removed.
Lisa: So again, if we think about chess, we could have a concept that is, space and time and material. You could add on a lot of other things.
Lisa: And while all 3 of these things, for example, are useful.
Lisa: it's a lot easier for humans to learn. One of these things as opposed to all 3 at the same time.
Lisa: And so the idea here is to sort of break down the knowledge into a form that makes it easier to understand. And so as concise as possible.
Lisa: So now that we have a definition, I'll start explaining how we sort of use that definition to actually find concepts.
Lisa: So in order to actually translate our definition into something that we can, you know, write up for formulate using math, we need to make 2 main assumptions.
Lisa: So the 1st assumption that we make is that concepts are linearly encoded in the latent space of the policy value network.
Lisa: So the previous work that I'd referred to way in the beginning, they'd actually already shown that they could find concepts
Lisa: in sort of the policy value network.
Lisa: And when I say latent space, I just mean, you know, if you take the layers and you take the post activation representations, those are the ones that I mean
Lisa: and the second sort of core assumption here is that we make a linearity assumption, right? But this is one that's commonly made in our field, and has shown some success in extracting concepts.
Lisa: You could also make other assumptions on the structure. But we stick with this one here.
Lisa: and the second assumption that we make sort of builds on this 1st one.
Lisa: So we assume that a concept can be represented as a vector and the second assumption that we make is that the more relevant a concept is for a position, the smaller the cosine distance to latent representation of that position.
Lisa: So this will come up later. But it's important for both in the way that we find concepts, but also finding examples of the concept.
Lisa: Now basis of these 2 assumptions, we can actually formulate concept discovery as a convex optimization problem.
Lisa: So if you remember our definition, we had that a concept is minimal.
Lisa: And so we can achieve this by trying to find a concept that is represented as a sparse vector
Lisa: which we'll do using the objective of the convex optimization problem.
Lisa: And we also want that a concept is useful, right? That it's something that we can use to solve a task. And we'll do this using constraints
Lisa: so sort of the the toy formulation looks something like this, right? So you want to minimize the norm of this vector
Lisa: subject to some constraints.
Lisa: Now, the question is.
Lisa: how do we get these constraints right? How do we write constraints to ensure that this concept is actually something that's useful.
Lisa: So the idea is sort of as follows, we say, okay.
Lisa: a concept of something that you know
Lisa: motivates a plan.
Lisa: So for example, in this position, which we also saw before,
Lisa: you know, there was like a structural weakness. And we can think of this sort of structural weakness induced in White's position as something that motivates us to choose a certain plant.
Lisa: So on the right hand side is basically sort of a very basic tree showing the calculations that you might make
Lisa: and the green sort of.
Lisa: you know, the moves that are circled in green
Lisa: show a rollout
Lisa: which the agent may select in this position. But what's also important about this rollout. Is that the reason we do it is because of the structural weakness. So we want to sort of leverage this
Lisa: concept in the position.
Lisa: whereas at a rollout which we don't choose
Lisa: may not actually leverage this concept.
Lisa: So maybe as a brief summary.
Lisa: the idea is that a concept is present in the rollout that we choose, and it's not present or it's not leveraged in the rollout that we don't choose.
Lisa: And so, using that idea.
Lisa: we can formulate or create sorry the convex optimization formulation.
Lisa: And so what we simply do is we require that the concept is more aligned with the latent representations of the positions in the green now rollout than it is in the Red Rollout.
Lisa: And so for every concept that we try to find we will formulate or sort of create one of these complex optimization problems.
Lisa: And essentially, all we need is actually an input position. And after that, what we do is we run Monte Carlo tree search to get these rollouts
Lisa: which we need to find. The vector.
pal.k: So I can. I can, I ask so and so like a concept is
pal.k: so so does that mean a concept is specific to a given board position.
Lisa: That's a great question.
Lisa: so it can be but we try to find concepts are that aren't. And so this comes up later, we sort of.
Lisa: we have a mechanism that basically filters the concepts to ones that
Lisa: generalize across different board positions.
Lisa: But so basically, if you'd want it.
Lisa: to generalize across different board positions.
Lisa: You can actually just combine basically these constraints and then aggregate them over different positions and say, Oh, this vector is relevant in this sequence and sort of add those constraints, and then just add more constraints for the next board position. You want it to be relevant, for
Lisa: if that makes sense.
pal.k: Yeah. So I guess so. So locally, a concept might
pal.k: only be
pal.k: relevant to
pal.k: a subsequence of the upcoming moves.
pal.k: for, like the the concept might not no longer be relevant after 10 more moves, or something like that.
pal.k: Is that right?
pal.k: But but it might be relevant for this. But yeah. And then eventually, hopefully, you're finding concepts
pal.k: that are relevant for a lot of different situations.
Lisa: Yeah.
Lisa: and sort of like
Lisa: I I don't know. I don't know. I think about like the parallels between chess and math, and sometimes I've
Lisa: I don't know. I found it an intuitive comparison. But like, if the concept is sort of like
Lisa: you know, taking derivatives or something like that. So you have sort of this
Lisa: way of taking derivatives like, you have, like certain rules and taking derivatives like, for example, for exponents. And those are the ones that might be a sort of more locally applicable to specific problems.
Lisa: But you also have sort of more general strategies, and I think
Lisa: so you could find both of ours. And there's a metric that comes up later which allows us to filter the ones that are more generally applicable.
pal.k: Thanks. what is the Z here? Is z here? The thing that kind of like results in that specific rollout in the Mcts tree.
Lisa: Yeah, thank you. That's a great question. Sorry I jumped off that very briefly. But see, is essentially just.
Lisa: The latent representation of the position in their rollout.
Lisa: So
Lisa: like, for example, Ztl, is the latent representation
Lisa: of the selected sort of of the position and the selected rollout at time t at a given layer, L.
pal.k: But all the rollouts are from the same board position right
pal.k: like like. Given it. Given a board position, what is the next move of the model.
pal.k: So
pal.k: what is the C actually,
Lisa: Oh, yeah, no. Sorry. This is a great question. So basically,
Lisa: maybe this is
Lisa: So like, you start in this position right? And then you have 2 moves that you could play. So one of them is Rook G. 8, and one of them is Castle.
pal.k: It is.
Lisa: The moves don't matter. But you end up in different states.
Lisa: And so basically, for these different states, we'll like find the latent representation corresponding to these different states, and then contrast them.
pal.k: Oh, okay. Okay. Okay. Great cool. Thank you.
Lisa: No, thank you for your question.
pal.k: And is there a way that you're finding the values for these States like like, How do? How do you?
pal.k: How do you make sure that one is a positive state, and one is an active state.
Lisa: Yeah, this is a great question.
Lisa: so what we do is when we
Lisa: when we find, like what we'll call the optimal trajectory. And this above optimal trajectory because we ran Monte Carlo sorry tree search. We have a couple of statistics that we get back. So we have what, for example, the estimated value is if we take these different trajectories, and that's 1 of the things that we leverage. So we ensure that the value is different
Lisa: between these 2 trajectories.
Lisa: Another statistics that you also have is number of times it was calculated, or sort of
Lisa: alpha. 0 looked at this position during its calculations.
Lisa: Which also gives some insight. But basically we ensure that the green one is always
Lisa: like the value is significantly or is different than the red one.
pal.k: Another question to the minimality constraint. You take the l 1 norm, and this kind of assumes that, like the directions
pal.k: or interpretable concepts are axis aligned. So multiple
pal.k: concepts that involve multiple directions.
pal.k: Multiple X's are
pal.k: bigger or like less minimal than ones that are access aligned.
pal.k: why is that? Why do you penalize the access alignment? Is there some like privileged basis that makes you think
pal.k: should be spice?
pal.k: I'm sorry. I'm just gonna say the word sparse like, why? Why? Why? The sparsity
pal.k: assumption is that basically what you're asking? Yeah.
Lisa: Yeah. That is a good question. I think that's an assumption.
Lisa: I think. Given more time, I would have actually liked to see what happens if we, for example, also consider different rooms. So I think this is a valid question. It's like,
Lisa: the design choice that we made that limited it.
Lisa: I do think so. Part of the reason
Lisa: is that
Lisa: like in the architecture it kind of corresponds to
Lisa: a chess board. So there is like an 8 by 8, sort of, or just as like an 8 by 8 board. And
Lisa: this is sort of continued throughout the network. And so
Lisa: when you sort of make this assumption, you're sort of saying like it focuses on certain squares.
pal.k: Oh!
Lisa: Or subparts of the board, which is something that chess players do.
Lisa: But yeah, so it is sort of like.
Lisa: yeah, I would have liked to play with the L. 2 norm. If I'm being very honest, I think that that's a great point. But I think in the context of chess. The l 1 norm also makes sense because you do get sort of these local patterns that you're picking up on.
Lisa: Oh.
pal.k: Thank you.
Lisa: Are there more questions?
pal.k: What one more question so is the is the probability for the policy like, does it usually like align with the value that's predicted. So like does a higher value give you a higher probability usually?
pal.k: Oh, right.
Lisa: Right? Okay? So the probability is the probability distribution of removes. So like, for example, here,
Lisa: like, you know, like Brook g. 8, it might give like, I don't know 0 point 2 2. And so on. And so like, basically.
Lisa: that's just the probability that, like a prior probability that assigns to the move. But at least that's the way I think about it.
Lisa: and it is, of course, sort of related to the value, because it's going to try to suggest moves that it thinks are better.
Lisa: And so
Lisa: you would hope that the higher the probability assigned to this move.
Lisa: the better the value is.
Lisa: or the the better the move actually is. So the better the estimated value.
Lisa: but I think
Lisa: it's not. Yeah. It's not necessarily always the case right? So sometimes what you'll do is you'll think a move might be an interesting move. But once you calculate it, you realize that there's a concrete reason that it's a wrong move, and so the value doesn't correspond to the probability.
Lisa: so there's a difference there. But there is also a relationship between the 2. Sorry this is not the best answer or the most clear answer. So please tell me if you have questions.
Lisa: because the value is sort of
Lisa: it's estimated, based off of the sort of final board position which takes into account the probability as well, because the probability determines the probability that you execute a move.
Lisa: So there's this weird relationship between the 2.
pal.k: I have one more question. So I'm not. I'm not sure if I just missed it. But, the the probability distribution over next moves that it picks are all of those vectors the same size every time. So so so I guess, would there just be a lot of zeros for moves that are illegal, or or or how do you deal with like illegal moves? Are those are those like elements of that vector or or does the size of the vector change, depending on how many legal moves there are.
Lisa: Yeah, so this is a great question. So the way that it's implemented it outputs it over all possible moves and then I think, filters out illegal moves, and then readjusts the probability.
Lisa: so you can think of it as zeros for legal moves.
Lisa: Are there more questions?
Lisa: I'll continue them. But please feel free to continue asking questions. They're great.
Lisa: okay? So we've sort of got into the convex optimization formalization.
Lisa: and
Lisa: yeah, because we are introducing a new method one of our 1st steps has to be, how well it works. This isn't a main part of the talk. If there is a specific aspect that you're interested in. Do let me know, because I can go over it in a little bit more detail.
Lisa: But we evaluated our method in the supervised setting.
Lisa: So in chess,
Lisa: because it's really sort of well analyzed by humans.
Lisa: what we have is a lot of data sets and specifically data sets corresponding to concepts.
Lisa: And so
Lisa: what we did, for example, is we tested 3 different things. The 1st one is we saw just whether or not sorry we could find
Lisa: new positions which weren't used in the convex optimization formula.
Lisa: They contain the concept and the way that we evaluated this is basically you can
Lisa: find a set of positions that contain a concept because we have sort of these supervised data sets.
Lisa: And then you can see if you find a vector using a subset of those positions?
Lisa: Can you identify the ones that aren't in the set?
Lisa: The next thing that we looked into is whether or not we could influence the model behavior using the concept vectors so essentially. What this meant is if you took
Lisa: the position and then you encoded it, and then you nudge the hidden state in the direction of the concept.
Lisa: Could we influence the behavior or the move that is selected
Lisa: and
Lisa: the way that we evaluated. This is, we looked at a chess puzzle set. So, for example, if you have.
Lisa: you know, a set of positions where the optimal move corresponds to
Lisa: for example, creating more space or obtaining more space.
Lisa: Then what we would do is we would take the position. We'd sort of pass it through the network. And then we'd amplify the space concept vector and then see whether or not
Lisa: Alpha zeros move changed.
Lisa: And what we found is that, like in terms of
Lisa: solving these problems, it got sort of 5 to 10% better if we nudge the embedding in the direction of the concepts. That was you or should be used to solve the position.
Lisa: And the last thing that we did is we evaluated our approach on data efficiency.
Lisa: So we found concept vectors using different numbers of positions. And then we looked at how the quality of the concept vector changed. And we found that we could
Lisa: find relatively high quality
Lisa: concept vector, is using only like 10 to 25 position pairs.
Lisa: so there's more details on this in the paper.
Lisa: But also just for time I will continue, because our paper is actually about trying to find
Lisa: new concepts so sort of more in the unsupervised setting.
Lisa: So so far, what we've done is we've formulated concept discovery as a convex optimization problem.
Lisa: And we verified their method works
Lisa: sort of well in the supervised setting.
Lisa: next, our question is okay, how do we
Lisa: sort of adopt this method or
Lisa: change it, not change it? But
Lisa: how do we use it in an unsupervised way?
Lisa: So recall
Lisa: our formulation. There were 2 parts. So
Lisa: what we really wanted to do here, or what we really needed to do is
Lisa: find positions and then run Mcts.
Lisa: The question, for example, is, how do you find these positions?
Lisa: Well.
Lisa: here, what we did is we leveraged that Alpha 0 uses self-play.
Lisa: so we had sort of 2 main steps. We 1st created a lot of self-play games from Alphazero.
Lisa: and to do that, we just simulated a set of games, and to obtain a diverse set of games, we just sampled different starting positions from the Chess Encyclopedia of openings.
Lisa: and then to filter the positions to find positions that might be interesting to look for concepts for.
Lisa: We actually leveraged Alpha 0 training history. So what we essentially did is we took a weaker version of Alpha 0, which was around 80. Elo points weaker than the strongest version
Lisa: and the strongest version of all 0. And then for every single position, we said, Okay.
Lisa: are they going to play the same move? And we also check that, you know, if they play different moves? Are they sort of meaningfully different in the sense of that, they lead to a different evaluation.
Lisa: So through, for every single self play game, we
Lisa: form sort of this filter, and then we found 4,000 positions.
Lisa: which was which we then start use to find concepts.
Lisa: So now we've sort of adapted the approach
Lisa: so that we can use it in an unsupervised way.
Lisa: I'm I'm.
pal.k: Curious how, how, how, how severe that filter was!
pal.k: Do you have a sense for you? You ended up with 4,000 positions out of?
pal.k: Is it like 1%.
Lisa: Don't know. It is in the paper.
Lisa: So this I think it is a pretty strict filter also, because the 2
Lisa: so the 2 versions of Alpha 0 that we took were relatively similar already.
Lisa: and I think we could have taken a weaker version. So maybe for context, they're both already like, very far into superhuman level.
Lisa: cool.
Lisa: But so this was a pretty strict filter. I think the filters that are coming up are also quite strict.
Lisa: cool.
Lisa: Okay.
Lisa: So you have our unsupervised approach. So the next question is, how do we ensure that concepts are in m minus H.
Lisa: So if you recall our definition of concept, we wanted a concept to be useful so that it can be used to solve a task. We wanted it to be minimal.
Lisa: And we add in
Lisa: 2 new aspects that we care about to find new concepts. So the 1st one is that we want our concept to be novel. We want it to be an M minus h.
Lisa: and we also wanted to be teachable. So we wanted to meet
Lisa: so that we can teach the concept to new agent or human. And this sort of relates back to an earlier question, right
Lisa: of is a strategy only, or the concept only applicable to one position.
Lisa: And
Lisa: teachability sort of ensures that it's something that could be more broadly applied.
Lisa: So let's 1st talk about novelty.
Lisa: so if you recall this diagram from like way before.
Lisa: This is kind of how I like to think about it like we want to find something that's in the shaded area. So m minus h.
Lisa: and the way that we go about it is
Lisa: pretty simple. We 1st want to kind of estimate the circle H, then we want to estimate M, and then basically, we try to find something that's more in m minus h, so it's more aligned with M than it is with the space H.
Lisa: And so how do we go about this?
Lisa: Well, essentially, first, st what we do is we estimate this page M. And H. So, for example, to estimate M. What we did is, we took all the Zeros games.
Lisa: and then for every single position we obtain the latent representation.
Lisa: And then we take all of these latent representations, and we stack it into a matrix. So the matrix is all the positions by the latent dimension.
Lisa: And then we just take the spectral decomposition of that matrix to find the basis. Vectors.
Lisa: you can kind of think about these factors as like.
Lisa: you know, like the units of knowledge of like
Lisa: Alpha 0, or sort of, like
Lisa: the language of Alphazero.
Lisa: and we do the exact same for human skins.
Lisa: And after we've done that
Lisa: for every single concept, we estimate whether or not that concept can be better reconstructed using the basis vectors of
Lisa: alpha zeros games or human games. So basically
Lisa: a sort of intuitive way to think about it is like, if we have this concept, can we better explain it? Sort of using the language of all for 0 or the language of humans? Whereas here the vocabulary are the basis vectors.
Lisa: And so we filter a lot of the concepts based off of novelty.
Lisa: And the second thing that we also cared about.
pal.k: So that's so to me, that's that's interesting, because it's really just your your concept here is just
pal.k: static board positions. There's no there's no tree search here at all in this like human concepts. Right? So you're not. You're not necessarily
pal.k: restricting it to concepts that humans might be trying to maximize, or that machines are trying to maximize in in either of these right? Or or am I missing something.
Lisa: no, that's a great question. So
Lisa: the novelty metrics somewhat. Yeah.
Lisa: doesn't.
Lisa: So the novelty metric just sort of looks at like the spaces where the concepts lie in. But the concepts, when they originally found.
Lisa: were based off of more sort of
Lisa: dynamic states. So they are like based off of a ruler.
pal.k: Oh, oh, so they! So they are! So so there's a there's a whole pipeline here where
pal.k: you're collecting a whole set of
pal.k: age concepts
pal.k: using this sort of tree search
pal.k: thing. So it's not like.
pal.k: I see, I see.
pal.k: I see. So you have. You have, like 4,000
pal.k: board states, from your
pal.k: synthetic case, you have some number of 1,000
pal.k: human games.
pal.k: And each one of those things corresponds to a concept.
pal.k: A concept vector is that right?
pal.k: Remember wrong.
Lisa: So almost so the
Lisa: we have a basically 4,000 positions for which we're going to look for concepts in. Yep. And then so we found 4,000 vectors. And then, after we found those 4,000 vectors, we say like, Oh.
Lisa: if we look at these vectors, do we think that they're like, is it sort of like a general vector, like a general idea that we are, as humans already know or not? And then what we do is
Lisa: for a different, like, just a very large set of Alpha zeros games and a very large set of human games. We say, like.
Lisa: is this vector sort of more aligned with alpha 0 games? Or is it more aligned with human games.
pal.k: I see, do they? Does it just show up in the game. So like. So the the basis that you're reconstructing it in doesn't have anything to do with tree search
pal.k: right? The basis is just
pal.k: board states.
Lisa: Yes.
pal.k: I see, I see. So this is like a this is a A, vector. That's more typical of that distribution.
Lisa: Yes.
pal.k: I see.
pal.k: So is the assumption kind of like when you get the both States. The concept is already kind of like baked in. So each you should be able to extract that with the basis vectors of the of the both States.
pal.k: Can you say it louder.
Lisa: Okay.
pal.k: Sure. So so you get the concept directors with the Mcts.
pal.k: And when you get the port States from maybe the middle of the layer.
pal.k: Is the assumption kind of like. The concept is already in the boards latent. So you are expecting to extract that concept when you
pal.k: extract that concept with the basis of the Board State.
Lisa: Yeah, so this is a great question. So in our paper, we actually differentiate between 2 different types of concepts. So the one that you're describing we call static concepts. So those are the ones that are already existing sort of in the position. So they're sort of
Lisa: they're there versus what we call dynamic concepts. So the dynamic concepts are the ones that would arise, and so they wouldn't necessarily be in the original state. But they will come at some point like in the rollout. So, for example, here, like, they might appear at T equals 3 or T equals 2, but they might not necessarily be there
Lisa: at T equals 0 or T equals ¥1. And there's also, if you look at the constraint, it's
Lisa: It's not a strict inequality.
Lisa: so equality is allowed. So it's allowed to, for example, in the beginning. Not be there in both rule sorry
Lisa: in the pairs of positions, but it can just arise later in one setting, but not the other one.
pal.k: So if if there is a dynamic concept that arises at like T is equal to 3, you have no way of extracting that at T equal to 0,
pal.k: right.
Lisa: Actually well, we would extract it with this right? Because if you think about the constraints, if it's not there at T equals 2 T equals one T equals 0. That doesn't matter, because as long as it's not
Lisa: in like, let's say the sub optimal rollout or the Red rollout. Because then you have sort of equality, right? It's like 0 versus 0,
Lisa: like, it's not there, it's not there. And that kind of continues.
pal.k: Oh, yeah, I understand that you can extract it with this, but you can't extract it when you when you do the when you do the Svd, when you actually work with the basis vectors
pal.k: because it only focuses on the current port state, not from the future.
Lisa: Yeah. And so
Lisa: that is true. But what we would hope is that.
Lisa: because we just sample a very, very large number of positions.
Lisa: that it would appear in one of them. But you are correct. There is like a shortcoming.
pal.k: Okay. Okay. Cool. Thank you.
Lisa: Thank you.
Chris Wendler: I would also have like a really short question, sorry
Chris Wendler: you embed the human games also, using Alphagos network.
Lisa: Yeah.
Chris Wendler: Okay.
Lisa: that's it.
pal.k: So when you was there a question that was continuing, and then I interrupted.
pal.k: oh, no, I think that's resolved, that was all. So if you, if you are taking the embeddings of the human games from the network.
pal.k: would that really give the entire edge space? Or is it just giving the intersection of H. And M. Because we are just collecting the knowledge of
pal.k: what the model thinks the human game is giving. So do you think it's mostly coming from the intersection.
Lisa: Yeah, I think that's a
Lisa: yeah, I think that's a valid point, I think. Yeah, it is probably mostly coming from the intersection.
Lisa: So these are, maybe, I think, a more accurate way to describe it is like.
Lisa: this is the view. Yeah, according to Alpha 0. So if we find concepts. They're new, according to Alpha 0. As
Lisa: if that makes sense. Yeah.
pal.k: And then I just have a technical question about like how you actually like what the actual metric for for like
pal.k: easier to reconstruct was, so you have these, you have all these vectors that correspond to board states.
pal.k: You put them in a matrix, you have one for the human and one for the alpha 0 games. And then you take the spectral decomposition of
pal.k: both of them, or or like exactly what are you doing.
Lisa: Yeah, so basically, like, if we take all for zeros games, we have, like the boards versus like the latent dimension. And then we take the spectral decomposition
Lisa: as we have a set of basis, vectors for all the Zeros games. Then we do the exact same for human games.
Lisa: and then we have our concept vector and we look at the reconstruction error
Lisa: if we try to reproduce it using the basis vectors of Alpha 0 and the same with, like
Lisa: the basis vectors found from human games.
Lisa: I don't have the equation. Sorry on the slide. It is in the paper. Okay? But what we.
Lisa: There is one thing we have to think about, which is that the the rank of the matrices are different.
Lisa: So we select a number of positions that's much larger than the latent dimension.
Lisa: But the ranks of these spaces are just inherently different. And so what we'll do is we'll
Lisa: we'll look
Lisa: sort of across like. Let's say
Lisa: we look at the reconstruction error, for if we take, let's say, 10 basis vectors, 20 basis vectors, 30 basis, vectors. And then we compare those.
Lisa: and it has to always be strictly better reconstructed using alpha zeros than humans.
Lisa: I have more questions. Thank you for your question
Lisa: enable. Continue.
Lisa: so that was a novelty metric. We also introduced a second metric, which is
Lisa: whether or not these concepts are teachable.
Lisa: so the deal is quite simple. It's basically a concept of something that is used to solve a task. But that's something that we assume.
Lisa: And so can we teach this concept to another agent? And can that agent apply the concept to solve the task?
Lisa: So we do this in the following steps. So first, st we find a set of concept prototypes.
Lisa: So this is a set of chess positions
Lisa: that epitomize the concept. So basically playing the optimal move in this position requires applying the concept
Lisa: and then we have to find a teacher. But given that, we found, you know our concepts in Alpha 0, we set Alpha 0 to be the teacher.
Lisa: We also select a student. So this is an agent that doesn't know the concept yet. And essentially, what we'll do is we'll just take an earlier checkpoint because we need an agent that can roughly play chess, but just can't apply this concept yet.
Lisa: And so what we'll do is we'll check whether or not the agent can make the correct moves in the position
Lisa: which require applying the concept.
Lisa: And then what we do is we teach a student. The concepts by knowledge distillation. So essentially what we do is on the set of prototypes that we've constructed.
Lisa: We minimize the kill divergence between the teacher's policy and the student's policy.
Lisa: and then we evaluate whether or not the student has actually learned the concept by seeing whether or not they can
Lisa: accurately or correctly find the best moves on a set of unseen positions.
Lisa: so this is sort of
Lisa: kind of what the metric ends up looking like. So this is an example of a concept that we did select in the end.
Lisa: where the student improves more when it's trained on the concept prototypes than when trained on randomly selected positions.
Lisa: So
Lisa: there's a lot of lines. Basically, you can kind of ignore the title that just says which layer the concept was found in
Lisa: but here
Lisa: the blue lines basically show when the agent was trained in concept prototypes and then evaluated on concept prototypes this dark blue line.
Lisa: So this answers the question, does it get better at playing positions which contain the concept?
Lisa: And then the green line shows when it was trained on concept prototypes and then evaluated on randomly sampled positions.
Lisa: So this checks whether or not I got better at playing chess more generally
Lisa: and the reason that we care about that is, we don't want it to get better just at the concept. But then forget how to do other things.
Lisa: And we benchmark our agent because
Lisa: you could also say, like, okay, this agent was trained, you know, through knowledge, distillation, with like a stronger model. And so it's always going to learn to play chess better. And so we train a second agent, which has the same starting level, but then is trained on randomly sampled positions, and we see how it does. And these are sort of the letter lines that you see at the bottom.
Lisa: And so what we find overall is that the student improves more when it's trained on the concept. But data, both in applying the concept and
Lisa: playing more generally. And you can see this by kind of the gap between the darker lines and the lighter lines.
Lisa: and we also find that that actually, the student is able to learn quite efficiently. So
Lisa: we didn't tune this experiment very much, but over the course of like
Lisa: 50 epochs
Lisa: you can see that the performance gain
Lisa: is roughly the same. As you know, the model would have learned
Lisa: during self play
Lisa: taking around 80,000 to a hundred 1,000 steps.
pal.k: Is each one of these? Is each of these graphs a single concept like this. This is an example of a single concept I see, so is it so? The point of this is not. Every concept behaves like this like are some concepts not teachable, and then you would discard them.
Lisa: Yeah. And so what this also requires is that the concepts?
Lisa: Because you have an invite like, you have to be able to apply the concept. In other scenarios. This does eliminate concepts which are only unique to one position.
pal.k: Oh!
Lisa: And so we want to find concepts that
Lisa: you know are you can think of them also kind of like a strategies that can be more generally applied.
Lisa: and are so, is something that like you can actually learn.
Lisa: And then it's useful for another agent, which is sort of what this filters out.
pal.k: I see, because your your teaching procedures has like this danger of overfitting like you're you're really only selecting this very tiny data set of positions where you know this concept is decisive.
pal.k: And you're just training. You're just doing gradient descent
pal.k: on that set of decisions, right or or or do you have?
pal.k: Or do you mix in other generic positions, or anything in the training.
Lisa: No. So we only it's a very short sort of
Lisa: training procedure.
Lisa: So we only sorry we only train on those concepts, but we know that it doesn't overfit, because you see, this improvement on sort of general positions, too.
pal.k: This is your test for over. But this is this is your test for over fitting. Basically, this is your, this is your thing here. So it's kind of amazing that there are any concepts like this at all because of your training procedure. Right? I mean, like, like, if a student came to me and I said, said, Oh, I'm gonna train like this.
pal.k: I would tell him.
pal.k: don't even try. It's just gonna overfit is nothing's gonna work. But you you sort of found a bunch of things that worked this way. Do you have a sense for like what percentage were teachable.
Lisa: No, so I it is in the paper. But I am going to be honest. I forgot.
pal.k: We'll look it up.
Lisa: But it is a very low percentage. It's a very like a
Lisa: I want to say it was less than 5%.
Lisa: But I could be off.
pal.k: Very interesting.
Lisa: I'll continue then.
Lisa: or are there other questions actually.
pal.k: I have some questions to get to make sure I'm on the same page. So
pal.k: are you labeling the concept by any chance or
pal.k: no? No.
Lisa: So the labels here like
Lisa: Actually sorry. Do you mean, are we labeling them when we teach the student the concepts? Or do you mean.
pal.k: In general, when you discover a concept, because you sometimes, when you're explaining a concept, you explain it as
pal.k: like the space strategy, like how much space it's take? Are you labeling like these discovered concepts in such a way? Or are you leaving them
pal.k: to be like just time to provide discounts.
Lisa: Yeah, so we're not labeling them. General, what we do is for every concept we end up with like a set of examples that contains a concept where, like the optimal play,
Lisa: requires applying the concept. And then.
Lisa: as humans, what we could do is we could look at them and try to find sort of
Lisa: what the common threads are, but it is very difficult to do so. So like
Lisa: these concepts are extremely difficult, because they were learned at the end of training. And so they're honestly a little bit beyond my level.
Lisa: and so that's like one thing that makes it very difficult to label, and I think the second thing that also makes it difficult to label is
Lisa: This is, I don't know. Maybe it's a little more of like a philosophical point. But like, whenever you're trying to describe them, you either have to say.
pal.k: Maybe.
Lisa: I'm sorry.
pal.k: No go ahead!
Lisa: You either have to like
Lisa: come up with new words for these concepts, but that wouldn't really help in explaining it, or you have to explain them, using sort of
Lisa: the language that we already have. And whenever we do the second thing, it sort of feels like you're simplifying it to, because you're constrained to the words that we have. And so then it always sounds like a recombination of existing concepts.
pal.k: And since, like the discovery has a constraint of being usable.
pal.k: and I feel like
pal.k: easier way to put, that is, if this concept improves in
pal.k: a better policy.
pal.k: Right? So I guess that's how you're evaluating this like on the the Y axis on the graphs. Here is
pal.k: student teacher policy agreement. Is that like, is it just better at
pal.k: like? Does it just have a better policy? Or what? What is that? Why access.
Lisa: Yeah, no, I think that that's like kind of a valid way
Lisa: of summarizing it. Because, like, we said, like in the beginning, our definition was like a concept is something that it's useful.
pal.k: Out of the.
Lisa: So like, we assume that concepts are useful for downstream tasks.
Lisa: And so in that sense, you could also argue that that's just saying it has a better policy.
Lisa: So.
pal.k: So what? What is the y-axis here? Though, like what is the what is student teacher? Agreement.
pal.k: policy, agreement? Is it
pal.k: clarity?
Lisa: No, so it's whether or not they select the same move in the position.
pal.k: Oh, I see. Who's that?
pal.k: Makes sense.
pal.k: Thank you.
Lisa: Graphic
Lisa: I do see, we're at 5 min so
Lisa: I think I will very briefly summarize
pal.k: We? We have a little time to go over. I don't know if you I know. I know that you might ha have to run. But but if you, if you have a little free time after we, we do have a little time to go over.
Lisa: Okay, no, I I'm happy to go over a bit.
Lisa: so maybe
Lisa: so the part that I still have to talk about is a little bit like the the human experiment, and then also
Lisa: the conclusion which I think would take somewhere around 5 to 10 min. Do we have any.
pal.k: That's perfect. Yeah, plenty of time.
pal.k: Alright.
pal.k: Well.
Lisa: Then the next part is the human experience. But maybe as a summary. So basically, we had sort of this 1st version of the method which.
Lisa: you know, found concepts that were useful, they were minimal. And then we also added 2 other aspects that we care about, which is that they're novel and that they're teachable.
Lisa: But the ultimate goal is actually like, okay, we found these concepts, are they thing concepts or ideas that we can teach to humans.
Lisa: And so we
Lisa: looked into whether or not we could teach chess grandmasters new concepts. And we're extremely lucky, actually, because we got to play with or sorry. We got to work with 4 brilliant players.
Lisa: Vladimir, Kramnik, Gukesh, D. Hui phone and Maxim, and they all happen to hold actually a world championship title, which is insane. And Gukesh is currently
Lisa: actually one of the world championship or world champion contenders.
Lisa: and so
Lisa: the setup
Lisa: was sort of as follows.
Lisa: it had like 3 different stages, and the way that I like to think about it is actually like, you know, when you were in high school, and you were trying to solve a math problem, or like, learn how to like.
Lisa: learn a new sort of method.
Lisa: The way that I would go about. It is like, you know, for, like taking derivatives, for example, I would go to the textbook. I try to solve a couple of problems.
Lisa: wouldn't always know how to solve it, and then I'd go to the back of the book.
Lisa: and I checked the solution and see like what I should have done
Lisa: and then kind of distill from that like, okay, like.
Lisa: what should I do next time? And then I'd go back and like, see whether or not. I understood it.
Lisa: and the procedure here is actually very similar. So, by the way, chess players.
Lisa: very often they train by trying to solve puzzles. So what we did is we had for each concept a set of puzzles that correspond to that concept.
Lisa: First, st we want to see whether or not you know, they could accurately solve these puzzles.
Lisa: So we got a baseline performance.
Lisa: Then, after that, what we did is we provided them with Alpha zeros calculations. So this is kind of like the solutions in the back.
Lisa: and we gave them some time to think about it and see what Alpha 0 would have played in these positions.
Lisa: And then the 3rd stage was that
Lisa: we gave them a new set of puzzles corresponding to the same concepts.
Lisa: and then
Lisa: the same sort of concepts that were shown in Stage one.
Lisa: And then we saw whether or not they got better.
Lisa: Oh.
Lisa: and so.
pal.k: Yeah, I have a question. How? How did you find puzzles that correspond to concepts?
pal.k: Yep.
Lisa: So basically, what we did is
Lisa: using sort of cosine similarity. So in the beginning, we had this assumption that the more relevant a concept is for position, the smaller the cosine similarity.
Lisa: And then what we did is we
Lisa: just going to go back to the formulation?
Lisa: Basically.
Lisa: we search for positions where these
Lisa: constraints would hold.
Lisa: So if you look at like
Lisa: we would, you know, find a position. And then we would check. Okay in the optimal rollout
Lisa: is this concept more relevant for the optimal rollout than it is for a suboptimal rollout.
Lisa: And we also did some filter checks sort of for the quality of the position which is detailed in the paper.
pal.k: And and how many, how many puzzles did you give the experts to look at.
Lisa: Yeah. So we
Lisa: I think so. We gave them each between 36 and 48 in total. We would have wanted to do more to be honest, but
Lisa: given that they are very strong
Lisa: and professional chess players.
Lisa: we had sort of a constrained amount of time with them.
Lisa: cool?
Lisa: Are there more questions.
Chris Wendler: So.
Lisa: M.
Chris Wendler: Thank you so much for your presentation. It was great also. The whole work is amazing.
Chris Wendler: I really enjoyed reading the paper.
Chris Wendler: but the the question that I had like when I read this part of the paper I was like.
Chris Wendler: Okay, but how would this compare to just the grandmasters play
Chris Wendler: against Alphago?
Chris Wendler: Do you have like any.
Lisa: Wait. So do you mean specifically if they would play against Alpha 0, or like their play versus Alpha Zero's play.
Chris Wendler: Let's you lock them into a room for 10 h to study these puzzles.
Chris Wendler: What would happen to their puzzle performance if they just played
Chris Wendler: against Alpha 0 for 10 h.
pal.k: Just generically.
Lisa: Oh, yeah, that
Lisa: would be interesting. And I think that that would be really fun to explore. I think
Lisa: I mean, in short, I don't know. I think that that would have been
Lisa: yeah, very interesting. I think the difference, though, is, maybe
Lisa: when you like, ask them to play one full game.
Lisa: there! There's some bias in which positions they select right, and we still go into like they can define it, whereas here
Lisa: I think you have, like a shorter experiment overall, because the game takes quite long, and, like
Lisa: you, select the position. So it tends to be more.
Lisa: I guess, informed by also, 0 like this is gonna be a weird way of.
Chris Wendler: I think maybe, like you would even outperform this baseline.
Chris Wendler: But it would be just so cool if if we could run this baseline.
pal.k: I have a or or a different suggestion would have been
pal.k: when you evaluate
pal.k: the idea would have been the ideal would have been to choose the same, to to choose 4 concepts, 2 of which were the same
pal.k: as the ones in the puzzle, and 2 of which were different concepts that were orthogonal
pal.k: to the others, and then you could see if there's a gap
pal.k: in the grandmasters
pal.k: ability to get better
pal.k: at solving puzzles and the concepts that they've seen.
Lisa: Yeah, that is actually one of my biggest regrets in the setup. Because I going back, I would go back and do that. To be honest, I
Lisa: I think we didn't do it because we want to sort of
Lisa: like maximize like we didn't know whether these concepts were teachable. And so we want to show them as many as possible and check which ones are teachable.
Lisa: But I think, from a learning perspective, the setup that you're describing would have been so much better. And we had like.
Lisa: Well, we did like a little pilot pre study just sort of see whether the setup made sense or not, and we had someone
Lisa: also a very strong player. And there they got a different concept in the beginning and the last stage, and we didn't see any improvement. But that was just based off of one concept. So we don't know.
pal.k: Oh, I I didn't! We didn't. Could you repeat what the the pilot study this your audio cut out.
Lisa: Oh, sorry about that. So in the can you hear me now? Again?
pal.k: Yes.
pal.k: Okay.
Lisa: In the pilot study. So we try to see like whether or not the setup made sense and get any kinks out of the experiment. But one of the things that we did have is one participant
Lisa: got a different concept in the 1st phase and the 3rd phase.
Lisa: and there we didn't see any improvement.
pal.k: Oh, okay. So you did have this experiment in Pilot. That's cool.
Lisa: Yes.
pal.k: So did did all of the participants get the same concepts. And you're looking at agreement or like, did they all get different
pal.k: concepts.
Lisa: So we gave them different concepts. I think both would have been interesting to do. I think it was just like our worry was like, oh.
Lisa: like, what if we give them? You know, a concept that's not teachable. And so
Lisa: we're just trying to maximize the number of different concepts seen by grandmasters.
Lisa: yeah.
pal.k: So did you find that all 4 concepts were learnable by the grandmasters.
Lisa: Oh,
Lisa: So
Lisa: we know all concepts were so. I think this isn't in the paper, but will come out hopefully in the next version.
Lisa: But we performed this analysis, and I think out of
Lisa: the concepts that they saw, the majority were learnable. But there was a small percentage like, I think 2 or so that weren't
Lisa: so.
Lisa: but we will.
Lisa: I think we're going to add that in the next version of the paper, the breakdown that we did have in the original paper was whether or not they improved in general.
Lisa: Which we did see
Lisa: programmaster
Lisa: I also had an example, but I will maybe skip over it, given the time.
Lisa: but this is in the paper.
Lisa: but maybe just to conclude the work. So
Lisa: in our paper. Essentially, what we try to do is, we found an unsupervised framework for finding new concepts that were minimal, useful.
Lisa: novel, and teachable.
Lisa: and then we showed that in the supervised setting
Lisa: the concepts that we found are data efficient
Lisa: and can also be used to find new concept, prototypes and influence model behavior.
Lisa: And then we also provided evidence that we can teach these concepts to top grandmasters.
Lisa: My personal takeaways from the paper.
Lisa: Is one of the things that it kind of.
Lisa: At least I had to think about is that it?
Lisa: It made me think about the way that we interact with AI systems. So if you think about a lot of the AI systems. So there's like chess engines or Llms. We often query these systems
Lisa: with prompts or chest positions. So we're asking them specific questions.
Lisa: And in this work what we try to do is, you know.
Lisa: Alpha 0 essentially gives us positions that it thinks is conceptually important, and this kind of removes at least to a certain degree, the bias of like only asking questions that we can think about.
Lisa: as a chess player. I think my key takeaway is that you can really see that office. 0 has a different prior over what concepts are relevant in a given position. So as humans, you know.
Lisa: in a lot of different aspects of life, we sort of.
Lisa: we already have to do these prior over which concepts are important or the way that we're supposed to interact or make certain decisions.
Lisa: But because alpha 0, you know, learned through self play, it has come up with a completely different prior over what's important in certain situations.
Lisa: There are also some things that made it difficult for humans to learn from Alpha 0.
Lisa: And I think one of the main ones was that there's this difference in objectives. So humans, for example, we play competitive chess, we will sometimes make slightly irrational decisions. So, for example.
Lisa: we'll take a slightly worse position if there's a chance that we can win as long as the risk is very small, whereas alpha 0 just tries to play optimally and assumes an optimal opponent as well.
Lisa: Oh.
Lisa: and
Lisa: don't know.
Lisa: There's supposed to be a slide after this, but it's not. Oh, yeah.
Lisa: so the future directions that I am excited about.
Lisa: So I like this project a lot because I thought it was really fun. It was very interesting to work on, but at the end of the day chess is a sort of toy domain where we can develop methods and verify them more easy, easily. But you know.
Lisa: ideally, we would do this in other fields like medicine.
Lisa: I think another aspect that I'm very interested in is sort of leveraging these interactions between humans and AI systems. So you already see this in Llms, where, for example, debate can be used to improve factuality.
Lisa: and improve reasoning.
Lisa: But I think another thing that would be very interesting is sort of to have interactive explanations. So
Lisa: a lot of interpretability methods are 0 shot in the sense of like, you know, we have a prediction, and we provide an explanation.
Lisa: But here, for example, I think our grandmasters would have benefited a lot from being able to interact with the system. And actually query, like, Okay.
Lisa: you know, my solution wasn't the same as the solution of
Lisa: Alphazero.
Lisa: But like what was wrong with mine. And so like having these explanations for why?
Lisa: Other decisions are incorrect. It's very useful for humans.
Lisa: And this also is another context. So, for example, clinicians, we find there in our medical work that
Lisa: when you have a model and it provides an explanation, and it doesn't use a feature that clinicians often use.
Lisa: they just immediately lose trust. And so, having this ability for clinicians to also sort of give feedback to the model of an explanation that's then used by the model, or at least sort of interact with models about explanations is something that's very useful.
Lisa: and the last direction I'm also very excited about is this sort of enhancing and blocking model capabilities?
Lisa: I think people talk a lot about enhancing capabilities and also blocking in the context of AI safety.
Lisa: But I think that blocking model capability is more generally something that
Lisa: can be leveraged
Lisa: in an interesting way.
Lisa: So
Lisa: think, for example, about the context of solving a chess puzzle or theorem proving
Lisa: what we'll do with humans is we'll usually try one approach when it doesn't work, we get negative feedback on why that approach didn't work. But we use that sort of negative explanation to then say, like, Okay, we're going to try a different approach. Right. So like, if we want to do proof by enumeration, it doesn't work.
Lisa: Then we might try to, you know. Write like an inductive proof, and if that doesn't work. We'll try a different proof style.
Lisa: In the same way, for example, with chess, we might try one approach or one idea, but once we know that that idea doesn't work. We block sort of.
Lisa: We don't consider it again. We don't keep repeating that, and I'm curious about whether or not
Lisa: we can sort of apply those ideas in other contexts to improve model performance.
Lisa: which
Lisa: is also related to this
Lisa: project, because basically with like
Lisa: with a chess human or in the context of chess, like humans. Also, like sometimes, you know, we try something, and once we've tried it, we block it. But we
Lisa: in Alpha 0. You can see that
Lisa: it very often retries the same idea.
Lisa: same
Lisa: anyways. Thank you so much for listening. And I'm happy to take any more questions.
Dmitrii Troitskii: So I'm really curious about your idea of knowledge. Discovery, like different models from Alphazero. Do you have any sort of vision for that, or how this method, or similar method would be applied in Llms.
Lisa: Yeah, I think there's like some components that can transfer or not.
Lisa: So I think, like
Lisa: the part about the specialty composition is something that you could also try in Llms.
Lisa: I mean in Lms. You also have, like other avenues, for example, like in the context of like sparse autoencoders you have, like, you know, the features you can label, and the features that you can't label which is maybe a very direct approach.
Lisa: I do think
Lisa: so. Maybe this context of like teachability is a little bit more complicated in Lms,
Lisa: but yeah.
Dmitrii Troitskii: I guess, like my my key question. There is, how do we deal with this? M. Versus H. Right? Specifically for Lms like, do we just limit our kind of like set of concepts that we would discover an Llm. To a certain domain.
Lisa: Oh,
Lisa: You could do that, I think, in the context of Llymps. I've been thinking about it.
Lisa: and there's a lot of different ways to do this. But, for example.
Lisa: if you think about it across different languages.
Lisa: is maybe like a way to do it.
Lisa: And then
Lisa: you might. For example, I didn't like
Lisa: one idea that I'm very specifically I've been thinking about for a while is like Lacune. So those are words that exist in one language or not another. And so you have this sort of concept that exists in one language and not another.
Lisa: and whether or not you can use that to identify patterns, to find new concepts
Lisa: as like. I said.
Dmitrii Troitskii: That's interesting.
pal.k: Yeah, cause. I guess the difference is that with Llms they're like essentially trained on the human distribution, whereas without Alpha, 0 is explicitly trained, not on the human distribution.
pal.k: But yeah, so I guess you would. I guess, though, you would have to do. You would have to find some some like
pal.k: I guess, like factorization, like different languages or something like that where, like one group of humans wouldn't know it
pal.k: and another would.
Lisa: Yeah, although you could maybe be like, I don't know in Lms. I've played around with them, and I've asked it to come up with new words, and it will.
Lisa: So maybe you can just get it to self-generated data set of like new words and then see what happens.
Chris Wendler: But also just because it was trained on the human distribution, doesn't pick up the same.
pal.k: Oh, shit. Yeah.
Chris Wendler: Concepts. Right? I mean.
Chris Wendler: that is kind of a strong assumption.
Chris Wendler: That is.
Dmitrii Troitskii: I guess, like another question is also so let's say we do find those like sort of different concepts, even with synthetically generated words. But then the question is like,
Dmitrii Troitskii: what? What is the objective? We are sort of like measuring this
Dmitrii Troitskii: concept against? Because, like here, you have a very specific objective, which is like winning the game right? But what do we do with the Llm. There like that? That's kind of why, I suggested. Maybe limiting to a certain domain.
Lisa: Yeah, no, I think that's a great point. And I think that's also
Lisa: why we kind of focus on like usefulness of concepts. Right? So like. They are sort of useful in a task. And I think, like with, for example, language, what you would have is like if you can have more efficient communication
Lisa: is like a weird proxy, maybe, or like
Lisa: oh.
Dmitrii Troitskii: That's interesting.
Lisa: If
Lisa: I don't know, like, for example, if that word can be used to solve or describe something, or based off of like.
Lisa: I don't know. Like, I think, like language is like information, communication, right? And so like, if I can create a word and based off of that word, you can then use that to
Lisa: solve the task correctly, which you could have only inferred if we had the same sort of.
Dmitrii Troitskii: Oh, I see!
Lisa: Something like that.
Dmitrii Troitskii: That's pretty cool. Yeah, so like, unique sort of
Dmitrii Troitskii: dense descriptor or something. Yeah.
Lisa: Yeah.
Dmitrii Troitskii: Right.
Lisa: No.
Lisa: yeah.
Lisa: no. I think language is a really interesting domain.
Lisa: Chris.
Chris Wendler: And
Chris Wendler: a question that
Chris Wendler: I had like. Did you think about applying your approach to other superhuman AI systems like Alpha forward, or something like that
Chris Wendler: where the domain is not as interpretable as like
Chris Wendler: chess boards.
Lisa: Yeah, no, I think I think that would have been really cool. I think.
Lisa: the hard part is like the verification at the end. But I think that that would be something that's very interesting to do.
Chris Wendler: I guess, all the way, until the superhuman part is kind of directly
Chris Wendler: your approach would be more or less directly
Chris Wendler: applicable.
Chris Wendler: like as long as you stay with the data sets, I guess.
Chris Wendler: and like properties that are captured by data sets. But then
Chris Wendler: the the next step
Chris Wendler: But there you needed the grandmasters anyways as well like, so the teaching to weaker versions would be still applicable. Filtering would be still applicable.
Lisa: Yeah.
Chris Wendler: But but just like the the final presentation to
Chris Wendler: the grandmasters
Chris Wendler: might have to look different.
Lisa: Yeah, I think that'd be really cool. I think you're right. I think up until that stage it
Lisa: could be a.
Lisa: The methodology could be transferred.
pal.k: So you have 1 1 difference, right in which is
pal.k: the the part of your presentation. You sort of flew through where you had a supervised
pal.k: sanity check of your method. So you actually, you have this unsupervised method.
pal.k: And then you actually sort of checked it on supervised data, which is very interesting, right? So like. In other words, you verified that the concepts that it was finding.
pal.k: You know, Matt, there was discovering a lot of concepts that you could have found supervised.
pal.k: Is that right?
pal.k: That we're.
Lisa: Previously, like.
pal.k: Enabled by humans.
Lisa: Yes.
Lisa: Yeah.
pal.k: And
pal.k: yes.
pal.k: I wonder how I wonder if that would be
pal.k: so. So, both in language we've talked about language and and something like Alphafold.
pal.k: I feel like our.
pal.k: you know, even in language that's been studied for so many years. Like, like, I have this sneaking suspicion
pal.k: that
pal.k: the number of concepts
pal.k: in language that we've labeled
pal.k: is this vanishingly small
pal.k: amount
pal.k: of the actual concepts that are needed to.
pal.k: you know, construct.
pal.k: you know, fluent language
pal.k: and
pal.k: and and and the sanity check might not even
pal.k: might not even work. Very well.
pal.k: Right.
pal.k: do or or do you? I'm I'm so it's an assertion of mine. I don't know if you feel like that's probably true, or.
Lisa: Sorry. I think you you broke up at the end. Oh, sorry.
pal.k: Right, we're having bad connection. So the so when when you did the sanity check for chess.
pal.k: you were checking with a lot of existing chess concepts.
pal.k: And did you? Did you have to have
pal.k: a large
pal.k: set of chess concepts to do this check?
pal.k: Or was it a small book?
Lisa: Yeah,
Lisa: I mean. So we checked it in a couple of different scenarios.
Lisa: I think it's like.
pal.k: Oh!
Lisa: I guess.
pal.k: Our audio is. Our connection is quite bad.
pal.k: Let's see, what can we do? It's good on my end. Oh, should I? Maybe I'll switch the room audio. Give me a second. Here. I'm gonna I'm gonna switch audio. Here.
David Bau: Okay, try try again. This is a this is a different audio thing. So can you, can you say, what is it that you did for the sanity check.
Lisa: Yeah,
Lisa: so I think so far as I need to check, we try to test it in different scenarios. Just
Lisa: because at the end of the day. It was sort of trying to understand whether or not our method functioned as we expected it to function and kind of
Lisa: I think so. We did it.
Lisa: I've forgotten how many concepts. And
Lisa: so we did it, like, I think on like 5 different types of concepts. In the end. But
Lisa: yeah, I think you could always go further and further with that. So I think it's
Lisa: this is maybe not the most scientific answer. But like, I think we try to like.
Lisa: test it sufficiently well to know that, like our method works as expected.
Lisa: but in terms of the number of concepts like I think more is always better. So it's hard to say exactly when to stop.
Lisa: Oh.
David Bau: Okay.
Lisa: Sorry. I don't know if that's a.
David Bau: No.
Lisa: You were asking.
David Bau: I'm just, I'm just. I'm more or less asking.
David Bau: is is this, is this something where chess is unique? Because so many things are given names and labeled in chess as certain types of things to look for, whereas even in language.
David Bau: or particularly in things like Alphafold, maybe the important concepts are not as well labeled or as well organized.
David Bau: and so you might not even know how to do the sanity check.
Lisa: Yeah, I think it definitely gets harder in other domains. But I do think we have enough in language. For example, right? Like we
Lisa: like. You could think about concepts and languages like a lot of different things like you could have, like just part of speech, as one you could have, like the different types of tasks and sort of the or like techniques, or
Lisa: I don't know how you'd describe it in language. But, like, you know, in translation. The ability to translate as like a sort of concept is also
Lisa: things you could analyze. And so I think.
Lisa: yeah, I feel like in language. It should be okay, and also like in
Lisa: I don't know and protein folding. So I know very little about this, but I could imagine that there's like certain things that are well known, and you could check whether or not like your method works as expected on those things.
Lisa: but that does require.
Lisa: I'm guessing more domain knowledge there, unless
Lisa: those data sets already exist
Lisa: great.
Lisa: Are there any other questions.
David Bau: Yeah, have you? So so has the has. Has. Have you showed this to the you know, to the chess community very much, and has there been sort of like a reaction within the chess community to this? To this work? Or they do they?
David Bau: But.
Lisa: Yeah,
Lisa: So I think
Lisa: I like, we've shared a little bit. It is under review. And so we're trying to be careful with.
Lisa: you know, avoiding
Lisa: I don't know we didn't. Wanna
Lisa: I think you have to be careful a little bit like when it's submitted to a journal with like not publishing articles about it elsewhere, or like talking
Lisa: about it elsewhere. I'm not exactly sure what the exact regulations are, so we've sort of not gone too much into that yet.
Lisa: but from the people who have read it and seen it, they were actually very positive.
Lisa: I think that, like
Lisa: most of them, found the concepts very interesting.
Lisa: yeah, I think, though, the
Lisa: one of the things that we would have loved to do given more time is also, you know, we, we found concepts that were learned by Alpha 0 at the end of training. Those are extremely difficult concepts by nature and by construction.
Lisa: And so it would have been really cool to see what concepts we could have found at different periods of training as well like earlier on. And so the ones that maybe, like, you know, like amateur players could have learned or like.
David Bau: Yeah, like.
Lisa: Closer to human level.
David Bau: I wonder if you look at the checkpoints that overlapped with people's Elo, like their particular Elo? If you, if you extracted check concepts from those checkpoints whether they would learn better right like, if you if you moved in a window along like model checkpoints. And you did this.
Lisa: That's real.
David Bau: Window, you know. Would people's learning, you know.
David Bau: get get better at their own Elo level.
Lisa: Yeah, no, I think that that's something that we would have loved to do, like, kind of like, create this like Alpha, 0 chess curriculum and see how it would have done. That would have been super cool. Yeah.
Lisa: yeah.
Lisa: okay.
David Bau: Well, thank you so much, Lisa, for spending so much time with us today. It was really enjoyable, and I think we learned a lot, I think.
David Bau: thanks, thanks a lot.
Lisa: Thank you all so much for coming and having me, and for all the great questions I really enjoyed. It.
David Bau: That's great.
David Bau: Yeah, Lisa, I'm gonna ask a greedy question, do you do you have? Do you have a couple of minutes? Just chat with me after afterwards.
Lisa: Yeah. Sure.
David Bau: Okay, great. I'll give you. I'll I'll I'll send you another zoom link over email
David Bau: and open. It's great.
Lisa: Thank you. Bye, bye.