David Bau: It's my mind.
David Bau: Either one would, whatever.
Core Francisco-Park: What do you prefer?
Core Francisco-Park: Can people in Zoom hear me?
sfeucht: Yes.
Core Francisco-Park: Awesome.
Core Francisco-Park: So, we're really lucky to have Corey to join us again, and, it sounds like…
Core Francisco-Park: He's at this transition phase in his career where he's moving from
Core Francisco-Park: being a, a researcher at Harvard? Well, I'm still there. Are you still there? I'm still there. I'm gonna be there for that. Being a research scientist at a new startup called Prior Computers, so it's, really exciting to…
Core Francisco-Park: see, startup activity in the Boston area. And so,
Core Francisco-Park: So, you know, I know that prior is going to be about making very human-like AI, but what Corey's been expert in
Core Francisco-Park: Is in setting, sort of the geometry of of, of representations.
Core Francisco-Park: In… in… in context reasoning and… and other complicated types of logic in LLMs.
Core Francisco-Park: And so, we're looking forward to hearing about his latest work here. Awesome.
Core Francisco-Park: Okay, well, thank you very much for the introduction. So, they'll be talking about, like, where do these representations come from? We talk a lot about, like, representations in neural networks, but then, like, maybe you should also wonder, like, where do they come from? Like, what's expected, what's kind of unexpected? So.
Core Francisco-Park: I gotta start a talk with…
Core Francisco-Park: this was a big question, I think… I don't know if it's still a big question, if do LLMs have world models? And then, I don't know, I think it's a vague question. And then I think a lot of the debate has, like, I'm sure probably most of you know this paper.
Core Francisco-Park: Where you train a transformer on Othello, just the sequences output, and then in the mobile internal, like the internal representation, you can read out the state, you can even do some causal interventions, like…
Core Francisco-Park: That's cool. And we also see some results, like.
Core Francisco-Park: LLMs, if you just, like, feed them in, like, landmark names, and read out the vectors, and then sort of try to linear probe for some geometry of longitude-latitude, you can read that out. Recently, Goodfire also had this nice post where you can take a DNA modal trained regressively, I think.
Core Francisco-Park: like, I think mask total encoder, actually. So, mask, mask.
Core Francisco-Park: mask language modeling, I think. So, you take a DNA model, and then from the internal representations, you can kind of read out the phylogenetic tree.
Core Francisco-Park: So…
Core Francisco-Park: These deep neural networks trained on a lot of data, like, represent some latent variable of the data generation process.
Core Francisco-Park: And kind of one question is, like, when this post came out, like, it was like, is this surprising, or do we actually expect this at this point? At this point, do we expect neural networks trained on vast amounts of data to just kind of obviously represent almost all features we know about data? Maybe… is that obvious, or is it still surprising?
Core Francisco-Park: And I guess, like.
Core Francisco-Park: myself, I didn't have a good answer to this question. So, first question raised here is, what are the origins of, like, these world representations? Like, where do they come from?
Core Francisco-Park: And the second one is…
Core Francisco-Park: GoPos should always move in AI, and LLMs, we need to put a word there, adaptive world.
Core Francisco-Park: So, what's an adaptive world model? So, now we know that chess… there's, like, superhuman chess-playing algorithms out there, and I think you can actually train them in within a day, or something.
Core Francisco-Park: And one thing is that one way to beat these chess algorithms is, at inference time, right before you play, you suddenly introduce a rule.
Core Francisco-Park: Today, your bishop moves like a horse, and your horse moves like a bishop.
Core Francisco-Park: And then suddenly humans generalize better.
Core Francisco-Park: Humans are better at this, and in fact, it's a little bit, like, even ill-defined how exactly to give this to, like, alphaChess, because, like, they're encoded with these representations already fixed in.
Core Francisco-Park: So it's a little bit even unclear what to do there. Maybe there's some… maybe interpretability can do something there, actually. I don't know.
Core Francisco-Park: So anyways, adaptive world models. So, the rules of the world changed. The world changed. Can you adapt? Whatever you know, can you adapt your representation? For humans, maybe you just need to swap some kind of in your brain representation of bishop and knight, and kind of try to apply the rest of the strategy.
Core Francisco-Park: So, can you adapt to a role change given right at inference time?
Core Francisco-Park: And I think, like.
Core Francisco-Park: people have been trying to study this, and I think, like, the classic example, this is studied in your lab, right? Like…
Core Francisco-Park: Bethel Tower is in Paris, and you kind of want to move to Rome.
Core Francisco-Park: And I think my kind of, like, personal thing about this is that, like, I think there's…
Core Francisco-Park: always this problem, how much do you propagate it? Like, do you now make this guy's name Gustavo Apel? Like, give it his little mustache there. Or… yeah.
Core Francisco-Park: Be careful, we have an Italian over here. Oh, that's alright. Okay, who is this? That's me.
Core Francisco-Park: Okay, where did Guidemo Passant? So this is a guy who, like, used to eats in the Eiffel Tower because he doesn't want to see the Eiffel Tower. Like, where does this guy eat lunch? Is this guy, like, French, Italian now, actually? So, some of these propagations are a little bit ill-defined.
Core Francisco-Park: So, this was a motivation why this is a small study we did, where we kind of defined these sets of new stuff, which are hypothetical, and not in the training data, because it just comes out of us right now.
Core Francisco-Park: And… but counterfactual, but not counterfactual, actually. It doesn't actually end, like…
Core Francisco-Park: break any knowledge you already have. So, same mathematicians defined application of X and Y as X plus Y times Y.
Core Francisco-Park: Can you use this new information, this new concept, to solve a…
Core Francisco-Park: problem, which requires this operation. And I think, like, most humans can do very well.
Core Francisco-Park: Sweet.
Core Francisco-Park: curated these kind of datasets where there's, like, a new mathematical definition. For example, like, I'm sure most of you have seen… encountered this bug where NumPy doesn't accept, like, axis. NumPy doesn't accept DIM, sorry. PyTorch uses DIM, NumPy uses Axis.
Core Francisco-Park: So, maybe they allow an update.
Core Francisco-Park: to kind of allow that alias. So, when these updates happen, like, you constantly adapt to the world, and can a model adapt to these kind of new informations?
Core Francisco-Park: And what we find is that models are actually pretty good at this if you give the new word in context. You can kind of expect them language models, I'm sure you all talk to them, and then, of course, they can just give this, like, counterfactuals, these, like, hypotheticals, and then they can reason on top of them.
Core Francisco-Park: But one thing which happens if you fine-tune on them.
Core Francisco-Park: is that if you fine-tune on these new facts, it integrates much less. So here, actually, what the accuracy we're measuring is that we're asking downstream questions, like, of the fact, like, propagated downstream.
Core Francisco-Park: And then, if you fine-tune, the models are actually much worse at integrating any of the knowledge and propagate them well downstream in their weights.
Core Francisco-Park: So, of course, like, here's one criticism is that, like, we're literally doing one-sentence fine-tuning, fine-tuning on, like, number of
Core Francisco-Park: The number of rows in the dataset is 1, and nobody in team learning does that, so of course you should augment the dataset.
Core Francisco-Park: So this is our best method. We try to augment the data set by saying we can't add additional information, we try to be fair. What we do is, like, we use the model to self-augment the data. So this is a common technique, which is discussed in many papers.
Core Francisco-Park: You give the fact, in context, you generate, like, you just ask the model to generate QA pairs, and then you take those QA pairs.
Core Francisco-Park: Train the model without the fact, needs to now answer the QA pair without the fact, so it must somehow integrate it into weights. That does a little bit better.
Core Francisco-Park: But there's, like, a significant gap, and I think, like, more than what the accuracy says, like, we tried to, you know, 5-check the modal, doesn't feel like the modal knows that application. Yep. Just a quick question, the ICR,
Core Francisco-Park: Is it just… the thing you're putting in context is just the description of the rule, or are you also giving examples? No, just, just a fact. Yes, just a fact.
Core Francisco-Park: just that we moved the Eiffel Tower to Paris, or to Rome. Yeah, for example, except that I think that's still counterfactual, even if you put it in context. Yeah, so these things are new definitions, so that, which are reasonable, the model, like, kind of can, like, generalize. There's, like, an answer to how you should generalize. Yes.
Core Francisco-Park: And for fine-tuning, is it, like, very localized to, like, a specific MLP, or is it, like, many layers? Yes, yeah, so here it's many layers.
Core Francisco-Park: And then, actually, yes, we did read that paper, mechanistic, like, what is it, unlearning? So, should he do… should he try to localize and edit those weights? We didn't try that, yeah.
Core Francisco-Park: Well, we tried data augmentation. Maybe this does something like that. I don't know.
Core Francisco-Park: And the question, too, here is, it seems like even though, even with all of this, it seems like it's brittle. Like, fine-tuning does not trigger the right updates, it just seems like…
Core Francisco-Park: gradient…
Core Francisco-Park: decent on, like, a small amount of data still compared to the whole pre-training corpus. Still very brittle to trigger the right updates.
Core Francisco-Park: So this is the second research question, and actually, they do seem a little bit separate.
Core Francisco-Park: What are the origins of world representations? And research question two is, why does fine-tuning not trigger the right updates?
Core Francisco-Park: So, this is a lab I come from, and we have, like, a lot of, like, tradition, almost, of building modal systems. What we do is, like, when we see, like, a phenomena where here we ask, like, an image modal to generate a blue square apple.
Core Francisco-Park: So that's, like, compositional generalization, generating something which is not in this world, but which uses concepts and combine them in new combinations. We develop, like, modal systems and generate findings. This is kind of our general approach, so I'm going to try to follow this to ask this kind of methodology to ask, like, answer these two questions today.
Core Francisco-Park: And so in our lab, we often do these stuff where, like, we perturb the data.
Core Francisco-Park: We do something on the data, train a model, and then see how the model changes.
Core Francisco-Park: But today, I think we need a new abstraction, because, like, the data is kind of like before, we used to just write scripts to actually just generate the data itself. But now we kind of, like, even modularize it further, so that there's a definition of a world.
Core Francisco-Park: Where things happen, like scientists deciding to kicking out Pluto.
Core Francisco-Park: And then there's data, somehow, that goes to the internet. There's a process which moves things, events which happens in the world, into the internet. Pluto, blah blah blah, or planets now blah blah blah. Something happens in the world, it gets moved into the data, and that's what we kind of trained our models on.
Core Francisco-Park: So we kind of needed an additional abstraction layer. So here's what we did. So…
Core Francisco-Park: we first defined our world is gonna be the coordinates of entities and cities, cities in the world. So, we're just borrowing the list of cities in the world, filtering them by some populations wrestled, and
Core Francisco-Park: I actually did this instead of, like, taking our kind of classic, like, nice Gaussian pure uniform data, because it actually had, like, a bunch of interesting properties, like, a bunch… most of the cities are actually in India and China. And then, like, there's, like, these sparse regions in the data, so I actually like the fact that it had these, like, interesting aspects of real data.
Core Francisco-Park: So… and also, I ignored all geometry and just, like, flattened it out to XY coordinates. Just, like, take the longitude-latitude and treat it as XY, because we're only borrowing the data, I'm not trying to do any geography here.
Core Francisco-Park: So, I'm gonna define that this is a world. Okay, this is a world. My world is basically a bunch of cities with XY cord.
Core Francisco-Park: And they have, like, their own names.
Core Francisco-Park: And I'm…
Core Francisco-Park: going to define the data generation process on top of it as geometric operations. So here, there's 7 tasks, so…
Core Francisco-Park: One is just measuring the distance between two cities.
Core Francisco-Park: So, distance between city 1, city 2 is some distance. Second is triangle area, so you need to measure the area of a triangle. You need to measure the angle between three cities.
Core Francisco-Park: Or you need to point towards which compass direction out of 8 directions, like a second CD8 compared to the first.
Core Francisco-Park: And this task is, like, you give a polygon and a city, and you need to determine whether the city is inside or not.
Core Francisco-Park: This is not a task parameter, you need to measure the perimeter of the polygon.
Core Francisco-Park: And then there's finally another task where you need to… you take two segments and determine whether they intersect or not.
Core Francisco-Park: So these are just, like, 7 geometric tasks, like, not sampled in any specific way. It just, like, came out of, like, me talking with AI, and also trying to make it… I tried to, like, make it diverse enough, like.
Core Francisco-Park: But okay, but actually, there's a property of all these tasks, is that
Core Francisco-Park: As soon as you give the world, these task definitions will not change. So these tasks, or geometric tasks, will not change. You can plug in any world, and it's gonna generate some data out of it.
Core Francisco-Park: So now you're able to come, like.
Core Francisco-Park: You're able to define a world update, and then you have a process which defines how that should update the data.
Core Francisco-Park: And one property of all these tasks here is that, we'll see later.
Core Francisco-Park: Even if you add any city to the world, say you add a city, which wasn't there in the world, so that's kind of a world update, it's kind of a new entity, new concept, you add a new city, none of the answers changes.
Core Francisco-Park: So in fact, we could also define processes where you add new cities, and the world changes consistently. For example, like, count the number of cities within a certain radius from a city. So that one is, for example, a rule where
Core Francisco-Park: you add a city, and then the answer changes consistently. But before that, we wanted to really go to the easiest thing, where you add a city.
Core Francisco-Park: you will… the added city will make more data, but, like, it won't touch… it won't affect any of the existing data. Okay, is this setup kind of clear?
Core Francisco-Park: Okay.
Core Francisco-Park: This is some, this is a dirty slide, but I think it's actually one of the most important slides, is how did I actually set up the details?
Core Francisco-Park: So… I actually tokenized in the dirty way, because I think
Core Francisco-Park: as we said about tokenization, I think we do it in a dirty way in natural language. So I actually, like, cut it in a dirty way, where I don't actually give one token per city. I actually make them numerical. So these are, like, 0, 1, 3, 8, or something like that. The ID is actually split it in this weird way, where you take a number and you just, like, slice all the digits.
Core Francisco-Park: And then it needs to output, like, an answer, like an angle between these cities, and that's also, like, split, not in a nice numerical embedding, but also, like, in this dirty way, where you take, like, a number 220 and cut it into 220. So the mobile has to figure out some arithmetic stuff internally.
Core Francisco-Park: And we're gonna be talking about representations.
Core Francisco-Park: So I'm gonna read out the representations by taking, this kind of prefix.
Core Francisco-Park: And then I'm gonna take the last token of the city, and then the comma, and then concatenate those representations, and that's kind of gonna be the city representation that I'm gonna be taking.
Core Francisco-Park: Any questions about this, actually?
Core Francisco-Park: Yep. What kind of model are you training? Oh, I'm training a transformer, autoregressive Standard Decoder on the transformer. From scratch, yes.
Core Francisco-Park: And is there a fixed dictionary of cities you have? Yes, so the dictionary of cities is… each dot there is a city, each one has an ID, C, something, something, something, ID, and it's, like, split into this, like.
Core Francisco-Park: The vocabulary of the modal is just alphabets and numbers, and these special… a few special tokens there.
Core Francisco-Park: So it's 10,000 points on a plane, or maybe… It's 5,000. 5,000. Oh, did I say something about… Oh, oh, so, okay, there's a reason that I didn't make them, like, linear to 1, 2. So there's actually, like, up to…
Core Francisco-Park: 9999. I allowed, like, all four-digit numbers, and randomly sampled there, because later, we're gonna add some cities. So, to make room for… to not make a distribution shift between what we add later.
Core Francisco-Park: Out of curiosity, did you put them on a sphere, or is it just on a plane? No, I intentionally did it on a plane, because I want to do a linear probe analysis. If I do a sphere, I need to do this
Core Francisco-Park: Yeah, Linear Pro and there's a discontinuity, so… yeah.
Core Francisco-Park: Yep. Did the size of your model matter? Not… so, it mattered in qualitative sense. It mattered in senses which
Core Francisco-Park: Might actually be more interesting for practitioners, because there are some interesting dynamics about how, like, earlier layers stop propagating representational changes.
Core Francisco-Park: So, actually, I think that is actually interesting in a practical point of view, but it didn't matter for the point of the main point I'm gonna raise.
Core Francisco-Park: What do I… oh, yeah, so I take the residual representation of the last token of the city ID, and then the comma.
Core Francisco-Park: And then I concatenated them. So I just did that because I want to make sure it's there, but actually, I think there's one experiment where I did, where I just take the last, like, IDN. Put it at the last layer. It just works. I take it at different layers. Oh, okay, so different layers. But…
Core Francisco-Park: layer… before the last layer, I just took that, yeah.
Core Francisco-Park: And for the last city before the parentheses, you take also the parentheses? For the last city… For example, CZZZZ. Oh, so actually.
Core Francisco-Park: So what I'm interested in is, after I train this, I'm just only interested in how the model trains, so I just use this template and swap the city out… Oh, I see. Yeah, swap that city with the 5,000 cities to get out the representation. Oh, I see, I see.
Core Francisco-Park: Yep.
Core Francisco-Park: Any more questions? Because I think the setup part is actually pretty important to get everyone grounded. Yep. You are training the model from scratch. Yep. So you could have just decided not to include these elements, right? Not to include BOS EOS. With a comma… Oh!
Core Francisco-Park: So that model figures out everything based on positions? Yeah, you're defining an angle. Yeah, could have. Yeah, yeah, yeah, exactly. Angle doesn't need to… it could be just A. Yeah. It can be just A, yeah. I guess it's just easier.
Core Francisco-Park: Yep.
Core Francisco-Park: I actually don't think the model cares whether I say angle or A, yeah, but… Yep, that's it.
Core Francisco-Park: You got a question? Okay.
Core Francisco-Park: Okay, setup's good. Okay.
Core Francisco-Park: Okay, so first question about world representations. This is going to be about pre-training.
Core Francisco-Park: So, here, the 30-figure.
Core Francisco-Park: I train only on the angle task, okay? So the data only comes from the task where I do this… this angle kind of data. Ignore the all other tasks. So, I think this is kind of standard, you see the loss going down, training succeeds, great. See, like, these induction head plateaus, stuff like that, these kind of, like, standard transformer stuff.
Core Francisco-Park: And as you train more, You see that there's, like, clusters emerging of different cities.
Core Francisco-Park: And you can actually see that if you PCA project out all the representation of the 5,000 cities, that already, like, clusters all the cities by kind of some geographic location, where close things are clustered close.
Core Francisco-Park: And then you can do a linear readout of the city coordinates. So we actually almost, like, reproduced from scratch, like, the West Gurney MX checkmarks results, where in LLM, they actually can extract out the city coordinates. But here, synthetically, we can kind of, like, see similar things happen here.
Core Francisco-Park: So, great, world representations emerged through autoregressive training. At this point, I don't know if this is surprising.
Core Francisco-Park: Okay, now… now I trained on different tests, yeah.
Core Francisco-Park: Why is a readout… oh, 3D. It's…
Core Francisco-Park: Doesn't need to be. It's 2D, actually. It's 2D, I'm just… I'm just showing another… the remaining residual principal component direction for no reason than I plotted everything in 3D, and then I just rotated in the best… in the XY plane, yeah.
Core Francisco-Park: So the readout is, yeah, the two coordinates you're seeing is in the just XY readout.
Core Francisco-Park: Which layer are you taking? Fifth layer. Yep. So this is a sixth layer transformer, and I'm taking it out after the fifth layer.
Core Francisco-Park: Yep, so you see representations of worlds emerge, and you can read out linearly some coordinates. Yep. The position.
Core Francisco-Park: Oh, position error, yeah. So this is just the metric that Can you,
Core Francisco-Park: set some cities in real… in, like, in actuality, like I said, like, 3,500 cities for a training set to regress towards X and Y, and then does that probe generalize?
Core Francisco-Park: To, like, does that probe generalize?
Core Francisco-Park: Here, actually, it's R-square, but basically, you can quantify also MSC distance, that also just goes down. You can actually, like, use this probe to… the probe generalizes to other cities. So this is like a regression probe, so it's a new regression probe, so you turn a regression probe on a few thousand cities, and then you test it on another thousand. Yeah, exactly. Yes, and then you can predict
Core Francisco-Park: oh, this is just the error on the task. The actual… the modal actually performs the task, the angle error goes down. And then actually, like, you actually do also get this, like, yes, regression. So here, I don't think R-square is actually a good metric. I should have shown the, what is it, the distance going down, but…
Core Francisco-Park: But yeah, but anyways, like, it kind of… not that surprising you can read out linear coordinates out of that, so…
Core Francisco-Park: Yep.
Core Francisco-Park: Okay.
Core Francisco-Park: So, things work.
Core Francisco-Park: And now, you're gonna treat on different tasks. So, here, the first one, distance, angle, compass, inside.
Core Francisco-Park: And they kind of give you different representations, where… and kind of it matches that, like, the inside task is a true-false task, so you just need to determine whether a city is in a polygon or not.
Core Francisco-Park: So, maybe that's why it's giving, like, these a little bit fuzzy coordinates?
Core Francisco-Park: And then, if you have more precise kind of tasks, which is, like, you actually need to get the angle exactly out, you get these more sharp representations.
Core Francisco-Park: One thing is that the PCAs all look quite different. The PCAs all look quite different, so this… which data generation process you use will actually kind of, like, shape some kind of geometry in your internals.
Core Francisco-Park: And no matter what, you can kind of still always
Core Francisco-Park: read out the linear probe. For the inside task, it's a little bit more fuzzy, but you can always kind of, like, read out the linear probe. But then the residual directions for the different tasks looks all quite different.
Core Francisco-Park: So, we're gonna quantify this.
Core Francisco-Park: Can you… Yep.
Core Francisco-Park: Third row. A third row is just rotating because here, just as before, I'm just showing the XY projection of the 3D PCA, so I'm just showing you some other direction. So this is the same… it's the same 3D object, it's just rotated in some other direction. I'm just showing you that, like, outside of the two…
Core Francisco-Park: linear readout probes, the dominant PCA just looks quite qualitatively different. Sometimes it's, like, collapsed on, like, a manifold, sometimes it's more fuzzy, sometimes there's, like, a separation of stuff.
Core Francisco-Park: That happens.
Core Francisco-Park: So yeah, this is how, actually, like, when the model trains, like, these representations form. One thing interesting I found is that, like, the representations, like, form, actually.
Core Francisco-Park: once here. The representation form exactly here, and then the global geometry is almost unchanged until the end of the training. So that's something I've found, and I'm actually kind of interested in, because one hypothesis I'm running for is, like, whether modal representations is this thing which has a small window to form, and then it kind of gets fixed in.
Core Francisco-Park: Need to confirm that. Do you, like, start all the trainings from the same seats, or… No, I do multiple seats. Wait, so…
Core Francisco-Park: all the initializations are the same, but the data is different, so I think it's effectively not comparable anyways, but in the paper, I do multiple seeds.
Core Francisco-Park: Yep. Is there one epoch of your… in the training? No, it's multi-epoch. Oh, okay, so do you know when in… can you go back? Yep.
Core Francisco-Park: To the training loss.
Core Francisco-Park: Yep. About… do you know approximately when in the tree? Let's see… so this is 6 epochs.
Core Francisco-Park: So… and this is log, so…
Core Francisco-Park: Okay. Hard to see, okay. Yeah. Okay, thanks.
Core Francisco-Park: Okay, yeah, so that's…
Core Francisco-Park: Now you can see that representations form, and these are… I'm showing different tasks, and then they kind of form different kinds of representations, and we can quantify this.
Core Francisco-Park: And there's this thing called CKA, which is you take two representations, and you compute the distance matrix, something like the distance matrix, because you subtract the mean, between all the points here.
Core Francisco-Park: and make a big city-by-city matrix, and you make a city-by-city matrix on different representations, and compare how those matrices themselves compare. So you do the CKA, and then you kind of get this thing where the distance test kind of hates
Core Francisco-Park: Everyone else.
Core Francisco-Park: the inside task also kind of hates other things. The thing you're seeing on the crossing task is because that task
Core Francisco-Park: just didn't train alone. So that one just doesn't train alone. It was this task where you need to figure out whether two segments intersect, and then I think it's related to the literature where Transformers, like, kind of struggling on these, like, two-hop composition tasks. So that one just didn't train if alone, so yeah.
Core Francisco-Park: So, different generation processes.
Core Francisco-Park: Create the frog geometries.
Core Francisco-Park: Now we're gonna do something interesting, which we're gonna actually pre-chain models, a game from scratch, a game from… yeah.
Core Francisco-Park: So, this is, this is, you know, what would, what would Jacob, who, or… Jacob Isla have to say about this? So, you know, they, they're like, hey, you know, you, you, you, you train.
Core Francisco-Park: image data… you train text data, you look at the CKA,
Core Francisco-Park: everything is, everything's the same as you scale up to more and more data. Yeah. You're like, oh no, this is, like, if you take something that's trivially the same, you take the same world, and you just take these slightly different
Core Francisco-Park: you know, posings of it, the CK is not looking that… that hot. So, do you think that your result is contradictory to, almost… almost as if you read the next slides, like, the… I'm gonna talk exactly about that. So, great introductions.
Core Francisco-Park: So, now we're gonna start training on multiple tasks.
Core Francisco-Park: So, for example, if you train on distance plus triangular area, which individually gave these kind of stuff, you kind of get this nice pumpkin.
Core Francisco-Park: And now, if you train on the inside task and the perimeter task, you kind of get this other kind of pumpkin, you might convince yourself if you actually, like, scroll the 3D PCA around.
Core Francisco-Park: And, okay, like, maybe not comparing by eyes, let's actually compute the CKA on these stuff.
Core Francisco-Park: And what you get is this. So here, each…
Core Francisco-Park: Row R column, the row is column, so it's trained on two tasks. DT means distance triangle area, ACO means angle compass, stuff like that.
Core Francisco-Park: So, the red triangles means that you have partial overlap. So these two have, for example, like, a triangle overlapping T.
Core Francisco-Park: So… but I want you to not look at those ones with triangle, because it's kind of expected that if you share some part of the data, half of your data, in fact, you might get similar representations. But what's interesting is that even if you don't share, even if you don't share tasks.
Core Francisco-Park: A test trained on distance triangle area and angle compass still has a higher CKA. These values are way higher than what we saw before here.
Core Francisco-Park: Nope.
Core Francisco-Park: So before you saw this, Now we see values which are much higher.
Core Francisco-Park: So yeah, so if you train on multitas, even if your model doesn't actually share the data, so if you share the data, I think it's kind of expected, if you don't share the data, even if you don't share the data, this starts kind of converging.
Core Francisco-Park: And if you add more and more tasks.
Core Francisco-Park: These three tasks converge even more, and in fact, one thing interesting is that, like earlier layers, the…
Core Francisco-Park: So this one, it's a hypothesis I'm working with on Andrew Lee, a postdoc at Harvard.
Core Francisco-Park: Is that, like, modals, representations forms at the end layers and then propagate kind of backwards into the earlier layers. You can kind of see that if you train on more and more tests, you can detect this kind of representational CKA going up even in the lower layers.
Core Francisco-Park: And this almost, like, as you said, directly points towards, like, the platonic representation hypothesis. In their paper, exactly, they have this hypothesis in there. Maybe this is because… maybe the reason representations converge is if you do large-scale training on a bunch of data.
Core Francisco-Park: you…
Core Francisco-Park: Out of the representation space, there's only a few representations which solves different tasks, so you're kind of converging there.
Core Francisco-Park: This is kind of a synthetic, like, example where, at least in the synthetic data, it seems like we're seeing something like that.
Core Francisco-Park: So, yup.
Core Francisco-Park: So… and then I did this thing where… which will nicely go to the next kind of chapter, but you can train on all seven tasks, so this is just performance. There's… some are accuracy, some are absolute error, they perform.
Core Francisco-Park: And if you take 7 tasks, and take the PCA, you're not doing any linear probes, you can already, like, see the world in the PCA. So, there's something interesting here where, compared… so these are single-task PCAs, so even in single tasks, you're able to read out linearly the world.
Core Francisco-Park: But here, like, you can actually just see it in the PCA right away, without training any probes.
Core Francisco-Park: So multitask learning, in fact, enforces this, like, the magnitude of this linear representation of your world, somehow.
Core Francisco-Park: And I'm sure that is related to, like, the multitask CKA going up, I'm sure.
Core Francisco-Park: So, kind of the first pre-training stage findings are, like, world representations emerge.
Core Francisco-Park: And then, if you train on different processes, you get different representations. But if you're training… start training on more than one task, your representations actually start converging.
Core Francisco-Park: So that's what we find, and then, one other thing we find is that, like, if you start training on many tasks, like, the magnitude of this representation gets bigger and bigger, and you almost see the linear subspace right in the PCA, some… if you train on many tasks.
Core Francisco-Park: Any questions before I go on to the next part? Yep, so the assumption on this last point is that, kind of like, if you have multiple tasks.
Core Francisco-Park: The way for representations to effectively
Core Francisco-Park: Encode this information that is… can be reused across tasks is linearly.
Core Francisco-Park: Yes, I think here… I think there's definitely something about…
Core Francisco-Park: You need to have strong representations at high magnitude, linear readout, if you have multiple tasks which want to read out from that representation.
Core Francisco-Park: I think that is the picture I'm thinking about. I don't exactly know why this is happening.
Core Francisco-Park: Could you go back? Yep. To the slide where you were showing single tasks?
Core Francisco-Park: Single tasks.
Core Francisco-Park: Like this one, yeah.
Core Francisco-Park: So here… Just trying to understand here. So the linear probe is able to separate out the… Oh.
Core Francisco-Park: Sorry, so the CKA…
Core Francisco-Park: The linear probe is just a linear… best linear projection of the XY coordinates of cities, and the CKA doesn't know about the linear probe, it's just you take the representations, you compute the raw distance. Both of them are on the fifth layer. Both of them are on the fifth layer, yeah. So, okay. So the linear probe is able to separate the data.
Core Francisco-Park: Which, able to separate the CDs? Yep.
Core Francisco-Park: But the CK is showing that the representations are not similar to each other.
Core Francisco-Park: Yeah, so CKA is actually magnitude dependent, so if you have, like, a bunch of representation, for example.
Core Francisco-Park: Okay, say you have, like, your mobile internal space is 4K dimensions, and then, like.
Core Francisco-Park: 3,998 dimensions are high magnitude and random, and you have two dimensions where your coordinates are perfectly linearly represented, then you can actually read out perfectly linearly, but your CKA will be terrible. Because, yeah. So CKA doesn't actually know about what is the relevance of space here.
Core Francisco-Park: So CKA just computes, like, the raw distance between, like, representations, and then the linear probe.
Core Francisco-Park: Even if the CKA is very bad, you can still probably read it out, because you are actually, like, finding for what is the right dimensions. And if you take the CKA of those dimensions? If you take the CKA of the linear probe dimensions.
Core Francisco-Park: So, I think that you're forcing it to converge, because you'll be taking CKA on two dimensions, vectors, which you force to converge to a map. So, I think, yeah, I think you're kind of optimizing towards high CKA, so… Yeah, I think my main confusion, why do… why are we seeing different…
Core Francisco-Park: like, difference in the representation similarity? Oh, yeah, yeah. Yeah, I think… Separation as well.
Core Francisco-Park: So we are seeing the separation as well… Separation. Separation. Separation? Separation, the same separation of cities. Oh, here, yeah. The second line. Oh, the second line, yeah. Yeah, you separate the continents, yeah. But you can't… but the similar… like, the representation similarity is not that high. That means the same kind of… It is…
Core Francisco-Park: not that low. It is not… not that low. And so actually, so I think the right… the right road to stare at when you look at CKA is a PCA plot, because these are the highest magnitude directions in the representations, which will affect the CKA the most.
Core Francisco-Park: So…
Core Francisco-Park: They are not that low, so that… because you can stare at these, like, clusters, at least, like, in everything, they're kind of clustered together, so the distance matrix will be roughly similar.
Core Francisco-Park: But then, even if you're able to separate here, if the magnitudes of these directions are not that big, it won't affect the CKA that much, even if you get the same map out here.
Core Francisco-Park: Yeah, so that one didn't… CR is not in the… oh, it's not tier, yeah. And also, like, I think I missed two other tasks, because I had to put it in the paper. Like, yeah. So… but… so when you say magnitude… Yep.
Core Francisco-Park: So, would it be accurate to say.
Core Francisco-Park: The linear probe is asking whether the information that you care about is present.
Core Francisco-Park: But CKA is asking about whether the dominant information is the same.
Core Francisco-Park: Yes. Yes.
Core Francisco-Park: Make sense?
Core Francisco-Park: Yep.
Core Francisco-Park: So, you said CR… the CR task? Yeah, it's some two-hop reasoning stuff, I think. So when you do multi-task, it definitely does get… Oh, yeah, yeah, yeah. As soon as you add any task to it, the… one of the tasks kind of makes you this nice word of representations, and then it uses it.
Core Francisco-Park: Well… I think that's the right way to… the right representations, the task becomes easier. Yes, yes, yes.
Core Francisco-Park: Definitely.
Core Francisco-Park: Which applies, like, some curriculum? Some curriculum, yes, definitely, yep, yep, yep.
Core Francisco-Park: Boring question, but, like, you didn't, like, try to optimize it because of the environment.
Core Francisco-Park: If you try to do a similar kind of thing? Well, I am optimizing.
Core Francisco-Park: Oh, oh, sorry, sorry.
Core Francisco-Park: Oh,
Core Francisco-Park: Oh, the CR task. Yeah, I tried hyperparameters, I think it's this two-hops thing. There are these weird tasks which transformers just don't refuse to learn, like parity, stuff like… there are some tasks which are kind of, like, known to be annoying.
Core Francisco-Park: Yeah.
Core Francisco-Park: Okay. We go to the fine-tuning section, though?
Core Francisco-Park: Maybe I spoiled a little bit of content. Okay, so now we're gonna go to a second question. Why does fine-tuning not trigger the right updates?
Core Francisco-Park: So, the reason I went to all this setup of, like, separating world and data generation process, so that now we can intervene the world.
Core Francisco-Park: I'm gonna add cities here. Atlantis, who knows the legend of Atlantis?
Core Francisco-Park: that there are a bunch of cities, or maybe one big city, which is in the Atlantic Ocean, which, like, got submerged. So I'm gonna add Atlantis there. So now that's part of the world. It's been discovered, there are a bunch of cities there.
Core Francisco-Park: Now, that's part of the world.
Core Francisco-Park: Okay.
Core Francisco-Park: So…
Core Francisco-Park: And now, the world has an update. And now all these tasks here are actually defined consistently. The world changed, you could have just generated the data on this right away. So, you don't need to change any of this, and in fact, in the code, you don't actually need to touch that part at all, you just plug in the world, and then the data comes out.
Core Francisco-Park: The world changed.
Core Francisco-Park: And now, how are you gonna update this to the models? If you kind of think about how we update models in the real world, there's, like, a world change that makes it into the news, it goes into the internet, it propagates information. You gather that and fine-tune your LLM, right? We're gonna do the same thing here.
Core Francisco-Park: We're gonna curate some fine-tuning data, where…
Core Francisco-Park: Atlantis is one of the two cities. It could be probabilistically.
Core Francisco-Park: like, both, but, like, we constrain the data so that it's, like, a rich, fine-tuning dataset, which always has Atlantis in one of the two cities.
Core Francisco-Park: And then we're gonna evaluate generalization, because
Core Francisco-Park: you're fine-tuning in, like, a few handful of cities, in reality, like, 100. If you… and deep learning needs data diversity, so I'm being very friendly to deep learning, because, like.
Core Francisco-Park: I could have done this thing where, I give 100 cities, and then if I expect perfect modeling of world, I could do this thing where I give it just enough information to triangulate out the city, but I'm gonna be more friendly to deep learning, and I'm gonna curate a big dataset.
Core Francisco-Park: Of, like, a bunch of cities interacting with each other.
Core Francisco-Park: And… but then I'm gonna ask deep learning to generalize.
Core Francisco-Park: to generalize to other evaluations. So if you have integrated this new city correctly in your representation, can you generalize it to other tasks? I only trained you on distance. If you know what a city is, if you have learned through all this training.
Core Francisco-Park: What a task is, and then what cities are.
Core Francisco-Park: By fine-tuning on this, you should be integrating this new concept, this new entity, into the right place.
Core Francisco-Park: And then generalizing on other evaluations. And then I'm gonna ask on, like, downstream tasks.
Core Francisco-Park: And I'm also gonna be friendly to team learning again here, because the model never sees the combination of Compass and this Atlantic City, so I'm gonna add a small, like, what is it, elicitation buffer, which is, like, I'm gonna elicit the modal a little bit, say that this compass task can go with Atlantis.
Core Francisco-Park: Okay, so I'm doing, like, things friendly for game learning.
Core Francisco-Park: So…
Core Francisco-Park: And what I'm gonna show is, I'm gonna show normalized improvement, where 0 means Atlantis scores as bad as before fine-tuning. So before fine-tuning… so these, again, are not new tokens. So that's an important part, which actually makes more knowledge to a real language model. These are not new tokens, these are just, like, different combinations unseen, the ID is.
Core Francisco-Park: Atlantis… so you can score, actually, Atlantis before fine-tuning, and the score is, of course, bad, so I'm gonna normalize whatever kind of score I get by 0 being as bad as before applying any fine-tuning, and 1 as Atlantis is indistinguishable from any pre-trained city.
Core Francisco-Park: So here's what happens. So, each row is a modal where I fine-tune Atlantis in using some data generation process.
Core Francisco-Park: And each column here is the evaluation of how it performs.
Core Francisco-Park: So some tasks, like the angle task, generalizes very well. So the diagonal has a small t on it because it's trained in distribution generalization.
Core Francisco-Park: And what you see is that some task generalizes pretty well.
Core Francisco-Park: And then there are some tasks we can kind of understand, because it's… it submits less information, crossing task is just true-false. Generalizes… kind of generalizes to some cases, less to others.
Core Francisco-Park: And then others, like its distance task, which is also a little bit CKA suspicious, actually is pretty bad at generalization.
Core Francisco-Park: Generalizing. Yes. Quick question. The pre-trained model is trained on all the tasks? Yes, the pre-trained model is trained on all seven tasks, and here the evaluation is only on Atlantis.
Core Francisco-Park: So it's… if you see one, it means that Atlantis is as good as, like, any pre-trained city. If you see 0, it means that Atlantis is as good as if you didn't do any fine-tuning.
Core Francisco-Park: So the best fine-tuning tasks are the ones that are green all the way across? Yes, that's… yeah, angle is the best one. You fine-tune on angle, you will automatically get 96% of the recovery on the compass task.
Core Francisco-Park: Yep. Gabriel, I think angles are better than distance.
Core Francisco-Park: Are you doing some geometry? Okay. I might miss this one. Is there a comparison between how the free fine-tuned model performed on a task versus, like, on stuff that's not related to Atlantis?
Core Francisco-Park: Oh, I see, okay. There's, like, a small catastrophic forgetting that you can mitigate by adding, like, pre-training buffers. Yeah, it's not, like, something I focused on, because, like, I think in reality, to make it, like, correct, you need to solve all these catastrophic forgetting issues, you need to, like, put a huge buffer of not forgetting. Yeah, stuff, yeah.
Core Francisco-Park: Okay, so… This is it, this is kind of… dirty, honestly.
Core Francisco-Park: So, what I'm gonna do is now I'm gonna directly jump to… oh, actually, I'm gonna plot… just show one thing, which is not too significant. One thing which is interesting is that you can plot these different points by their CKA at pre-training stage. So, okay, so this is…
Core Francisco-Park: A relation which I see, but I don't want to push too strongly. It's a weak relation.
Core Francisco-Park: And what I see is that I think the CKE between
Core Francisco-Park: like, the CKA values between tasks are weakly predictive of how well you can generalize, and I think what that kind of means is that, like, maybe that there is something intrinsic. As soon as you set, like, architecture and the data.
Core Francisco-Park: maybe there is kind of a generalization kernel, which is kind of, like, set by setting that up, and then your model, like, that is fixed by your architecture and your task, and still remains even though you did join pre-training. Okay, but would you state the hypothesis again? The hypothesis is that
Core Francisco-Park: You…
Core Francisco-Park: you compute the CKA between models which are pre-trained, only on, like, one task, another. So the CKA we saw during, like, the first chapter.
Core Francisco-Park: you get those CKA values, that's your x-axis, then you're arguing that
Core Francisco-Park: Tasks which didn't agree that much when individually pre-trained.
Core Francisco-Park: Doesn't, generalize well during fine-tuning as well.
Core Francisco-Park: That's a hypothesis.
Core Francisco-Park: And then we see a weak relation there.
Core Francisco-Park: And… which is pointing towards something, I think.
Core Francisco-Park: So, but here, I think there's a lot of analysis to do to see, like… I think… I'm trying to get through the mass. Is there a mathematical way to kind of get a kernel out here? Some gradient kernel I'm changing kind of stuff? I don't know.
Core Francisco-Park: So, we see some relation here. This is a smoking gun.
Core Francisco-Park: We're gonna jump to two-task fine-tuning.
Core Francisco-Park: So just as we did two-task pre-training, we can also do two-task fine-tuning, and again, I think the base model here is trained on all seven tasks on all pre-trained… all pre-training cities. All cities without Atlantis is trained on all seven tasks. We're just using two tasks to fine-tune in Atlantis now.
Core Francisco-Park: So, this is dirty. I'm not even gonna ask you to look at this and, like, like, say anything. Okay, this is hard.
Core Francisco-Park: What we're gonna do is, we're gonna set up this best teacher model. So, what you can expect when you train on two fine-tuning tasks is, I think, a very reasonable assumption is you do as well as the better teacher, is something you can expect. So here was the first, like.
Core Francisco-Park: One task fine-tuning.
Core Francisco-Park: And for example, say you're… now you're gonna fine-tune your 7-task model on the Compass and Insight task. You're gonna use Compass and Inside to fine-tune in these Atlantic cities, and you're gonna evaluate on crossing.
Core Francisco-Park: So this hypothesis is saying that, if this is true, if you're learning from your best teacher, you're saying that you can take the fine-tuned one on Compass and inside, you go to the crossing column, you take those numbers, 0.59 and 0.65,
Core Francisco-Park: And you take the best teacher, better teacher, so basically crossing, when fine-tuning on these two, should perform, 0.65. So, so this is… this is… I'm not doing any experiments, this is just a modal I set up. So, if you're fine-tuning.
Core Francisco-Park: takes is as good as taking the better teacher of the two tasks, okay? Right.
Core Francisco-Park: And I'm gonna subtract that modal. I'm gonna just get the deviation from that model, which means that everything you're seeing here, if it's… if the pixels are blue, it means that you actually did better than your best… better two teachers. So somehow, learning from two teachers managed to generalize better on an untrained task than learning from one teacher.
Core Francisco-Park: And red means interference.
Core Francisco-Park: Red means that training on this task actually harms.
Core Francisco-Park: training on two tests, somehow they fought. The teachers didn't agree or something, so you actually got a degradation. You might have as well just not… just removed one of the teachers. So that's what you're seeing in red.
Core Francisco-Park: And, in fact, you can see that all fine-tuning tasks, like, involving the distance task, anything involving the distance task, you can see that, like, this red row, which means that whenever you involve the distance task, it harms your fine-tuning. It harms. It's not just that distance doesn't contribute enough, it actually harms.
Core Francisco-Park: Okay. Yeah, so distance is, like, you say DIST, you put in city 1, city 2, it outputs you the distance between the two, those two cities.
Core Francisco-Park: No, it, the task is trained autoregressively to predict exactly the output. The… so, yeah, kilometers, yeah. So it's, like, zeros, for example, 385.
Core Francisco-Park: So yeah, that's the task.
Core Francisco-Park: Okay.
Core Francisco-Park: So you actually find that there are some of these tasks where it actually shows interference. And actually, whenever the distance task is involved.
Core Francisco-Park: It sabotages, like, multitask fine-tuning. So it actually, like, harms. It harms you.
Core Francisco-Park: Any questions here?
Core Francisco-Park: Is the distance the one that is by far the hardest to de-tokenize? E-tokenize.
Core Francisco-Park: Yes, because the numbers get kind of low.
Core Francisco-Park: Oh, there's this task called… triangle area, where the numbers get huge. Oh, okay. Yeah.
Core Francisco-Park: If anything, like, it might be the inverse, where the discus is somewhat easy, especially within a transformer to compute. Sum up some…
Core Francisco-Park: some, I don't know, some dog product, something, I don't know.
Core Francisco-Park: But then, like, there's an easier one, I think, which is Compass, which also involves only two cities.
Core Francisco-Park: So, yeah, so I'm actually, like, hunting for maybe there's more, like, divergent… I call these divergent tasks because they kind of don't like everyone else.
Core Francisco-Park: But of course, Chris's real question is why?
Core Francisco-Park: Yes. YP. Yes.
Core Francisco-Park: What? I do not know, but…
Core Francisco-Park: I was gonna say the next slide! Well, yeah. So, this chart is showing the deviation from the max of the two individual performances, but for the distance rows, just out of curiosity, how does that compare to the performance of just the distance factor?
Core Francisco-Park: So… Let's see… so this already means that it does worse, because it's taking the maximum of
Core Francisco-Park: only fine-tuning on distance, and only fine-tuning on angle, and… oh, I see, I see, I see what you mean, yeah. You're doing worse than on the angle. Oh, okay, I see what you mean, yeah. What is it doing better than fine-tuning just on distance? Like, is the DA there? Yeah, yeah, yeah, I think it…
Core Francisco-Park: I think it could be doing very well, because distance basically did zero on everything, so… let's see, DA, 0…
Core Francisco-Park: to 1818 ADate.
Core Francisco-Park: Well, okay, I can't tell. Okay, I can tell from here, I guess.
Core Francisco-Park: DA. So… Yeah, it just deviated negatively, but it did way better than distance, I think.
Core Francisco-Park: Yeah, so okay, but I do not know why, but at least, maybe this is expected, at least I looked at the representations, what happens in there.
Core Francisco-Park: So, if you take a good…
Core Francisco-Park: nicely agreeing to tasks. And now you ask the modal, you get a modal at the end. You do the same thing, you get the representation of… you put in Atlanta cities where they are, and of course, initially, they start, like, randomly. They're distributed all around the globe.
Core Francisco-Park: And if you take a model where they kind of agreed, you see that the Atlantis cities actually integrate well, so here the probe is not trained on Atlantis. You can kind of see that the Atlantis is reconstructed kind of where they should be.
Core Francisco-Park: If you take tasks where The diversion task was involved.
Core Francisco-Park: your representations are just there, and I think that's, like, the dataset mean of, like, all the representations. And basically, your representations are just not integrated in the right place. We're gonna look at, kind of, a
Core Francisco-Park: Where this is the data… there's, like, a data manifold, and you see Atlantis there? There, there, there, there, there.
Core Francisco-Park: So you can see that if it's well integrated, it goes in well there. But if it's not well integrated, what happens is that it's floating outside of the data manifold there.
Core Francisco-Park: Somehow outside of the data manifold. So this is, like, pointing towards something, at least, that
Core Francisco-Park: Diversion tasks, like, actually disrupt representational integration, so they kind of, like, put them somewhere else.
Core Francisco-Park: So this is it, and yeah, thank you very much, and actually, I did this fun thing where a lot of this code, like, I directly supervised cloud.
Core Francisco-Park: to, like, write the code, and then, like… I actually, like, managed to… I think I actually managed to make my research more rigorous, because I reproduced the whole experiments 3 times from scratch, which is something I would never do before AI, so…
Core Francisco-Park: But anyways, if you scan this, like, QR code, you can actually see the whole research process divided into, like, when I'm… when I just, like, do raw work of coding, when there's a major pivot in the project.
Core Francisco-Park: And then, like, when there's ideation and moments, so, like, you can have fun scanning this and staring at this. What is that tool? So, is this just a GitHub? No, no, no, no, this is…
Core Francisco-Park: this is all custom. It's like, I interact with AI, there's a logging system which logs all of that data into a… Yeah, and then… and then… Vibe coded a meta vibe coding interface for you to do Vibe research. No, the Vibe coding… The Vibe coding is just cloud. I'm collecting all the logs, so that one day I can fine-tune a model to replace myself, and… Oh, I got it.
Core Francisco-Park: Yeah, but… But actually, like, okay, I mean, there's a whole other talk
Core Francisco-Park: maybe I should prepare about this is because I really think continual learning is the next problem after looking at this. No, I love it. Because this is a lot… I realize that research is way, way, way, way more than a modal's context length. Like, way, way more.
Core Francisco-Park: Anyways… Anyways, like, scan this, it's gonna be pretty fun. And…
Core Francisco-Park: And yeah, actually, I'm gonna show some of the… what is it? Can I… I'll show the whole screen, and I'll share some of the…
Core Francisco-Park: What is it? I'll just show… these stuff, where…
Core Francisco-Park: what happens in the PCA is that if you have an ill-integrated case, so here's the process of fine-tuning.
Core Francisco-Park: Oh, sorry.
Core Francisco-Park: Oh, I… Let's see… Worth.
Core Francisco-Park: Oh, oh no.
Core Francisco-Park: Hmm.
Core Francisco-Park: Or if you pull down, wait, why, why does it… PowerPoint might be owning the thing, if you stop PowerPoint.
Core Francisco-Park: I see, okay. It might… Create up the…
Core Francisco-Park: Then you have to choose mirror access. Oh, what does choose mirror for?
Core Francisco-Park: Should I just share, or… No.
Core Francisco-Park: Let's see here… Oh.
Core Francisco-Park: Like, up here.
Core Francisco-Park: And then… Oh, this. Yeah, this thing. Is there… Do I remove it? Yeah, choose content? Yeah. And then the entire screen. Entire screen.
Core Francisco-Park: Yep. Okay, great.
Core Francisco-Park: Yeah, so this is the fine-tuning process, and if the… if whatever is… ill-integrated.
Core Francisco-Park: If these new Atlantis stuff is ill-integrated.
Core Francisco-Park: Do you see that there's, like, a data manifold of cities, and then it's, like, floating in the middle there?
Core Francisco-Park: Unfortunately, this plotly doesn't respect what's in front, so here's Atlantis.
Core Francisco-Park: It's, like, floating in the middle there. Yeah. I think it's sort of spherical. Yeah, it's sort of spherical, because the, what is, activation magnitudes are constrained by regularization.
Core Francisco-Park: And… here…
Core Francisco-Park: If you see… you can see that when it's well integrated, it's kind of there where it should be. Like, it's still… there's still a little bit of, like.
Core Francisco-Park: Difference between, like, the pre-trained cities, but it's kind of there.
Core Francisco-Park: And if you try to, like, forcefully,
Core Francisco-Park: Okay, this… I think this one is the more interesting one.
Core Francisco-Park: Okay, so here, now you're trying to, like, forcefully, I don't care, get me the linear… linear probe, including some…
Core Francisco-Park: Some Atlantic cities are in the training data, Of the probe.
Core Francisco-Park: And… In the well-integrated case.
Core Francisco-Park: You can actually train a probe, including Atlantis, in the probe training data, and then it's, like, right there, Atlantis.
Core Francisco-Park: And… but in the ill-integrated case, you can still say the same thing, where, yeah, still give me the linear direction where Atlantis goes where it should go.
Core Francisco-Park: And…
Core Francisco-Park: If you go to the XY coordinate, it's kind of there, Atlantis. It managed to put it there, right? Between, like.
Core Francisco-Park: Europe and North America.
Core Francisco-Park: If you look outside, it's, like, floating out there.
Core Francisco-Park: So, there's some… there's something wrong about these Atlantis representations. You try to train a probe on it, kind of like, you have a lot of dimensions, you can find a dim… you can find some dimension which kind of, like, projects it to the right thing, but then it clearly is causing, like, some problems in the representation space.
Core Francisco-Park: Should I just push it up onto the… Oh, like, like…
Core Francisco-Park: Like… Yeah, no, I haven't tried that, that's… is that profession of this lab? Like… Representation surgery? Yeah, surgery, that's, yeah. That's a good idea. Yeah, yeah, there… I don't know, I was, like, stuck with, where do I push it? Do I push it at layers 3, 4, 5, all of them? Oh. If I…
Core Francisco-Park: Yeah.
Core Francisco-Park: Yeah, I was, like, just, like, sitting, should I do this? And then I was like, oh no, deadlines, not that. I'm wondering if, yeah, the representations are causal.
Core Francisco-Park: Yeah, yeah, yeah, so I think… That's, I think, the implicit hypothesis, right? Yes, yes, yes, definitely.
Core Francisco-Park: Yes.
Core Francisco-Park: I think at the high level, like, my biggest takeaway here is that if you… even if you do everything friendly to deep learning, so here, you created a world, you generate a diverse set of data.
Core Francisco-Park: like a multitask data, you generate diverse, you train on a lot of these diverse data sets, you get, like, these nice representations which emerge even in the PCA.
Core Francisco-Park: So, my mental model was that, obviously, the network has this picture of
Core Francisco-Park: Your cities form a representations, and the task reads out from that nice representation space to perform stuff.
Core Francisco-Park: And then, even if you get this nice forward representations, I feel like, oh, backward pass can still be not that ideal, is kind of, like, what I take away, so…
Core Francisco-Park: And maybe there's some learning rate which magically enables, like, maybe there's some gap in the lost landscape. You need to use a high learning rate so that it goes over some local minimum or something, but…
Core Francisco-Park: That's… That's in high-dimensional hyperparameter space, so…
Core Francisco-Park: Yep. I feel like I know the answer to this, but I just want to ask anyways, did you do any training where the loss was geometrically inspired? Like, where it wasn't just, like, oh, it was right or wrong? No, no, yeah, I intentionally stuck to the LLM paradigm, yeah.
Core Francisco-Park: Yep. Did you try, like, integrating a new function, like, fine-tune a new function afterwards? Oh, yeah, because I can do the inverse where I don't fine-tune a new entities, but a new function? Yeah, no, I didn't do that. Like, I had that in mind, they didn't do that.
Core Francisco-Park: I'm also interested in, like, functions which are, one, stochastic, like random walk within some distance, where functions, I think, even harder. I think there's a lot of challenges awaiting if the model does everything ideally. There's, like, a lot of goalposts moving to do, where, like, you can ask for stuff like
Core Francisco-Park: circular stuff, like, count the number of, count the number of, like, cities in a nearby circle, and that actually changes if you integrate new entities. And that's actually more like the questions we see in real life, where there's, like, a new event happening in the world, it doesn't affect this one thing, it affects a bunch of different stuff in your life, so… yeah.
Core Francisco-Park: Do you think this can connect to… are you aware of, like, the causal representation learning literature? A little bit.
Core Francisco-Park: They also care about disentanglement and learning the right latent variables and stuff.
Core Francisco-Park: They have their own, like, conditions of when that happens. Oh, cool. I should read that. It kind of might relate to this multitask setting. I see, I see. Yeah, I should read that, yeah.
Core Francisco-Park: I should read that. No, I don't know that much.
Core Francisco-Park: Yep.
Core Francisco-Park: I'm wondering if you know this O.A. Neves paper where they…
Core Francisco-Park: Do you know what I'm talking about already?
Core Francisco-Park: Maybe. If it's a famous one, I think I do. So what they do is they, like, take an LLM. Yeah. Ow? They've given a new token. Oh.
Core Francisco-Park: And this is, like, token ID, you know, 5201.
Core Francisco-Park: a new token, randomly initialized. Then they pretend that this new token is some city, let's say Paris.
Core Francisco-Park: Oh, I know, yeah, I know this one. Out of context, yes. Yeah, out of context, I think. And then, but if I remember correctly…
Core Francisco-Park: they train on distance examples. So, this token is, the distance from this token to Dubai. Yep, yep. And then they give it the distance from Paris to Dubai, and they do this with a bunch of cities. Yes. And then, to test generalization, they say.
Core Francisco-Park: I'm in… Bob lives in blah, tonight they're eating…
Core Francisco-Park: And then it'll, it'll say baguette. Yes, yes, yes. …and represents Paris. Yes. I'm wondering, like, it seems like, from your work, the distance…
Core Francisco-Park: task might not generalize so well, but… Yeah, yeah. Yeah, so… so I'm sure, like, large language models, how they represent distance must be very different. Here, I think they're implementing something more direct, because they're almost given direct supervision for the distance task. I'm sure the circuit is much simpler here than the language models.
Core Francisco-Park: Second thing is I do think that, like, fine-tuning new tokens and fine-tuning existing concepts using current tokens are actually pretty different in optimization space, is one thing.
Core Francisco-Park: But I think at the, like, L-level, I think in generality, even here, like, we're seeing that most things actually journalize pretty well. Most things journalize pretty positively once you
Core Francisco-Park: satisfy all the things DeemLearning, like, data diversity, like, elicitation buffer, once you, like, satisfy all of the stuff DeemLearning likes, then generally things… and then, in fact, in that paper, I think they have, like, 1 million rows of data where they do that.
Core Francisco-Park: they have a lot of, like, data. And then, I think in general, deep learning generalizes pretty well. I'm kind of showing that there's these very, like, sharp bugs sometimes, yeah. So I think, yeah. I think the same, like.
Core Francisco-Park: Definitely, in general.
Core Francisco-Park: things work, yeah. Could you say more about this distinct intuition that you have about, like, new tokens versus… Yeah, so when you do a new token, like, you're basically initializing a new embedding vector, and then that
Core Francisco-Park: That, I'm sure the lost landscapes looks different, because if you're optimizing, like, this new vector.
Core Francisco-Park: It just must look different from optimizing the weights of the model, which process already a bunch of tokens which have, like, embeddings which support many different tasks.
Core Francisco-Park: Which, which I, yeah, I'm sure are strongly optimistic. Yes, you're optimizing… Yes, exactly, it's a different… and yes, and I'm sure dynamics are a little bit different, so…
Core Francisco-Park: Not sure exactly how, but…
Core Francisco-Park: I think maybe, like, the model could do something, like, look at, like, for this ID, it could have an MLP neuron that's, like…
Core Francisco-Park: only lights up on that ID, and then it could effectively write the, you know, custom embedding just for that ID or something like this, that they would be roughly equivalent, but…
Core Francisco-Park: Yeah, could be.
Core Francisco-Park: Nope.
Core Francisco-Park: Okay, so I have two questions which might actually be related.
Core Francisco-Park: Did you look into early layers? What are they encoding? What are they representing? Early, yeah. And the second question is, so I think one of the… one of the slides, you showed that the…
Core Francisco-Park: performance of ICL was better than the pre-fine-duty one. Yep.
Core Francisco-Park: So the reason for that, I don't think so I got that. Oh, I see, I see, yeah, yeah, great, great, great, great questions.
Core Francisco-Park: early layers' representations don't move that much in general. So, don't move that much in general. And one thing is this structure,
Core Francisco-Park: the structure, even if you train for a while, the later layers, layer, like, 6, 5, 4, gets this word representation structure in some form or the other very quickly. Layer 3 is very slow, and I'm sure Layer 2 will at least start showing representations after layer 3 is already very good.
Core Francisco-Park: So I think basically never here, because… and Layer 3 is very slow, and…
Core Francisco-Park: I don't know why, actually. I don't know why. I discussed this with a few friends at Harvard.
Core Francisco-Park: We actually think that maybe… maybe this should be fixed, I don't know.
Core Francisco-Park: And about the second question, yeah, so here, basically, there's no ICL, and in fact, like, one of the
Core Francisco-Park: most promising thing, I think, to do is, in fact, introduce ICL here. And by that, I mean, like, define some cities which are only given an inference time, which never… which there's no pre-training information on.
Core Francisco-Park: You define a city in context using some specification, and then the model needs to perform something about that city, and then the pre-trained cities, or between even in-context cities.
Core Francisco-Park: And if we see the nice amount of, like, generalization we see in large language models, which generalize much better in context compared to fine-tuning, I think that kind of, like, can lead a way of, like, presenting the same information in context and gradient descent somehow, like, one generalized much better than the other. Why? So…
Core Francisco-Park: Yeah, and indeed, there's no experiments about in-context learning here, but yeah.
Core Francisco-Park: So that would be, like, another setting case study.
Core Francisco-Park: Yes, I think it's gonna be… I think the world can be maintained. The world can be maintained. I think you just need to define in-context cities, which are only… which only appear during, yeah, the data generation process in context, and there's no pre-training information, yeah.
Core Francisco-Park: Which means, like, inconsistent across rows or something, yeah. Not the same city across different rows.
Core Francisco-Park: That means, in context, so… Yep.
Core Francisco-Park: Yes?
Core Francisco-Park: What was the distance, the divergent tasks that made the Atlantis come out? Yes. And…
Core Francisco-Park: Do you have any intuition around, like, why…
Core Francisco-Park: That task in particular didn't seem to, like, Yes, I…
Core Francisco-Park: I'm suspecting force is too easy. I'm suspecting Force is too easy. The model can just represent things, do a dot product that can read out. Not sure why easy would mean anything like divergent during fine-tuning.
Core Francisco-Park: So, yeah, I don't know. Did you have, like, any stopping criteria for the sign-tuning, or did you just do, like, okay, like,
Core Francisco-Park: I create my dataset using my pipeline, I use the same parameters as for training.
Core Francisco-Park: Yeah, the fine-tuning is, like, I did a small learning rate sweep, but that's all.
Core Francisco-Park: Yeah, I did a learning rate sweep where I started with, like, 1 tenth of the learning rate, then did, like, perturb it 3 times more, 3 times less, so it didn't do anything special. Maybe I should use a crazy high learning rate, and things will change.
Core Francisco-Park: Oh, and by the way, of course, like, if this distant task is… if Atlantis is include, of course, included in the pre-trained data, then everything works nicely, yeah.
Core Francisco-Park: So it's really something about optimization there, because if you have this data scattered in your pre-trained data, things totally work.
Core Francisco-Park: Yeah, but… because, in fact, I say fine-tuning, but actually there's more Atlantis… Atlantis is actually over-represented, if you look at the pre-training set plus this fine-tuning set.
Core Francisco-Park: It's, like, compared to number of cities, it's actually overrepresented, which means that if you actually mix everything, it behaves all fine, and it actually does, that's… there's… yeah, this is… well, you can see that the geometry isn't that special, like, there's no reason, you know, electrics will just hate that, so…
Core Francisco-Park: For a training task overall, like, let's say that you are doing the distance task, and that you have two distances, like, you have your target distance that is gonna be, like.
Core Francisco-Park: a four-digit number, right? Okay. And, maybe, like, 1111, right? And your, actual distance that gets predicted by the model is, like, 9111, right? Which is, like, 800 kilometers away. Do you penalize one token?
Core Francisco-Park: So… Like, it's kind of like it got 3 right, 1 wrong, or does it get 8,000 as the signal for, like, oh, you got 8,000 wrong?
Core Francisco-Park: it gets one wrong. It's cross-entropy loss, just like language. Yeah, so it's like, yeah, it's not actual geometric loss. Yeah, but that's interesting, right? Yeah. Because it would be, like, intuitively, actually, if you're measuring distance, that would be actually a huge error, right, to get the first one wrong, rather than the last one.
Core Francisco-Park: Yes, exactly, yes, yes. I think loss functions being… Yeah.
Core Francisco-Park: Yeah, totally. It is a setup where, like, the loss function isn't aligned towards the metric, but then the metric actually behaves fine.
Core Francisco-Park: But all of your tests are set up that way, so it's not, like, the thing that explains why… Yes, all the tests are set up that way. That would make a difference if you're doing something like RL, but… Now, your linear probe readout is not set up that way, that's the linear… when you… The linear probe, yes, it's… yes, the linear probe is actually…
Core Francisco-Park: linear regression towards the values. So it penalizes the 8,000 a lot more.
Core Francisco-Park: Yes, except that linear probes are just trained on XY coordinates, but… Right. Yeah. But yeah, those are actual coordinates, actual numbers. Did you play this model size? Yes, but I didn't find anything.
Core Francisco-Park: So it was bigger than that. Not, not, not… yeah, I didn't try crazy modal sizes, it was just, like, 128 tenant dimensions, 6 layers.
Core Francisco-Park: Can we go this way? Yeah. So I'll start. Okay, the plot that you have behind the Zoom screen…
Core Francisco-Park: is if, oh, you can share a screen so that people can see the business.
Core Francisco-Park: Oh. Questions? Oh, yeah, you're not, you're not sharing anymore.
Core Francisco-Park: I guess I'm not sharing anything right now.
Core Francisco-Park: Okay.
Core Francisco-Park: Should I share something? There's some questions in the chat. Oh, okay, I see.
Core Francisco-Park: I'll share this for… Nice visual there.
Core Francisco-Park: Okay, I'll answer this question after your question, and then go for it. Yeah.
Core Francisco-Park: So, this is when the distance doesn't work.
Core Francisco-Park: Yes, this distance is included in the fine-tuning tasks, yes. Right. So, for a single fine-tuning task, where everything worked really well, is it compass? The single fine-tuning task where
Core Francisco-Park: It generalized pretty well. Anything which doesn't involve distance generalizes reasonably at first task, there is that 30 matrix, and then as soon as you have two tasks, they generalize very well. So if you look at their representations… Yep.
Core Francisco-Park: It would be perfectly fitting into the…
Core Francisco-Park: Yes, yes, so that's actually, like, precisely here.
Core Francisco-Park: Oh, no, but this is a duo list.
Core Francisco-Park: Sorry? This is a dual multi… multi… This is two tasks, yeah. Oh, even if one, yeah, they fit in pretty well, yeah, they fit in pretty well, even, like, for example, the angle, which did really well, they fit in almost as good as, like, the other, like, two-task fine-tuning ones. I see. So, is this…
Core Francisco-Park: Is the distance observation something you pinpoint to say that there's brittleness in finding? Yes, yes, yes, it's something specific about this distance task, and I'm hunting for more tasks.
Core Francisco-Park: I see, yeah. So, like, you can say it depends on what it is that you're fine-tuning, that would help.
Core Francisco-Park: what task? What task it is that you're fine-tuning. Even, like, just to, like, project it to language models. Yes, yes, yes. Exactly. It's gonna be… so, basically what this says is that it's gonna be terrible if, like.
Core Francisco-Park: You have new news about some stuff in the real world, some new concept.
Core Francisco-Park: And it's gonna be terrible if there's just, like, some things in the data that your mobile hates, and actually doesn't integrate, like, your concept well to whatever else the mobile has. So, yeah.
Core Francisco-Park: And yeah, in real life, probably it's hiding somewhere, but yeah. My question is, it's not the fact itself.
Core Francisco-Park: It's the… This sort of data that represents the fact that might cause the brittleness.
Core Francisco-Park: Yes, yes, yes, yes, exactly. So, there's nothing inconsistent about the synthetic at all. Yeah, it's something about, like, yeah, something about the task.
Core Francisco-Park: Is the first PCAs you are showing calculated across all tests? Sheridan, do you want to ask your question verbally?
Core Francisco-Park: Sharon, I know you're there?
Core Francisco-Park: Do we have any mic on here, though?
Core Francisco-Park: Oh, I guess I must have my mic on. Okay. Maybe share an answer. Oh, wait, wait.
Core Francisco-Park: It's the first PCAs you're showing calculated across all tasks, and the second slide has… this is, like, from, like, the pre-trained experiment, right? For each task separately, or… wait, these are all separate… oh, these are all separate models, yeah. These are all separate models. Question?
Core Francisco-Park: Yeah, yeah, I had this… I didn't, I didn't manage to run the whole suite of experiments, because I saw it, like, but I, yes, I also put something in the Pacific Ocean called Pacificus, and it was the same thing.
Core Francisco-Park: Question of the…
Core Francisco-Park: So you mentioned, this is some optimization thing, probably, because if you sprinkle things into the pre-training… Yeah.
Core Francisco-Park: Then everything goes. So your setup was like, you have…
Core Francisco-Park: A pre-trained model where on the normal world data, you're training on, like, 7 tasks, or, like, all the tasks.
Core Francisco-Park: And then during fine-tuning for Atlantis, you stick a bunch of new Atlantis points, and then you're training on one task or two tasks. Yep. So when you say, like, you sprinkle into… if you were to sprinkle it into the pre-training data, it works fine. Does that mean if you sprinkle the Atlantis points.
Core Francisco-Park: just on the distance test. Oh, I see, okay, yeah. You have the rest of… for the rest of the world, you have all seven kind of views from the data. Does that generalize well? No, I didn't test that, yeah. That should be tested, yes.
Core Francisco-Park: Wait, can you repeat the question? So, like, the setup right now is you have, like, your pre-training… Yeah, that might not work, I don't know. …on all seven tasks, on the, like.
Core Francisco-Park: world.
Core Francisco-Park: After pre-training, you have only one task for the Atlantis points.
Core Francisco-Park: So…
Core Francisco-Park: One question is, like, if you sprinkle in that one task Atlantis view into the pre-training, and just intersperse it into all the data, but you have… still have those seven views of the rest of the world, will that generalize better?
Core Francisco-Park: Yeah, yeah, that's… I think that might not work, actually, yeah, that's true. I… when I sprinkled it in, I sprinkled it in with all data as well, so… Yes, yes.
Core Francisco-Park: So, like, specifically distance tasks? Yeah, yeah. If for, like, some subset of the data you only have one view of that data… Yeah.
Core Francisco-Park: Is it a pre-training versus fine-tuning thing, or is it just actually limited to this, like, restriction? Yeah, great question. Should test that, yeah.
Core Francisco-Park: Yeah, just… same task.
Core Francisco-Park: Well, the data is different here, so in what Andy's suggesting, so I think…
Core Francisco-Park: I think it might make a difference, yeah. Is Andy's data also different from the fine-tuning data? Sorry? Is Andy's data also different from the fine-tuning data? No, it's the same. Yeah, just take the fine-tuning data, yes, just… But in the fine-tuning stuff, do you sprinkle in pre-training data, like, to prevent catastrophic or getting cycles? Oh, I do that, but I think the question is more about
Core Francisco-Park: the Atlantic cities, whether they go well. Not about the construction we're getting, but…
Core Francisco-Park: Yeah, so in the fine-tuning, I definitely sprinkled the pre-training data in, but basically, like, the inverse, yeah. Oh, oh, wait, do I do that? That was my question. Oh, oh, oh, there's a new question, okay, yeah. That's done. Okay, okay, okay, okay. The fine-tuning is like the pre-training that anyone…
Core Francisco-Park: But not from random, we don't initialize with random, but with… pre-print model.
Core Francisco-Park: Right, yeah. Post the representation format.
Core Francisco-Park: Yep.
Core Francisco-Park: So, this is kind of more like a… Like, a loose, fluffy question?
Core Francisco-Park: About, like…
Core Francisco-Park: Your thoughts on the epistemological validity of, like, transferring, like, in our personal understandings of these results onto, like, the real-world question of, like, how do actual language models incorporate, like.
Core Francisco-Park: New news about the world. Yep.
Core Francisco-Park: The reason why I see, like, a potential disconnect there is that…
Core Francisco-Park: When new events arise in the world, Like, yes, they, like…
Core Francisco-Park: they're… they're new, and they're, like, out of distribution of that SaaS. Yep. But I don't see them as, like, totally exogenous in the way that this introduction of, like.
Core Francisco-Park: oh, there's just this entirely new undiscovered area called Atlantis, in my world's model. Like, it's more just, like.
Core Francisco-Park: I… here's… here's one way I would put it.
Core Francisco-Park: Concretely, is if you, instead of,
Core Francisco-Park: instead of creating, like, an entirely new region called Atlantis as the source of your fine-tuned data, if you drew your synth… if you… if you created your fake cities by matching their distribution to the original distribution.
Core Francisco-Park: of…
Core Francisco-Park: So, if you just scattered your new cities so that they roughly matched the distribution of the original data, what do you think the result is? Oh, that actually I…
Core Francisco-Park: That actually, a reviewer asked for it, so that I actually did, and then it does the same thing of collapsing, so it's definitely something about, like… So it's still bad. Because, yeah, and that was very expected for me, because, like, I don't think Atlantis really has anything special about the map, like, Australia is weirder, so,
Core Francisco-Park: So, that I actually wasn't expecting, but I, like, definitely see a…
Core Francisco-Park: point, a disconnected real world, where in the… well, actually.
Core Francisco-Park: I mean, that result actually is kind of enough to convince
Core Francisco-Park: I see. Well, in the real world, I see… I just see a disconnect because of the scale of everything, but, like, yeah, but yeah.
Core Francisco-Park: I don't know if I really thought, so…
Core Francisco-Park: The tokens for Atlantis are, like, totally new, uninitialized… No, they're just, like… so these are, like, C1234, and there's, like, unused IDs, like C2321, and that can… that get assigned, yeah. Definitely.
Core Francisco-Park: we're doing… They're just, like…
Core Francisco-Park: random embeddings in our… in our embedding matrix that haven't been touched by… No, the embeddings have been touched, because it's just like all the alphabet has been touched by language models, but it's just like inventing a new word, and that new word hasn't been touched, so that multi-token combination hasn't been touched… hasn't been touched as a combination yet. Yeah.
Core Francisco-Park: I guess I'm just wondering if, like.
Core Francisco-Park: it's getting mapped to some really weird spot. Yeah, it gets mapped all around the globe. It's starting in some crazy spot, and it has to find its way all the way over to the manifold. Yes, yes. And so, like, if you, like, initialize it as, like, random… like, they're all clustered in some random, like, average of all the…
Core Francisco-Park: Yeah, they're clustered all around the… actually, like, I think you can kind of just see it here.
Core Francisco-Park: You can kind of inverse undo the fine-tuning here.
Core Francisco-Park: And they're kind of, like, just scattered here. Like, they're just, like, in the middle.
Core Francisco-Park: It's all scattered there. Yeah, I don't know what the best… I think this is the best angle, so…
Core Francisco-Park: They're kind of just, like, scattered there. Yeah, the color is… Is PCA all we need?
Core Francisco-Park: I don't think so. Actually, yeah, is there, is there, what is it? Is there some…
Core Francisco-Park: causal PCA, give me the three most dimensions this task cares the most about.
Core Francisco-Park: I mean, you have, like, UMAP or something like that that is nonlinear, right? And that preserves distances that it could be, maybe? Still uncorrelated. Yeah, I want something like, okay, here's my task, like, do something with gradients to give… Some space.
Core Francisco-Park: Yeah, PCF gradients… I don't know if it's…
Core Francisco-Park: Yeah, I don't know if it's the method, or I don't know if it's me, but…
Core Francisco-Park: Oh, it doesn't give you pretty stuff. It was weird, but maybe it was me. I didn't try too hard, so maybe it could definitely be just me. I've had trouble with analyzing gradients. Yeah, I had… I'm not good friends with gradients. At all.
Core Francisco-Park: What about averages of gradients?
Core Francisco-Park: Those sounds better. Those sounds a little bit better.
Core Francisco-Park: Yep.
Core Francisco-Park: I wanted to extend on…
Core Francisco-Park: Sorry? I want to extend on these questions. Yep. Yeah, it made sense to me, and I was thinking about how, like, for example…
Core Francisco-Park: real data often doesn't just completely change. Like, the ample tower doesn't switch to Paris, but there's, like, little attribute updates, so, like, maybe, like, an analogy would be, like, content threat. Like, you know, Australia starts to, you know, move a little bit, and, like.
Core Francisco-Park: does the, you know, maybe we did another measurement of, like, the locations. Yeah, yeah, yeah. You know, what happens if Australia moves, like, you know, a bit over? Oh, yeah, yeah, no, I think that would work much better. If you're, for example, like, detaching Europe and then bringing it slowly to… I think that would work much better, especially if you do it slowly. Yeah.
Core Francisco-Park: But then the real world has these updates, which are… Nope!
Core Francisco-Park: Yeah, but then the real world has these updates, which are… actually, I think the real world updates can be pretty crazy in concept space. Like, like, you get sometimes these news, which are surprising, like…
Core Francisco-Park: wouldn't mention what, but, like, you sometimes get news which are just like, wait, what is going on in the world? Like… That should be normal. Yeah, yeah, what's happening? Like, and these newses actually, like, cause, like, big updates into, like, people's, like.
Core Francisco-Park: Decision processes, which means that actually humans are able to process these very out-of-distribution stuff, because they have a world model, which actually they have, like, a specific way they generalize, so… yeah.
Core Francisco-Park: by NVIDIA.
Core Francisco-Park: Yeah.
Core Francisco-Park: I have a question. When you were fine-tuning on the divergent task, was the performance of the other, like, cities, impacted at all? Yeah, they all are impacted a bit because of catastrophic we're getting. Deep neural networks are just not…
Core Francisco-Park: nice one about Catch for getting. I saw that, like, going away as I add more and more pre-training buffer.
Core Francisco-Park: But decided that I'm not gonna solve this problem, forget it for now. Between training with the diversion task and not training with the diversion task? Oh, was it… was it especially worse? No, it wasn't especially worse, it was kind of similar. The forgetting on the pre-training stuff.
Core Francisco-Park: This is a LAMA model.
Core Francisco-Park: No, this is all from scratch.
Core Francisco-Park: Oh, wait. Oh!
Core Francisco-Park: This was the Quen architecture, actually, Quen 2.
Core Francisco-Park: Probably could be similar. Similar-ish, yeah. Oh.
Core Francisco-Park: rope embeddings. I think that's the thing which might matter. So, yeah, rope positional embeddings.
Core Francisco-Park: Well, thank you very much, Corey.
Core Francisco-Park: Thanks, everybody, Emily.