David Bau: Here, I think I got that one out.
David Bau: So, today we're very lucky to have Gaein Park, who's a master's student in Korea, to talk about some of his recent research for us today. He's over Zoom. So,
David Bau: So, so, yes, and he's gonna be… he's interested in looking at PhD programs next year, so he's looking for opportunities to share his research.
David Bau: So, take it away, Anne.
Yein Park: Okay, I will take in here, and thank you for giving me this chance to present at your lab seminar, Professor Bao. And today, what I want to present is my recent paper, which is named as Thinking Sparks and Emergent Attention Heads in Reasoning Models During Post-Training.
Yein Park: So, today's contents will be like these, and I'm gonna start with short bio of myself, and then start my presentation of research. And also, if we have the time, we can get some question and answering time, too.
Yein Park: So, for the first double, you can, the sound of the voice is right? It's okay, or is it need to be routed?
David Bau: It's fantastic.
Yein Park: Okay, thank you.
Yein Park: And so, this is just a short fire for myself. I'm currently a master's degree student of Korea University, and I'm going to prepare in PhD this year, yeah, for the next forced admissions.
Yein Park: And what I'm interested in is,
Yein Park: Right, and that's engaged processing, and especially for interpretability, and some… also, AI safety or post-trainings. And what I, most interested part is, like, the highlighted part.
Yein Park: From interpretability to post-it training or alignments.
Yein Park: That is like these pages. Yeah. I always had questions that how the internal mechanisms of LRMs handle specific tasks
Yein Park: So I'm… I started from the insights of research on knowledge, what I presented at iClear 2025, and it named it chronology, which, targeted for temporal knowledge of evidence, which, changed across the…
Yein Park: time passed. And then, I…
Yein Park: expand the approach to… with interpretable methods, like using circuits, to find out, is there any specific attention tests, or MLPs, or the other residual streams inside the LLMs has a specific…
Yein Park: effects for those specific tasks, especially for temporal tasks, because I did, similar research before. So that is the name that's temporal head, which should be showed in the later presentation, too, because it is also the basis of my, this
Yein Park: presentations, too.
Yein Park: And so, I want to design an interpretable analysis to trace models' internal computations, so that is why I use a circuit.
Yein Park: And then I'm going to aim to inform and refine subsequent training and alignment methods. So, today is, topic is about the post-training, and…
Yein Park: Like the board text, I do circular level analysis with diverse post-training methods, rebuilding that emergent reasoning-specific attention heads for complex tasks among various modules series.
Yein Park: And… this is the abstract and overview of my paper. Like you can see the picture that, I make some prompt for making circuits, and then trace all the…
Yein Park: checkpoint circuits, and then found that some, attention tests are emergent in circuits, and then I collected it to find out what exactly each post-training shows in this kind of circuit analyses circuits, yeah.
Yein Park: So, I traced the reasoning circuits, and then, after pre- posterally activated attention heads, also implies its performances, too. That is what I found.
Yein Park: So it is a brief description of today's presentation. Complex seasoning abilities in LRMs are largely by SFT and RL improvements running. However, the architectural mechanism behind these games, remain still OPEC.
Yein Park: And so I did a con… I conduct circular analysis to repair emergence of functional specialized attention heads.
Yein Park: And… I'm also preaching… I also want to bridge some kind of micro library circuit analysis with macrover performance differences, so I do, ablation analysis, and…
Yein Park: Checking out how each, has effects for its performances, too.
Yein Park: So, key finding is… it is like these pages, like, there is some… I picked some, famous.
Yein Park: plus training methods, like distribution, and SFT, and then GRPO, and then I also do some, circuit, making circuits for thinking on a photo, which is QN3, in this case.
Yein Park: And the findings in each category is, like, the… Text below the…
Yein Park: Each category, like the distillation and SFT, induce a large number of emergent attention heads, mostly in early to mid layers for distillation, and mid-to-rate layers for SFT layers.
Yein Park: Also, they instill complex reasonings, but can introduce confusions or redundancy among those heads, too.
Yein Park: And also for GRPO, it engages in a dynamic search for reasoning heads during reinforce learning training, reinforce,
Yein Park: I want training.
Yein Park: And also, it performs minimal yet high-impact edits.
Yein Park: Optimizing the use of existing ability without creating new pathways. It also, some URIPS papers, background for my research, also shows similar insights, too.
Yein Park: And the last one, thinking on off, shows that thinking on Model X exclusive heads, but triggers low-buster racing activities.
Yein Park: But, however, thinking of what activates compensatory heads, which improve short-term performances but reduce reasoning, robustness, too.
Yein Park: So, this is the motivation, like, the paper that I mentioned is a temporal head, which is, research to find out some kind of temporal knowledge-specific attention heads in…
Yein Park: evidence. So, I created two comparative set of template, prompts, which is with the temporal, temporal context and without temporal context. Then I create barrier circuits for these, temporal knowledges and.
Yein Park: temporal invariable knowledge. Then I found out some specific attention heads only come with these temporal knowledge. That is what I call the temporal heads.
Yein Park: And it affects highly for temporal knowledges, like, it can change the entity, the object of each temporal knowledge,
Yein Park: And when it is ablated, the temporal knowledge is highly affected, like, goes to the temporarily long object, not effect… not, entirely effectual alone.
Yein Park: And also, we can edit some wrong temporal objects to right temporal object using the value of these temporal heads. I do some, representation engineering.
Yein Park: And then, this… her work would be the background of my approach for this ca- this research, too.
David Bau: So, can I ask about that previous research? Is it okay if I interrupt you, Yan?
Yein Park: Yeah, it's okay.
David Bau: Yeah, so, so in this work, were you able to… so if you ablate all the attention heads that, transport
David Bau: the year.
David Bau: Does it go… does it go to the prediction for the current time, like the most recent, or is there some default that it goes to?
Yein Park: It's not quiet for the listen or, later, but it, the model computes when these temporal heads are oblated, so it cannot correctly answer the light temporal object for that
Yein Park: Knowledge, but it also, the object is not completely wrong, actually, because it…
Yein Park: It goes to another year's object.
David Bau: I see. But it doesn't go to some constant… Yeah.
David Bau: Sure, sure. Just, it goes to some random…
Yein Park: Yeah, yeah, render me ears.
David Bau: I see. And… and then, when you… when you patch… Are you able to patch…
David Bau: Let's say you have a year specified in one sentence about one subject.
Yein Park: Yeah?
David Bau: And then you have a sentence about a different subject.
Yein Park: Yeah, yeah, sure, sure.
David Bau: Are you able to patch?
David Bau: the year across, so if you ask about… South Korea in 2009.
David Bau: Could you patch that over to a question about Russia in 2009?
Yein Park: Oh, yeah, yeah.
David Bau: Does that work?
Yein Park: Oh, yeah, yeah, sure, sure. I use the value of attention heads for this president of South Korea cases to the president of Russia, and then it goes to the corrected temporal objects, like the changes from Putin to Matt Badip.
David Bau: I see, so the… so the encoding of the time is independent of the other, context.
Yein Park: Shut.
David Bau: You can… you can bring it over, and it applies.
David Bau: To a different question, and brings the same…
Yein Park: Yes, right.
David Bau: Excellent.
David Bau: Another thing I'm making… sorry, another thing I'm making sure is that, you do everything at the last token position, right?
Yein Park: Yeah, yeah, sure, sure.
David Bau: Wait, so this is the token right before you predict?
Yein Park: So, yeah, yeah, like, you can see the prompt, I, as the circuits that I use, like, it is an EAP, IHE-based circuit, and it predicts the next token right after the.
Yein Park: input prompt, for example, like, in 2009, the name of Russia was… and then they predict the object token, the first of the object token, and then, yeah, it traces back to, what internally affected most for these knowledge cases.
David Bau: Right, so you're patching states at the token was?
Yein Park: Yeah, I patch it at the, level of time, like, in 2009, at the patch at that point, and then the object goes to, correctly.
David Bau: Oh, you patch it at the time token, like 2009, right?
Yein Park: Yeah, yeah, yeah, sure, sure. It was the most effective, yeah, yeah.
David Bau: So is that… is that vacuous? Because you could just patch the embedding, and it would change the…
David Bau: It would change the year. Yeah.
Yein Park: Oh, yeah, yeah.
Yein Park: it would be, another, like, possibilities, too. But, yeah, like, it was, simple, admission addition, task, so I just picked some.
Yein Park: Tokens randomly, starting from the end to the west, and then, the time position, like 2009, at the end of the time position, was the most effective for these, testers.
Yein Park: And, yeah, still there exists some other possibilities, too, yeah.
David Bau: So, can I make sure another thing about the setup? So, the time always appears before the subject, which is the president of Russia, right?
Yein Park: Yeah, yeah, sure.
David Bau: Okay, yeah, that I… yeah, that was just my question. Did you… right, you didn't… you didn't test the other order, right? The president of Russia in 2009?
Yein Park: I also checked it in the ablation analysis part of my paper.
David Bau: And so is that the same attention heads for those as well?
Yein Park: I checked it, yeah, the same temporal heads are appeal in that cases, but, the reason why I add this, some kind of prefix thing… prefix lead?
Yein Park: for this temporal context, right before the subject is… it was the most effective when I do some
Yein Park: previous, preliminary research with, based on loan? What does that, loan approach like? Who can…
Yein Park: And if I… Yeah, yeah, check the residual difference.
Yein Park: Yeah, so, yeah, yeah. It was the most effectively, reconstruct all the internal structure of LRAMs.
David Bau: Okay, okay, good.
David Bau: Thank you. Thank you.
Yein Park: Yeah, go ahead.
David Bau: I do. Yep.
Yein Park: Okay, so today's, topic is, a little bit different, like, we use circuits for, circuits for the same lead, but in this case, we use it in post-training methods, so it is the basic frameilary for my research.
Yein Park: as you… most of you maybe know about circuits, because, yeah, Professor Pao are so famous about this topic, so, yeah. I treated transformer as treated, cyclic wrap, and I used, specifically APIG-based, circuits.
Yein Park: And then, in this case, I selected some model series, which can show various,
Yein Park: posted trainings among, same based models, like, I select base model, trend 2.5, MS 1.5B, and then it has also deep-seq R1 version, and it can be used with GRPO or
Yein Park: SFT, and it also has a similarly context series of models, like QN3, which can, on and off the thinking.
Yein Park: in its chat templates. So that is why I choose the QN series. And then I select some datasets, like for training, I used OpenRMS and GSM Power… GSM8K, and then I do the evaluation with AIME and AMC, or, yeah, the kind of famous reasoning…
Yein Park: Benchmarks.
Yein Park: This is the, prompt setting, like, you can, like, before the temporal knowledges, like, the… I sampled some answers of, previously.
Yein Park: base models answers, and so I selected some samples right after the ink token, and then I also select some samples of baseline models answers, like we will use Python, or that kind of things.
Yein Park: And same approach with… in here, like, the circuit would predict via the right adapter, the first token of the ink token, or right after the…
Yein Park: same length of the next token, like we… and then predict the use in these cases.
Yein Park: And so, this is the full point of my method. Like, I'm first mapping the circuits with active computation graph for both baselines, and then post-trained models. And so, baseline model's claim prompt would be, like.
Yein Park: its original answer, like, to determine the molecular or other things, and then the claim prompt, the targeted prompt provisioning model would be the DINC, and then the right next to the Dink tokens.
Yein Park: Then I identify some emergent components, which means I compare all the…
Yein Park: circuits between baseline models and the reasoning models, and then I found out some specific heads are… only come up with the reasoning motor circuits. So I named it as some kind of reasoning heads, and those that are active in the post-trainings, but not in the baselines.
Yein Park: And I do the causal variations with ablations to confirm costly responsibility, and check the…
Yein Park: variation of performance among various motors. And then also, I do some scaling up or down for these Riesling heads, and it shows, some interesting insights, too.
Yein Park: Okay, and… can I go to the next point, or is there any other questions for this?
David Bau: I think we wanted… interested in the experiment results.
Yein Park: Okay, thank you.
Yein Park: And this is the list of the…
Yein Park: attention heads from various models, and I reached it up, and you can see that, like, distillation and SFT has more heads than GRPOs.
Yein Park: And then, I also drew some, very interesting, figure. It named this map of reasoning. It would be a little bit, confusing, but yeah, I just,
Yein Park: rested all the attention tests that I found from various circuits, like, all the checkpoints from GRPOs and all the checkpoints from SFTs, and I colored it.
Yein Park: And you can see that various, distillation heads are found early to mid-rare, and the SFT heads are usually focused on the mid-to-rate rears. And GRPO heads are separated around all the rears.
Yein Park: And… Who can I ask you a question?
David Bau: Can I ask a question? So, so you, so you take the…
David Bau: The Quen 1.5B base model.
Yein Park: Yeah, sure.
David Bau: And then you've… you've used, 4 different methods?
Yein Park: Yeah, yeah, sure, sure.
David Bau: To achieve reasoning, is that right? Is it 4, 3, or 4?
Yein Park: It would be 3 for this case, because think or not, it would be, yeah, QN3, yeah.
David Bau: I see 3 methods. And then… and then now…
David Bau: I don't see a category here for heads…
David Bau: that appear in all three methods. Are there no heads?
Yein Park: Oh, yeah, yeah, sure, sure. Like, there was no head for these cases, the… all the three metals has, same link, yeah.
Yein Park: So that is why I also have some, interesting points, because every post-training would be reacted differently, like, because they don't have
Yein Park: Or the common heads.
David Bau: So even though the external behavior is presumably kind of similar, because you're changing all…
David Bau: With the same goal in mind.
David Bau: what happened here is… is very different. Do you have… did you run… did you ever consider running…
David Bau: Like, a baseline where… You just…
David Bau: used the same method twice? Like, if you use SFT twice.
David Bau: and then… and then train it, and you repeat your whole procedure, would you get the same heads the second time? Do you mean, like, two separate SFT regimens? Yeah, that's right.
Yein Park: Excellent.
David Bau: You start the base model, and then you shuffle the training data, or something like that.
David Bau: And then you do.
Yein Park: Oh.
David Bau: Again, would you… would you get the same heads, or would you get different heads?
Yein Park: Oh, in this case, I just run the single post-training approaches for baselines only, so I didn't do the kind of, like, twice accept these, or… yeah. So maybe it would be, issues a little bit different.
David Bau: Yeah, it'd be interesting just to know, how much is… how much is due to gist.
David Bau: random.
David Bau: Randomness of training, and how much is due to… The different method.
Yein Park: Yeah, yeah, sure, sure. It would be very interesting too, yeah.
Nikhil Prakash: Wait, I have a question.
Yein Park: Yeah, sure.
Nikhil Prakash: Wait, so I'm just trying to understand what is the difference between the two plots that you have. What does headshotted mean?
Yein Park: Head sorted means it's just, like, number, yeah, yeah, not the IDs of each head, it just shows, how much these heads are up here, so it is just, number of these heads, yeah.
Nikhil Prakash: Sort of, like, frequency?
Yein Park: No, no, not that kind of sort. This figure is based on, like, some similar figure in IOI papers, and they just sort it, and then, just count all the…
Yein Park: Hats down size, like, for the, downsize from Raiders, and then the number just goes up, and it doesn't like bidding.
Yein Park: Any more else than that of things.
David Bau: Aquila, I think he's just putting blue underneath green underneath, yellow underneath.
Yein Park: Yeah, yeah, yeah, sure, sure. It's just tech, it's just tech or the heads categories, yeah.
Nikhil Prakash: I see, I see, okay. Okay, okay, yeah, that makes sense, okay.
David Bau: Great. Let's, let's continue. Oh, one other question.
Yein Park: Oh yeah, yeah, sure, too.
David Bau: Are you doing the training yourself, or…
David Bau: Because from the… at least the… like, are… or are you just using, like…
David Bau: Quinn 2.5 base compared to Quinn 2.5 math, where they already did the training for you?
Yein Park: I, for only for the distillation version of DeepSeek, I use the, that, checkpoint of, DeepSeek, released, and… or the other, like, SFTs or GRPO, I run every experiment in myself.
Yein Park: Yeah, then I saved all the checkpoints from, like, 100 to 3,000, and then make all the circuits for each checkpoint. Then I collected all the heads from these circuits.
David Bau: I see. And what's the difference between, like, the SFT that you did and the distillation that they used to train, R1 distill?
Yein Park: It would be, like, some difference from, like, RND steel, it would be the end of the end checkpoint for…
Yein Park: knowledge from DIPSIC model, DIPSIC R1, and the… it would be a little bit different, because I… for SFT, I used the OpenR1 datasets and collected it with,
Yein Park: each collected heads from each checkpoint, and also some landing rate, and performance would be different, too. So, yeah, that is why I,
Yein Park: differently categorize the SFT and distillations in here.
David Bau: Okay, so both the distillation and SFT are SFT, but they're on different, like, data sets that you're training on? Would it be accurate… would it be accurate to say that
David Bau: The distillation probably is a…
David Bau: I don't know, naturally.
David Bau: Yeah, that would…
Yein Park: Sure, some, like, other researchers also mentioned that distillation is not quite the same as SFTS2, yeah, that is also I knew.
Yein Park: So, yeah, it needs more… a little more process than just simple SFT, maybe.
David Bau: Well, distillation's a different loss.
Yein Park: Yeah, yeah.
David Bau: Right.
David Bau: Because you're matching all the logits. Alright, is that what they…
David Bau: But I don't think they do that, do they not do that? Because it's a different model, right? They're distilling, like, DeepSeek, which is, like, a 600, bill parameter, like, the 1.5… But if they share that, is it… is it the shared tokenization, or… I don't think Gwen and Llama, and DeepSeek share the same tokenization.
David Bau: I think… I think you just SFT it to still work down. That was my understanding, but I could be wrong.
Yein Park: Yeah, yeah.
David Bau: Oh my god.
David Bau: You don't happen to know the difference, do you?
Yein Park: Yeah, like, I also don't… yeah, yeah, yeah, yeah, sure.
David Bau: Okay.
David Bau: Yeah, that's interesting, because they're… the histograms are so different between these very similar cases.
David Bau: Right, so it seems… Right?
David Bau: Yeah, it's a good observation. Any other questions? Good question. For the different post-optimization technique, did you find overlapping emergent traits?
David Bau: For instance, I think in 27…
David Bau: Layer 27 and at 3 and 4. GRPO and GRPO and SFT seem to be close together. I don't find a pattern in the graph, but I'm just curious.
David Bau: Or different optimization techniques where they're, like, same heads from the same layout that showed emergent behavior or something.
Yein Park: So your question is, like, is there any… this optimization shows different, circumstances, or… yeah, yeah, I, I can… sorry, I cannot hear, yeah, yeah.
David Bau: It's the other way around, for, like, different optimization techniques where there's same emergent hits in the same behavior.
Yein Park: It would be, like the upper,
Yein Park: Trends would be similar, but not…
Yein Park: the exactly same heads would be appears, maybe, yeah, because, when I use, this SFT or JRPO with different data sets, they show some, common heads too, but also they, they show different heads for different… with
Yein Park: Those different data sets, so yeah, it would be not quite the same for all the types.
David Bau: Let's, let's go on.
David Bau: Thanks.
David Bau: You can continue.
Yein Park: Okay, thank you.
Yein Park: So, let me start to the in-depth analysis. So, the distillation has strongly affected performances, like it induces new consistently activated reasoning heads, not present in basic circuits.
Yein Park: And these tests are enhanced reasoning by building on pre-trained structures, not replacing them. Deactivated emergent reasoning heads also lead to drops of performances, and also, it's, I guess, distillation-induced heads are crucial, but also…
Yein Park: Because computation erasing, which means, like, if we update some of the heads, or even increase the performances. Like, you can see there is, models, and then each method, each, oblated categories, and each performance of benchmarks, which is a past one.
Yein Park: K in these cases, and the performance, would be dropped for this deep-seq R1 and QN2.5 mass for 1.5B continuously, especially affected
Yein Park: For its original categories, like reasoning heads affect, strongly for deep-seq model, and base model heads affect strongly for QN2.5 mass base models. But also, I saw some interesting points, is that, like, deep-seq R1 distill, 7B,
Yein Park: And also, QN base model of 7B shows,
Yein Park: Performance increase when it oblated some nut-obulated heads. For example, like, it, shows performance gains, highly with base model heads are ablated.
Yein Park: And… The QN2.5 SMB also shows similarly-like oblated reasoning heads, increased its performances.
Yein Park: And those, the bottom 3BR head QAE head, which is, which comes from circuits using 3BRQAE.
Yein Park: And I saw it as a representation of simple basic knowledges, and it ablate… if it is ablated, then the whole knowledge process is harmed for… especially for this 1.5 big models.
David Bau: So it's very surprising to see any performance increase anywhere.
Yein Park: ablation. Can you slow down and explain which…
David Bau: Like, what you think is happening there?
Yein Park: For me, that is why I, named as, like.
Yein Park: confusing points, like, if, even though those models are highly, trained with highly reasoning datasets, or distal from, higher performances models, they're also confusing or do some, like.
Yein Park: too much long reasonings, or too much wrong calculations for each case, math problems, or other benchmarks. So that is why some of the performance would be degraded when it is, checked, and…
Yein Park: For me, in this experiment, I saw some quantitatively analysis for these cases, and it shows that, the performance drops, from those kind of, too much wrong reasonings or, long of calculations, and if we ablated some abundant.
Yein Park: heads, then maybe the model would be focused more for only specific tasks, and that would be one reason for
Yein Park: this performance increase.
Yein Park: Yeah.
Yein Park: So, in this experiment, I do some, I, do… do this kind of find… Found out this kind of findings with this, qualitative and quantitative analysis, but it need be… it should be need more experiments, too, for the, for… with the next research, yeah.
David Bau: Do you have any qualitative examples?
Yein Park: Oh… I'm sorry, but, not this…
Yein Park: today's presentation, but yeah, in my papers too, yeah. In my papers, appendix has some qualitative, analysis cases, and in this case, in this presentation, I just, bring some, like, plots of performance difference.
David Bau: Okay.
Yein Park: Yeah, you can find out in my papers.
David Bau: Thank you.
David Bau: I guess the weird thing is, like.
David Bau: a blade, so I'm just looking at the second-to-last row. Like, a blading for AIM24 goes up, but AIM25 goes down, like… A lot, you know? Like, just the opposite, so…
Yein Park: Yeah, yeah.
David Bau: It just… it doesn't seem consistently doing the same thing.
Yein Park: Mmm, yeah, yeah, yeah, sure. It would be not consistently, or yeah, the model, seems to differently…
Yein Park: For this, AIME 2024 or 2025, yeah.
David Bau: Is there a difference in the level of reasoning that is required to solve some of these problems? Like.
David Bau: Is that one quality, maybe? Like, the difficulty level? Is that possible?
Yein Park: For each benchmark, I know that AAME 2025 would be the most difficult questions for, and especially for mathematics reasonings.
Yein Park: And the AEMC would be the easiest one for these reasoning LRMs. It is also mathematics, but would be much more simpler than other AIMEs, yeah.
David Bau: Okay, interesting.
David Bau: Thank you, Ian.
Yein Park: Okay, let me go to the next pages. And also, some… SFT introduces large amount of attention heads, like you can… you saw the map of reasoning.
Yein Park: And removing 10 meter rate SFT heads drop all the benchmarks across almost zero, and over-activation reveals trade-offs, some task improve.
Yein Park: But others degraded, like, you can see that MESS or AMC datasets, MES datasets increase its performances, but…
Yein Park: AMC datasets decreased its performances, and qualitative analysis shows some overthinking, like I said before.
Yein Park: They chose, too much wrong algebraic passes, even though we can just simply add to
Yein Park: to the answer. And also some new heads, improved structure, but reduce strategic efficiency, and causing busy solutions for some of the questions.
Yein Park: And it is the interesting figure for my research, and…
Yein Park: You can see that some emergent ads are continuously added, not disappeared, and it is usually focused on the mid-to-reader, like the colored blue at the…
Yein Park: Plot B.
Yein Park: And this guy, yeah, this shows, the…
Yein Park: overall trend of SFT, and I checked all the checkpoints from 100 to 14,000… 1400, and then, yeah.
Yein Park: The number of attention heads also are larger than other GRPS hats, too.
Yein Park: And for the GRPO, it helps to find the optimal resisting passes. It is the most interesting point of my paper. Like, unlike accepted static installation, GRPO dynamically activates and prunes wristling heads in response to reverse signals, and has rise during training, like it reflecting reverse fluctuations and accuracy trends.
Yein Park: And final JPO heads are fewer targeted and distinct from SMT heads, showing unique specializations. Need embeds, exploits some kind of exploit and explore trade-off in reinforcement runnings.
Yein Park: And scaling activation improves mass scores, but lowers some other performances, and scaling down reduces AIM sharply, while slightly boosts AIS and AMC data sets. It shows some kind of trade-off between those heads.
Yein Park: And qualitative analysis also shows some, enhance of artist reasoning, but also it shows some overpitting and instability, like, it
Yein Park: Also, so it shows similarly with SFT, like, too much wrong reasonings for trading agility, for structure reasoning precisions.
Yein Park: So, this, plot is quite different from the plot, like you saw before for SFT. Like, you can see some of the heads are disappearing, like the born to step, step point, step…
Yein Park: 10.300 is disappeared, maybe 2000 standpoints, checkpoints, and…
Yein Park: the overall number of, attention heads are also smaller than SFTs, and it is, largely focused, across all the layers, and the overall trend is quite different from SFT, so…
Yein Park: I saw it, it's one of the things that what makes GRPO different from SFTs.
David Bau: Can I ask a question?
Yein Park: Yeah, sure.
David Bau: What does it mean? So, like, could you explain the scores in the…
David Bau: Bottom left plot, like, if an attention head has a score of 1 versus negative 1, how, like, but, yeah, how do you compute the score?
Yein Park: Scores for, what? Okay, can I ask you again for…
David Bau: Yeah, plot, plot.
Yein Park: Yeah, yeah, of course, it is just some kind of, yeah, temperature, like, the lead one is the base motor's head, and then the blue one is the… how much…
Yein Park: How many dose heads are emergent in this head? So, some light blue hats are only, emerged in small number of circuits, and the,
Yein Park: Darker blue ones is, the most, iffy,
Yein Park: frequently, emergent heads among those attention heads in the circuits. Yeah.
David Bau: And the red… and the red ones mean…
Yein Park: Outlet once means the baseline hits.
Yein Park: Base mode to assess. Yeah, base mode success.
David Bau: And then those ones don't appear in the circuits in the fine-tooth.
Yein Park: Yeah, yeah, yeah.
David Bau: And then, in the plot on the left, like, how do you determine
David Bau: Whether, like, what does it mean for an attention head to be born?
Yein Park: like the… yeah, yeah, it is born in circuits, only for the circuits, not in the model's architectures, yeah?
Yein Park: Because water has already all the attention heads, and for all the rears, and it… I just, found out some emergent heads in those circuits for each checkpoint.
Yein Park: So,
Yein Park: Some more, some newly appeared attention heads are found in those, each checkpoints, like 200, 400, and 600, and that is why I called it emergent.
Yein Park: in… Of attention heads for each circuit, yeah, not the, yeah.
David Bau: I have a question about the, Red Hat in the base model. Are they, are they…
David Bau: Die, or they stayed in the final checkpoints?
Yein Park: Also, yeah, like, you can see the plot A, the maintained node, which is a lead dot client, and it means that, all the circuits of each checkpoint, maintained, this number of.
Yein Park: base motor's attention heads, and some of the attention heads are not anymore appeared in other cases, like you can see in SFT. The number of,
Yein Park: maintain those are a little bit fluctuated, but in GRPO cases, it's always maintained, and some of the specific, emergent heads are appeared in circuits.
David Bau: Thank you.
Yein Park: Thank you.
David Bau: And so does… does red… I'm sorry, I misunderstood. So is it… are the redheads the ones that died, or are the redheads that were the ones that were present at the very.
Yein Park: No, yeah, yeah, it is, yeah, I also said wrongly, so, sorry. Yeah, yeah, it maintained, like, you can see the board of the… it would be a little bit smaller, but, in plot C, some of the blackboard line for the boxes is the…
Yein Park: Maintained, note.
Yein Park: maintained, attention heads for the last checkpoint. So, yeah, all the attention heads emerged in baseline motor circuits are still maintained to the end of the.
David Bau: I see. Checkpoints, yeah.
David Bau: Thank you, Anne.
Yein Park: Okay, let me continue.
Yein Park: And it is the, qualitative, quantitative analysis of this GRPO. Like, the performance changes, like, applying structure to step-by-step reasoning, increases the performances, but sometimes, this connection between describing solutions and actual computing.
Yein Park: Make some performance degradations.
Yein Park: And for the difference, different benchmarks, like AMC, it shows systematically decompose
Yein Park: problem with mathematical principles, but also it signs of some kind of overfitting to only specific, solutions of mathematics, so it can make some errors, and then the performance sometimes degraded.
Yein Park: Yeah, and like the MS is the most easiest benchmarks, and it, almost continuously, enhanced compensation reliability when it is trained with GSM8K for GRPO.
Yein Park: And then… but sometimes, cost of stability, such as calculation mistakes, are degraded its performances when it is, trained with mass 220 datasets.
Yein Park: And the last one is thinking on a model, like, I selected QN3 for these cases, and it activates and deactivate
Yein Park: is thinking with chat templates. And think of compensate performance through enormous header emerging, which means, like, turning off reasoning triggers a broad and resource-efficient set of attention heads, which indicates through only efficiency and modern compensate by selecting optimal computational routes.
Yein Park: And the research overhead intervention shows some removing or scaling thinking overhead affect hard benchmarks.
Yein Park: Like AIME, performance goes down, and oblating lead-nd heads improves focus, but may reduce adaptability on broader tasks, like other benchmarks, too.
Yein Park: And performance difference against coverage comparison shows that baseline thinking of models perform better with large sampling coverage as its large number of active attention heads, which facilitates exploration of diverse computation pathways, like
Yein Park: When we increase the sampling coverage, then the models can found out various ways to find out the answer. So, baseline thinking of models enormous attention heads to this kind of work.
Yein Park: And the performance of the ablation is like the table. You can see that performance is very, fluctuated.
Yein Park: Wind…
Yein Park: Most of the time, like, the thinking of weight ablation, performance goes up from 30 to 36.6.
Yein Park: And sometimes the, just, thinking of mode, usually has, lower scores.
Yein Park: But sometimes, a scaling down an ablation also shows popomestic radiate.
Yein Park: And this is the plots that I want to show for a coverage of
Yein Park: coverage of solutions like Pascade, if we increase the Pascade to 1 to 64, then the thinking of model always wins the other, like, oblated one or scaled-down one. But,
Yein Park: But sometimes when the sample is very raw, very small, then the thinking of model, shows less, a little bit degraded performance than oblated one. But, right after some samples increased, then the
Yein Park: Thinking of mode compensate this, performance difference with its environmental attention test to find various solutions.
Yein Park: And, the new ribs?
Yein Park: paper that I mentioned before shows some similar, like, insights, as they showed that if the samples are increased, the base model's performance wins the GRPM model's performance.
Yein Park: And this is also connected with the previous insight, like the GRPO, only instills some specific attention heads for the models, and it…
Yein Park: In… before the…
Yein Park: Before the number is increased, it shows, higher performance than base models, but after some coverage of… increases, then the…
Yein Park: base model, in the… Reinforcement… reinforcement learning… reinforcement running models.
Yein Park: And… okay, and… huh? Is there any questions for this one? Like…
Yein Park: Okay, I can go to the next point, like…
Yein Park: Yeah, yeah. So, this research would be some interesting points. It has… it surely has some interesting points, but has limitations too, like the model generalizations currently implemented only for QN model series, and it further tests… it needs to further test into other model architectures, so I re-implemented
Yein Park: for this, or the process with Lama Motors.
Yein Park: for generalizability, and because we are, I am, I am currently,
Yein Park: Going through the reboot tool process of…
Yein Park: rebooted process, so I recently got some harsh review from some reviewers, and it… yeah, yeah, sure. It shows some constraints of circuit analysis, like the analysis all relies on prompt-based circuits, so it demands precise setup and calibrations, and also some…
Yein Park: vulnerability to polysynasticity, and it would be a challenge to various interpretabilities, but in this case, we can use SAE-based circuits, but like you can see, there is an enormous
Yein Park: experiment that I need to do, but it would be very cost… costly for all making all SAEs for each checkpoint, for each model, so that is why I just chose the simple circuit method with EAPIG.
Yein Park: And so, in this study, yeah, that would be… I thought it would be a practical way to compare various methods.
Yein Park: And, conclusion, and I want to expand this approach to, like, mechanistic understanding to post-training, like, we can resolve that SFT administration in the stable reasoning heads while GPU dynamically repines them.
Yein Park: And I think on updating reveals selective head usage and some kind of compensatory activations, but deep reasoning improves problem solving, but some overthinking problems would be… surely need to be solved.
Yein Park: So that is why I want to make it into some attention head-specific post-training methods, like increased targeted head activation, rather than uncontrolled head growth.
Yein Park: And use of reverse shaping to jointly optimize brand quality and calculation reliability. And I would want to make it further to, like, reverse per head influence estimate.
Yein Park: to guide scientists trainings, yeah. So, that is why I hope to make some kind of reliable and low-risk executions.
Yein Park: And thank you for listening to my presentation, and if there are any questions, please feel free to ask me.
David Bau: Thank you, Ann.
David Bau: Now, we asked a lot of questions during the talk already. Any… are there any other questions that people have?
David Bau: I have a question.
David Bau: So, like, let's… like, if you find a… a head that is, like, most…
David Bau: different between, like, reasoning and base, and you take this ahead, like, did… did you, check, like.
David Bau: try and… try and ask or figure out, like, what is this head doing? And, like, kind of check, I don't know, its attention patterns, or…
Yein Park: Mmm.
David Bau: like, your temporal… your slide-on temporal heads was very nice. You could see the temporal heads, like, attending back to the 1999. I'm wondering if you did…
David Bau: Any similar, like, microscopic analysis on… on these heads?
Yein Park: Sure, like, you mean the, each specific head's, influence for these models, like the temporal heads?
David Bau: Yeah, just, like, whether, like, you can look at ahead that is, yeah, useful for reasoning and try and understand, like, what function it plays, or what it's doing at some level of explanation.
Yein Park: It would be much more helpful, but in this case, like, you can see that it, it shows very…
Yein Park: much larger number of heads when it is compared to temporal heads, so it would be a little bit hard to distinguish the specific, like, points for each head, so that is why I can't…
Yein Park: do that kind of, analysis in this experiment, but maybe later I would be figured out, but, I also worried about some polysymptistic problems, and also as these models are much later models than
Yein Park: simple, like, without group query attentions. So, it would be quite a little bit different from, like, models pre… before the group query attentions, like the LAMA2, so, I… it would be a little bit harder than distinguish the difference between those,
Yein Park: Categories, heads, list.
David Bau: Other questions?
David Bau: So, Diane, it sounds like you submitted this to ICLR, is that right?
David Bau: Yeah, yeah, yeah, sure. You know, maybe you can't tell us, I don't know if we have any reviewers in this room. But yeah, we're all going through the same process. Probably there's a couple students who are also going through the rebuttal process.
Yein Park: Yeah, yeah, sure.
David Bau: Thank you for taking some time out of your busy rebuttal. I'm sure that you have a bunch of writing and work and thinking that you're in the middle of
David Bau: To come and present to us today. I didn't realize when we, scheduled this
David Bau: this meeting that would be right in the middle of rebuttals for you. So thanks a lot for joining us, sort of at this sensitive time.
Yein Park: It was my pleasure, yeah, yeah.
David Bau: Yeah, it was… it's a… it was a very… yeah, it was a very nice, interesting, study of, what's going on inside reasoning models, just so important to understand. I think it's still…
David Bau: Still a pretty big mystery, how these… how these complex models are working, and it's good.
David Bau: To have people looking inside it.
David Bau: Yay, and thank you so much for joining us.
David Bau: Thank you. And good luck with your rebuttals.
Yein Park: Haha, thank you.
David Bau: Okay, great.
Yein Park: I also, want to make a good note for all of you.
David Bau: Yes, it's great to meet you. Sometime, you know.
David Bau: you know, sometime when you're… if you're… if you're in the United States, you should… should come and… come and visit us.
David Bau: Okay. But it's really great to have you here over Zoom.
Yein Park: I also had a pleasure time with you to present my own work.
David Bau: That's great.
David Bau: Okay, thank you very much.
David Bau: Alright.
David Bau: Yep.
Yein Park: Okay.
Yein Park: Okay. And… so it is the end of the presentation. Yep, let's see right now.
David Bau: We can close everything down here now. Thanks a lot, Yan.
Yein Park: Thanks a lot, Nico.
Nikhil Prakash: Thank you.
Yein Park: Thank you.