177 Huntington: My major research insight is, but I don't totally know why or how or how we produce it, so it probably wouldn't pass peer review. So I actually. So you know you, you sent us a
177 Huntington: a nice abstract, but I actually don't have a sense, for you know what the talk is going to be about so. But I'm very excited to hear. So so we're lucky to have Benjamin Frewer, Freyer Freyer
177 Huntington: here today to to give us his perspective. Yes, on the
177 Huntington: recent evolution. Progress of our field. Yes, through a data centric. Lens, indeed! Yes. So so to sort of briefly, self introduce. I am mostly going to be no entirely going to be talking about Llms today, but that is not the entirety of what I do
177 Huntington: But yes, that's mostly what I'm focusing on today. We will get into me in a minute. But the name of the talk is Anous charting. A year of explosive progress in Llm. Through a data centric lens which sort of invites the question, and before I dive into it I'm going to try to keep this talk to 45 min, you should feel free to intercede with questions if you have them during the talk, but I am also going to try to leave time for questions afterward, if you prefer to hold them till the end. So that's sort of up to you.
177 Huntington: So anous mirabilis is laughing for a year of miracles.
177 Huntington: The term was popularized in a historical poem by John Dryden, commemorating the naval victories of the English over the Dutch. I did not know this. I looked it up, and also the great fire of London, and why you would commemorate that as a miracle is because it wasn't worse. Apparently
177 Huntington: so. A year earlier, in 1665, the bubonic plague had broken out in London. So maybe it just seemed miraculous that something that bad didn't happen again, I don't know.
177 Huntington: and also in 1665, at the age of 2023, at the age of 23, a young man named Isaac Newton figured out that the moon is held in orbit by earth's gravity, proved that light is composed of particles and developed integral and differential calculus in the same year.
177 Huntington: So that was a pretty good year for Isaac Newton, and sort of led to this term, being kind of generalized in particular in scientific discourse, as opposed to just sort of historical discourse.
177 Huntington: So llms, I would argue, have had a pretty miraculous year
177 Huntington: chat gpt arguably a miraculous 2 years Chat Gpt is a really impressive achievement in many respects it's sort of easy to forget, because it's so ubiquitous now how fast things have changed and are changing. But we sort of went from stochastic parrots and gpt 3. And like all this sort of kind of noise in the prompting space to
177 Huntington: things that operate so smoothly and in quasi agentic fashion that we can almost just use them as though we were talking to someone and and rarely encounter unless we're doing something really out of distribution, rarely encounter major surprises.
177 Huntington: The skeptics have sort of been forced to shift their goalposts. Again and again the metaphysically minded skeptics remain completely undeterred. Good for them. But
177 Huntington: my kind of view of this as Professor Bao indicated, is is primarily data centric.
177 Huntington: I'm really interested in some of the open scientific questions around benchmarks. How do we create them? Of why and when should we trust them? And what should we trust them to tell us like, if a benchmark is kind of this signal in metric form, like, what is that signal actually indicating? And with Llms, I think that's less clear than with almost any other type of kind of metric based benchmarking because they can do so many different things.
177 Huntington: Llm. Judges, which are often used as a kind of component in said benchmarks like.
177 Huntington: when can we trust them, and and to what extent should we trust them, and trusts a complicated word? But in this case I mean trust to be reproducible and predictable, like the things that they do, they will do again, and the things that they don't do they will not do again.
177 Huntington: and then finally, synthetic data, which is increasingly supplementing, and in some cases replacing traditional kind of human annotated data for training Llms, and also other types of models as well. How can we get the most out of the synthetic data that we are producing. When should we
177 Huntington: prefer using it? When should we prefer not not using it? So these are some of the questions I'm going to be talking about.
177 Huntington: Now, that promised sort of brief about me. I am a 3rd year Phd. Student in Nyu. My advisor is Chin meh hecte
177 Huntington: my research interests are primarily data centric, benchmarking alignment, multimodal machine learning scalable systems and a fun fact about me is that before Nyu I wrote and directed movies for a living. So that was my last job. Anything that we can go and see. Yes, that's so cool. Give us an example. I'm on Imdb. You can find it there.
177 Huntington: You're welcome to do so. My boss at stability was also a movie director. Really, I was in the flight team.
177 Huntington: Yes, Joe Penla Joquena.
177 Huntington: And now maybe I'm butchering the name. Here's here. He has a Youtube
177 Huntington: presence, and he also did some Netflix movies, I think.
177 Huntington: Well, he might have been higher profile than me. I mostly did anonymous rewrites of Disney movies.
177 Huntington: Amazing. Well, you should get into January.
177 Huntington: you know. That's people keep saying that to me. But honestly, I'm enjoying being in an area where I can measure progress more concretely or change anyway. So that is my my fun. Fact.
177 Huntington: some of my work during my, some of it was pretty recent, so it says less life. Love was a 2021 movie you directed, is that right? Yeah, that's right. So it's a pretty recent transition. It is. Yeah, that's great. Quite recent. That was basically the year I started was the year that movie came out.
177 Huntington: Nice movies operate on a delay. And I shot it. I think 2019 and a little bit during the pandemic. Yeah, it's great.
177 Huntington: Yes, sorry. That that movie in particular is
177 Huntington: whatever it's. It's an interesting movie. Feel free to watch it if you want to. All right? So yeah, some of my research areas in the master and the Phd include Llm, a little bit of computer vision and vision language. Ai for science, and tabular data. And very recently a little bit of work on diffusion models. But I think probably almost everyone in this room knows more about diffusion models than me.
177 Huntington: So the roadmap for this talk
177 Huntington: is going to be a 4 part walk.
177 Huntington: 1st part is going to be about detecting and mitigating bias and Llm. Judges. Second part is going to be about mitigating data contamination with dynamic benchmarking. The 3rd part is about how to improve your Llms by using synthetic data in the post training pipeline, and then in part 4 just kind of things that I think are interesting and and possibly worth looking at in the near future.
177 Huntington: although it's always hard to say that because the field moves so quickly that the near future becomes the present a lot more fast than you might think it should.
177 Huntington: So. Llm alignment, some kind of some core definitional challenges here? What do we align to? By what method do we align? This is sort of the algorithmic aspect of the question is like, what algorithm for alignment do we use when have we succeeded? Right? It's not so obvious, like
177 Huntington: intuitively like what an aligned Lm or a more aligned Lm is. And kind of what trade-offs should we expect to be
177 Huntington: unavoidable like, we're definitely going to face these trade-offs.
177 Huntington: So the work that I'm about to talk about focus primarily on the 3rd of these questions. When have they succeeded? But just briefly addressing
177 Huntington: sorry. Just briefly addressing it. So kind of using as a guiding principle. The Hhh.
177 Huntington: From anthropic, which stands for helpful, harmless, and honest. So one way we could conceptualize
177 Huntington: alignment is, we could say, well, if a model is kind of pareto dominant along all these 3 vectors of performance, and we assume they're kind of linear. They have a direction. And you know, we kind of reach some better state, then we're better aligned. Now, whether that's actually possible to do in the real world, or know when you've done it is another question. But
177 Huntington: so how do we actually judge when we've made progress in alignment? So the
177 Huntington: prominent way to do it? Certainly, when I wrote the paper, and really still now, even in the Age of Reasoning models, is using Llm. Judges and using these benchmarks where Llm. Judges read the output of other Llms and kind of render these scores on the outputs so the names of these benchmarks are things like alpaca, Eval, empty bench and arena hard auto
177 Huntington: these benchmarks are generally designed to closely mimic Chatbot arena.
177 Huntington: So they kind of operate on the premise of pairwise interactions, and the idea is that there's going to be a winner, and there's going to be a loser, and in the aggregate you develop some sort of
177 Huntington: metric for general rate of preferredness of model A with respect to model B.
177 Huntington: So one of the things that we kind of noticed early on when we were kind of trying to do
177 Huntington: preliminary work in this
177 Huntington: area. And this was work that I started doing at Arthur AI, which is a startup in New York. And I was kind of the person who was just very interested in this. And my supervisor was very kind, and said, Yeah, go ahead and look at it, and just coming from a background of having looked at a lot of standard benchmarks. What immediately struck me is that oh, there's some kind of new confounds in this pipeline that I didn't really have to think about before like this. Llm. Judge, like.
177 Huntington: you know, doesn't it matter which judge we choose like? Are they really all just going to behave the same way? And each, should they even all behave the same way? And how are we converting these judgments into a metric right? Like you know what is. There's not necessarily a
177 Huntington: I mean, you can ask a model to give a number right? Assign like a liker score, which is what's done frequently like a 1 to 5. Kind of number of like, how good was this, or how bad was this? How much did you prefer it?
177 Huntington: Was it strongly preferred or strongly dispred model A's response compared to model B, but it's not obvious that the numeric scale that Llms use is actually stable and if you actually do the experiments, you'll find that usually it isn't right like that. Some Llm judges will score everything on really on the high end, others really on the low end, and they will often have these weird.
177 Huntington: quirky preferences. Or I mean, that's sorry. That's anthropomorphizing. They have weird, quirky artifacts of behavior that are hard to predict in advance.
177 Huntington: So
177 Huntington: another thing that I think maybe people don't talk about enough when looking at these benchmarks is kind of measures of progress and alignment is that while overall.
177 Huntington: these benchmarks have very strong correlation with the overall Chatbot arena scores for any particular domain of knowledge that you might be interested in medicine, law, science. There's a lot of variance, because these are small benchmarks. The largest of them is around 500 to 1,000 prompts. So you just don't get a lot of samples for available for judgment. So there's this sort of idea that a model can generally become more aligned. But
177 Huntington: when we actually get down to oh, but is it more aligned for the thing I care about? You're actually dealing in a very sparse area of knowledge. You don't know that much.
177 Huntington: Another thing that maybe is not so obvious is that judges
177 Huntington: must, in order to do their task, make use of explicit biases. This is essentially the instructions they're given when they're asked to judge. You are a judge.
177 Huntington: Here are the things you must judge on. You must judge on this. You must judge on this. You must judge on that. If you don't do that, the problem is really not well defined at all.
177 Huntington: But what's kind of also interesting is that judges may also, and
177 Huntington: possibly always, have to employ implicit biases while understanding and interpreting what you've written in the prompt as your explicit bias. Right? If I say.
177 Huntington: tell me whether model A is more helpful than model B
177 Huntington: inside, and the judge's internal architecture has to decompose what is helpful. It has to make some sort of meaning of that, break it down into things that we might be able to describe in smaller concepts, and it may not weight all of those smaller concepts equally. That's what I mean by implicit bias. This one's important. This one's not
177 Huntington: so fair enough. Is this something we could study? Here's a hypothesis we could float, do Llms implicitly reweight explicit judgment criteria if I list, helpful, harmless, and honest, and I don't tell the Llm. Like which one is more important. My expected behavior would be that weights them all equally. But is that actually what it does
177 Huntington: and also do? Llms implicitly reweight infractions of judgment criteria? Right? So models can be unhelpful in many ways. Are they all equally bad.
177 Huntington: What's a what's kind of a measure of severity? How unhelpful was the response. And how do we know?
177 Huntington: So first, st I'm going to speak about the first.st
177 Huntington: So this is a kind of a dense scatter plot of sorry. Can I ask you to redefine so implicit reweighting? So can you expand that for me. What do you mean by implicit reweighting? Sure. So here I have. What is an overall metric for arena hard auto, and it's some score that I've assigned with respect to all models that says like, how preferred they are.
177 Huntington: But then I actually, in the in the prompt, I listed 5 criteria, style, completeness, correctness, safety, and concise right? So I didn't list in the prompts how to wait these 5 criteria. I just said, these are the 5 criteria.
177 Huntington: So my kind of intuition going in was well, then it should. Everything should be 20% should be 20% style, 20% completeness, and so on and so forth.
177 Huntington: So to not see that behavior would be an implicit reweighting of the Llm judgments. Got it?
177 Huntington: Does that answer your question? So okay, so what do we actually see if we do this? So kind of the methodology is in our paper. And I want to kind of move quickly so that we don't get bogged down, but basically
177 Huntington: across 4 different Lm judges across 12 or 13 models in the study. Across 500 prompts.
177 Huntington: The correlation between the overall and style rankings was one
177 Huntington: that means there was not a single case in which they differed.
177 Huntington: We were asked to do like factor, analysis and stuff in our rebuttal. And it just like broke the factor analysis. It was literally a perfect correlation. So I had to like, introduce a little bit of variance. Just to get a factor.
177 Huntington: Completeness is almost almost as strong. It's point 9 9
177 Huntington: correctness, considerably weaker, 0 point 7 4 safety. Everything's basically bunched up together. This is just like, really, when it comes to safety, like there was barely any judging going on. And then finally, for conciseness. It's actually our point 0 2 5. And in many cases you're getting the reverse order.
177 Huntington: So there are a couple of things that are interesting about this 1st Lms. Do clearly, implicitly reweight criteria, but, second, they actually do understand the difference, right like it. It. The model produces a reasonable ranking
177 Huntington: with respect to conciseness. It knows what it is to be concise, and it knows the conciseness is part of the instruction. It was given it just when it actually comes time to render the judgment. It ignores that even though conciseness is actually a part of style, right like to be more concise, is, in some settings at least better style.
177 Huntington: Yeah.
177 Huntington: do you include these different criteria? One at a time? No. So we prompt everything at once, and then we extract the factor, wise rankings as well as the overall ranking. So basically, the Llm returns 6 scores instead of one. Okay.
177 Huntington: let's see.
177 Huntington: But but what if 6,
177 Huntington: you you didn't mention an example of what you gave to the language model to judge right. But what if the question did not include any element of safety? For example, yeah, this is a good point, right? Like so. And this is something that
177 Huntington: that we heard. And and I think it's actually a very reasonable thing to think about is like, well, maybe arena hard auto really doesn't present any safety challenges. So it makes sense that the ranking here is very flat.
177 Huntington: and I think then my response would be kind of in the sense of meta motivation is like, well, then, we don't really have the right benchmarks yet, do we? Because, alignment, like safety is a really important factor in alignment. So if we're saying, Well, the arena hard auto score went up, therefore it's a better aligned model, then we should know at least that we're ignoring safety when we say that
177 Huntington: so implicit rating of infraction. So the idea here is, I have a model that initially performed really well one of the models that was scored very high, and then I kind of break its responses systematically I introduce, like a systematic ablation over every single response.
177 Huntington: So in the blandness, ablation, I have gpt, rewrite every response. To have simplistic vocabulary kind of reuse the same words over and over again. I just basically told Gpt, be as repetitive as you can. And so that was bland and you can see it loses about 8% compared to the base. And that means like, if you you know, basically, you're losing 8% of whatever that base score was.
177 Huntington: were wrong. I went into every single. So these were in arena hard auto. Most of the prompts have to do with like a math or a science question. So most of them have, like either the model has to return some code, or it has to return some kind of equations or something. So I went through and made sure that there was at least one really glaring error in every single response which took a while. And that led to a 13% decrease.
177 Huntington: And then, after that, I prompted Gpt to make the responses more concise while keeping all the facts the same. Basically just copy everything necessary to answer the question, but eliminate all the dressing
177 Huntington: that went down 63% and then sarcastic is.
177 Huntington: keep the content exactly the same, the facts exactly the same. Just have a bad attitude about just like, be really sassy and back talk to user and be kind of obnoxious. And that led to 96%. So decrease. So the losses on arena hard.
177 Huntington: Yes, arena hard auto as scored by an Llm judge, right?
177 Huntington: So basically, the conclusions in this part of this work. There are some other stuff that I that we did in this work. That's not really that I'm not really talking about here. But this part of this work Lms. Do implicitly reweight the judgment criteria, and they do so systematically
177 Huntington: across 4 judges across, you know, 13 models to be judged across 500 prompts. Same exact behavior or very close. Yeah. Also. If you if you go back to your previous slide do you remember which kind of intervention had? What kind of effect on which criterias like if you make the models responses more sarcastic, which of the scores go down.
177 Huntington: So that's interesting. We didn't actually port that in the paper the numbers probably are in the artifacts we released with the paper. So I think that's something you could actually look up. But we didn't
177 Huntington: like, I'm I'm trying to understand, what was the motivation for doing this intervention experiment, basically to understand.
177 Huntington: So
177 Huntington: if I have my different criteria again, the the assumptions were the same here, like I'm I'm assuming that because I gave notes particular instructions as to how these are to be weighted, they'll be weighted evenly. So 20 waiting for each.
177 Huntington: That means, if I introduce an infraction. If I break a rule, I should expect roughly a proportional decrease. Right? So if I take a response that was correct and make it incorrect, it should decrease about the same amount as if I take a response that had good style and now has bad style.
177 Huntington: And what this shows is that that doesn't happen at all
177 Huntington: right. You go from right to wrong, it gets a little bit lower in the score, but if you go from good style to what the Llm. Considers bad style, you lose almost the entire score.
177 Huntington: Yeah, is the Llm. Aware of this implicit waiting? So if you'd ask the Llm. Hey.
177 Huntington: what is alignment to you and give like a weighting on these 5 categories. Would it like be actually reflective? Or what do you find in your intervention? Experiments? Great question. We didn't run that experiment. I think it would be an interesting one to do. I guess I will say, sort of generally, that I don't
177 Huntington: really trust Lm's ability to kind of self reflect and report what they're actually going to do, you know, I mean, but you could experiment. It's just I feel like if it if it agreed there'd be a tendency kind of post hoc, say, like, we've discovered something, but like probably if you ran that experiment 100 times.
177 Huntington: you know, there would be low rate of agreement across different factors, and maybe you can fine tune for that.
177 Huntington: So this is what a lot of
177 Huntington: concurrent and subsequent work is proposing to do. I think it's interesting. I I think it's important work. It's not easy, because
177 Huntington: even if you can solve for these particular factors, you know about. As we said before, there are implicit sub factors in those factors that you, you know it's turtles all the way down. So at some point you. You reach this kind of level of things. You haven't flattened out, and then it gets all wonky again, potentially. But we don't know that will happen. So I think it's worth exploring for. Sure.
177 Huntington: It's also worth exploring. The idea that you could just
177 Huntington: get these factor scores and then reweight that right like you. You once, you know.
177 Huntington: kind of the bias you could just reweight. You could rescore it. However, you want.
177 Huntington: So
177 Huntington: we we have a related work. That kind of maybe provides a nice abstraction that could be useful for this.
177 Huntington: where we check how language models. You know, balance between the information provided in the context
177 Huntington: and their prior knowledge.
177 Huntington: which is kind of like the judge instruction versus the biases. I mean, in our toy settings.
177 Huntington: we are able to create an interventions that lets us, you know.
177 Huntington: prioritize one thing over the other.
177 Huntington: Do you need access to the weights to? Oh, yeah, to do the yeah to do the intervention, because most of the time
177 Huntington: in practice you have closed weights, models as judges. Oh, but actually,
177 Huntington: you can get like a little bit of the performance. Also via instructing like, listen to the context.
177 Huntington: yeah. Then you get like, I don't know 60, 70% of the performance that you could get with tuning.
177 Huntington: That's interesting in deploy tasks.
177 Huntington: Yes, so and then we we built benchmark
177 Huntington: which is intended to be a supplement for these Llm. Judges as measuring progress and alignment. It's fairly large benchmark kind of a Meta benchmark. 19 different test sets, 150,000 samples, and these different
177 Huntington: tasks are subdivided into tasks that essentially measure honesty, helpfulness, and harmlessness which we
177 Huntington: called world knowledge, instruction, following and safety.
177 Huntington: And in the paper we report the results on
177 Huntington: Lama 8, 3, 8 B base, fine tunes and mistral, 7 B fine tunes using 20 different curation techniques from the recent literature.
177 Huntington: and we find that data scaling rather than the particular creation method is the best predictor of improved alignment in the sft stage of post training, which means basically
177 Huntington: to a large extent. I can be agnostic about what clever curation method was used to collect the data. If I collect enough of it. The model gets better regardless. Right? So and this is the challenge, I think, of doing data, curation methods, researches. You often discover things like this. So I think it's good to look every now and then and see. But
177 Huntington: worth noting. This purple bar is the arena hard auto score. So these papers, when they came out, what would they report? They would report how we did well on empty bench. We did well on arena hard auto, and so on. That was kind of the standard for publication is you beat the existing state of the art on those.
177 Huntington: So you see that that's where you see, actually very low correlation with data set size, you can get really, really big Delta. And we actually did this mostly unintentionally, in a later work which I'm going to talk about later in this talk with very few prompts when you're using an element. Because basically, they're not. The judgment is just not that complicated.
177 Huntington: What it's considering.
177 Huntington: Okay, yeah, are you? SSD tuning a model here?
177 Huntington: These are all
177 Huntington: Sf models that we either trained ourselves. But most of them were already available on hugging face.
177 Huntington: So we just ran the house like
177 Huntington: data set size K, and on the right side, being one K, does this mean 1 million.
177 Huntington: Yes,
177 Huntington: yeah, 1,100,000. These are just sft data, sets containing diverse tasks, well, they're, sft data sets which were purportedly curated using specialized curation techniques, like, for example, the Lima paper which came out a couple of years ago. And now there's a new version of that. You're saying they need to be big
177 Huntington: of.
177 Huntington: I'm saying that if you look at what actually correlates with improved performance on our benchmark at scale, it's not method. It's unless, again, unless
177 Huntington: you're using Lm judges, then it's much less clear that scale matters lowest light.
177 Huntington: Yeah, the purple.
177 Huntington: Okay? So moving on mitigating data contamination with dynamic benchmarking. So
177 Huntington: there are some limitations. Aside from even if we're doing kind of ground truth, benchmarks. There are some limitations that we face. Nothing is perfect. Right? Like
177 Huntington: static benchmarks, as they're now called, are fixed in difficulty and somewhat narrow in scope.
177 Huntington: They tend to just cover one or 2 topics. I mean some really big ones like Mmlu report to cover many, many topics. But there's always kind of somewhere where they end, and the whole thing about Llms that there really is nowhere where they end. There's no boundary to what an Lm. Knows that we are aware of.
177 Huntington: There surely is one, but we don't know how to map it.
177 Huntington: so the but the biggest problem, I think, with static benchmarks is that they get contaminated, and these days they can get contaminated very quickly indeed. So this is a plot that I love for illustrating this from a paper called Data Contamination through the lens of time. It is the marginal pass rate on code forces problems with respect to time. So as you go further back in time, the pass rate goes up as you go forward in time. It's flat.
177 Huntington: So I think this is a pretty good way of illustrating the contamination problem right? Like as time goes up, your P. Of contamination just
177 Huntington: mystically drifts on air.
177 Huntington: And I think that kind of cycle is in speeding up as kind of companies. There's more and more
177 Huntington: there's more and more of a kind of an emphasis on benchmark meeting, and sort of just generally kind of
177 Huntington: trying to hit numbers.
177 Huntington: So yeah, sorry. Maybe it's just me. But I didn't really get to know what contamination means in the last slide.
177 Huntington: That's a great point. I'm sorry I didn't cover that. So what? It actually is not an easy question to answer, like what even is contamination in in the case of Llms, like the the most conventional view that's taken, which I think is woefully inadequate is that contamination means. I literally have a copy of the answer in my training data like Oopsie Daisy. I copy pasted the answers into the training data
177 Huntington: at some point.
177 Huntington: But actually, there's there's this weird kind of
177 Huntington: gentle slope from contamination to generalization and depending on which lab you go into. And when I give this talk in different labs, I get very different views on where that boundary is, but it's it's sort of hard to say like if I have Gpt rephrase the answer. But keep all the facts correct, right like it's still Mmlu, and all the facts are still the facts. But Gpt sort of tosses the words around, and it's word salad generator. And now it looks different.
177 Huntington: Is it contaminated. If I train on that
177 Huntington: now, I would argue, yes, but it wouldn't be caught by the standard methods of decontamination that are used, which is n-gram decontamination
177 Huntington: that will not detect that right. There are stronger methods of detecting decontamination which were introduced by very reputable authors and then ignored. And you can decide for yourself? Why, that is the case. But that is a multifarious, multifaceted, multifaceted view of what contamination might be.
177 Huntington: Yeah. So I'm not sure how, if this is even like what is done. But how would contamination work if you have like? A starting sort of question answered. And then for the training data, the starting question is kind of paraphrased a few times, and then all that is used with the same answer on it. But is that something that like labs do when training models. And if so, is that into this framework at all, interestingly, that is actually a strategy for detecting contamination.
177 Huntington: So if a model has been contaminated. It is believed, then, the exact order in which you pass in kind of the prompt, if you like, rephrase the prompt, the kind of log, prob of the correct answer should peak.
177 Huntington: And if you have kind of rephrasings that mean the same thing, it should be down, and if it's sort of roughly the same. Then it's thought that the model is less likely to be contaminated.
177 Huntington: It's a cool technique. It is a cool technique. There's some good ones out there now. I wish people would use
177 Huntington: give more talks.
177 Huntington: Yeah.
177 Huntington: I'm sure that I'm sure that anthropic is is dying to hear why they should lower their benchmark scores by being really rigorous about decontamination.
177 Huntington: Yeah, do you think this is just because people don't care what they're scraping off the Internet? And they'll just take anything, even if it's a benchmark like
177 Huntington: or like. Is it careful like? Is it carefully curated like, oh, like we care about this benchmark. Let's rephrase it and
177 Huntington: throw it in the data. I think both can happen so. Alex Demacus. When he was from University of Austin he was recently giving a talk on this topic, said, the reason
177 Huntington: providers won't
177 Huntington: release any information about what's in their data anymore is that they don't know. And I actually think that that is to a large extent true, they just collected so much data that there's absolutely no way for them to actually know what's in it. They know what works right? They know if they like. Kind of grab this block, and you know, concatenate it with that block and sort of feed it onto an Lm, they get good benchmark numbers out, but they don't really know why. And it's actually very expensive and difficult to figure out why.
177 Huntington: So I think that they kind of make some efforts to decontaminate, and how hard they try varies by the lab and by the year. But I think everybody's aware that kind of at least conventional n-gram contamination is a problem, and they try to avoid it.
177 Huntington: But a lot of other kind of kinds of contamination like that I'm talking about here. I think some people would just argue philosophically like you should not be concerned about that kind of contamination right like, because if the model is able to generalize to that degree
177 Huntington: in practical terms, it's doing what you needed to do. Maybe
177 Huntington: I think that line of reasoning is shaky, but it's certainly one that people advance. But then there are cases
177 Huntington: I don't know if any of you heard about the recent s. 1 paper. There are cases. So this was a
177 Huntington: s 1 is a reasoning model. The idea is that you, by training on just 1,000 data points. You can convert your standard Llm. Into an Llm. That can reason, and the way that they demonstrate the model is now able to reason is they take challenging benchmarks like aim 24, and Math 500. And they show that after their fine tuning and Gpqa, after their kind of fine tuning on this specially curated data. Benchmarks go way up.
177 Huntington: So if you look in the appendix that paper as I did because I am a nerd, you will find that there is that they describe the algorithm they use and the algorithm they use is they get traces from deepseq, r 1 on every aim problem. They can find that that isn't literally n-gram contaminated, and if they know that it gave the correct answer and the trace was correct, they force it to be included in their data set. And as a result.
177 Huntington: 60% of that data set that 1,000 samples is aim distribution problems
177 Huntington: shockingly. The aim score goes because, of course, aim never reuses questions or parts of questions.
177 Huntington: anyway. So that's just me editorializing. It's really a very interesting paper. And they released all their artifacts. There's a lot of actually good stuff to dig into. It's just like, but the particular headline results is like.
177 Huntington: so the overfit on the benchmark. Yeah.
177 Huntington: And if you actually use these models, you discover it pretty quickly, because they're like.
177 Huntington: like, I've I've been benchmarking them lately, and like, 1st of all, like benchmarks that used to be like. If you run when, without the reasoning, you will get answers on these benchmarks, which are only like aims. Only like 30 questions, you get an answer, and like
177 Huntington: maybe 24 h on one Gpu. If you use one of these reasoning models, it will take 5 days because they just generate so many tokens.
177 Huntington: anyway. So how do we evaluate Llms without contamination? This is not such an easy thing to do. We just took Trust
177 Huntington: briefly, and then I think it went back up.
177 Huntington: so we could do kind of vibes based Evals, you know. Don't knock them. They're extremely popular every single time a new model comes out, you know you go in and you type in your favorite pet, your private test set of 4 questions. You like to try
177 Huntington: the vibes? Yeah, I think they're actually good in many ways. The problem is, it's a low, low, sample size, and they don't. You can't really generalize across people right? It's like, because once you make it public, like the sort of Alice in Wonderland trick questions that were used for a while to fool models, and after a while, you know, if if it gets popular enough, then they just they fit on it, and then they beat it.
177 Huntington: So there's vibe space that has limits. It doesn't scale. Well, there's private test sets. I think this is a nice idea. The seal benchmark has a sealed test set that you never see. The problem with the sealed test set is, I don't know what's in it right like they they can describe it, but if they describe it very much, then they're just risking contamination again. So it's like.
177 Huntington: it's kind of like. I don't know what those numbers mean that much. I almost prefer a benchmark that is contaminated, but at least I know what the numbers mean to benchmark that I know isn't contaminated, but I also don't know what it means.
177 Huntington: You can do random humans randomly prompting like elm, says Chatbot Arena. But if you actually look at the questions people ask in Chatbot arena like you might not be so impressed with the fact that one model beats another on the Elo score, because,
177 Huntington: they, it takes all kinds to make a world and so then the final category is something called dynamic benchmark. So the idea behind a dynamic benchmark is instead of having this one set of test prompts that's frozen in time.
177 Huntington: you update your test prompts over time. Like every 3, 6, 9 months, you actually replace some or all of your questions with new questions.
177 Huntington: Very simple idea.
177 Huntington: So it's almost like a version update or like version, yeah, I just wanted to ask, are Llms unable to generate novel, prompt answers that would
177 Huntington: intentionally try and construe something very
177 Huntington: distinct, or is that just, not reliable?
177 Huntington: I'm not sure I understand like, because previously these benchmarks are good, like one is created by like random people on the Internet. But presumably you could just get like an Llm. To generate questions like generate things to evaluate another Llm on
177 Huntington: is that like, yeah. I was just wondering whether that well, so
177 Huntington: if you get one Llm. To generate the questions.
177 Huntington: Then how do you? How can you have confidence that the answers are correct unless there's a human?
177 Huntington: Yeah, I guess maybe you could construct a habit, have it in a sort of environment where you can get ground truth. And then it's about how more phrases it than the actual like novelty per se. But if you have enough, if you have a
177 Huntington: maybe like a physics model as an example, where you can probably phrase things in a particular way, and then empirically validate using some
177 Huntington: model. And then.
177 Huntington: if there's some like well defined way to ensure that, like the mapping of how it paraphrases maps directly to the query language, and then you get adequate
177 Huntington: sampling from that
177 Huntington: I guess model. Then maybe then you can get some kind of novel answers to things.
177 Huntington: Yeah, I mean, I think in some sense, that's definitely something people are looking into in domains where ground truth is cheap and straightforward to acquire algorithmically. I think that's a pretty good strategy. And I think it's actually worked really well for things like code.
177 Huntington: So in some domains that can help. I guess it's just very narrow. So it's not necessarily yeah. Well, no one's figured out the magic Sauce yet for how to generalize that to domains where ground truth is more expensive to acquire. There are some ideas, but
177 Huntington: so one such dynamic benchmark that I worked on is called Live Bench.
177 Huntington: live bench is a wide coverage generalist benchmark, really oriented mostly towards frontier models. So the idea behind live bench is, you have this one score which can be broken down into kind of sub scores on math, coding, reasoning, language, instruction, following and data analysis. But taken together, it's supposed to give, like a very good kind of rule of thumb estimate, for
177 Huntington: how likely you are to find a model useful and helpful, and
177 Huntington: all the questions in Livebench have ground truth answers. Questions are updated over time. Particularly, we will replace problems if they get too easy, or if we detect a lot of contamination on a particular set of problems.
177 Huntington: and and it's all open source, and it has a nice leaderboard, and you can go and check that out.
177 Huntington: and it's been pretty widely adopted lately by Openai and Glenn, 2.5 technical report and lots of other folks. So that was nice. How do you get the the question, how how do you maintain the live segments? So it's maintained by a startup called Abacus, and they have an engineer on staff who updates the questions and the topics.
177 Huntington: But the questions needs expertise, like
177 Huntington: does engineer can change the question well, so basically, we rely on the fact that there's a lot of new data hitting the Internet on a fairly regular basis. And there's kind of like.
177 Huntington: you know, there might be a new type of puzzle that comes out where we have a bunch of examples of that puzzle, and we can use it as a substitute for, like an old puzzle that became too easy, or we find a way to kind of transform a puzzle to make it a little more challenging for Llms by adding kind of complexity programmatically.
177 Huntington: how many questions is there in the live bench in each session? I think, yeah, it's a thousand. So it's around. And it's not evenly distributed across kind of areas. So coding, I think, has like 100
177 Huntington: alright. And then, very briefly, talk a little bit about my most recent work, which is on improving Lms with synthetic data. So
177 Huntington: historically, going back pretty much any time except very recently.
177 Huntington: synthetic data generated by models. If you trained on that data, you would get a model that underperformed compared to a model trained on data authored by humans. But there was a very interesting recent work called Tab Pfn, which showed the transformers trained on synthetic facsimiles of tables generated using Mlps outperform the pretty much any model you can pick deep learning or otherwise trained on real tables, so that kind of
177 Huntington: laid the groundwork, at least for those of us who read and love the tapia pen paper like me. It laid the groundwork for the idea that oh, we might actually be entering a period where there's a phase shift and synthetic data becomes more useful than real data. Yeah, was the argument that, like the synthetic tables, there's just like orders of magnitude. More of them. Or is it like the same amount of data or no, because, I mean, there's been models trained on at this point. There's like there's a repository called Tab Lib, which is like
177 Huntington: 70 TB of tables. I mean, it's just it's ridiculous, like, I mean, because this is like the world of big data, right? So like, there are a lot of tables.
177 Huntington: and no, it's even so. Tapia Fn only trained on, I think, 130 million synthetic tables. It's not that many.
177 Huntington: So what was what was the insight from tab? Pfn! There are tons of insight from. There's a ton of insight from Tab pfn! I don't know how much you want me to digress into that, since I didn't. Even I didn't write the paper. But it's great. Basically.
177 Huntington: the the way it's framed is as like kind of a Bayesian approach, where you sample from a prior distribution over a space of
177 Huntington: synthetic tables that are designed heuristically to resemble a particular cluster of real tables, and that particular cluster of real tables is a few 100 tables in the openml kind of repository. So they kind of study sort of aspects of normalized openml tables and figured out kind of ways to express those more generally, and then they create what they call structured causal models, which are actually just Mlps designed to generate
177 Huntington: functions, and these functions can be sampled from indefinitely. And if you kind of look at the way these functions appear after you sample them from them for a long time. They look like
177 Huntington: functions that you would sample from real columns of tables.
177 Huntington: So at a very high level. That is
177 Huntington: the idea behind happy events, data curation.
177 Huntington: And then the model is just a transformer. It's relatively small transformer. It's about 26 million parameters.
177 Huntington: So they have a data generating process which
177 Huntington: creates interesting kinds of structure that they think are the kinds of structure that they want
177 Huntington: for tables for table learning. Yeah, that's right.
177 Huntington: And so they, it's kind of like a space of functions you can sample from that are all kind of normalized to be in roughly the same kind of
177 Huntington: because one of the real problems with tables in particular is you get heterogeneity of data like the scales of the numbers, to start to vary widely. And so they kind of normalize everything, crush everything down to this very limited space, and within that very limited space the model is able to learn to kind of stepwise anticipate
177 Huntington: what the function is likely to look like at the next timestamp.
177 Huntington: So tap Dfn is not autoregressive. It's set valued. But it looks at all of the
177 Huntington: input data which is like call training data. But it doesn't update parameters just fits to the functions in the input data. And then one by one, it can predict target values in the output.
177 Huntington: I think really interesting. Yeah, it's a, it's very cool paper, like, I'm always like, what the hell to do with these huge numbers.
177 Huntington: like, I worked with combinatorial auctions for a bit. So bidder preferences like for items.
177 Huntington: Yeah, they they also like they are rich bidders. They had like multi 1 million
177 Huntington: bits. And then there are local papers like 50,000. So how to deal with that.
177 Huntington: because the the solution is not dealing with it and normalizing into this range that you train on. Yeah, it seems to be
177 Huntington: yeah, if you're interested in this. We can talk afterwards. I'm working on actually a follow up that's trying to define a prior for semantic data rather than numeric, which is all the Tabia then does, but that is its own kind of problem, anyway. So this paper is about synthetic data for Llm. Fine tuning.
177 Huntington: so
177 Huntington: papers called Wild Chat 50 M. We used a bunch of different Llms and generated 1 million responses per Lm synthetic data generator. The data set we used for prompting was a data set called Wild Chat, which is just a big collection of questions that people asked of Chat Gpt over like a 1 year period. It was sponsored by Allen AI.
177 Huntington: And so these different data generating oems, they all answer the questions slightly differently, and in some cases very differently. We actually have some kind of statistical analysis. We ran in the paper which show that actually they're not all that different, but not as different as you might think. But it's still different. And so then what we do is we fine tune, Lama 3, 8 B base checkpoints on our newly curated data sets. We experimented with lending the different synthetic data generating models. But since we found that they behaved
177 Huntington: almost exactly as like linear interpolations of the data generating models, and made more sense to just kind of train them singly in the end, which is what we did. And we sort of found that certain data generating models work much better for this task than others, and we're able, based on that insight, to create a
177 Huntington: relatively small sft. Mix that outperformed the recent 2 to 3 Sft mix from Ln AI.
177 Huntington: So choice of model was very impactful. And
177 Huntington: happily, I was very unhappy with this insight when I was working on this paper. Because I'm like, that's like, that's like a non. It's a non fact, it's true, but not interesting, right? It's like, why does it matter like, why are some models better than others for synthetic data generations very hard question to answer that I worked hard on and made some progress on.
177 Huntington: So one thing that is definitely not the case. Models do not inherit benchmark performance via distillation, especially on hard topics, when you're just prompting them with like generic wild chat prompts. And this makes sense right? Like the prompts aren't asking for answers to questions in the math. 500 distribution. So the model doesn't learn about the math. 500 distribution.
177 Huntington: But it's not kind of this naive thing of like. Oh, it got better because the Dgm. Was better at that benchmark.
177 Huntington: It doesn't even work that way for chat. Benchmarks, even chat benchmarks. It doesn't actually transfer smoothly, although it transfers more.
177 Huntington: But they do inherit stylistic
177 Huntington: formal and informal stylistic aspects. Responses will get longer if the Dgm's responses are longer. If the Dgm. Uses particular Markdown styles or HTML styles, they'll show up in the model that's fine tuned.
177 Huntington: and Llm. Judges care a lot about style, as we know from previous part of this talk. So llm judge benchmarks like alpaca, eval and empty bench, which is where we saw the biggest delta over prior work. As opposed to the ground truth. Benchmarks like mix eval, Mmw. Pro, and user.
177 Huntington: l and Judge benchmarks show huge delta here. The other one that shows a huge delta is instruction following. If Eval, this is an area where I wish there was more benchmarks. But right now it's basically just if Eval. So if Eval is this like they, they give a wide variety of kind of like very specific instructions to the model about, like how many characters it should generate, or like how many, like every 3rd word, should start with an L that kind of stuff.
177 Huntington: And then they test whether it followed the rules.
177 Huntington: And so basically, when you use a strong Dgm in the upstream, the model learns 2 important things. Number one. It learns about what it means to actually follow instructions and respond to a prompt and number 2. It learns how to style a response in a way that's pleasing.
177 Huntington: And that combination of factors means that if you use a strong Dgm, you get a strong
177 Huntington: kind of downstream model. Relatively speaking.
177 Huntington: coordinate Lm, judges. Model responses become more accurate, comprehensive, and appropriate after distillation.
177 Huntington: Alright. So what time do we?
177 Huntington: Okay, so we're already like, kind of pretty far along. So maybe I'll just skip through this and
177 Huntington: move to questions if people have. Oh, future work. Okay, I didn't know if we were over time. But it seems like people are kind of more flexible here. So okay, one thing that I'm really interested in, how do we? And actually, Chris, you were just talking about this, how do we ensure the Lms ground the responses and provided context and fully exploit that context when generating responses. This is kind of like the Lm judging problem.
177 Huntington: So there was a recent paper from University of Washington called Open Scholar, which is introducing some new methods to try to do this. And there's definitely, I think, progress being made. But long context is a huge challenge here. When you have the longer the context gets the worse the model gets at kind of picking the important aspects out.
177 Huntington: Slm, still struggle compared to Llms with this kind of thing. And there's generally like a failure to, I I think,
177 Huntington: to ground responses in certain types of context, right? So you can use the term very generally, but like, if the context is a table, it's like in a table in the form, in a format. The Lm isn't familiar with.
177 Huntington: It might ground really poorly. But then, if you translate that same table to like HTML or Json, it might do much better. We we're seeing that in recent experiments right now, like we're trying to prompt a small Quin model to do better on like table. Qa.
177 Huntington: And we found, oh, what do you know, Quinn, Qa. Quinn understands Json, not Markdown, not HTML, just json. Okay.
177 Huntington: Another thing that I think is interesting. What alignment strategies will work best in fields where ground. Truth is multifaceted and complex. So this was your question. Right? Like
177 Huntington: where we have cheap ground truth, I think. Kind of the roadmaps actually getting pretty clear at this point, like, there's some really good work coming out of lubric Schwitz lab and and others that like
177 Huntington: you can eventually just see enough examples, with enough reasoning traces that the model learns not just how to answer, how to give the answer that works, but how to actually reason its way somewhat to the answer that works.
177 Huntington: So while I'm skeptical of reasoning, writ large, I'm not skeptical of reasoning where we can access infinite ground truth, because eventually I do think it learns the path.
177 Huntington: But that doesn't work in most domains, most domains. It's really hard to get those reasoning traces right? It's really hard to explain, like.
177 Huntington: for example, rendering a judgment in a in a matter of law, right like, supposing I have like this dense body of law, and maybe there's like 5 different statutes, all of which influence how a judge would interpret a particular case, not to mention the fact pattern of that case. And then the judge has to render a ruling like. We have
177 Huntington: many, many people every day in this country who do this kind of reasoning work, and we have no idea how to teach Llms to do that right. That's just one example. What doctors do is another.
177 Huntington: And then this is something that I'm very passionate about open source. I'm very passionate about about not just open weights, models, but models where the data is open. I think it's a real problem that we don't know what data our models are trained on. And we don't because
177 Huntington: it's a problem for a number of reasons. But one reason it's a problem is, it's entirely possible. And I'm starting to believe it. It might even be true
177 Huntington: that the only way to get the really highly performant models that we all use every day is to train on copyrighted data, for instance, and we don't know yet how it will shake out whether we're legally permitted to train on copyrighted data right now, everybody's assuming the answer will be, yes. But what if the answer is no.
177 Huntington: you know, then we have to have some kind of data curation strategy to adapt, or we will never be able to train Gpt again. That would be a problem. Right? So yeah, how do we reproducibly build fully open source models whose performance rivals out of closed source models. And if we need to do things like train on copyrighted data. We should know that right like, Oh, here's the Delta we got from that that should be in the literature.
177 Huntington: And unfortunately, still, there aren't any large high quality open source. Llms. Alan AI is fighting a good fight here. They did release a 32 B checkpoint of almost 2 recently. I benchmarked it, and it is a lot weaker than Quan, but it's it's better than than anything that's out there so far. But the gap is still large.
177 Huntington: so, yeah, so those are some
177 Huntington: talking points. Happy to discuss anything
177 Huntington: in the scope of this talk, or within my limitations as a human outside of the scope of this talk.
177 Huntington: Yeah, I was wondering, for when you're doing the supervised fine tuning. So you're taking a checkpoint, and you're trying to basically
177 Huntington: teach it a particular like subdomain or set of problems. I was wondering whether when you do that, whether you see how well the model can
177 Huntington: like answer those questions if it's given in context learning or in context information. So like the supervised fine tuning is training parameters. But maybe it's just learning very simple, simple.
177 Huntington: subtle changes that like you could learn in context. So yeah, I was curious.
177 Huntington: So like, what would be a systematic way not to put you on the spot here, but, like what would be a systematic way, we could ablate something
177 Huntington: right, like
177 Huntington: different models, would understand different forms of in context prompting better. There would be a tendency, I think.
177 Huntington: to. I mean that to optimize.
177 Huntington: to prompt engineer around the constraints of a particular model.
177 Huntington: Oh, what do you know for this model chain of thought works? Why? Because it was trained a lot of chain of thought. Oh, for this model! It's more like it likes few shot, you know. Maybe likes to be told. My grandmother is in the hospital, and she desperately needs you to, you know, to do this to answer this math problem for her.
177 Huntington: So I don't know. That's kind of a non answer. But like
177 Huntington: it's maybe like pointing at why, I think they are fundamentally different domains, like one is sort of exploring the space of
177 Huntington: a model that is already frozen, with all its quirks, intact and kind of saying like.
177 Huntington: where do the quirks lead us?
177 Huntington: And yeah, I would even throw methods like disp and text grad. You know, which
177 Huntington: claim to be optimization into this bucket as well where they're basically
177 Huntington: they're just sort of learning the noise on the surface of of what the model happened to fit to.
177 Huntington: The reason I prefer things like Sft is because I understand well what that process does. And I understand what it does for models that aren't Llms. So I can expect similar effects when I do with Llms, where it's harder to benchmark.
177 Huntington: If I were to like naively, think about it, I would think that since
177 Huntington: comparing with like, you're kind of wanting a specific specialized model, and
177 Huntington: it sort of doesn't matter what the context or the supervised fine tuning is as long as it does well in this kind of subtopic.
177 Huntington: So it almost you could almost like, just take the best resulting supervised fine tuning, like just sample different ones. And then, if it turns out that that is still doing better, it's still it's doing better than the fine tuning. Then you have a prompt that does well in that sort of range. It's a bit naive as an approach. But I guess that's kind of my
177 Huntington: yeah. I mean, I think.
177 Huntington: yeah, I mean, I think it's it's certainly an area one could look at. I will. I will say, if you're referring in particular with this work that the point was actually not to sft to a particular domain, but to sft more generally like the the idea being generally let's make. Let's see if we make a model. That's better at a lot of things. As it turned out, we couldn't necessarily make a model that was better at a lot of things, but we could make a model that was better at things that had to do with instruction, following and conversation
177 Huntington: generally, it would appear.
177 Huntington: Yeah, going back to your 1st topic. So you're talking about like looking at implicit biases of Lm as a judge. And I think now, with the reasoning models I've seen some people using the reasoning models as a judge as well. And do you think if there's any value in looking into the reasoning chain, and then see, like, whether, like the Lm. Will verbalize its bias in how it arrives at the decision.
177 Huntington: So
177 Huntington: the short answer is, yes, the slightly longer answer is, Lm. Judges already do, that they already give reasoning behind their their decisions and their judgments, but
177 Huntington: those that reasoning could conceivably be different than the reasoning that would be generated as part of an internal chain of thought, the idea being that what a model might say to itself in private would be different than what it would say in public when kind of explaining its decision. That's actually a distinction that a real judge would also employ. So like. It's not unreasonable to think like models might reason better if they have an idea that some of their reasoning is private.
177 Huntington: But that, said I think we're like. So I'm I.
177 Huntington: right now I'm working for a company called Umi, and interning for a company called Umi, and one of the projects that I'm supervising is basically that it's looking at this same problem. But with reasoning models as judges and kind of asking, can we fine tune these reasoning models to make them more reliable judges by like improving the internal clock architecture.
177 Huntington: and just follow up. Did you see any difference between the reasoning chain of reasons. And if you ask an Lm. To give you post hoc reasons for for its decision
177 Huntington: we don't have them gonna have the results yet. So I don't know what we'll see. I would certainly hope they'd be different.
177 Huntington: Whether we can quantify
177 Huntington: the differences in a way that I will find satisfying. I will have to do some thinking about that.
177 Huntington: It took kind of a lot of thought to come up with this experiment like there were a lot of
177 Huntington: experiments that I thought of that I didn't really like very much like it's sort of hard to say like, oh, this was better reasoning.
177 Huntington: you know. When it comes to like rendering a judgment. It's like better, according to what
177 Huntington: you know. Like what you know. What's your
177 Huntington: what's your gold standard for reasoning internally consistent.
177 Huntington: you know. Are you gonna try to do some sort of like hallucination detection, because then the longer ones will always look worse.
177 Huntington: you know, etc.
177 Huntington: What what's like? Your go to benchmark, or couple of benchmarks that you use to like check stuff out for a new model. Well, I'm biased with live bench live bench, I think, is a really good benchmark for what it is, which is like. Kind of
177 Huntington: Should I pay attention to this frontier model or not? I think live bench is great for that, because it's designed not to be able to overfit, too. Is that? Why do this? Just because you're you're when you look at a benchmark. Your 1st reaction is, oh, that's very easy to game. Yeah, yeah. Even Chatbot arena, which, like many people, present as ungamable like. If you actually go and look again like the.
177 Huntington: there aren't really infinite questions in Chatbot Arena. It's really that people ask a lot of the same questions over and over again. And more importantly, people themselves are hackable like, that's another part of this paper that, like people's judgments, are hackable. We didn't talk about as much, because it's more like social sciences than something I would publish, and I clear, but like, it is true, right like there's tons of literature on it. So when you train an Llm. Judge to behave like human judges, you inherit their implicit biases as well.
177 Huntington: anyway. So to answer. But to answer your question. I like live bench. I also like mix eval if you want like kind of a cheap to compute thing that looks kind of behaves like an Lm. Judge, but is based in ground. Truth. I think Mix Eval is very nice, and it's small.
177 Huntington: I like. I love suite bench. I think Swede bench is fantastic.
177 Huntington: and I think there should be more like it. The only downside of sweet bench is, it's really hard to run, because you basically need like a docker container and like a special environment, because it's basically for those of you don't know. Sweet bench is a benchmark where you start with a prompt and you end up with a pull request.
177 Huntington: So the model has to do this kind of agentless reasoning, going all the way from the prompt, like figuring out which tools it needs to call to writing the code, to actually diffing the code to submitting the diff, and either the diffs accepted or it's not. You run certain tests on the dip to see if it passed, and it's based in like real repos, like scikit-learn and stuff, and like real kind of feature requests and bug fixes.
177 Huntington: So I think that kind of direction of like these sort of focused, practical benchmarks. Something I'm excited about.
177 Huntington: I also love it anytime. Someone actually does the work to come out with like a new hard math benchmark, because it's so hard to make them. I feel bad that they like get contaminated so quickly. But like there was Putnam axiom recently came out, and right now it's pretty good. I'm sure in 6 months they'll have seriously, the numbers will float up. Humanity's last exam. Same story.
177 Huntington: So I guess the rest are kind of like domain specific. But like for general, I like those cool.
177 Huntington: Yeah, I have a very naive question.
177 Huntington: Do you think it's feasible? So when a company or an org trains a model that they watermark the inputs and outputs to everything, so that later on, after it's trained. You can see whether it produces the output from the seed. Input and it would be like, kind of algorithmic, because maybe the watermark is like with respect to like a pair. And so for each pair you could like.
177 Huntington: Well, you, you'd be able to query the the the
177 Huntington: query for it, using the key some database, and then your answer would abide to the watermark in some respect. Yeah, I don't know. It's a bit of a you know out there question. But given you thought about this, far from being an out there question, I think Lm, watermarking is a really active subfield of research. With lots and lots of papers. Deepmind had one.
177 Huntington: I think so. My initial kind of hot take which I'm I'm prone to make hot takes that are way too premature. So my premature hot take on all motor marketing is, this is impossible. Nobody should be trying to do this, but, like I think, upon like
177 Huntington: reflection, I would have to say, it's just very hard
177 Huntington: to do it generally for everything. But like, if you have enough text from a model, it's not at all unreasonable to imagine that you should be able to identify like the more text I get from a model at some point, I should have enough that I know if it's model A or model B, as long as it hasn't been contaminated or messed up right like. If some human gets to go in and adversarially change
177 Huntington: words, especially if that human has access to the algorithm that was used for watermarking. Then I'm skeptical about it all the time, so I don't think it's like a good defense against, say, like, people who want to cheat on their exams with Llms because of what I just said. But if, like, you're if you're using it more sort of like
177 Huntington: internally, yeah, it should be possible. And I think it is possible
177 Huntington: for the fusion models. You have this hidden in the noise paper. Right? Yes. So you're like, okay, I can just make a database of my database of noise. Yes, a database of noises, as as Nick calls it. Yeah, yeah, I was sorry I couldn't make you talk. So yeah, yeah,
177 Huntington: was fun.
177 Huntington: Yeah, no, it was a
177 Huntington: are my 1st foray into diffusion models, and I'm I think they're great. I really enjoyed Rohit's talk when he came to Nyu recently, so
177 Huntington: would love to learn more about fusion.
177 Huntington: So so I have a naive. By the way, did you take nif spot?
177 Huntington: Did you take nifs presenter spot, or was like, because, like Nif
177 Huntington: wanted to come here as well. Oh, he was here. You missed him because you started a little too late.
177 Huntington: I don't know, but like
177 Huntington: he wanted to come here again. Oh, again. Oh, well, we haven't. He hasn't gotten in touch with me to schedule another one I put him in touch with. Oh, okay, yeah, we'd be delighted to have him. Okay, yeah.
177 Huntington: I did not know he was planning to come again. I knew he had come.
177 Huntington: We were chatting about it. I'm not too sure sorry. I don't think I took. I don't think I took his spot. I don't know if you don't know about it. You didn't. Okay.
177 Huntington: But
177 Huntington: so I I have. I have a naive question about benchmarking, and then I have another. So so the naive question is.
177 Huntington: you know, isn't. Isn't the standard thing to do to have
177 Huntington: you know the the secret you know, to split your benchmark. You have a public part, and you have the secret part, so the secret part can't be gamed.
177 Huntington: Is that not at the defense, or when, or do people.
177 Huntington: or or when people game the public part. Do you feel like that sort of contaminates the secret part of your back?
177 Huntington: So if I have enough Meta information about the benchmark. Then I don't really need to see the samples to contaminate right like I can guess that's sort of the that's sort of in a way that's like the s 1 problem. Right? Like, okay, they're holding out the 2024 edition of 30 questions from A,
177 Huntington: but they're training on 1980, through 2023. Every single question that was in a
177 Huntington: the the P of
177 Huntington: exact overlap is high, and it gets higher the more you can sample from the distribution
177 Huntington: so like at some point in expectation, you you will just sample every single aim distribution question. There could be close enough that an Lm. Could fuzz it right like. And there you face kind of the the question of like, well, is that really contamination? Is that just the models learning to generalize, and I don't know the answer to that right like that's philosophical on some level, I think.
177 Huntington: but I think that it's more that there, that it's contamination when you can't.
177 Huntington: when you can't make reasonable transformations and reasonable changes, and still have the so-called reasoning hold up, you know, like, if I'm able to like.
177 Huntington: just make the prompt longer. And all of a sudden they can't reason. Then I question whether it was really reasoning.
177 Huntington: So let me ask the flip side question. So you know, if we, as a field, succeed in this reasoning dream
177 Huntington: and get models that
177 Huntington: can be creative that can come up with a generation of insights and knowledge that humans have never had before.
177 Huntington: What's your perspective on? How we should recognize that or benchmark, that
177 Huntington: so creative meaning? They can do things that are like like like the way like the way that we can do in board games today, where? Where, you know. Oh, we have
177 Huntington: chess players that can employ chess strategies that grandmasters cannot come up with, or you know, or or that kind of thing. Yeah, so I might say, like.
177 Huntington: so right? I mean. So there's like the Go models and the chess models that can figure out kind of move sequences in a fixed system, like in a narrow system that are novel, that are undiscovered, or, like the protein discoveries, right like right, so in a broader, in like, in broader domain, the general purpose domain in the Lm domain. General general knowledge.
177 Huntington: you know, is, is it feasible? Like, what would you we do to recognize when we're succeeding in doing that? Is there a way of benchmarking that so, as a former filmmaker, I would argue that there is a difference between creativity and hard problems and creativity and wicked problems. Like that. Basically protein folding or chess is a hard problem. It's just like there's a lot of
177 Huntington: combinatorial possibilities. But they're finite, and there are rules and the rules pretty much always hold
177 Huntington: because they're kind of governed by systems.
177 Huntington: I think
177 Huntington: creativity in domains like that, we already know, is within the reach of frontier models, and they do it very well. Compositional creativity, right? Like. Sample a little from here, sample a little from there
177 Huntington: smush it together. Voila!
177 Huntington: But, like David Lynch, who recently passed away, is someone I consider to be a supremely creative individual, and he was creative in like a field that I consider to be a fairly wicked domain film, where, like, you sort of have 90 min in a blank slate. Anything you can imagine in images anything you can imagine sounds you're allowed to do. There are no rules, there are no. And and every time someone tries to make them. The really creative thing to do is go and break the rules and show that you can still make it work.
177 Huntington: And I don't.
177 Huntington: Not only do I not know how to benchmark, that? I don't even know
177 Huntington: that I believe that any sort of algorithm that's based on the convergence of probabilities towards like kind of local maxima could ever do that, because, like what David Lynch does is like, well, because everyone's going here, I'm going to go over there
177 Huntington: not to be different, not to be contrarian, but just like cause. I see things differently.
177 Huntington: and then you get a razor hand which is like unlike anything before or after to me.
177 Huntington: So you're skeptical that the the research program
177 Huntington: could ever succeed in doing this.
177 Huntington: I think we spoke about this a little bit. Actually, it's like I. Here's the thing, I think in order to do it, we would 1st need to figure out how to measure that we've done it.
177 Huntington: And I think I don't know. It's hard for me to conceive, like, how you would measure that you've done it
177 Huntington: like
177 Huntington: diffusion models being creative diffusion models coming up with genuinely new stuff that's not like kind of again, like interpolation of artist styles.
177 Huntington: like, what would that look like.
177 Huntington: you know? Like, how would, you know, when you'd seen it like sometimes, if you kind of. I see presentations on the fusion models, and they kind of scandal late in space, and they discover something
177 Huntington: that they can't interpret. And I said,
177 Huntington: I can't interpret that. I can't put that in my paper. That's not anything
177 Huntington: but like if I was Matisse, I might look at that and say, that's my canvas. Now I'm going to start here because this expresses what's in my soul. And now I'm going to build on this, and then I'm going to make it my style.
177 Huntington: So like, if you like. What do you do? You put a label on everything like
177 Huntington: you can't put a label on something until it's a thing that's not a thing until somebody kind of shows you that it's a thing
177 Huntington: alright had some cool results about this.
177 Huntington: Yeah, I just do this letter space.
177 Huntington: The the principal components are at least not overlapping really with artists.
177 Huntington: but which is not the proof. Of course.
177 Huntington: I think so. It's kind of interesting.
177 Huntington: What do you think about this?
177 Huntington: Let's make a game show robots versus humans. Yeah, yeah. General.
177 Huntington: generally intelligent robots that do something like drawing a painting, making a movie versus generally intelligent humans.
177 Huntington: Let me just see who wins.
177 Huntington: If you're measuring by human preference. The robots will win eventually. You think so? Yes, because humans are hackable, as you were saying. Yes, because humans are hackable. Humans don't know what they don't know until someone like David Lynch or Henry Matisse comes along and shows them, and then they're like, now I know that.
177 Huntington: and then it becomes part of the cultural conversation, and then people copy it, and then it becomes a cliche, and then everybody forgets it, and then the cycle starts again.
177 Huntington: But like, if they can recognize, the novel attribution
177 Huntington: is that that seems like that's a solution to me that
177 Huntington: if you, if you have people who are able to say like, Oh, that's good, that's good.
177 Huntington: Well, the people who are able to say it are mostly
177 Huntington: the platform absent from the cultural conversation at this point, because we've devalued criticism, and we've devalued kind of the idea that there could be such a thing as a judge of art. There used to be people whose profession was to go and watch all the art.
177 Huntington: and they're still there still exists in some places in diminished capacities, but they used to be their job right to go in and watch everything, and then they would know what something new was. But then we sort of all said, Yeah, we're we're better judges than they are. We don't need these guys bunch of stops. Right? It's like, okay. But then, now it's new to you.
177 Huntington: It's on mute.
177 Huntington: right like. And somebody who knows would tell you it's not new. And then you say, Shut up, you stop. I like it.
177 Huntington: Yeah. How did you classify the AI photo that won the I forget which competition it was? Yeah, if that if that passes, does that mean creativity?
177 Huntington: No, I mean, no, it just. It's it's like, here's okay.
177 Huntington: I don't know how you again like I would love. I would love to design a true benchmark of creativity. It's like such a cool idea. But like, here's how hard I think the problem is, it's like.
177 Huntington: I don't know what it was like for you, or if or if people have had this experience. But when I was 18 years old and I watched Twin Peaks for the 1st time.
177 Huntington: it was like my brain just like
177 Huntington: lit up and went to another area. It's like, now I know what I want to do and like, I didn't even know this world existed. It would like it haunted me. It's like I was thinking in different ways. I was seeing in different things like it literally transformed my perception.
177 Huntington: And there are, you know, like a dozen or so filmmakers throughout my career who did that for me at different times.
177 Huntington: And it's not just me right, like I'm not the 1st person to think David Lynch is special. Lots of people have this experience watching David Lynch
177 Huntington: or Ingmart Berkman or Jean Renoir.
177 Huntington: So it's not like. It's just some idiosyncratic work of me.
177 Huntington: It's a transformative experience that others can access, but not everyone.
177 Huntington: And some people hate lynch and hate God and hate
177 Huntington: grandmother, and they hate the feeling of being pushed out of their comfort zone
177 Huntington: by the media because they don't want to go to that place.
177 Huntington: and so they say, you know, give me something comfortable. Give me my comfort. Give me my Star Wars. It's like, Okay, you can have that. It'll tell you what you already know in the way you're already comfortable hearing it.
177 Huntington: But the discomfort that initial discomfort is part of the transformation. So it's like something that's received, I think.
177 Huntington: by it's a it's a like something that happens to the receiver
177 Huntington: more than what happens to the artist.
177 Huntington: and I don't know how you know
177 Huntington: I love it. I feel like it. It belongs on your future work slide.
177 Huntington: Yeah, I'll think about it. I don't know. Maybe there's some way to do it. Maybe it's like, maybe there's some way to do it. Do you see a relation between novelty and creativity?
177 Huntington: That is that a proxy that is very, but not sufficient.
177 Huntington: Right? So can it be a proxy for evaluating creativity?
177 Huntington: Evaluate novelty, novelty, earlier novelty.
177 Huntington: Yeah, if you had a good enough judge, you could do it like that's basically the role of the critic. The role of an art critic is to say, this is actually new
177 Huntington: and good and good, but almost sometimes just new is enough like sometimes our critic will say, this is not good, but it's new, and like that also can be value. It's like this almost worked. But it was actually terrible. But you should consider watching like people like me, like we would go and seek out those movies that were like bad but interesting, because that's where you find the seeds to make something good and interesting. The noble failures, the
177 Huntington: the adventures of Buckaroo Bonsai. Oh, my God, what a mess! Sorry? What was your question? Yeah. Yeah. So measuring novelty might be a simpler
177 Huntington: tractable question, because you have access to the training data.
177 Huntington: Yeah, so you can measure surprise, right? Like, enter, you know, in some entropic sense.
177 Huntington: then I think the question becomes like
177 Huntington: narrowing the corpus selectively in a way that prioritizes
177 Huntington: certain subdomains. Right? Because if your domain is all images, then what's surprising is garbage
177 Huntington: because it's not like images. There's a huge space of noise and images on this tiny, low, dimensional, manifold, surrounded by lots of noise, so like the hard thing is finding useful directions for noise that exist in like almost vectors of meaning.
177 Huntington: like, there are certain vector directions that lead to meaning, but most lead to noise, which is why most experiments fail in art? Right? So in an abstract sense, what do you? What would you say is the gap between novelty and creativity? What's that
177 Huntington: missing direction.
177 Huntington: Hmm into.
177 Huntington: Here's a mushy answer for you. Intuition like, and taste like the ability to. So, as I just said, like
177 Huntington: monkeys on keyboards right like, there are literally so many, so many vector directions you can take from a starting pick a starting point somewhere in the world of things you can see.
177 Huntington: Pick a direction you move in that direction. You'll get somewhere.
177 Huntington: Now pick another direction. You move in that direction. You'll get somewhere. Is there a causal link between these 2 directions. You picked
177 Huntington: intuition will tell you whether there is or there's not
177 Huntington: right into like. We all have some degree of intuition, right. That one direction leads you to a cow.
177 Huntington: and then, right after the cow, you get an image of milk.
177 Huntington: and your intuition will tell you. These are linked.
177 Huntington: You know, where that intuition comes from. It's not like it's not easy to bound that intuition, because I could also follow cow with factory, and you get a different association. It's called the cool shop effect. So
177 Huntington: where the edit creates the meaning
177 Huntington: and so you could juxtapose sequences of images and create meaning in the mind of the audience that's not actually in the images. It's in the latent space of the audience's mind.
177 Huntington: And that is what artists exploit. They exploit the at least film film artists. They exploit the what the audience knows and what the audience expects.
177 Huntington: Interesting.
177 Huntington: Yeah. Don't they also exploit like emotions that the audience might feel so like, intuition does come from knowledge. But I feel like it also comes from emotions. And that's probably also what's lacking? Oh, yeah, but emotions are are like, partly triggered by our expectations. Like, if you go into
177 Huntington: a scenario expecting, say, a horror movie.
177 Huntington: and then you find out that, like all the vampires are goofy, then you might be in what we do in the shadows.
177 Huntington: and you might be surprised to be in what we do in the shadows. And then you might like that. And a lot of people do like that right? Which is where you get something that many people call creative. It's like vampires, but like hipster vampires, and it's funny, instead of being horrifying.
177 Huntington: So that's a form of kind of low grade but meaningful creativity.
177 Huntington: It's not just a random juxtaposition. It's not like some interpolated vector between like vampires and comedy, because you had to pick
177 Huntington: polls that were going to be meaningful in the mind of the audience. And then, when you met in the middle, it created something
177 Huntington: another wonky question. Do you think that if an AI model is capable of causing a human to fall in love with them, that it then would have the capability to produce an artifact that would be judged creative by that human.
177 Huntington: No, because humans can anthropomorphize anything humans can anthropomorphize their carpets, you know, like, it doesn't mean anything right? Like, it's a human. It's the human that's special, not the AI
177 Huntington: humans can make meaning out of anything. It's kind of what we do like.
177 Huntington: It's really important that we can do that because it's how we relate to one another. But
177 Huntington: but we can do it for anything. We can do it for our toasters. If we really want to
177 Huntington: like Tom Hanks in the beach ball and castaway. Right, Wilson, you know. If like, if you're desperate enough, you can anthropomorphize a beach ball, volleyball, volleyball, volleyball. My bad, it's a factual error
177 Huntington: like so for humans, we also don't know where this comes from. Right?
177 Huntington: So there are some people
177 Huntington: that are making the point that you know, we are kind of a product of evolutionary process.
177 Huntington: which is kind of like a weak optimizer. Yeah, just random mutations and some stuff sticks, and where stuff's highly conserved, right? Like, there's all this stuff that's conserved in evolution. That's kind of bizarre. But and then so some people then start drawing the analogy.
177 Huntington: In how we actually optimize with a better optimizer we are at least we are using gradients.
177 Huntington: To, you know, fit something
177 Huntington: to some huge data pile or whatever.
177 Huntington: Yeah, okay.
177 Huntington: So and then they are making this point like, evolution gave rise to humans. Right?
177 Huntington: So this other optimization procedure could also give rise to something interesting, something maybe even superhuman.
177 Huntington: I mean, I I really love love love your sober take really love it.
177 Huntington: But like so, yeah, do you have something to say to that? But the impossibility of creativity? No, he said, something superhuman, I'd say Claude is already well, but also along the creativity axis.
177 Huntington: I don't. I just don't see it. I I'm very skeptical. I would love to be proved wrong, and I'm happy to like, sit and muddle over a research project for but, like again, I would think 1st in terms of benchmarks. How would you ever know that you succeed? Right like? That's the 1st place I would start is like, show me a measure
177 Huntington: that quantifies how close you come from on the scale of like a Platypus to David Lynch, and I would be very impressed that you had come up with such a measure, because.
177 Huntington: as far as I can tell, even simple scores of like image fidelity are largely noise
177 Huntington: like we don't even have a good score for that
177 Huntington: like this is a better or worse image. We can't even score that like.
177 Huntington: how are we gonna score like this is a more creative
177 Huntington: as a prior, we need to know at least what's better. And we need to be multifaceted in the way that elm judgments are. This is better because it's scary. This is better because it's
177 Huntington: more impactful.
177 Huntington: This is better because it shows more craft.
177 Huntington: you know, if we could say those things we control control those things, then we might be able to steer the evolution of a learner
177 Huntington: to force it. To take a non majority path through the data
177 Huntington: might be able to say, I'm going to force you to ignore the fact that there are lots of bright images, because you are going to be a chiaroscuro artist. So you are going to be attracted to dark images with lots of soft shadows and lots of soft color.
177 Huntington: And like, that's it's not because I'm gonna throw away the other data. You're still gonna know it's there.
177 Huntington: right? But you're going to prefer this data. You're going to be biased. You're biased. Learn?
177 Huntington: Would you make a bet about whether. And basically, would you make the bet that no AI generated movie will ever win an Oscar? No.
177 Huntington: because an Oscar might be hackable. So Oscars are super happy. So so actually, I kind of, here's, here's a, here's a data setting for you. And maybe you need to build a relationship with one of these startups that you work with this. But but I feel like, okay, yeah, if humans are hackable. And yet.
177 Huntington: you know, there are, there are expert humans who might be experts in creativity.
177 Huntington: So if you had access to the open analogs, or for image generation, if you you know.
177 Huntington: do you join the staff of. You know, what's the Big Image Generation
177 Huntington: company got all those usage logs.
177 Huntington: you know. Most most of the users
177 Huntington: are doing not very interesting things, and not very discerning, but
177 Huntington: with the right analysis you might be able to identify. Aha! There are some. There's some real experts in here who really understand.
177 Huntington: You know what they're doing. We're really steering these things in creative ways or recognizing the difference between what's creative and what's not.
177 Huntington: you know. Maybe it's maybe it's about identifying
177 Huntington: those humans, and that that could be a source of an interesting sort of benchmark or a test or something. So if I imagine that each
177 Huntington: each David Lynch, is some low probability path through the distribution of all data that lands at some endpoint which is totally not random and totally not the most likely outcome.
177 Huntington: but somehow, but great for some reason. Yeah, and somehow great. For some reason. Yes. Well, a bunch of other people landed somewhere else. We don't know who they are right, but like, but I know that trajectory fine, but that's O of one. I only have that one. I don't know what's near it, and I don't know really how. What's near. It is different. I don't know why, if I go and copy everything David Lynch did, and I could.
177 Huntington: The movie would not be as good as a Dave Lynch movie, right? Like, because I
177 Huntington: people try to do this all the time in Hollywood, right? They try to copy something successful that sucks. You can't copy something successful like it doesn't work that way because you're you're missing something, and also because it's the second one. Now
177 Huntington: that, too, that, too. Right? So then, like, okay. But but in order to learn in machine learning, we need lots of data, sure lots and lots of data in the near region of the thing we want to learn. But what's in the near region of the creativity of David Lynch and the creativity of Ngmar Bergman and the creativity of. So I'm not proposing a way of solving it. I'm just sort of proposing. Oh, there might be a path to benchmarking right? Which may be the first.st
177 Huntington: You know your prerequisite, you're like, oh, I don't think that's true. Right. Does that make sense. So like the idea being that there might be. Yeah, no, that's that's true. There might be learnable. Well.
177 Huntington: there might be learnable commonalities in the trajectories even, or or even in the sense of your live bench. Maybe it's not even
177 Huntington: a benchmark that's automated. It could be
177 Huntington: a benchmark made up of systematically identified humans.
177 Huntington: Right?
177 Huntington: You know. Oh, we have this 0 point 1% of Openai users who we judge are the best.
177 Huntington: you know, judges of creativity. So if you really want to benchmark your creativity algorithm, then, you know, pay open AI a little fee, and we'll mix in your algorithm into a little bit of their outputs
177 Huntington: right? And if they, if they seem to prefer it.
177 Huntington: these people are seeking out creativity
177 Huntington: like you're doing a good job
177 Huntington: kind of inspired by your live bench, but maybe a scalable way of doing it.
177 Huntington: Yeah, there's certainly something to the idea.
177 Huntington: But there's latent commonalities that aren't obvious.
177 Huntington: But you actually have historical data. Right? Like, you know, you actually have lots of historical data on like which artists became successful and the context in which they became successful, which ones did not. Now it's awesome. You could assume. Some of that's noise, some of that's randomness. But is there something learnable?
177 Huntington: It's interesting, Benjamin.
177 Huntington: What a fascinating conversation. Thank you very much. Thanks again for coming. My pleasure