j.fiotto-kaufman: Here.
j.fiotto-kaufman: Okay? So so yeah, so welcome. Welcome, Jaden. Yeah. So I'm Jaden. I'm the principal engineer on the national deep inference fabric.
j.fiotto-kaufman: And then so what is the Mdif
j.fiotto-kaufman: So it's a set of tools and hardware enabling people to do research on very large neural networks.
j.fiotto-kaufman: And why
j.fiotto-kaufman: are we specifically doing this? It's because probably doing research on the model that actually affect our lives is important, like, you know, Chatgpt and Claude, and then there's a nice figure here from my paper on the right, showing the the research people are doing is not on the largest model. So like here all these publications on the right hand side in these, like blue bubbles.
j.fiotto-kaufman: marking different pieces of research done on open source models, and then on the right, you know the exponential kind of increase of model size over time, whether it be closed or public.
j.fiotto-kaufman: Yeah, I don't know. There's not really any incentive for large companies to be doing this work. So kind of we have to whether they believe
j.fiotto-kaufman: it's not useful, because.
j.fiotto-kaufman: you know, as long as you get inputs and outputs and they work. That's fine, or that you can't do it right. It's a black box. There's numbers flying through the model. I'm sure this lab obviously doesn't agree with that.
j.fiotto-kaufman: Yep.
j.fiotto-kaufman: So where are we now? So like one of the biggest things that's happened recently is right. We're finally on Delta, you know, I'm pretty sure, even if you oh, you have to explain all these things right. I was, gonna say, like, even if you're in this lab, and you have nothing no idea about Ndif. You probably have heard the word delta, and Michael complaining about it, or I complaining about it. So that's our partner who is at Ncsa. And they have a large Gpu cluster that they just built.
j.fiotto-kaufman: You know, we originally get this going to get this grant for about double the money, and the other half of the money would have went to hardware. But they said, You know, work with our partners who we just funded for 10 million dollars in hardware. Because, you know, you guys are software guys, we give you the money for hardware. So you started somewhere else.
j.fiotto-kaufman: local or something like where our machines were. Yeah, we had some machines here at Northeastern. They were actually pretty nice machines. They were like 8 by 8 100 s. Which could fit one of the models that we're trying to run. But if you want to expand right, we need a whole cluster.
j.fiotto-kaufman: And so you know, the primary model we're working with, right is 4 or 5 B, you know, there is research on 70 B and 8 B, but like 4 or 5 B is this huge model where there really isn't any research, and it's like kind of the standard. For, like the largest open source model.
j.fiotto-kaufman: and so we're running that with our pilot program right now, where we have people submit applications to do research on Llama 4 or 5 B. We selected about 10 of them to start, and they've kind of been our like guinea pigs of like seeing how they're using the system all the different, you know, problems that they had running things. And the different use cases, right? Some people want to do training. Some people want to just get do inference and kind of tailoring our future
j.fiotto-kaufman: service based on that and we're already looking to like. Add more people, I think now we're, you know, go up like 30 or 50.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So we're finally learning about like how our system behaves with some observability. Meaning that just like, you know, we can see the metrics of how the system is processing people's requests, how long requests are taking which models are being requested which is important, you know. But when we're just kind of looking to get off the ground really fast, you know, we didn't really have time to do that. But now that we're here that's kind of gonna guide us forward and how we change the system.
j.fiotto-kaufman: so and then, you know, we're going to be releasing insight 0 point 4 at the end of the month, and that kind of adds a lot of different features that people will be able to do some of the experiments that they want to do via the proposals. Yes. So that's the Python library that people use to interact with Nda. It's a custom Api format. You can't really interact with Ndi after over Http directly without using insight. Yep.
j.fiotto-kaufman: yeah. So I'm gonna visually kind of go through the entire architecture of the system.
j.fiotto-kaufman: I'm not sure what speed I should do this at, but we'll see as we go.
j.fiotto-kaufman: So I'm going to be building up to this figure by piece by piece. And so this is kind of how ndif works at, you know, a global level.
j.fiotto-kaufman: just like for the legend, real quick anything you see like in blue, that's like a single docker container. So it's kind of like one service. And all these services are different micro services that interact in order to fulfill people's requests.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So to start right? Like, I said. So insight is how people interact with ndif it
j.fiotto-kaufman: takes people's experiments and represents them as an intervention graph which I'll talk about in a second. And there are kind of 2 components to the ndif backend that processes people's requests.
j.fiotto-kaufman: There's the openstack kind of front end, which is all the services that don't have to do with executing people's requests on Gpus. So it's like, you know, for receiving requests, storing things in a database and then responding to you with updates. And that's on the Massachusetts screen high performance computing cluster, which is in Western Massachusetts.
j.fiotto-kaufman: And then the other half is on Delta AI, which is the cluster from Ncsa with all the Gpus.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So like, I said, before, insight takes experiment code, you want to do some experiment on a large model. You want to see how some part of the model is behaving, or you want to say, inject some of your own logic to like, influence it and see what changes that's represented by a computation graph which we call the intervention graph.
j.fiotto-kaufman: And so that's what insight generate. Generates, that is, representing the entire experiment to be sent to the server.
j.fiotto-kaufman: So I assume that those are
j.fiotto-kaufman: very different for each model, because the architecture of each model is potentially quite different. Yeah. So like, one thing will be just like the name, right? So like one model will have, like,
j.fiotto-kaufman: the name of like an Mlp. Layer be like Mlp. Or some will spell it completely. But most of the transform architecture actually pretty similar. And so, like some of the experiments you might want to do on them are actually very similar. You just want to get some values and then run a few functions on it and to do that over different models. All you have to do is like change like this string saying which part of the model you want to experiment on? Yeah, eventually, we're looking to expand to like very different types of models. But right now they're pretty all similar. Yeah.
j.fiotto-kaufman: so first, st we need to serialize the intervention graph, meaning. We just need to turn this into a format that we can send over the wire. So I don't know if you've used python or like pickling, but you can pretty much convert any object into bytes, and then send that over the wire to re generate it on the other end. But Pickle is really known for its insecurity. So like, yes, I was hoping you weren't going to say that I was not using
j.fiotto-kaufman: because you pretty much can't prevent it from running arbitrary code. So we have a custom serialization format where we define like a list of certain operators you can use and certain data types which, even with just limited set, we have like pretty much expand to like all the experiments we've seen so far, you know, tensors, lists.
j.fiotto-kaufman: dictionaries.
j.fiotto-kaufman: And then you know all the torch operations and all like the basic adding and subtracting operators.
j.fiotto-kaufman: And you didn't use a standard, Json or Xml, or yes, so it's it's a custom. Json format. Yes.
j.fiotto-kaufman: right? And then at the very end. So right? So this intervention graph represents all the things you want to do in a computation graph. And you know, as you're executing your interventions right? Each one of these nodes. These little circles represents like an operation and kind of the center of the nodes represents, like the value that results from their execution. And so in insight, you have an operation called saving, which is essentially saying, like after execution. I want these values back to me.
j.fiotto-kaufman: and so, when you save certain nodes right at the very end of execution, some nodes will still be filled in. But normally they'd be garbage collected because you don't need them anymore for memory. And so what happens at the very end? Right is, you take all those saved values and you inject it into your local version of the intervention graph. And so now you have, like a completed local intervention graph when the entire thing was actually executed on our servers.
j.fiotto-kaufman: Yep.
j.fiotto-kaufman: so the 1st part of the front end is the Api right? That's the gateway in actually interacting programmatically with the nda system. That uses Fastapi, which is a very common python library to do. Http servers.
j.fiotto-kaufman: and there are 4 endpoints of note here
j.fiotto-kaufman: to actually interact with Naf. So of course, the request one is where the actual serialized intervention graphs are sent to and that's where some other things happen like
j.fiotto-kaufman: validating your api keys and then routing the request to the back end, where it actually gets executed
j.fiotto-kaufman: a status endpoint which it tells you what's currently up. And you know that's what you hit when you go to the insight.net slash status to show you all the models and current configurations.
j.fiotto-kaufman: Stat, I'm sorry that status of the system as a whole, not of your particular request. Yes, yeah, yeah. Right now, there isn't a lot of observability for like users to see their own request as they're going through the system other than like specific updates that they get indirectly.
j.fiotto-kaufman: is a is the request
j.fiotto-kaufman: Apis that fire and forget you, don't you guys, you you get back a response saying, I got your request exactly, and yes, everything else happens after that. Yes, so yeah, yeah. It does not wait for like the whole thing to be done before returning to you. That, hey? We just got it, that's all. It's saying.
j.fiotto-kaufman: I'll skip that one. So yeah. So you know the request endpoint just tells you we got it right. And then so once you've known that your job is completed, you have to go back to the server and kind of ask for the response. And then that will return to you. All the values that you've saved, like I previously told you about, you know, streamed back to you as a pickled bite, because we don't really care about you being concerned with our
j.fiotto-kaufman: arbitrary code execution. But we're concerned with using desk. So we send you a pickled response of like anything that was saved during execution.
j.fiotto-kaufman: And so how do you actually receive updates about where your job is in. The process is, you have a Websocket connection with the server at all times. So that's kind of a live connection that you have on your local machine with the back end. And so if you use insight right, you'll get some updates saying we received your job. Your job's been approved, your job is running and your job is completed. And so that's all happening to the Websocket server.
j.fiotto-kaufman: yeah, can I ask about the result being pickled? Yes, if I remember correctly, which I may not. There have been some issues with like pickle format changing or pickle format being effectively different if it's a and big Indian versus little Indian machine. That sort of thing. Is that something you've well, definitely like. So if your python, even like minor versions, are different. Sometimes the pickle format will be different, and it won't work, so you will have to like inform, like the
j.fiotto-kaufman: pickled version of what we had on the server when we originally pickled it. Do you have any plan to
j.fiotto-kaufman: change to Json or something, so that it's stable? We could convert the same saved objects to our custom format right and send it. But that would like
j.fiotto-kaufman: again limit the things that we're sending back. So we're obviously we're limiting things you could send us. But ideally, we wouldn't limit things. People can retrieve from the actual execution of the model, because that might be like some object called like Gpt. 2 output, you know, which is some custom object that only exists for this one model.
j.fiotto-kaufman: and is definitely in some python format or something already. Yeah, right. It's just some python object. So if we wanted to add our format, we have to like, have a specific like, here's Gp to do output. Here's long output. Here's a bunch of like thousands of different things.
j.fiotto-kaufman: right?
j.fiotto-kaufman: Yep. And then so to actually communicate because we have a bunch of website servers the one that receives your connection has to be the one that responds to you, and so to do that kind of intercommunication. There's a message broker service which runs. Read this.
j.fiotto-kaufman: yep.
j.fiotto-kaufman: there's a layer in front of the Api. So before you actually request, or any you know, Http request gets sent to the server. It goes through Caddy, which is just this service that we use for a few reasons, just to reverse proxy your ndif dot dev, you know, URL, to send the request to to our local port on server, and then this also does rate limiting too, so you can only send like, say, 2 requests a minute to the server.
j.fiotto-kaufman: Oh, yeah.
j.fiotto-kaufman: And so they're actually a bunch of different copies of the Api running at all times. To like consume different requests, but everyone 12 but we'll probably wanna increase that as we get more users
j.fiotto-kaufman: so we have an object storage service. Which just runs minio, which is an open source
j.fiotto-kaufman: object storage application, which is runs the same protocol as like an Amazon s. 3 bucket. We just run it ourselves. And so this is where actually all you know, all the responses are, you know, ended up being stored, and that's what's retrieved when you hit the result endpoint.
j.fiotto-kaufman: Your Api keys are stored in Google's Firebase. And so that's what's used to validate your Api key on the request endpoint
j.fiotto-kaufman: and then so here is kind of our observability stacks of how we kind of know about how the system is working. We have some aggregator service, which is Prometheus, which is a very common
j.fiotto-kaufman: metrics aggregator that people use. And that's why we kind of use it because it's popular.
j.fiotto-kaufman: Which takes in logs from the Api and the back end to see how people's requests take during the system. Like, how long they take at a certain stage or how much Gpu memory your request is taking that gets sent to the aggregator, and then we have a dashboard, which is Grafana, another popular one. And so we can go right now to Grafana, the end of Thatus, and we can see the different requests coming through the system and their different properties. And maybe what different bottlenecks in the system are taking too long.
j.fiotto-kaufman: Okay? So yeah, the back end. The back end uses this Python library, called Ray and Ray is a library to kind of build micro service architectures.
j.fiotto-kaufman: that are kind of auto scaled to your needs, meaning that you get to define a piece of code you get to define what type of resources that code needs. Say, like this needs to 3 gpus and 2 cpus
j.fiotto-kaufman: and then you run a ray node application on all of your different machines. So I have a machine over here which has no gpus and I have another machine over here which has 4. I say to Ray, I want to run my application one, and it knows it takes 3 Gpus and 3 gpus are free, so it knows what machine to put it on. And so that's kind of ray kind of handles. All this auto scaling for us. Which is nice, you know. It's pretty with kubernetes, if you've used that.
j.fiotto-kaufman: is it built? Is that what Delta is providing? Or is that a separate thing? That's Delta's not providing us. We're using it. And then Delta's providing us the cluster which we run all the rate nodes on. Okay? Yes.
j.fiotto-kaufman: So each of these light blue boxes are these different deployments and deployments are exactly the thing I just said where they're pieces of software. We want to run somewhere on the cluster.
j.fiotto-kaufman: And so the only one we specified array we want to run is the controller, and the Controller's job is essentially to build all the other deployments that we want with this very specific type of configuration. So on the right is a very simple configuration of what like, you might want to run the entire service on which is just saying we would like to run llama 70 billion. And that's it. Slama, 70 billion requires one gpu, and we want one copy of it for the number of replicas.
j.fiotto-kaufman: And so the controller ingests that and uses ray to auto serve that to any part of the system that can handle a 1 Gpu model
j.fiotto-kaufman: and one of the other services that it creates is the request service.
j.fiotto-kaufman: Which is lightweight, requires no Gpus, and that's the actual service that ingests the request to then route it to the correct model.
j.fiotto-kaufman: So right now it's really all this this service is doing. It's just taking it. Your request, looking at the model you're asking for, and then sending it to the right deployment running the model. But in the future we imagine that. It'll do some more analysis on like the priority of your request. So like, you know, you're a special user who, you know, some thousands of requests a day.
j.fiotto-kaufman: Maybe you get like, put back in the queue. Somebody else can run their request, or your request is really big. It might take a few hours, maybe we should let in some other requests that take just a few minutes first.st
j.fiotto-kaufman: yeah, so that might happen there in the future.
j.fiotto-kaufman: And then, of course, the model deployment itself. And that's the thing that actually loads and runs the Pytorch model.
j.fiotto-kaufman: So in this case, 70 B
j.fiotto-kaufman: on initialization, it loads the model.
j.fiotto-kaufman: and then it, just like the request service right? Maintains a Websocket connection to the server, which then maintains a connection to your local clients. So that's why like when you, when a request makes it to the model, and it knows that started running, or it's been completed. Right? It sends that response over the Websocket connection with the server to then tell you the user. Hey? Here's where you are during.
j.fiotto-kaufman: your job is processing.
j.fiotto-kaufman: Yeah. Question in the example you gave for like, how Ray does all this distributed work like
j.fiotto-kaufman: you said it like, Oh, this model needs, you know, one gpu or 3 gpus
j.fiotto-kaufman: But is that? Is that? Does that distribution happen on the basis of just like an estimate of how much vram it's actually going to use? Or is it like a like a hard coded like a llama 70. B is typically, so, yeah, so I mean, we. So we hard coded essentially in that service config file is that we're saying with one Gpu I. You can also specify custom resources, and I have it. So it also. Each node tells you how much vram it has. So you could also do it based on the vram
j.fiotto-kaufman: but in our case, right? All the gpus are the same size. So we're just doing like, oh, we already know this, Gpu, because the case, the thing I'm imagining is like, well, what if somebody's experiment like requires a lot of gradients which would be like would explode the amount that yeah, so right now, we kind of just use like a 15%,
j.fiotto-kaufman: just heuristic of saying, like, we would like 15% of
j.fiotto-kaufman: the Gpu to be free for people's requests. But I guess probably in the future there might be like maybe one. They do 2 deployments of the same model, one with more vram, free one with less, and depending on your request. Right? It sends it to the right one. Seems a little tricky to tell. But I guess you, as long as there's enough padding, it would. Right? There's a whole like, probably multi year work stream of like analyzing people's intervention graphs before they come to the server, and determining some like heuristics based on that about to do. Yeah.
j.fiotto-kaufman: So once the your request and the intervention graph makes it to the model deployment. Right? That's we finally deserialize it and turn it back into the intervention graph. And then we just use insight like you would locally on your local machine to just execute the model with the intervention graph.
j.fiotto-kaufman: Yeah. And then, like, I said before, after executing your computation graph, the intervention graph, some of the nodes will still be filled in, which means you must have saved them, because otherwise it would have been garbage collected, and it is those values that have been pickled and then sent to the object store. And then at that point, right? A final websocket
j.fiotto-kaufman: response is sent to you, saying, Hey, your job's been completed. Please come back to the server with your request, Id, to get the results, and that's how it then gets injected locally.
j.fiotto-kaufman: Yep, and like, I was saying, there can be multiple replicas of the same model. And those are all round, Robin, consuming from a queue where they all process requests one at a time.
j.fiotto-kaufman: Yep. And so, you know, this is a cluster of nodes, and of course we're not just running one machine the current configuration uses 5 different machines, each running for Grace Hopper, gpus.
j.fiotto-kaufman: and so we can run a ray note on each one of those, and then so Ray obviously knows to serve our models to the correct
j.fiotto-kaufman: node, which has enough Gpu capacity.
j.fiotto-kaufman: Yep.
j.fiotto-kaufman: So let's talk about a different type of model, a distributed model. And so, you know, we have some pretty big machines here, right for 90 GB each Grace Hopper gpus. On each of these nodes, but even that isn't enough to run 4 or 5 B on half precision. So we'll have to run this model on multiple machines at a time, and so to do that we're going to distribute parts of the model onto each gpu and then communicate, but onto each node and then communicate between them in order to process people's forward passes.
j.fiotto-kaufman: What does half precision mean in this case? So representing the parameters of the model, not as float. 64,
j.fiotto-kaufman: 1432. But it's 1216 60, yes, right?
j.fiotto-kaufman: So on. So just like with the other model deployments.
j.fiotto-kaufman: specifying that you want a distributed model starts up a distributed model deployment, and then its job is to kind of do all the other work, to build its kind of own like distributed subcluster. So it is the thing that spawns the other shards, also running a piece of the model. So yeah, so shard 0 creates the other. Shards
j.fiotto-kaufman: depending on how many machines we want to split it over. So that is again like a hard coded value. So we say, like we want to split this model over 16 different gpus. We don't really care.
j.fiotto-kaufman: You know how many different machines they're on, or how many different nodes. We just care like 16 Gpu. Somewhere.
j.fiotto-kaufman: But the controller or Ray had to know in order to
j.fiotto-kaufman: to find available free machines right? Right? So like to even like start up this model. 0 model shard 0 deployment. We had to have 16 gpus free somewhere in the system. Yep.
j.fiotto-kaufman: right?
j.fiotto-kaufman: Right? So after
j.fiotto-kaufman: the shard 0 comes up, sees that it needs 15 other deployments, tells 15 deployments to go start up somewhere in the system. Each one of those deployments is gonna load a single 1 16th slice of the entire model. So that means like at every single matrix, right? So Mlps and attention. Only 1 16th of the rows and or columns will actually be filled in
j.fiotto-kaufman: and then so it's that Char's responsibility for executing that part of the matrix
j.fiotto-kaufman: and then so to actually do all these distributed communications which I'll talk about right? There's nickel, which is a protocol for doing communications between Gpus. distributedly
j.fiotto-kaufman: that runs on the model shard 0. And then all the different shards are communicating over this nickel protocol to do the processing and so, you know, there's like tons of these distributed operations that have to happen right? So like, if you've broken up one of your Q or K, or v matrices into 16 chunks
j.fiotto-kaufman: before and after every single time you process 1 16th of these right? You need the values of the entire tensor to even go to the next layer or the next matrix. So after processing just the Qk or V, you need to actually, then go back to all the different machines, collect the full tensors and then bring them back.
j.fiotto-kaufman: Which is, you know, a pretty costly operation. But Gp, memory wise, right?
j.fiotto-kaufman: just, you know. One person's forward pass, like hidden states, is not nearly as big as like all of the parameters in the system.
j.fiotto-kaufman: Is there any special networking hardware associated with that to? So it's not on the same network as all the other traffic or no? No, yeah. At some point we need to work with Delta to figure out their whole like setup, and like
j.fiotto-kaufman: cause, I did kind of build their setup as like being for distributed training. So
j.fiotto-kaufman: I'd like to think that they have some special way of doing things. But we don't know. We don't know yet.
j.fiotto-kaufman: I'm recalling Vidya having
j.fiotto-kaufman: something special hardware for networking. I don't remember what it's called yeah. So so these gpus on the cluster are special, and that they're Grace Hopper Gpus, which means a couple of things, is their CPU and Gpu memory are in the same Numa domain, and that has complications for reasons. I'll talk about later. And then also, apparently, it's supposed to be more of a direct connection from the Gpus to the network cards. So there's less transfer of data. But
j.fiotto-kaufman: so far it seems things are slower. So I don't know. Yeah.
j.fiotto-kaufman: yeah, the Gpu's have their own network cards. Yeah, right?
j.fiotto-kaufman: And so, yeah, shard 0. That does all the other stuff right? It's the thing that sends the data to the results. Object store communicates with you about the different statuses. The other shards don't do that.
j.fiotto-kaufman: and then, of course, shards can be on other workers. So you can have multiple nodes talking to each other over multiple GPS processing events.
j.fiotto-kaufman: Yeah. So now that like, that was a lot of pieces, right? So nice pause for questions. Ask questions.
j.fiotto-kaufman: I've asked questions.
j.fiotto-kaufman: Do you have several instances of mama? 4 or 5 d. Or like, why is there these little? You could have several licenses of it? So anything like with those little like, you know.
j.fiotto-kaufman: copies of
j.fiotto-kaufman: the part of the figure is like you could in theory tyrolyze it over multiple things. Yes.
j.fiotto-kaufman: and and what was the deal between having multiple worker groups for one big distributed model?
j.fiotto-kaufman: Like you have. So you have some shards in one worker, and I guess you could have other shards in another worker. Exactly. Okay, yeah.
j.fiotto-kaufman: And was, was Ray the one that's doing like implicitly. It's got a job queue or something like that. It's yes. Yeah. So when you like, you submit something to a deployment via Ray, it's putting on a queue. And then all the replicas of a deployment are consuming from my queue.
j.fiotto-kaufman: So is there a reason that, like the llama, 70 B. Isn't in its own worker.
j.fiotto-kaufman: the head does some special stuff right? The controller requests. But below that, right? It's kind of just another worker, so it itself can be run on a machine with Gpus, and so it itself can serve models from it. So the only difference between the head and the worker is this, the controller? Exactly. The extra deployments. Yeah, Controller requests. And so, worker here and head just mean a machine that's available in your cluster. Yes, right?
j.fiotto-kaufman: That managed by Ray. Yeah. And so just one of them happens to be the head. I guess a couple extra terms on it.
j.fiotto-kaufman: Do you need to think about patching or anything like that when you build. So we are thinking about it and the reasons why? Obviously like, because you can batch requests. Then you have to batch intervention graphs. Right. You have to say, like, you're interviewing this list of the thing. Which actually, we have working with. Vlm, oh, yeah, like, I said to you, so if we hosted the Vlm version of the model, then we could start thinking about actually how to do batching.
j.fiotto-kaufman: yeah. But in theory, so what one other thing that like for non vlm models, that kind of confuses the batching stuff is vlm, vlm.
j.fiotto-kaufman: It's a popular open source library to do. Inference on language models, and they've kind of done all the state of the art nice to have stuff for executing things really fast and batching multiple people's requests together.
j.fiotto-kaufman: then it's not anymore.
j.fiotto-kaufman: No. So they they use force distributed, which is using nickel. Yeah, yeah.
j.fiotto-kaufman: is there, do they? How? What's the relationship between Ray and the other end.
j.fiotto-kaufman: So Vlm itself uses Ray to distribute different processes onto the different gpus on your machine.
j.fiotto-kaufman: I see. So that so that somehow is like the same as the way that we use. Ray, is that right? Yeah, they're not using ray serve. Which is this, you know, like for deploying things on multiple nodes. But like, even if you ran Vlm on one machine with 3 gpus, right, it uses Ray to like spawn 3 processes which then each communicate with each other over at nickel.
j.fiotto-kaufman: Yes, ray, like kind of organized by the same person. I think it's the same like that. That's so. Yeah. So so I'm just trying to understand, like we're using Ray. And they use ray.
j.fiotto-kaufman: And is that like, I don't know anything about how to set up ray. Is that like the same ray deployment, or is it like a nested deployment? Right? So so there's the Ray Library which does this kind of like
j.fiotto-kaufman: making processes communicating between the 2. And then there's another land type of race serve. And that's more for cluster, wide stuff. That's what we're we're using to like, actually have deployments. Okay? And for auto scaling and serving and stuff like that, I see. But the Vlm stuff uses base right. Just the communications. Yes, I see
j.fiotto-kaufman: but like I was saying before, you know, one thing that complicates batching on normal requests, right? Is like, what if you ask for a different like, say.
j.fiotto-kaufman: temperature parameters right for your execution, or you want a different way of tokenization or something? It's like hard to batch those. So I guess, for non vlm requests for batch, I'm thinking, just like we'll have one set of like. Here are the default parameters. And if you're using these, you can get batched. If you want to do a different one, then you can't get batched, you know.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So like, even for this super giant model.
j.fiotto-kaufman: you have a big benefit from batching.
j.fiotto-kaufman: Yeah, ideally, you don't fully utilize the Gpu's
j.fiotto-kaufman: by just forward passing a single sequence
j.fiotto-kaufman: you sound like. If we were to batch, then we'd be using the GPS more.
j.fiotto-kaufman: He's asking if that's true. Yes, I do, though, that we don't have enough users right now for that to be practical right. The probability of 2 users hitting at the same time right now is pretty low, and then the probability of the thing being batchable is probably also low. So, and the wait time to wait for people's like requests to be batched together versus the wait. Time. They would, you know, incur by not batching is probably way less. How long is a typical request? How long does it typically take to process them or
j.fiotto-kaufman: the range. Are we talking minutes to days? Or, yeah, it depends like. So like one forward pass maybe takes like
j.fiotto-kaufman: 5 seconds. But like some people's requests will have like a loop wherever they came up before it passes.
j.fiotto-kaufman: And so, depending on that. Yeah, what if I do? If I have it. I usually have data sets right?
j.fiotto-kaufman: And I do kind of the same interventions on each data point, do you batch those?
j.fiotto-kaufman: Yeah. Yeah. So so just like, you know, you're defining batching things in your local script right? Instead, accepts the same kind of like, okay, framework. Yeah, yeah. Okay.
j.fiotto-kaufman: That's why we have a max timeout of about an hour on the models now, because, like we were looking at, you know, the logs and somebody's doing like a data set of like thousands of things looping over and over and over again, like what is going on. They're saying, multiple requests each one. Yeah, like, you can even see the the model was not listed anymore, because the thread to like check the status of it was blocked right? Right? That's great. Yeah. It was for hours and hours and hours and hours. It's crazy. Yeah.
j.fiotto-kaufman: these Phd students, when they want to get something done, they they have shoot.
j.fiotto-kaufman: Let me ask again. No mercy on your system.
j.fiotto-kaufman: Okay.
j.fiotto-kaufman: Somebody mining bitcoins. Oh, no! Oh, no. I want somebody to try. And then if they can, I can change things.
j.fiotto-kaufman: Yes, okay.
j.fiotto-kaufman: so I have an animation showing the life cycle of request that goes to the system. So you guys can further understand how things work. Right.
j.fiotto-kaufman: So on the left right, you have an idea represented by insight code, which is experiment, which you can convert into an insight intervention graph.
j.fiotto-kaufman: right that gets serialized.
j.fiotto-kaufman: turns into this request object which takes your key. Yeah, which model you're using right? And then the session id, which is like how to respond to you over Websocket
j.fiotto-kaufman: gets into the server.
j.fiotto-kaufman: That is invalid data that you have that Api key via firebase.
j.fiotto-kaufman: It becomes a backend request.
j.fiotto-kaufman: Then we tell you, hey, we've received your requests
j.fiotto-kaufman: that gets sent to the right back end via the request deployment
j.fiotto-kaufman: upon seeing that your model is actually one of the models we're hosting. We'll then tell you. Hey? Your job been approved?
j.fiotto-kaufman: Now, let's say you requested Lama 4, 5 B. Well, then, the request object will be sent again via Ray to the worker, node and deployment that is running 4 5 b.
j.fiotto-kaufman: We will tell you that your job is running.
j.fiotto-kaufman: So let's say the the user has to close their workstation. You know their laptop or lose power. Can they re establish connection to all it did they have to have saved
j.fiotto-kaufman: session? Id somehow, etc, so you can run your insight request in a non blocking mode. Which just means send the request, and then immediately stop, and then the next time you start your script right, it'll check a local save job Id to find out the current status. Okay? And then, if it's completed, then it knows to go to the server and grab the response. Yeah.
j.fiotto-kaufman: But so if you're running in the default mode. We assume that you'll be like waiting for your requests.
j.fiotto-kaufman: But if you're one of these Phd students who've sent thousands of requests? Not so much exactly.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: wait. Is there a time limit of how long these responses are saved, or something? Because you said like, we can finally get the second right now. No, I mean they're deleted the second. You retrieve them. So if somebody retrieve them, they would say that forever. Yeah, we do need, you know, some cron jobs that I can check in once in a while.
j.fiotto-kaufman: Yep, shard 0, then
j.fiotto-kaufman: duplicates the request. Send it to all the shards, you know. So like serial deserialization happens on all the different shards. Not just on shard 0.
j.fiotto-kaufman: We deserialize your request back to an intervention graph.
j.fiotto-kaufman: Then we start executing the intervention graph with the model. So that'll do. The nickel communication between the different parameters, the models execute, and that not only that is, there's a layer on top of insight for distributed models. Right? So if you say, get the output of a query,
j.fiotto-kaufman: block in your model, and that happens to be a distributed tensor. There's a layer in between that and the intervention graph on the distributed model that says, I'm going to check for any distributed tensors and make sure I actually recollect them into the full tensor. So anything that enters the intervention graph is made sure to be a full tensor. So we don't have to worry about, like, you know, only intervening on certain slices or something. It, you know, brings the complexity way down. Does does that mean that for those things where you need to merge the intervention graphs just being executed on Shard 0.
j.fiotto-kaufman: Or do you execute on all shards? Okay, so you. So you merge everything on all shards, and it's redundantly done. Yes, yeah, I mean, well, most of the time it isn't redundantly done right, because all shards will need the full tensor.
j.fiotto-kaufman: right.
j.fiotto-kaufman: I see. But if somebody but if somebody like writes, I'm I'm just trying to think of something weird, like. If somebody like writes a random number.
j.fiotto-kaufman: or like, writes the current time, or something like that, into into the tensor.
j.fiotto-kaufman: then there might be inconsistency with the different charge. You might get different random numbers. If you're doing that, then yes.
j.fiotto-kaufman: okay, just there's no random numbers. Okay? So so it's a little bit of a leaky abstraction, just something to be aware of.
j.fiotto-kaufman: Yeah, I don't even know.
j.fiotto-kaufman: How do you solve for that?
j.fiotto-kaufman: Well, if you were to do the work. Only on the charge 0,
j.fiotto-kaufman: could could you do the work only on charge 0? Is there any reason you couldn't given that? Everything's being brought together there. Well, so like operations need to happen on all the different machines right back out. But like, yeah. But distributing, how do we know? Distribute? Like, let's say you're just using like the random
j.fiotto-kaufman: library in Python. Right? How do we know like that needs to be distributed to all the different machines?
j.fiotto-kaufman: I think I'm I'm I'm missing something that because it sounds like at at each.
j.fiotto-kaufman: You know, everything that gets distributed then has to be brought back, you know, if it's been sharded.
j.fiotto-kaufman: then you have to. Every other shard will eventually need to know the output values
j.fiotto-kaufman: from, you know, shard one and so, and that all goes through chart 0.
j.fiotto-kaufman: So that sounds like you could do your interventions.
j.fiotto-kaufman: All, you know, after everything's being calculated, you know. Find out, you know, record what the the output value was from all the even number charts.
j.fiotto-kaufman: Well, that could all happen on chart 0 rather than
j.fiotto-kaufman: spread out, just so am I missing. So what am I missing?
j.fiotto-kaufman: Oh, so like you're saying like the the output values themselves. But like, let's say you have like an in place operation that you do on a tensor.
j.fiotto-kaufman: That would have to happen all the different charge right? Because there's no way for us, I guess. Like, if you were to cut out a node, or I mean a weight, or you know, yeah, or like an index and array and change it to something else. Okay, got it. So the so interventions as opposed to recording so interventions have to happen where you know, distributed right? So we could differentiate between, like, you know, in place operations which we might want to do anyway, for other reasons, and then only
j.fiotto-kaufman: do work on all the machines. If we think that you're doing an in place operation that affects like sensors. Yeah, okay.
j.fiotto-kaufman: got it. Thank you.
j.fiotto-kaufman: yep. And then, after execution. Right. You have a partially filled intervention, graph where a few of the nodes have been saved, represented by the green circles
j.fiotto-kaufman: which is then sent to the object storage.
j.fiotto-kaufman: Then you are informed that your job is completed.
j.fiotto-kaufman: and then you go back to the server and say, Please give me the response or the result.
j.fiotto-kaufman: And then, now that results in a completed intervention. Graph on your local machine, as if you were to execute it locally yourself.
j.fiotto-kaufman: and before I move on any questions about
j.fiotto-kaufman: yeah, I'd be inclined to have at least a
j.fiotto-kaufman: some amount of grace period between reading the result and it being deleted because you don't know whether the
j.fiotto-kaufman: just because you wrote the bytes to the network doesn't mean they made it to the other end for sure.
j.fiotto-kaufman: Yeah. And so
j.fiotto-kaufman: so much has been learned over the past year, right? Just going from me, working on this to a team of now like Michael and Adam and then all the kind of the pis professors coming in and starting to do more things so much has changed.
j.fiotto-kaufman: one of the biggest things that I guess I didn't expect, although I didn't really have any preconceptions about how long it would take to move our system onto Delta. I kind of just thought, oh, it's more Gpu machines, right? We'll just put it on there, and everything will be fine
j.fiotto-kaufman: but it's been quite difficult to have to like go through another entity and not have, you know, pseudo access to all their machines. And
j.fiotto-kaufman: right? So kind of it's a people problem really to like work with them, to communicate what's important to us and kind of the severity of we need this done now or
j.fiotto-kaufman: right?
j.fiotto-kaufman: And then, like, I was saying, with the the new technology so the Grace hyper machines have had quite
j.fiotto-kaufman: many problems in getting our systems around them. So one problem is right because they share a new domain. Just like you would your machine normally caches things. Let's say, if you just copy a file from one spot to another. Your machine would normally cache that well on Grace Hopper machines because it thinks the Gpu memory is part of the CPU memory. It's just going to use Gpu memory to copy something. So you'll copy a file and you realize, well, your vram is totally filled up.
j.fiotto-kaufman: and so to get around that we have to kind of do a bunch of like hacky stuff in terms of like before we copy anything, we have to run a program to just make a big tensor fill up the gpus. Don't let you use it for caching.
j.fiotto-kaufman: Is that really a a Linux shortcoming? I assume this is running on Linux. That it doesn't have.
j.fiotto-kaufman: It doesn't have the knowledge baked into it that
j.fiotto-kaufman: this memory space that looks like it might be, you know, uniform is is really not, and therefore, you know, some portion should be, you know.
j.fiotto-kaufman: used for for the I can't remember what they call the but
j.fiotto-kaufman: the page cache portion. But it sees it all as one thing
j.fiotto-kaufman: and you know, at least from Delta's perspective. They're saying like, Oh, well, that's just kind of how it is. And there's not really a way around that. Yeah. So
j.fiotto-kaufman: that that is to say, they don't care about the problem. It's not hurting them. Yeah. Well, well, and then they'll say, like, Oh, you know, Nvidia had these type of problems for the past 2 decades, you know 2 decades. But then, you know, they're the ones who invested like how many millions of dollars in Nvidia chips. So I don't know.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: another thing that's really important is like having a consistent environment. So like, you know, to really care about observability or security, or to like get a good process of how to deploy your system. Those are all kind of hard things to do when your system's constantly changing and composed of like a bunch of different types of deployments. So you know, before we were on Delta, right, we were on some of our machines. We were on some of the Nagoya machine. We were on some of the machines at Mgpcc. We were on some of our machines in his lab, all distributed
j.fiotto-kaufman: is like one cluster with like different types of operating systems, and different, like other people using the machines at the same time, for, like their own experiments.
j.fiotto-kaufman: and then introduce, you know tons of different weird errors which I hope would go away on Delta. And it seems like we have. So that's good. But yeah, I guess the the main takeaway here is that like
j.fiotto-kaufman: to really get like the good process stuff going in terms of like having a consistent deployment and understanding how your system behaves. You need a consistent environment over multiple months ideally to understand that.
j.fiotto-kaufman: One thing that's been annoying is maintaining multiple versions of insight with Ndif. Right? So, you know, for all Ndif releases if they coincide, you know, with a similar time, with an insight release. We kind of have to have 2 versions of both. So the Ndif code here and then the Insight library. So right now, right there's an 0 point 4 branch of Ndf and an 0 point 4 branch of insight. And they both have, like kind of different ways, the same type of update needs to happen.
j.fiotto-kaufman: So this is where I would definitely recommend not using pixels, because, in fact.
j.fiotto-kaufman: you know that my experience is, it's really useful to be able to support multiple versions. Because, you know you're doing, you want to do incremental rollouts, for example, to make sure that things, you know are not going bad to the inside version might change and change how it's serialized. And that consistent is probably important. Right?
j.fiotto-kaufman: Another thing is just kind of balancing between going fast and then stopping and then thinking about what we're doing now. You know, we just
j.fiotto-kaufman: had so many points in the past year where we just had to get something out right. We needed to get 4 or 5 B. We needed to get on Delta. We needed to get the pilot program users
j.fiotto-kaufman: situated. Which you know, doesn't really give you a chance to kind of say, like, okay, is our system architected correctly, you know, should we be focusing on observability stuff? Right now I think we can answer that the answer is always no, first, st because it was based on what you thought at the time. And now you have new information. Exactly. You always have regrets. Yeah. So you have to give yourself a break on that. Yes. Oh, for sure. It's yeah. It's
j.fiotto-kaufman: having having spent like 3 years. The last 3 years in that environment, like is the only way to do things honestly is just to like, do something. Your project manager sweet in your ear, saying how like everything should be a thousand Github tickets, and you know, told exactly how it's gonna take for the next 4 years. I feel like with these things especially you can't anticipate how you can try to anticipate how people are gonna need it. But it's always wrong. So
j.fiotto-kaufman: Another important thing just looking at from managerial perspective of like learning how to be a manager, right? Is like, it's really important to find. Like, if you're
j.fiotto-kaufman: bring somebody new to the team to like. Say, like, Oh, here's your part of the project that you're responsible for. Here are the exact kind of like success metrics that you can like look toward. Because, you know, without that people can get lost. And
j.fiotto-kaufman: that's never good.
j.fiotto-kaufman: so yeah, doing, going going forward, doing? That is probably quite the priority. No, I'd say it. It very much depends upon the employee that you're bringing on, because
j.fiotto-kaufman: and people can be vastly different in whether they need that guidance
j.fiotto-kaufman: and yet be both be very successful contributors to a team. So somebody who
j.fiotto-kaufman: doesn't feel comfortable without a very concrete charter, could still be a brilliant member of your team, as well as the person who. You just point them in the vague direction, and they will succeed. Yes, you know I'm the vague direction type of Guy, but I guess it's good to have a concrete charter ready to go, you know, regardless of like, if you are or not, you know, I mean so it certainly helps with a new person to have a starter project. Yes, that gets their feet wet. And then maybe a starter project. After that, you know. Yeah, because, let's say, if you're on a new project.
j.fiotto-kaufman: like all the different things you have to think about and like kind of the explosion of different problems that you think you maybe need to solve when you actually just really need to like, get an Api key into the dashboard, you know.
j.fiotto-kaufman: yeah.
j.fiotto-kaufman: Yeah. Telemetry. Yeah. Telemetry. We need more telemetry. I guess it would have been nice
j.fiotto-kaufman: to have that earlier to debug some of the things in the system. But, like we were saying before, it's kind of hard to focus on that when you actually just need to get the thing working so people can use it for experiments as fast as possible. I would say most of the postmortems I was involved with at Google involved. Add new telemetry. Yeah. So
j.fiotto-kaufman: there's always something. Oh, I wish we'd known that. What was was any of the postmortems like, Oh, we wish we had like delayed development a little bit to work on telemetry, or is like we wish we could have done both. Well, yeah, I mean, that's I was. Gonna ask you, you know what your bigger challenges are which you're not showing us. And I was. But I was gonna put the caveat, besides needing more time, because you always need more time.
j.fiotto-kaufman: yeah,
j.fiotto-kaufman: We always wish we had had more time for features and for making it run better. Yeah, yeah, I can't think of a time when that wasn't the case. Yeah.
j.fiotto-kaufman: especially right now, we're like we're trying to get our foot in the door right, you know, like, make ndif necessary as fast as possible. Yeah. You want people to be complaining
j.fiotto-kaufman: that that because they need the feature. Yeah, right? So do you have people complaining? Oh, yeah, I mean, where's the kill?
j.fiotto-kaufman: Yes, for sure. But you don't want them to say I didn't get my Phd. Because.
j.fiotto-kaufman: you know you don't. They couldn't get there. We wanted to push for this conference to get the 4th heavy results. But, like, you know, it was down over the executive.
j.fiotto-kaufman: That's the biggest thing.
j.fiotto-kaufman: Yep.
j.fiotto-kaufman: so, looking forward you know, where is Ndif? What would be doing for the next like, probably like 6 months.
j.fiotto-kaufman: Michael's been working on like having a consistent appointment set up. So you know, it's not kind of black magic on how to deploy the system. Really doing good, like site reliability, engineer practices
j.fiotto-kaufman: to have like kind of a a turnkey system where, you know I, or Adam, or even you know David, who maybe to know how the system architected, could just go on to the system. If it's down and press a button and restart it.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: metrics collection, like I'm talking about. And there's kind of 3 reasons why we need a lot of metrics. Probably the primary reason is for the Nsf. So we need to go back to the Nsf. And say, because right now we're running on Delta right? But we only have so many gpu hours that's been allocated to us. And that was kind of a very rough estimate of like we want to run the system over the next 3 months.
j.fiotto-kaufman: Here's some numbers, but next time we go back and ask for hours, we want it to be a kind of more principled and like, well, people are using these models this often. So if you want to continue that service and expand
j.fiotto-kaufman: actually over the next 6 months.
j.fiotto-kaufman: Here's what we need. And they expect a more principled approach next time. Yes.
j.fiotto-kaufman: and then, obviously, internally. So we understand, like how to change the system, and you know what's going on where the bottlenecks and performance. And then, like, I was saying before, like for users, right like it'd be nice if they could go to a dashboard, see their jobs, see all the results that they have. That would definitely make them feel more comfortable about like sending their experiment off, and you know who knows where it's going or what's happening to it.
j.fiotto-kaufman: Yep, and then alerting, you know.
j.fiotto-kaufman: it'd be nice if we got a notification to our phones, saying like systems down, please restart it instead of kind of waiting for somebody on discord to go. Hey? Please restart the system. Yeah.
j.fiotto-kaufman: we're gonna be like building a test harness to test all these things, to see load balancing, how the system behaves with different types of insight requests for like looping, or, you know, doing, streaming back and forth, which is a new feature to insight.
j.fiotto-kaufman: and Adam's gonna be kind of taking point on that for the next 6 months.
j.fiotto-kaufman: Then, once we know metrics and we could test the system. Well, then, we can tune the performance, because, you know, people have been reporting that the new delta version of the system is slower. We have no idea why could it be the Gpu communication? Could it be the Http communication? We'll have to find out.
j.fiotto-kaufman: Do you have a way to deploy your software like on local hardware, the same as if it were on Delta effectively the same right? Because, you know, on Delta, there's like to communicate between the front end and the back end. We have to set up some like H. Ssh forwarding, which it doesn't exist on a local machine, and that might be a big source of slowdown.
j.fiotto-kaufman: but in terms of the base system itself. Yes, we have a way. Yeah, there's a whole repository that I'm building for doing that. So you can basically use one repository to deploy in any scenario. Essentially very good.
j.fiotto-kaufman: And and then so like one of the ways we've shifted in looking at in Ndif is kind of building it around these specific applications. So you know.
j.fiotto-kaufman: once Ndif exists and is usable, right? We want kind of higher level abstractions on top of that for people to use in the classroom. Right? So if you want to learn about a model, we want the best way to do that to be. You load up this, you know, logic, lens application, which is some web page which shows you for if you don't know what logic lens is.
j.fiotto-kaufman: it kind of shows you as a language model is processing your prompt what it's thinking about along the way at the different parts of its brain. Essentially
j.fiotto-kaufman: and that's a really good like, first, st heuristic on understanding, maybe like what it's thinking about.
j.fiotto-kaufman: Yeah.
j.fiotto-kaufman: So getting a lot of those applications up and running and then having them be useful in the classroom is really important at proving ndi. If it's useful.
j.fiotto-kaufman: and then we want to standardize. Ndif is like a docker container, so like, if you yourself wanted to run your own version of Ndif, you could do that, you know fairly easily on different types of systems, on like aws, or wherever you'd like to run. Ndif, it's like 3 different startups. Now, right?
j.fiotto-kaufman: You're using. They are making different variations of right ndf. Sort of on their own. Yes, and then one of them has like express sentiment that they want to help and contribute back. So that would be nice to like help. Other people just like add pull requests and stuff. Locally. They have to legally obligated to
j.fiotto-kaufman: either. The State of California has told them they have to do it, or or or their or their customers are asking them, why don't we would like? We would like warning before Skynet starts, you know.
j.fiotto-kaufman: And then just some open questions about like things. I, you know, are pretty far in the future. But I don't really know how to solve right? So the race Hopper stuff.
j.fiotto-kaufman: It's kind of shocking how little
j.fiotto-kaufman: Delta knows about the problem, how other people are using Grace Hopper. Gpus, know about the problem? I guess the best way how much Nvidia selling these things right? How much or just one chip, you know, like they just keep on every every Nvidia event, they say, by Grace Hopper, they were just at
j.fiotto-kaufman: They were just advertising this new $3,000 machine at. Yeah, that's another great software with this, with this unified memory architecture business. So as soon as you like, load a file, you're not going to be able to use the Gpu anymore. Right?
j.fiotto-kaufman: Is the context around there, like conversations around Grace Hopper. Specific, if you're like, oh, if you're doing like training, or is it like, just like you should be using Grace Hopper for any machine learning application? I guess they just want everybody to use Grace Hopper because they capture more of the
j.fiotto-kaufman: margin on base opera machines because you don't have to buy an intel chip. It's all in video all the way through.
j.fiotto-kaufman: Yeah, it was. Yeah. It's Nvidia's arm, right? Right?
j.fiotto-kaufman: Yeah. Like, I was saying before, like evaluating the the performance bottlenecks. Right? So we don't know
j.fiotto-kaufman: you know how the system's working? What parts of the system do we actually not know anything about in terms of how to run things so like distributed is a pretty new library. The torches developed. I imagine that. You know there's a thousand different environment variables and stuff like the things to tune, and things like that.
j.fiotto-kaufman: Like, I was saying before the safety of the inside Api. So right, we specify these only. You can use only these objects on these operators. And in theory, right? You can only do things like look at hidden states and interact with them some way. But maybe you can build a Bitcoin mining with the inside Api. I don't know so probably maybe at some point getting like a fellow to like red team. I hope not.
j.fiotto-kaufman: Yep.
j.fiotto-kaufman: When you can run language models
j.fiotto-kaufman: right, and language models are turned completely. If a language model can be Bitcoin mining, and then that's such a waste of
j.fiotto-kaufman: maybe maybe they will come up with a more efficient way to produce the next Bitcoin. Oh, I see. Then you'll be in trouble, or like the what was it
j.fiotto-kaufman: open AI system for
j.fiotto-kaufman: playing chess, which just changed the internal state to say, it won. All right. Yeah. Just change one state to like find the right Bitcoin like, yeah. Well, just to tell everybody you know, update your, you know, distributed ledger to say I invent. You know I have all the bitcoins.
j.fiotto-kaufman: That's second on the list after making the paperclip factory. Yes, right?
j.fiotto-kaufman: Yep. And then just how to manage our Gpu hours better.
j.fiotto-kaufman: So if we had infinite hours right? We'd run all the models in the world all the time.
j.fiotto-kaufman: but we want to decide like, especially if somebody asked for a model that isn't up at some point we wanted to like automatically, add a deployment for your model. Have you kind of have exclusive access to it for a few hours. And that will really kind of open
j.fiotto-kaufman: I guess. In theory you couldn't. But like, if you're asking for one model, and I guess a bunch of people have. The other request comes in and fills it up for like the 2 h that it's up.
j.fiotto-kaufman: I guess it's not really great for your one experiment. Yeah.
j.fiotto-kaufman: So you talked about a 1 h time limit?
j.fiotto-kaufman: Why, let's see, would it be possible to split that
j.fiotto-kaufman: request processing that request up into
j.fiotto-kaufman: 10 min batches or whatever, and then every 10 min say, okay, somebody else can come in, and I'll resume when your time, your time slicing of the something I'd love to do. I thought a little bit about it. In like you can have to like save the current state of the intervention graph.
j.fiotto-kaufman: and then, you know, one other thing complicates. That is like, let's say, you're running the model. It's it's a python program, right. It's like running a model halfway through. How do you like resume a python program? Essentially
j.fiotto-kaufman: right? So there might be ways to maybe not do it at arbitrary points, but at like any point where, like, you're at an insight note or something. Right? Then you can checkpoint something. Yeah.
j.fiotto-kaufman: yep.
j.fiotto-kaufman: And then also some way of like, maybe asking the community what models they want offered. Right? Yeah. At some point, I like to put a diffusion model up there just to see how it would behave. A diffusion model. So a text image?
j.fiotto-kaufman: Yeah, I mean, you have. They're they're pretty small. Right? You have the space.
j.fiotto-kaufman: Yeah, how much free, Gpus, do you have none? Yeah, you know, we plan on adding more nodes. But where they are they're all being used in the sense that you have deployments for them. Yes, what percentage utilization is there, like actual utilization of the Gpu is probably very low, right? Because, like. Only when a request is processing, is that the Gpu being utilized so very low?
j.fiotto-kaufman: so you could.
j.fiotto-kaufman: you know by time slicing, for you know models. Well, how long does it take to bring a large model up on a node? Yeah. So for the 4 or 5 B, it takes about an hour and a half to bring it up. So you don't want to do that very change for 5 B. You know it's kind of like the whole proposition with Ndf, is that like it's worth it for?
j.fiotto-kaufman: Not. Every individual user needs to like be loading the right right? No, I get that.
j.fiotto-kaufman: So you want to figure out, you know, what?
j.fiotto-kaufman: What percentage of time do you want to
j.fiotto-kaufman: spend on loading? You know? Let's say, 5% versus using. Yeah.
j.fiotto-kaufman: yeah, but that's all. I had any questions about the last few slides.
j.fiotto-kaufman: What? What are the usual reasons? It goes down. It gets restarted. A lot of the times in the past couple of weeks. It's gone down because
j.fiotto-kaufman: Delta forgot to give us more time on the reservation
j.fiotto-kaufman: and then Michael will have to email them at late Sunday night and ask for that again.
j.fiotto-kaufman: so sorry you have. They have a bunch of Grace Hopper machines. Yes.
j.fiotto-kaufman: and you're getting not machines, but slices of machines. Or we're getting full machines. We're getting 5 full machines. Okay. But we have to ask for we want to reserve these ones for like a week or 2 weeks in a row, not permanently, or for the 3 month period, or we'd like to do that. But you know, this all goes through slurm. So it's like
j.fiotto-kaufman: we're we're trying to balance between like us being a special case and them being treated as a normal customer. This is what they kind of want to do.
j.fiotto-kaufman: Yeah, that's why they know Slurm and the long running job.
j.fiotto-kaufman: Slurm. Oh, Slurm is a software for allowing many people to submit jobs to a cluster of many nodes, so like a lot of universities will have slurm clusters where they expose a bunch of Gpu machines, and then you'll submit a job, saying I want 3 machines with my gpus, and they'll be put on a queue and auto serve like right as
j.fiotto-kaufman: right. And then so we're going through that system. So like, we have to submit a job saying, Please load all the different models we have and the different nodes. And the default. Max. Reservation time is 2 days. So we have to like, ask for special treatment and say, Please extend this, and we have to keep asking for that every week.
j.fiotto-kaufman: Very similarly, for 1 point. O, so like a some drivers like a text file with a big comment at the top of it. Sounds like you definitely need to get them to
j.fiotto-kaufman: have a model, for
j.fiotto-kaufman: you are a service provider on top of this. And so and I have, they at least acknowledged. That's a thing they should be doing.
j.fiotto-kaufman: Yeah, I mean, well, like we're you know we are getting some special treatment in the sense that they are allowing us to do reservations for longer than 2 days. But it's still hassle
j.fiotto-kaufman: well, but it sounds like it still fails. Yes, if he just forgets, you know, this is very like human point of failure. Mode right now.
j.fiotto-kaufman: so you need an auto sender on email every Monday morning. Please extend our alright
j.fiotto-kaufman: reservation for 2 weeks, or whatever. Yes, that's terrible.
j.fiotto-kaufman: Yeah, I am familiar with this, though. Oh, you are familiar with. Yeah, cause that was the
j.fiotto-kaufman: for doing for running tests of the
j.fiotto-kaufman: You know the cluster software it we would indeed. You know you get it for
j.fiotto-kaufman: 24 h, and you can extend it.
j.fiotto-kaufman: but you have to remember to extend it. So if your test was running when you forgot, you know. Remember to request it. Yeah.
j.fiotto-kaufman: let's see, right? Because I think we needed for a Gc
j.fiotto-kaufman: zone region, you know to do a test of that you needed.
j.fiotto-kaufman: It was maybe 125 services running on a whole bunch of machines and and some some dedicated mostly not, but some dedicated.
j.fiotto-kaufman: So you
j.fiotto-kaufman: there, you know, there's just a limited number of such machines, right? And you know a thousand engineers who all would like to run their tests.
j.fiotto-kaufman: Yeah, yeah, I. You know, I open a year. Once we have a lot of performance numbers like, Oh, the GPS are being utilized. We're having tons of requests. And they say, Okay, it's, you know, worth it to give you special treatment. But if you're using the gpus, then, like, I understand their hesitation. Right? Yeah.
j.fiotto-kaufman: So how many different models do you have that are getting used on roughly a daily basis? So 4 or 5 B. Right now, you know, gets like, maybe like 10 requests a day. But like llama Ap. And 7 dB, will get like
j.fiotto-kaufman: 500 600.
j.fiotto-kaufman: Maybe all from like a few users, probably. But yeah, so people are using the open
j.fiotto-kaufman: access ones pretty frequently right after.
j.fiotto-kaufman: Yes. Why do you think 4 or 5 B is so much just because you have, it's only 10 people can use it. Yeah, yeah.
j.fiotto-kaufman: Only 10 people. Only 10 of your clients are allowed to use it. Yes, those are special pilot program users. Yeah, the other ones are totally anybody can. Api key and use.
j.fiotto-kaufman: It's like of the 10. Is it just Nikhil? And yeah, some guy in, maybe Vermont
j.fiotto-kaufman: is not on that project.
j.fiotto-kaufman: He he does have a 4 or 5 B program. Oh, application. I forget what he's doing definitely. Yeah. Or maybe he did one with the multilingual
j.fiotto-kaufman: But we we also have Kevin, right? Like the
j.fiotto-kaufman: theory of mind. Yeah, experiment. So once Kevin starts, I think it's because he is a grinder.
j.fiotto-kaufman: So it's gonna be a lot.
j.fiotto-kaufman: I wonder if you know you need to either
j.fiotto-kaufman: increase the number of people in your pilot program because I'm guessing you have some who signed up and aren't using it.
j.fiotto-kaufman: yes, or rescind theirs. Or, you know, say, you know, people who use it at least once every 2 weeks get to keep it. Yes, yeah. Yeah. So we have expanded. And yes, we have, like tool, like gone back to the 1st 10 and said like, did you actually use the system? If not, then? Right? There's no reason like once a week, just for like one run, you know, you have a paper that uses a bunch of different models, and you want to like show. Oh, it generalizes, maybe, or works better on a larger model. Just one request might do that. It might be an interesting result. Yeah.
j.fiotto-kaufman: which makes me think that maybe rather than having
j.fiotto-kaufman: A very restrictive pilot program having a
j.fiotto-kaufman: restricted, you know. Queries per day. You know, it's like you can have one a day if you're not in the program
j.fiotto-kaufman: just
j.fiotto-kaufman: oh, and then, being in the program, maybe lifts your quota right? I see. I see, so that anybody can try it.
j.fiotto-kaufman: and then, if they like it, then then they can sign up. That'd be cool. That's a good idea. How hard is that technology
j.fiotto-kaufman: showing a few, you know, fields to the firebase.
j.fiotto-kaufman: That's a good idea.
j.fiotto-kaufman: We had.
j.fiotto-kaufman: you know, you talked about having rate limits and so we on the the Google. Every every server is, you know, has a set of Api endpoints, you know, Http. Endpoints, and most of them
j.fiotto-kaufman: were set so that they have a rate limit of, you know.
j.fiotto-kaufman: like one query per second for anybody who isn't on the team, you know. So you every time I say, Google, my phone wakes up, which is problematic at my house. It's it said, a fair amount.
j.fiotto-kaufman: but you know. So you know, people on the team can then go and look at the status page, or whatever as much as they need in order to manage the system. But somebody who's not on the team is just trying to figure out, what does that server do? Can just go look, and they can see. But they they can't. They can't take us down by, you know. Excessive queries
j.fiotto-kaufman: makes sense for sure.
j.fiotto-kaufman: That's cool. Yeah, yeah, right now that we're at the point where we have like summary limiting, and some time out, right? One person can't totally take on the system. And that's kind of what we just wanted to get to.
j.fiotto-kaufman: Yeah, that's a good idea
j.fiotto-kaufman: questions.
j.fiotto-kaufman: It's fantastic congratulations for getting to this point. Janet.
j.fiotto-kaufman: being able to take the whole system end to end and put on slides. Right? So do you feel like this architecture?
j.fiotto-kaufman: Do you? Do you feel like
j.fiotto-kaufman: a year from now? If you were explaining
j.fiotto-kaufman: how and it works to somebody that
j.fiotto-kaufman: basically the same slide deck. Yeah, I mean, in the sense that you know, we.
j.fiotto-kaufman: the abstraction of that computation graph is executed remotely. I think that has to stay the same.
j.fiotto-kaufman: I think that
j.fiotto-kaufman: serves models to different Gpu machines based on how many resources they need. Just as probably each one of the hops. Yeah, we'll still have it, every one in the hops. It'll just add to it. Not necessarily change it.
j.fiotto-kaufman: One thing might be so we use ray, right? For all this, like micro service distributed stuff
j.fiotto-kaufman: which is pretty new, although it's pretty popular within the AI community. It seems fairly
j.fiotto-kaufman: restrictive on like how much you can customize it. And then, like, like one thing right, I want to see whatever like distribute queuing system. There is right for the different models I'd like to be able to like look into the queue and maybe change things around but at least from like what I've seen so far and my questions asked to the right team, like, you know, exposing that
j.fiotto-kaufman: programmatically. Yeah, it's not a priority for them, right? Because, like people want to use Ray as like a mid level abstraction to just like serve some model, to like classify images. And you're like, you know, your little company not necessarily like, build something like this?
j.fiotto-kaufman: so either we might have to get rid of Ray, or maybe change Ray or ask them to change ray? Or.
j.fiotto-kaufman: yeah, is this a python system? And every all this stuff is pretty much all python.
j.fiotto-kaufman: You kind of build it yourself right? It's just like distributed executing code from a remote machine. So you can kind of build right from celery if you really wanted to. And maybe we'll go back to that.
j.fiotto-kaufman: Maybe I have no idea. Yeah, yeah.
j.fiotto-kaufman: But you don't want to invest too much in that. If you don't have to putting up with
j.fiotto-kaufman: some other systems idiosyncrasies worthwhile, if you can focus on the features your users need. Ideally, we make that decision sooner rather than later, before we get like too much into the ray ecosystem. So maybe like
j.fiotto-kaufman: a job would be to actually figure out what's possible isn't. Now, yeah.
j.fiotto-kaufman: yeah. Cause it's like a whole sub team.
j.fiotto-kaufman: You know, the team that's building Ray.
j.fiotto-kaufman: They're mainly around building Vlm and things like this. And but they have this whole separate project right? Right?
j.fiotto-kaufman: And so, you know, you can imagine
j.fiotto-kaufman: getting into a situation where we are. We're in a similar situation where
j.fiotto-kaufman: you know, we're trying to build end in. But then
j.fiotto-kaufman: we have our own job management system, and it's like.
j.fiotto-kaufman: requires this whole separate team to create them job and the assistant.
j.fiotto-kaufman: or perhaps if you're thinking about changing
j.fiotto-kaufman: trying to architect your system so that you use abstractions
j.fiotto-kaufman: on top of Ray, so you could change Ray out if you need it.
j.fiotto-kaufman: but not, you know. But don't have to. Right right? You know, kubernetes would do a similar thing. Right? Yeah.
j.fiotto-kaufman: Interesting?
j.fiotto-kaufman: Well, cool.
j.fiotto-kaufman: Thank you, Jayden.
j.fiotto-kaufman: How's morocco? Are you in Morocco already? Adam.
a.belfki@northeastern.edu: Yeah.