Schedulers are Boring Dino-Tech
A short rant about why schedulers aren’t boring but also about so much that is missing from them and how much better they could be.
But Hamza, schedulers are boring. Old Hat. Why do you keep talking about them? What’s so exciting about software that was written 40 years ago and even the newest ones are more than five years old. That’s ancient dino-tech.
Let me guess, you can probably vibe code one this afternoon instead, right?
The fact that they are old is exactly why I want to talk about them.
Listen, what we need from a HPC scheduler has evolved beyond something you can cook up on a Friday afternoon or even in a couple of weeks. What modern HPC applications need isn’t met by today’s schedulers. Not without bells and whistles attached and entire cottage industries devoted to making them functional. And when I say modern HPC applications I mean everything from training AI models to running financial models with accelerator cards.
The simple truth is that scheduling HPC workload is no longer a one dimensional problem. It’s an N dimensional problem and one that quite frankly we should be using ML to solve. Except that we can’t. We don’t have the requisite input data to do so. So instead, we make do with the current paradigms.,
It’s a problem that needs better integration and (hopefully) standardised and open APIs between the functions of not only job and resource scheduling but also infrastructure control planes, data catalogs, historic workload meta data, benchmark results, observability, topologically aware caches… and… and… and breath!
The way I see it is you have a couple of choices. Keep making do with what we have and tacking bits on to it. Or wait for some new fangled, vibe coded, half baked techbro excuse of a product that’s actually worse but has all the right buzz words, is probably marketed to AI model companies but your boss will foist upon you anyway as it’s the new shiny.
Or start thinking about and demanding what you really need.
This is why schedulers are exciting. Because there is so much that could be done. It could be so, so much better. I don’t see boring old tech. I see potential for something that can improve and accelerate scientific research, weather and climate modelling, and yes even AI training and financial models.
Oh and sorry for the departure from my usual writing style. I think I’ve listened to too much Ed Zitron this week 😁
