Elastic Compute in the Cloud
Elastic compute in the cloud? An old idea but is it viable? How much work is involved?
We often think of supercomputers as monolithic and in continuous operation. Flipping bits across thousands of CPUs and GPUs and churning your data through RAM all while not melting your face off. Most centre directors won’t do much to dispel that idea either 😉
Honestly though, the reality is that many supercomputers don’t run a single large workload that consumes the entire machine. Well not once they’re done running their Top500 submission anyway 😆. Often, they will be used by a number of smaller jobs and utilisation may vary over time. The shape of that variation is often a function of the industry with some, such as financial services, having nice regular predictable patterns and others such as academia being much more capricious.
Your friendly cloud solution architect knows this. Of course.
There’s a certain amount of logic then, in potentially moving some of the workload to the cloud. But only if you limit your cloud capacity to when its actually being used. I mean it makes no sense to run your cloud 24x7 unless that matches your workload.
You could even do that across multiple clouds. Selecting the vendor based on price, performance, suitability for a particular workload or data locality.
All sounds great right? And it is!
Suddenly though your supercomputer needs a way to control cloud resources. But that’s ok. Its 2026. Code is cheap. Claude or Codex can sort that out for you in pretty short order.
Problem with code, especially when it has to run real production systems that people depend on, is that it has to be maintained. And the cloud providers have this nasty habit of continually improving their offerings and providing new ones. Keeping up with that can end up being a full time job in itself.
You could do that. Or you could just make that someone else’s problem. There are both commercial and open source solutions in this space to make your life a bit easier. So do you want to do some science (or financial risk or meteorology) or play dev ops?
Want to learn more? Come along to HPC Club.
