Chocolate & Peanut Butter @ HPC Club

Chocolate and peanut butter. That’s how a friend likes to describe the worlds of HPC in academia and finance. Domains that feel completely distinct and that you perhaps wouldn’t think to combine, but actually, go together quite well!

Despite the very different technical starting points (classical MPI based supercomputing vs HTC on highly commoditised hardware) I see an increasing convergence of requirements and solutions. Since I’m already freely borrowing from the works of friends, I think what we’re seeing here is the convergent evolution of supercomputing across rather different domains.

I think, in large part, that is being driven by AI (and probably not for the reasons you think) but really, that process started before ChatGPT. More like 2008 or so with an increasing requirement around counterparty risk and other forms of portfolio risk calculation. This moved the game on from embarrassingly parallel compute to varying levels of additional complexity (depending on the bank) but essentially an acyclic graph of some form.

Increased AI adoption has significantly accelerated this convergence. Partly because running an LLM looks the same regardless of what your prompt contains. Partly because whilst banks used to enjoy a preferential status with technology suppliers due to their spend, these days unless you’re a hyperscaler or neocloud no one else comes close so academia and finance are more similar than different as far as big tech is concerned! And partly because, quite frankly, they always had more in common than first met the eye.

Both run shared supercomputers having to provide access to competing users. Both have surprisingly similar organisational and power structures. Both have compute that bottlenecks on IO before CPU even if they don’t like saying that out loud (and let’s be honest that’s been true for most supercomputing in the real world for a long time).

I’ve been trying to keep a foot in both worlds for a little while and I think it’s time that we put you all in a room together. The next HPC Club not only will we have Lawrence Livermore National Lab sharing their experiences but also a large hedge fund. Chocolate and peanut butter.

I’ll let you decide which is which.