HPC in Finance and Academia: Chocolate and Peanut Butter
HPC in academia and financial services has a surprising amount in common and I don’t just mean the obvious challenges of operating at scale.
Over the past few months while building out the Supercomputing Strategy Group and also at the HPC Leadership Institute last week I’ve had numerous conversations with centre directors and heads of national labs. It has been fun to discover just how similar some things are between academia and the finance world.
Before you jump on me, yes, I know there’s no MPI in financial services and academia is about research and improving humanity not making money. I get it. Aside from some obvious technical similarities around operating at scale, there are some other fun parallels.
The organisational and power structures are remarkably similar. Academia has star researchers that call the shots and get the lion’s share of the compute resource. In financial services those are the traders that happen to be making the most money. This was particularly apt pre 2008 when those traders also happened to be the ones dealing in complex derivatives that demanded huge amounts of CPU hours to value and calculate risk for.
Then you have the RSE folk, their equivalents would be the risk system developers and lastly you have the centre directors who are replaced by the heads of grid computing in banks instead.
Courtesy of Conway’s Law, the similarities in organisational structure and power dynamics also mean we end up with very similar architectural patterns and solutions.
The evolution of HPC across the two is analogous too. Financial services often started with multiple clusters of machines serving each trading desk or business area. Over time these slowly consolidated into larger bank wide compute grids (or didn’t in some cases!). A similar played out in academia with multiple university departments (reluctantly sometimes) pooling compute resources for shared time on a single supercomputer. The establishment of RSE as a discipline in its own right is also somewhat paralleled by the increasing role played by risk system developers in finance.
I’m not sure traders or quants or will have much to share with academic researchers but I think centre directors and heads of grid computing certainly do. Probably RSEs and risk system developers too. Y’all should speak more.
