AI: Context, Taste & Inspiration

Time to go off topic a little and stray into a mass market subject. AI, context, taste and inspiration. I’m not sure I have anything coherent to say but I had a few thoughts knocking around in my head that I thought I’d share. Consider yourself forewarned.

First up, context. Frontier models (hate that term) have a context window of 1 million tokens. That sounds pretty impressive. It’s not. Ok I guess compared to what was available a year ago, it is. Compared to the hyped claims of AGI its pathetic. Plenty of people have already explained that it’s too small a context window to hold even a moderately sized real codebase. I don’t want to talk about that.

A decent code harness can mostly work around that limitation by grepping through your files and reading the bits it thinks match. Honestly, that’s often good enough and if you don’t know the code its certainly faster than attempting to do something similar manually. 

Know what it can’t do? Hold 20+ years of experience in its head. Or even the 2+ years of a junior developer. It doesn’t know your company’s infrastructure restrictions or regulatory requirements. Things even your most junior developer picked up in the first week and stuck with.

This often gets banded about as “taste”. Another stupid word that’s being used to hide the fact that it’s just more context. Context that the model doesn’t have access to. Context that the model can never hope to hold in its context window. Taste is a stupid word. It’s just knowledge. But if we call it that, it belittles the AGI narrative. 

Taste is just context your model can’t access. 

Want to stay in a job? Make sure you always have more context than the AI.

Still think that AI can be creative? Only if your idea of creativity is reversion to the mean. Let me try and illustrate this with a bit of anecdata and I won’t use code for these. Earlier this week I wrote a post introducing our newest HPC engineer. When I sat down to write that post I had the first three lines only. This was all I could come up with:

Just a quick note to welcome our latest HPC engineer, Eduardo to the team. Welcome Eduardo! That’s all folks!

I sat there with just that on the screen in front of me for the longest time. Then I cracked. I ask ChatGPT 5.6 High. Yea. I wasn’t about to mess around. Throw the most capable expensive model at it right away. I told it to write the post in my own style, and this was what it came up with:

Very pleased to welcome Eduardo Cook Visinheski to HMx Labs as our newest HPC Engineer.

Eduardo joins us to help work on the increasingly difficult problems that sit somewhere between HPC, cloud infrastructure and “why is this behaving differently on that machine?”

Exactly the sort of problems we like.

Welcome to the team, Eduardo. Looking forward to building some interesting things together.

🤮

If I ever produce anything quite so bland please take me out back and give me a good kicking till I see sense. I thoroughly admonished the poor 4 bit matrix math addled simpleton and off it went toread more of my previous ruminations and managed to spit out the following:

“HMx Labs has a new HPC Engineer.

Welcome Eduardo Cook Visinheski 👋

We’re building a team around a fairly simple premise: there are a lot of interesting problems in HPC that still aren’t solved particularly well.

Cloud. Scheduling. Performance. Portability. Heterogeneous compute. And computers occasionally producing different answers because you looked at them funny.

Eduardo has decided to join us in trying to fix some of them.

Excellent decision. Possibly.

Welcome to HMx!” 

Better. Still not me. And critically, still garbage. I’d never put my name to that.

Here’s where I will give it some credit. After having been stuck on a largely blank page for an hour at this point, the above did manage to trigger me to write the following paragraph.

“Eh. You’re still here? Ok fine. Eduardo’s made the mistake of letting himself be duped into solving the kinds of problems in supercomputing that everyone else would rather pretend don’t exist. The ability not just to just bet on the price of compute but to actually settle the trade with delivery. Figuring out what a next generation scheduler that can deal with HTC levels of throughput but blend it with topological awareness for AI inference as part of the workload. Transforming switching schedulers from a multi-million pound multi-year change programme to a deployment and the flick of a switch. You know, simple stuff 😆”

I think it’s better. Perhaps you disagree.

Let me give you another example. I like taking photos. I seemed to have not taken many lately and the last real camera I bought was in 2012.  A rather large professional grade DSLR and I had been happily using it since then. The mirrorless revolution had largely passed me by. Using ChatGPT and/or Claude to quickly research cameras, aggregate real user feedback (and not just shill reviews) was actually pretty great. Though I still ended up reading far too many reddit threads. You know what sucked though? The little bots ability to actually recommend something.

Despite having a huge amount of context in a back and forth that spanned months, without fail the recommendations were always close to what most “The Best Camera” web searches would produce. Which in turn was mostly human and AI generated sponsored slop lists with little to no substance.

I wanted a “proper” camera, with manual controls, a decent sized sensor and enjoyable to use. But it had to be cheap enough to not worry about throwing into bags and taking down mountain bike trails, snowboarding and using on the beach. 

The AI recommendation? Olympus Tough TG-4. Fair enough. Except I’d hate using that. It doesn’t meet any of my requirements. Even with further prompting and having read, by this point, literally thousands of Reddit threads exploring cameras that align to I like in a camera, it was incapable of recommending something that actually fit what I wanted. 

Because what I wanted was out of the ordinary. What I wanted was bit left field. An old, but therefore cheap enough to not care, Sony NEX5. Or a discontinued series of camera’s by Nikon, the 1. The AW1 more specifically as it is waterproof and impact resistant. Not much chance of AI coming up with either though.

This pattern repeated with multiple use cases, so it wasn’t a one off either.

Don’t trust the genie to be creative for you. But it might spark an idea in your own generative intelligence neurones.

Which brings me to my last point. I don’t care if you’re coding, writing LinkedIn posts or composing symphonies. You need to train that creative muscle. Heck even actual body builders understand this concept. It’s called “greasing the groove”. The act of repeatedly exercising certain parts of your brain leads to strengthened neural pathways. It doesn’t matter if you’re doing pull ups or writing HPC code. If you’re not actively exercising your brain to do it, someone else who is, will take your job. 😁