AI: Are we Confusing Intelligence with Eloquence?

Is ChatGPT actually smart? Or is it just playing us with its good looks and way with words?

This last week I’ve been a little sick causing my ability to speak to range from completely mute to a speak-n-spell monotone coupled with short words and long pauses. It was genuinely scary how robotic I sounded at times. I cancelled most of my meetings naturally, but I do wonder how smart the odd few people I did have to meet (such as the doctor) thought I was. Scratch that, I was judging my own verbal output to be on par with a brick. There’s definitely no risk that anyone meeting me in that state was about to think I’m smart.

There’s a fairly well established halo effect that leads us to believe that people who are attractive or eloquent and articulate are also intelligent (one example). This led me to wonder, well what about AI?

Do we think AI is smart just because it is well spoken? Humans also have a tendency to project and anthropomorphise. For example, we will often refer to cars as having character or a soul (I mean dinosaur juice powered cars of course not those the electron powered kitchen appliances with wheels). While the default language output of ChatGPT might be fairly corporate vanilla (real Madagascan vanilla I grant you) it is still probably more articulate than the majority of the English speaking population.

I’m sure this is no accident and the smart folk at OpenAI, Anthropic and Google et al spent a long time tuning the output to get to this point. What if it wasn’t though? Google search has (or at least used to have) the same wealth of knowledge that ChatGPT does, we never attributed any kind of intelligence to that. How smart would we think ChatGPT is if its output sounded more like Joey from Friends instead?

LLMs are fantastic word calculators. They have no understanding of the output they generate. When you read reports like the recent spate about AI’s trying to escape, you’d do well to remember that. And maybe ask the question, who benefits if people believe the lie?

Not convinced? This is from part of the ChatGPT conversation to generate the header image

That was the fourth attempt to generate it too. I gave up and went back to Photoshop...