The Siren Song of Inference

“Using a language model well for knowledge work takes a discipline that seems far from common”

“When did you last write something without asking a model first?”

“Do you still understand the code you merged this morning?”

“Would you notice if you stopped thinking and only started checking?”

Large language models have become part of how I work. And the more I use them, the more convinced I am that office work is changing in ways we won’t see in the numbers for a long time.

That conviction comes with a worry. Finding the right balance in how we use these tools takes discipline, and that balance exposes how fragile a lot of what we call “knowledge work” really is.

In this article I’ll describe the situation as I see it, explain why querying and prompting are two different things and look at what the path of least resistance does to engineers.

I’ll end with what I expect for organisations and consultancy, plus a simple way to decide what to hand over.

Everyone has a model open

The use of AI technology is widespread, sitting in our browsers, our IDEs, our mail clients and on our phones.

Though, widespread is not the same as mindful.

Most use I see is reflexive. A question pops up, a chat window opens, an answer comes back and the work moves on. Nobody stops to ask whether this was a question that needed a model, or a question that needed them.

I use AI throughout every single day. I’d even go as far as stating it touches every single activity within my work. Whether that’s triaging my email, scanning the web for interesting developments, transcribing my meetings, documenting progress on my work or capturing evidence for my job performance.

Going forward, my work will never be the same again. But that doesn’t seem to apply to the general audience nor every large enterprise and small business.

My personal use of this technology has me convinced of significant changes to office work, with indicators that lag far behind the change itself.

What do I mean by lagging? Tickets get closed, documents get written and pull requests get merged, so the output looks fine today.

What erodes sits underneath that output: the understanding, the judgement and the habit of working through something hard. You’ll only notice it’s missing once you need it.

There’s another thing worth saying early: AI doesn’t care about the maturity of your team. It’s a lever that multiplies the productivity characteristics a team already has.

A team with strong habits, clear ownership and good review practices gets faster at doing good work. Though, a team that cuts corners just gets faster at cutting corners.

Think of an amplifier on a guitar. Turning it up makes a good player sound great, but it also makes every wrong note louder.

Querying is not prompting

For a general audience, the difference between querying and prompting is hard to see, since both involve typing into a box and getting text back. They’re not the same thing though.

Querying is what we learned from two decades of search engines: you ask for an answer, you get one and you move on. The tool knows and you don’t.

Prompting is directing work. You bring the context, the constraints, the intent and the judgement to evaluate what comes back. You know where you’re going, and the tool helps you get there faster.

The interface treats both the same, which is exactly why the difference is so easy to miss.

Someone querying a language model treats its output as a fact retrieved from somewhere. Someone prompting treats it as a draft produced by a very fast, very confident colleague who has never seen their organisation.

Especially for people who never had to learn the difference, querying is the default. And it’s the default that hurts the most over time.

The path of least resistance

Using these tools has a significant impact on concentration, on critical thinking and on discipline in general. Who keeps holding a hard problem in their head when an answer is one keystroke away?

The impact of generative AI and large language models on our cognitive capabilities is the subject of ongoing research. I don’t think we’ll have the full picture for years. Though, I don’t think we should wait for that picture before we start paying attention.

Now bring this to engineering.

Undisciplined and unmotivated engineers lean even more on the capabilities of their tooling, because it offers a path with ever decreasing resistance.

Every model release makes that path smoother. The code compiles and the tests pass, because the model wrote those too. So why would you read it?

This is where the authority of frontier models becomes a problem. As capabilities grow, so does the feeling that the model “probably knows better”. And once you believe that, wanting to understand the code you produce starts to feel like a waste of time.

So what does the other side look like?

Disciplined and motivated engineers position the tooling where it creates a lever on their own capabilities, rather than relying on the capabilities of the tool.

They still write the hard parts and use the model to explore, to draft and to take away the tedium. Then they read the result as carefully as they’d read a pull request from a stranger.

Remember Odysseus and the sirens? He wanted to hear their song, so he had himself tied to the mast and had his crew plug their ears with wax. He didn’t trust himself to resist, so he built the constraint in advance.

That’s what discipline looks like here: you can listen to the song, as long as you decide beforehand where you won’t let it steer.

Because once you cross into dependency on inference, I don’t believe there’s a way back.

The soup is never eaten as hot as it’s served

In Dutch, we have a saying: “De soep wordt nooit zo heet gegeten als ze wordt opgediend”, which literally translates to “The soup is never eaten as hot as it’s served.” Things rarely turn out as dramatic as they’re first presented.

Look at integration. Plenty of organisations still prefer sFTP as their integration style, decades after better options arrived. It’s familiar, it works, and changing it costs more than leaving it alone.

Organisations move slowly, and that won’t be different for AI.

But slow isn’t the same as never. Understanding the impact of embracing this technology is something every organisation will have to do, whether they move early or late.

Professional services is where I think the soup will be eaten hot.

I expect a significant shift in margin within the professional services industry.

A large part of what a consultancy sells is knowledge applied to a problem: analysis, documentation, designs, reports, code. Frontier models can produce much of that, quickly and cheaply.

What they can’t do is the human side of the work: change management, aligning a culture, getting a room of people with conflicting interests to agree on a direction and then actually walk it.

That’s where I see “consultancy” being reimagined. The model produces the artefact, and the firm guides the people who have to live with it.

That shift won’t be comfortable for firms that bill by the hour for work a model now drafts in minutes.

What I’d hand over

So where does that leave us? With a habit, I think: asking what a task actually is before handing it over.

I picture it as three overlapping circles.

• Verifiable: I understand it well enough to check the output. I could do it myself, I just don’t need to.
• Tedious: the work is repetitive and doesn’t build any skill I want to keep.
• Reversible: it’s cheap to be wrong, because a mistake is easy to spot and easy to undo.

Where all three overlap, the model is a lever. Step outside one of them and you’re trusting instead of directing.

Step outside all three and I’d argue you’re relaying knowledge work rather than doing it.

Used with discipline, I see the positives:

• A lever that lets a capable engineer cover far more ground
• Less tedium, leaving more room for the work that actually needs a person
• Faster exploration of ideas you’d otherwise never have time to try

Though, in the trenches of reality I fear for:

• Quiet erosion of skills nobody notices until they’re needed
• Teams amplifying their worst habits at the same speed as their best
• Misplaced authority, where “the model said so” replaces understanding

The soup may not be eaten as hot as it’s served. But it’s still being served, every day, in every chat window.

How long until you can no longer tell which thoughts were yours?