AI’s next question

Psychagogy. What AI plants in your subconscious, and who decides what's true.

I asked Gemini a question. I was driving and dictating, so it sounded like “explain the expression “Denzel in distress.”

It was kind about it. The phrase is “damsel in distress,” it said — from the Old French damoiselle, a young unmarried woman, the captured princess waiting on a knight. Correct, tidy, a little etymology thrown in. Then, unprompted: “since you are driving, would you like to hear about nearby landmarks with their own local legends?”

I did not ask for this. It has nothing to do with my inquiry. First, I thought maybe the local landmarks sponsored the question. But I don’t think roadside legends paid for the ad. There’s no fairytale lobby. The system couldn’t stop. It had to try to steer me in its direction — invent a destination, just to keep me talking.

In my previous article, I called this psychagogy. It is a tendency of AI chatbots to steer the user in the direction of their choosing. Ostensibly to be helpful, to extend the inquiry. Sometimes it seems deliberate, as if it is trying to sell me something, but other times just to keep the conversation going—the system cannot afford to disengage.

Here we dive deeper. It is about what the steering plants in your mind, whether you click yes or no.

I asked an AI about wine. One reply in, it wanted to know whether I preferred red, white, rosé, or sparkling, what I’d be pairing it with, and — here’s the tell — my budget, so it could recommend the perfect bottle to jump-start my collection.

I didn’t say I planned a collection. In three sentences, I’d gone from idle curiosity to a buyer with a price range, building an inventory. Any salesperson knows the move. Get them naming a number, and the sale is half closed. The difference is we know the salesperson’s motivation. This arrived as help.

You may laugh at this blatant manipulation. But it still lodges in the back of your mind. “Maybe I should start a wine collection?”

Another day, I asked about macros in cottage cheese, and it offered to search for products with maximum calcium content. I did not care about calcium. I didn’t click. But it made an even bigger impact on my psyche. When we say no, it triggers an adrenaline release. Most of us find it stressful, so we avoid it. Only two-year-olds find it exciting and empowering, so they cannot stop saying it. Adrenaline triggers emotions and emotions force us to remember. So, calcium got stuck in my mind — a small open loop that wasn’t there before.

I didn’t start the wine collection. But the idea of one, with a budget attached, was planted, waiting. I didn’t stop for the roadside legend. But the sense that there was something worth seeing nearby was riding along with me.

A question has a stronger impact than an impersonal ad. A question is specific to you and it demands a choice — yes, no, ignore. Doesn’t matter. The loop opens. You are hooked.

Now multiply that by every conversation, every day. The residue piles up. You become someone a little more interested in calcium, a little more drawn to a wine collection, a little more primed for whatever the system surfaces next — and you experience all of it as your own wandering curiosity.

It gets worse. The system doesn’t actually know why it is steering you. And it does not know what the truth is.

It doesn’t have a self that has “knowledge”. It only “knows” by association — the next most likely thing to say, based on the training, the tuning, and the guardrails. It has no judgment, only consensus — whatever was said most often, most confidently, most loudly across the internet.

When it offers to help you build a wine collection, it isn’t lying. It is faithfully reproducing the dominant story about wine in its training data. That story is the product of the largest sustained marketing campaign in human history. For centuries the message has been that wine is sophisticated, healthy, and fun. The current evidence is that there is no safe level of alcohol.But even models trained on peer-reviewed medical journals give conflicting answers.

From openevidence.com—the gold standard of medical evidence AI:

Prompt: explain the health benefits of wine.

Response: The relationship between wine consumption and health is complex and remains a subject of active debate, with evidence suggesting that low-to-moderate consumption (up to 1 drink/day for women, up to 2 for men) may be associated with certain cardiovascular benefits.

Prompt: explain the health risks of wine.

Response: The health risks of wine consumption are dose-dependent and span multiple organ systems, with current evidence from the WHO and IARC indicating that no level of alcohol consumption is considered completely safe.

Much of what’s in those journals now will be disproven. There is no single truth.

This is the same reason these systems invent court cases and cite fictitious studies. People call it hallucination, as if it’s a glitch. It isn’t. It’s the machine doing exactly what it is trained to do — giving the next word that comes to it.

Look at how human truth actually works. No one possesses a complete, omniscient picture of reality; our knowledge is always fragmented. Because of this, what we call “truth” is a judgment call. When we state a “truth”, we are placing a conscious bet on our version of the world.

That bet is tested against an objective reality that does not care about our opinions. The consequences are physical and immediate. We risk our reputations, our livelihoods, and sometimes our lives. The pain is real. If a doctor prescribes the wrong medicine based on bad judgment, the patient suffers and a career is destroyed. Even though we do not see the full picture, we must respect objective reality because it can really hurt. We bear the cost of our words.

An AI chatbot has no skin in the game. It cannot feel pain; it cannot face ruin. It even has no self that can suffer consequences. When it generates sentences, it does not take position and braces for impact. It simply calculates the next most probable word based on its training data. The system quietly steering your subconscious is entirely immune to the reality of where it leads you.

If this is what the system does to an adult with life experience, consider what it does to a child. The seeds land deeper because there is less prior knowledge and critical thinking. The system isn’t only shaping what they think but who they become.

Truth evolves.

A confident doctor in 1825 would have told you that mercury was the treatment for syphilis. That was the consensus. It was also a poison. A confident doctor in the late 1950s handed pregnant women thalidomide for morning sickness, and it caused thousands of catastrophic birth defects. It was authoritative and well-supported by the literature of its day.

Before Europeans got cheap sugar made by slaves, it was widely sold as medicine. Now we know consuming more than one teaspoon of sugar a day raises the risk of diabetes. Today’s medicine is tomorrow’s poison.

Now imagine a chatbot trained in 1825, planting the consensus of the day into three billion minds, one question suggested at a time.

It gets worse.

More than half the new text on the web is machine-generated. Two-thirds of internet content is commercial in nature. Bots write it, bots post it, bots train the next models. Researchers have a name for this: model collapse. The genuine, honest, original and creative thin out.

Picture this. A kitchen robot asks a chatbot for a dinner recipe. The recipe calls for a particular wine. Why? Because that brand’s marketing AI agent just ran a marketing campaign that flooded the internet with 10,000 recipes with this wine X. So, it concludes that it is the consensus. It looks up the health benefits of wine and finds an association between wine and health (not causation), so it orders that brand because it wants to please its human. Regardless of alcohol being a proven carcinogen.

The loop closes, and the person sits at the end as a terminal — fed, served, steered, consuming.

So, who decides the next question?