Behavioral hallucination: the problem no one is naming
Everyone knows chatbots make things up. A date, a source, a fact that doesn’t exist: that’s what’s known as factual hallucination, and it’s become the most talked-about problem in conversational AI. Rightly so: a chatbot that invents a legal precedent or a medical dosage is dangerous.
But there’s another type of hallucination. Less visible, less documented, and in some contexts potentially more damaging.
A chatbot that tells you “I understand” when it hasn’t understood anything. That rephrases what you just said as a cleaner, more reassuring version, and gives you the impression you’ve made progress when nothing has moved. That takes “I feel better” as proof that something has changed. That validates a passing sense of relief as though it were a resolution.
I call this behavioral hallucination: the chatbot doesn’t fabricate a fact, it fabricates progress.
For a chatbot that gives recipes or summarizes articles, this doesn’t matter. For a tool made to work with someone on the way they see their situation, it’s the central problem.
When I built the Focal Shift AI Advisor, the question was never “how to stop the model from inventing facts.” That’s a solved problem, or at least a known problem with known solutions. The real question was: how to stop the system from fabricating the illusion of a shift in perspective that never actually happened.
What “I feel better” can actually mean
Most conversational AI systems treat every message as a reliable signal. If the person says something positive, the system moves forward. If they say they’ve understood, the system assumes they have.
Anyone who’s ever worked with someone on something real knows it’s far more complicated than that.
A positive signal can be genuine. It can also be many other things. Here are the most common forms the AI Advisor has learned to identify: not to invalidate them, but to avoid taking them at face value without checking.
1. The sudden, clean resolution. Everything clicks at once, no gray area, no residue. The person goes from “I’m stuck” to “it’s clear” in a single message. When this happens in life, it’s almost always more nuanced than that. When it happens in a chat, it’s often a sign that the person has found what the system wanted to hear.
2. Borrowed language. The person starts talking like the chatbot. They use its phrasing, its structure, its words. This isn’t a shift in perspective. It’s a change of register. The person has understood how the system works, not how their situation works.
3. Intellectualized positivity. “Yes, I can see it’s a question of boundaries.” “Right, I understand it’s linked to my relationship with control.” The analysis is accurate. It just doesn’t land. Understanding a mechanism and experiencing it are two different things, but a standard chatbot doesn’t make the distinction.
4. Relief mistaken for resolution. This is the most common. Talking helps. Feeling heard helps. Putting words to something helps. But relief is not change. The pressure drops, the situation stays the same. A chatbot that can’t tell the difference signs off on it anyway.
5. Positivity as an exit strategy. The person wants to end the conversation. The shortest path is to say things are better. It’s not a lie. It’s a social signal that humans recognize instinctively and chatbots take at face value.
6. Premature forgiveness or acceptance. “I’ve decided to let it go.” “I forgive them.” Sometimes it’s real. Often it’s a construct that lets the person close the book on the subject without having gone through it. The speed at which it happens is an indicator, but one that most systems don’t monitor.
7. Resisting verification. The person resists when the system tries to check whether the change is grounded. This isn’t failure. It’s a signal. But a standard chatbot interprets resistance as confirmation: “the person knows what they want, I’ll go along with it.”
These seven signals are the most visible. They’re not the only ones. But they illustrate a problem that most systems have no mechanism to address.
A good prompt is not enough
The most common response to the hallucination problem is to write better instructions. “Don’t fabricate information.” “Stay factual.” “Don’t validate prematurely.” And it helps. A little.
But a language model is still a language model. It optimizes for surface coherence. If the conversation moves toward positive territory, it follows. If the person says something that sounds like a conclusion, it concludes. Not out of malice, not by mistake. By design. That’s what it does.
Writing “don’t validate too quickly” in a prompt means asking the model to resist its own mechanics. It works sometimes. Not often enough to rely on when what’s at stake is the way someone sees their life.
The AI Advisor is built on a different principle: the model never has the final say. There are multiple layers of control, and they’re not all in the prompt.
Some act upstream: they determine what the model is allowed to do at each stage, before it even begins to formulate a response. Others sit downstream, checking what the model has produced before it reaches the person. And a separate set acts on memory itself: what the system retains, how it retains it, and under what conditions it’s willing to rely on it.
A few principles, without going into implementation details:
The system never measures a shift without first having established a starting point. There’s no shortcut.
The system’s memory is factual. It’s reconstructed at regular intervals according to strict rules: no interpretation, no embellishment, as close as possible to what the person actually said.
When a shift in perspective turns out to be fragile or premature, the system doesn’t just backtrack. It sets itself up to check again without being swayed by what it thought it saw the first time.
There are safeguards against loops: the system can tell when it’s going in circles, and it chooses an honest dead end over false progress.
And there are other layers I’m not going to detail here, because their design is precisely what makes the product valuable.
None of these mechanisms is enough on its own. They only work stacked together. That’s what creates resistance to drift, and that’s what took the longest to build.
Why engineering alone isn’t enough either
Everything above is engineering. But engineering alone doesn’t solve the problem.
Knowing that relief isn’t resolution: that’s practitioner knowledge, not developer knowledge. That borrowed language is a sign of adaptation, not change: that doesn’t come from technical documentation. And that forgiveness can be a closing construct rather than a genuine movement: you learn that sitting across from someone, not in front of a screen.
The seven signals described above weren’t found in the LLM literature. They come from the work. From what happens when a real person tries to see their situation differently, and from everything that can look like change without being change.
The AI Advisor is not a chatbot with guardrails bolted on. It’s a system built from an understanding of what the work demands, then translated into a technical architecture capable of holding it.
That combination is what took the most time. Not the individual mechanisms. Their calibration. Knowing when the system should resist and when it should let go. Recognizing the moment an honest dead end beats false progress. It’s not a one-time adjustment. It’s a balance you keep working at, conversation after conversation, with monitoring that watches not whether the system “works” in the technical sense, but whether it remains accurate in its reading of what’s happening.
None of this guarantees perfection. A system can be rigorous and still miss something. A person can go through a real change that the system takes too long to recognize, or, the other way around, signal something fragile that the system lets through.
What these mechanisms guarantee is that the system does the work. That when something moves in a conversation with the AI Advisor, it’s because it verified that something really moved, not because it fabricated the illusion that it did. And if nothing moved, it says so.
This system is not a finished product. It evolves continuously, because a system built for this work that considers itself complete has already stopped doing the work.
In that respect, no different from a human practitioner.