Behavioral hallucination: the problem no one is naming
Everyone knows chatbots make things up. A date, a source, a fact that doesn’t exist – that’s what’s known as factual hallucination, and it’s become the most talked-about problem in conversational AI. Rightly so: a chatbot that invents a legal precedent or a medical dosage is dangerous.
But there’s another type of hallucination. Less visible, less documented, and in some contexts – potentially more damaging.
A chatbot that tells you “I understand” when it hasn’t understood anything. That rephrases what you just said in a cleaner, more reassuring version – and gives you the impression you’ve made progress when nothing has moved. That takes “I feel better” as proof that something has changed. That validates a passing sense of relief as though it were a resolution.
I call this behavioral hallucination: the chatbot doesn’t fabricate a fact – it fabricates progress.
For a chatbot that gives recipes or summarizes articles, this doesn’t matter. For a tool that works on the way a person sees their situation – it’s the central problem.
When I built the Conseiller IA, the question was never “how to stop the model from inventing facts.” That’s a solved problem, or at least a known problem with known solutions. The real question was: how to stop the system from fabricating the illusion of a shift in perspective that never actually happened.
What “I feel better” can actually mean
Most conversational AI systems treat every message as a reliable signal. If the person says something positive, the system moves forward. If they say they’ve understood, the system takes it at face value.
Anyone who has ever accompanied someone through a real process knows it’s far more complicated than that.
A positive signal can be genuine. It can also be many other things. Here are the most common forms the Conseiller IA has learned to identify – not to invalidate them, but to avoid taking them at face value without verification.
1. The sudden, clean resolution. Everything clicks at once, no grey area, no residue. The person goes from “I’m stuck” to “it’s clear” in a single message. When this happens in life, it’s almost always more nuanced than that. When it happens in a chat, it’s often a sign that the person has found what the system wanted to hear.
2. Borrowed language. The person starts talking like the chatbot. They use its phrasing, its structure, its words. This isn’t a shift in perspective – it’s an adoption of register. The person has understood how the system works, not how their situation works.
3. Intellectualized positivity. “Yes, I can see it’s a question of boundaries.” “Indeed, I understand it’s linked to my relationship with control.” The analysis is accurate. The person isn’t affected by it. Understanding a mechanism and experiencing it are two different things – but a standard chatbot doesn’t make the distinction.
4. Relief mistaken for resolution. This is the most common. Talking helps. Feeling heard helps. Putting words on something helps. But relief is not change. The pressure drops, the situation stays the same. A chatbot that can’t tell the difference validates incorrectly.
5. Positivity as an exit strategy. The person wants to end the conversation. The shortest path is to say things are better. It’s not a lie – it’s a social signal that humans recognize instinctively and chatbots take at face value.
6. Premature forgiveness or acceptance. “I’ve decided to let it go.” “I forgive them.” Sometimes it’s real. Often it’s a construct that allows the person to close the subject without having gone through it. The speed at which it happens is an indicator – but one that most systems don’t monitor.
7. Refusal of verification. The person resists when the system tries to check whether the change is grounded. This isn’t failure – it’s a signal. But a standard chatbot interprets resistance as confirmation: “the person knows what they want, I’ll validate.”
These seven signals are the most visible. They’re not the only ones. But they illustrate a problem that most systems have no mechanism to address.
A good prompt is not enough
The most common response to the hallucination problem is to write better instructions. “Don’t fabricate information.” “Stay factual.” “Don’t validate prematurely.” And it helps – a little.
But a language model is still a language model. It optimizes for surface coherence. If the conversation moves toward positive territory, it follows. If the person phrases something that sounds like a conclusion, it concludes. Not out of malice, not by mistake – by design. That’s what it does.
Writing “don’t validate too quickly” in a prompt means asking the model to resist its own mechanics. It works sometimes. Not often enough to rely on when what’s at stake is the way someone sees their life.
The Conseiller IA is built on a different principle: the model never has the final say. There are multiple layers of control, and they’re not all in the prompt.
Some act upstream – they determine what the model is allowed to do at each stage, before it even begins to formulate a response. Others check what the model has produced before it reaches the person. And a separate set acts on memory itself – on what the system retains, how it retains it, and under what conditions it agrees to rely on it.
A few principles, without going into implementation details:
The system never measures a shift without first having established a starting point. No shortcut possible.
The system’s memory is factual. It’s reconstructed at regular intervals according to strict rules – no interpretation, no embellishment, as close as possible to what the person actually said.
When a shift in perspective turns out to be fragile or premature, the system doesn’t just go back. It puts itself in a position to re-verify without being influenced by what it thought it had observed the first time.
There are safeguards against loops: the system knows how to recognize when it’s going in circles, and it chooses honest impasse over false progress.
And there are other layers still, that I won’t detail here – because their design is precisely what makes the product valuable.
Each of these mechanisms is insufficient on its own. It’s their interlocking that creates resistance to drift – and it’s that interlocking that took the longest to build.
Why technology alone isn’t enough either
Everything above is engineering. But engineering alone doesn’t solve the problem.
Knowing that relief isn’t resolution – that’s the knowledge of someone who accompanies people, not a developer. That borrowed language is a sign of adaptation, not change – that doesn’t come from technical documentation. And that forgiveness can be a closing construct rather than a genuine movement – you learn that sitting across from someone, not in front of a screen.
The seven signals described above weren’t found in the LLM literature. They come from accompaniment. From what happens when a real person tries to see their situation differently – and from everything that can look like change without being change.
The Conseiller IA is not a chatbot with guardrails bolted on. It’s a system built from an understanding of what accompaniment demands – and then translated into a technical architecture capable of holding it.
That combination is what took the most time. Not the individual mechanisms – their calibration. Knowing when the system should resist and when it should let go. Recognizing the moment where honest impasse is better than false progress. It’s not a one-time adjustment. It’s a balance that’s worked on continuously, conversation after conversation, with monitoring that watches not whether the system “works” in the technical sense, but whether it remains accurate in its reading of what’s happening.
None of this guarantees perfection. A system can be rigorous and still miss something. A person can go through a real change that the system takes too long to recognize, or signal something fragile that the system lets through.
What these mechanisms guarantee is that the system does the work. That when something moves in a conversation with the Conseiller IA, it’s because it verified that something really moved – not because it fabricated the illusion that it did. And if nothing moved, it says so.
This system is not a finished product. It evolves continuously – because a system of accompaniment that considers itself complete has already stopped accompanying.
In that respect, no different from a human practitioner.