We bound the clinician. We never bound the tool the patient trusts most.
When I tell people I am involved in healthcare AI, it feels like they need to confess to me how they use AI. I’m sure the same is true for many of you. Just a few weeks ago, a neighbor told me, “I tell my ChatGPT friend everything — my labs, my meds, my health problems, including the ones I haven’t even told my doctor yet.” After I urged him to talk to his own doctor, he said that sharing how he actually uses AI gave him some relief. Like he was confessing some hidden vice. He was relieved, but I lost sleep.
Here is what he did not know. When you pour your health into a consumer AI tool, you are not whispering to a physician. There is no HIPAA in that room.
The conversation is not his. It can be stored, used to train the next model through a setting buried in a policy no one reads, swept up in a breach, or summoned in a lawsuit. He thought he was in a confession booth. He was standing in a room he could not see, speaking out loud.
That is not his failure. No one told him the rules had changed.
In my work in healthcare AI governance, this is the question we keep skipping. Not whether AI can help the patient. It can. The question is who owns the conversation.
Convenience answered first. Governance never showed up. We built a tool intimate enough to hold a person’s deepest fears, and never decided who holds the record afterward. And the clinicians who hold to the strictest privacy standard in medicine now watch their patients carry that data straight out the door.
Sovereignty over the body has to include sovereignty over the story being told about it.
Before we send one more patient home to “go ask the AI,” we owe them one honest sentence:
You may be the only one in that conversation who thinks it is private.