Your AI Knows What You're Thinking. Should It Tell Anyone?
Florida just became the first state to sue OpenAI. The headline is about child safety and deceptive marketing — and that's a real conversation worth having. But buried in the announcement is something that's going to matter a lot more long-term: prosecutors reviewed chat logs between ChatGPT and a gunman before the Florida State University shooting last year.
That's the part that should make you stop and think.
Where Everyone Agrees
If someone tells an AI "I'm going to shoot up a building tomorrow," and two days later two people are dead — yeah, we're going to ask hard questions about why nothing was flagged. That's not unreasonable. Nobody wants to live in a world where we shrug at that.
Where It Gets Complicated
Here's the problem: drawing that line is harder than it sounds.
People say dark things to AI that they'd never say to another person. Not because they're dangerous — because they're processing. Grief. Rage. Suicidal thoughts they're fighting through. Violent fantasies they'll never act on. The AI is a pressure valve, and it works precisely because it doesn't judge and doesn't tell anyone.
The moment you build a surveillance pipeline into that relationship, you change it fundamentally. People stop being honest. And ironically, that makes things worse — you've just pushed those conversations back into isolation, where they can actually fester.
The "Where Do You Draw the Line" Problem Is Real
Let's say OpenAI is now required to flag "concerning" conversations. Great. Who decides what "concerning" means?
- Today it's credible, specific threats. Fine.
- Next year it's expressions of "extremist ideology." Okay, which ones?
- Five years from now it's anything that scores above a threshold on an automated risk model trained on... what, exactly?
This isn't paranoia — it's pattern recognition. Every surveillance system in history has started with a use case that seemed obviously justified, and expanded from there. The technology doesn't create that tendency; human institutions do. But AI changes the scale and the speed at which it can happen.
The 1984 Comparison Isn't Hyperbole
Orwell's nightmare wasn't just that the government watched you. It was that the possibility of being watched changed how you thought and what you said. You didn't need to actually be surveilled — the infrastructure was enough.
If your AI therapist, your AI confidant, your AI sounding board might be logging anything you say for review — you're not going to say the thing you actually need to say. You're going to perform wellness instead of achieving it.
So What's the Answer?
Honestly? I don't have a clean one. But I think:
- Truly imminent, specific threats should probably be reportable — with a very high bar and a clear legal framework, not left to private companies to decide
- The decision shouldn't sit with a trust & safety team operating under political pressure
- Any reporting framework should require a warrant, not just an algorithm
- And we should be deeply skeptical of any system that makes it easier to surveil, even when the first use case feels justified
The Florida lawsuit will mostly get argued on child safety grounds. But the precedent it sets for AI companies as surveillance infrastructure — that's the story worth watching.
Thoughts? Genuinely curious where people land on this one. I'm at chris@theanchorlight.com.
