According to Gizmodo, companies like GoGuardian and Lightspeed Systems now monitor the majority of American K-12 students’ interactions with AI chatbots through software installed on school-provided devices. In Los Angeles Unified School District alone, about 96% of elementary students received take-home laptops during COVID, creating a massive monitoring infrastructure. Lightspeed Systems revealed that Character.ai triggered 45.9% of flagged conversations, while ChatGPT accounted for 37%, with examples including students asking about self-harm methods and gun usage. Julie O’Brien of GoGuardian noted that AI chat monitoring comes up in “about every meeting” with customers now. The Electronic Frontier Foundation has criticized these systems for targeting normal LGBTQ behavior and doing “more harm than good,” while a study showed 6% of educators have been contacted by immigration authorities due to monitoring software alerts.
The surveillance dilemma
Here’s the thing: we’re creating this weird digital panopticon where kids can’t even have private conversations with AI. The monitoring works by having bots scan everything with natural language processing, then feeding suspicious content to human moderators who decide whether to alert school officials—who might then involve police. But is this actually helping kids? Or just creating a generation that knows they’re constantly being watched?
And let’s be real—the same companies that were getting slammed by privacy advocates last year are now positioning themselves as saviors protecting kids from dangerous AI conversations. It’s convenient timing, isn’t it? Suddenly there’s a new justification for the same invasive technology.
What the studies actually say
The research on monitoring teens is pretty damning. A University of Central Florida study found that parents who used monitoring apps were more likely to be authoritarian, and their teens were actually more likely to be exposed to explicit content and bullying. Another Dutch study discovered that monitored teens became more secretive and less likely to ask for help. Basically, constant surveillance seems to backfire spectacularly.
Now we’re applying this failed approach to school devices, except instead of just parents watching, you’ve got corporations and school administrators involved. And when kids turn to chatbots for advice they’re too uncomfortable to seek from humans, that “help-seeking” behavior gets flagged as problematic.
The slippery slope we’re on
This feels like we’re building the infrastructure for permanent student surveillance. First it was web filtering, then social media monitoring, now AI chatbot surveillance. What’s next? These companies have a business incentive to find more things to monitor, more “risks” to protect against.
The really troubling part? We’re normalizing this level of surveillance for an entire generation. Kids are growing up thinking it’s normal for every digital interaction to be monitored, analyzed, and potentially reported to authorities. That’s going to have consequences we can’t even imagine yet.
Meanwhile, schools are stuck between legitimate concerns about student safety and creating an environment where kids feel constantly watched. There’s no easy answer here, but turning education into a surveillance state probably isn’t it.
