A feature Google demoed at its I/O confab yesterday, using its generative AI technology to scan voice calls in real time for conversational patterns related to financial scams, has sent a collective shiver down the spines of privacy and security experts who’re warning the feature represents the skinny end of the wedge. They warn that, once client-side scanning is baked into mobile infrastructure, it could usher in an era of centralized censorship.
Google’s demo of the decision scam-detection feature, which the tech giant said could be built right into a future version of its Android OS — estimated to run on some three-quarters of the world’s smartphones — is powered by Gemini Nano, the smallest of its current generation of AI models meant to run entirely on-device.
This is actually client-side scanning: A nascent technology that’s generated huge controversy in recent times in relation to efforts to detect child sexual abuse material (CSAM) and even grooming activity on messaging platforms.
Apple abandoned a plan to deploy client-side scanning for CSAM in 2021 after an enormous privacy backlash. Nonetheless, policymakers have continued to heap pressure on the tech industry to search out ways to detect criminal activity going down on their platforms. Any industry moves to construct out on-device scanning infrastructure could due to this fact pave the way in which for all-sorts of content scanning by default — whether government-led or related to a selected industrial agenda.
Responding to Google’s call-scanning demo in a post on X, Meredith Whittaker, president of the U.S.-based encrypted messaging app Signal, warned: “That is incredibly dangerous. It lays the trail for centralized, device-level client side scanning.
“From detecting ‘scams’ it’s a brief step to ‘detecting patterns commonly associated w[ith] in search of reproductive care’ or ‘commonly associated w[ith] providing LGBTQ resources’ or ‘commonly related to tech employee whistleblowing.’”
Cryptography expert Matthew Green, a professor at Johns Hopkins, also took to X to lift the alarm. “In the longer term, AI models will run inference in your texts and voice calls to detect and report illicit behavior,” he warned. “To get your data to go through service providers, you’ll need to connect a zero-knowledge proof that scanning was conducted. It will block open clients.”
Green suggested this dystopian way forward for censorship by default is barely a number of years out from being technically possible. “We’re just a little ways from this tech being quite efficient enough to comprehend, but only a number of years. A decade at most,” he suggested.
European privacy and security experts were also quick to object.
Reacting to Google’s demo on X, Lukasz Olejnik, a Poland-based independent researcher and consultant for privacy and security issues, welcomed the corporate’s anti-scam feature but warned the infrastructure could possibly be repurposed for social surveillance. “[T]his also implies that technical capabilities have already been, or are being developed to watch calls, creation, writing texts or documents, for instance seeking illegal, harmful, hateful, or otherwise undesirable or iniquitous content — with respect to someone’s standards,” he wrote.
“Going further, such a model could, for instance, display a warning. Or block the power to proceed,” Olejnik continued with emphasis. “Or report it somewhere. Technological modulation of social behaviour, or the like. It is a major threat to privacy, but additionally to a spread of basic values and freedoms. The capabilities are already there.”
Fleshing out his concerns further, Olejnik told TechCrunch: “I haven’t seen the technical details but Google assures that the detection could be done on-device. That is great for user privacy. Nonetheless, there’s way more at stake than privacy. This highlights how AI/LLMs inbuilt into software and operating systems could also be turned to detect or control for various types of human activity.
This highlights how AI/LLMs inbuilt into software and operating systems could also be turned to detect or control for various types of human activity.
Lukasz Olejnik
“To date it’s fortunately for the higher. But what’s ahead if the technical capability exists and is inbuilt? Such powerful features signal potential future risks related to the power of using AI to manage the behavior of societies at a scale or selectively. That’s probably amongst probably the most dangerous information technology capabilities ever being developed. And we’re nearing that time. How can we govern this? Are we going too far?”
Michael Veale, an associate professor in technology law at UCL, also raised the chilling specter of function-creep flowing from Google’s conversation-scanning AI — warning in a response post on X that it “sets up infrastructure for on-device client side scanning for more purposes than this, which regulators and legislators will desire to abuse.”
Privacy experts in Europe have particular reason for concern: The European Union has had a controversial message-scanning legislative proposal on the table since 2022, which critics — including the bloc’s own Data Protection Supervisor — warn represents a tipping point for democratic rights within the region as it will force platforms to scan private messages by default.
While the present legislative proposal claims to be technology agnostic, it’s widely expected that such a law would result in platforms deploying client-side scanning with a view to find a way to reply to a so-called detection order demanding they spot each known and unknown CSAM and in addition pick up grooming activity in real time.
Earlier this month, tons of of privacy and security experts penned an open letter warning the plan could lead on to thousands and thousands of false positives per day, because the client-side scanning technologies which are prone to be deployed by platforms in response to a legal order are unproven, deeply flawed and vulnerable to attacks.
Google was contacted for a response to concerns that its conversation-scanning AI could erode people’s privacy but at press time it had not responded.
We’re launching an AI newsletter! Enroll here to start out receiving it in your inboxes on June 5.
