AI, user data and the asymmetry of understanding

Each time users belatedly discover that a man-made intelligence feature has been drawing on their data in ways they didn’t fully grasp, the response is commonly an instinctive sense of violation – of trust, consent and privacy.

Accusations and outrage have all the time followed potentially invasive AI integrations, with examples starting from email content used to tell model training and large on-device models embedded in on a regular basis software to voice assistants retaining snippets beyond explicit commands and default settings that enable cross-product activity to tell AI responses.

Even when such changes are technically disclosed, awareness doesn’t necessarily follow. Updates arrive one after one other, and settings default to “on,” putting the onus on users to navigate a labyrinth they never asked for. The cognitive gap between what organizations understand about how their systems use data and what individuals can reasonably expect to grasp seems to widen each day.

Most users don’t mind disclosing chat, clickstream or location history if it serves their purpose, but firms on the opposite side may even see training data, embeddings, personalization signals, safety-tuning inputs, fraud-detection features and future product capabilities in those messages.

Regulators are already acknowledging how upstream data decisions persist downstream. In late 2024, the European Data Protection Board updated its opinion on problems with anonymity, legitimate interest and AI models trained on unlawfully processed personal data, noting that this could affect whether such models may be lawfully deployed unless properly anonymized. The U.K.’s Information Commissioner’s Office also stresses the necessity for organizations to elucidate AI-assisted processes and decisions to those affected.

Burden on the user

 s it realistic to expect individuals to reverse-engineer opaque data ecosystems from privacy notices? Most individuals are simply attempting to use products. To think in regards to the downstream flow of their data, its implications and routes is overwhelming, to say the least.

In practice, the obligations should fall more heavily on firms. They design systems and understand their downstream uses. Also they are the one actors positioned to scale back the complexity on the source. Meaningful transparency can’t be simply reduced to shorter privacy policies; it must be contextual, specific and genuinely actionable.

This isn’t only a theoretical concern. Across the General Data Protection Regulation, ICO guidance and the EU AI Act, there’s a recurring recognition that transparency must transcend disclosure to turn out to be something people can actually understand and act on. Additionally they push for explanations covering how data is used, who’s responsible, and what consequences follow.

The EU AI Act is adding further transparency duties for certain AI systems, aimed toward helping users recognize once they are interacting with AI or exposed to AI-generated content in order that they will make informed decisions.

The catch in ‘manage your preferences’

Privacy responsibility is ceaselessly redistributed toward users through interface design and rhetoric. The tendency to confer an impression of “control” through settings and toggles is more likely to persist in a single form or one other, including dark patterns which will proceed to lurk inside interfaces. It’s a low-friction way for systems to signal compliance and user empowerment without actually changing the underlying distribution of power or reducing organizational discretion.

The first responsibility should rest with those that shape the system’s architecture. Users should still have rights and controls, but firms are those deciding the defaults, retention periods, data flows, vendor relationships, and increasingly how models behave in practice.

That is ultimately an issue of where responsibility is placed in systems that not follow easy, linear paths. If privacy risk is structural, it will possibly’t be administered through settings and preferences alone. Accountability should be on the architectural level because that’s where real decisions are made.

Onur Alp Soner is the co-founder and CEO of Countly Ltd, a digital analytics and in-app engagement platform. He wrote this text for SiliconANGLE.

Image: Wikimedia Commons