Skip to content

GOOGLE 2026-10-05

Read original ↗

Open and Emergent Problems in Agentic Privacy and Security: A Contextual Angle

Summary

A Google Research workshop report (co-authored with 50+ academic and industry participants from the late-2025 Contextual Agent Privacy and Security (CAPS) Workshop in New York) that frames agentic AI privacy and security around the theory of Contextual Integrity (CI) — the idea that privacy is "appropriate information flow" governed by established social norms, not secrecy or access control. The report's central argument is that autonomous agents, unlike deterministic software, combine broad access to personal data with the ability to take consequential actions, under three new conditions — unstructured natural-language inputs (hence prompt injection), probabilistic generative control flows, and autonomy/delegation (hence "confirmation fatigue") — so appropriateness must be reasoned about and enforced at runtime. The proposed answer is a multi-layered approach: a contextual policy engine inside a supervisor layer that evaluates whether a data flow or action is appropriate before it executes, complemented by system-level sandboxing, model-level appropriateness reasoning, user-centric controls, multi-agent collusion guardrails, and ecosystem-level norm governance — all measured by new multi-agent "Agent Gym" benchmarks. (Source: sources/2026-10-05-google-open-and-emergent-problems-in-agentic-privacy-and-security-a-contextual-angle)

Key takeaways

  1. Agents differ from traditional software on three axes that break the old playbook. (1) Unstructured interfaces and input ambiguity — natural language and images make "expected behavior" hard to define and open the door to adversarial manipulation like prompt injection. (2) Probabilistic control flows — generative planning yields execution paths traditional testing can't secure. (3) Autonomy and delegation — long tasks and sub-agent delegation make per-action human oversight insufficient, risking "confirmation fatigue." (Source: sources/2026-10-05-google-open-and-emergent-problems-in-agentic-privacy-and-security-a-contextual-angle)

  2. Privacy is reframed as "appropriate information flow," not secrecy or access control. Grounded in Helen Nissenbaum's Contextual Integrity, a norm is defined by its actors (sender, recipient, subject), information types (e.g. medical, financial), and transmission principles (e.g. confidentiality, reciprocity). The report extends CI from information sharing to contextual security — the appropriateness of an agent's actions, not just its disclosures. Illustrative example: you'd share a gift-shopping list with a shopping assistant but not with family and friends.

  3. LLMs make machine-readable, context-dependent policy feasible for the first time. Historically there was a "semantic gap" between high-level norms ("protect my data while organizing my travel") and low-level system permissions; manual permissions and expert-written policies "cannot scale" to many concurrent long-running agent tasks. Because models now begin to understand CI, the gap can be bridged with machine-readable policy — related to the wiki's policy-as-data idea.

  4. The proposed enforcement primitive is a contextual policy engine in a supervisor layer, evaluated before any data leaves the workspace. A dynamic policy-generation loop operates in real time to tailor policy to the user request and to open-ended, dynamic contexts — including new tools and capabilities discovered at runtime — and "evaluate whether a requested data flow is appropriate before any information leaves the user's workspace." This is the Google-Research framing of an out-of-band / pre-execution policy boundary (cf. the wiki's Out-of-Band Policy Engine and central-proxy choke point).

  5. Defense must be multi-layered across the whole stack. The report lays out five layers: (a) system-level sandboxing — dynamically limiting agent capabilities, establishing agent identity, and authorizing/revoking data access based on context changes (vs. OS-style static sandboxing), mapping to the wiki's capability-based sandbox; (b) model-level reasoning — disambiguating under-specified prompts and judging appropriateness under changing norms; (c) user-centric controls — replacing "Notice and Choice" (which assumes foreseeability) with dynamic, contextual, personalized controls matching user mental models; (d) multi-agent interactions — guardrails to stop agents colluding to violate norms (a new failure mode for multi-agent systems); (e) ecosystem governance — collecting/sharing norms across agents and resolving conflicts, negotiating norm changes, and verifying compliance.

  6. "Notice and Choice" is declared obsolete for agents. The traditional consent framework assumes individuals can make fine-grained, foreseeable decisions about each action on their data; autonomous agent behavior is "inherently probabilistic, high volume, and generative," so static per-action consent collapses — the same force that produces confirmation fatigue in human-in-the-loop designs.

  7. Measurement needs dynamic, multi-agent benchmarks ("Agent Gym"). The report calls for standardized, open-source sandbox environments that simulate complex, cascading multi-agent interactions over extended periods, to establish a shared privacy/security/safety baseline across academia and industry — static, single-turn safety evals don't capture the emergent risks.

  8. Call to action, not a shipped system. This is a research-agenda / workshop report (open problems + proposed directions), not a product launch or a benchmarked production system. Its value here is the framing — Contextual Integrity as the organizing lens for agentic appropriateness — and the explicit multi-layer decomposition of where enforcement must live.

Systems / concepts / patterns extracted

  • Concept — Contextual Integrity (NEW): privacy/appropriateness as norm-governed information flow (actors × info-types × transmission principles); extended here to "contextual security" of actions.
  • Concept — prompt injection: the concrete manifestation of "unstructured interfaces and input ambiguity."
  • Concept — AI agent guardrails: the model-level appropriateness-reasoning layer; the report argues in-model guardrails alone are insufficient and must be paired with an out-of-band boundary.
  • Concept — capability-based sandbox: the "system-level sandboxing" layer, generalized from static to dynamic context-driven capability grant/revoke.
  • Concept — human-in-the-loop: the "autonomy and delegation" axis and "confirmation fatigue" failure mode.
  • Concept — policy-as-data: machine-readable, context-dependent policy the policy engine evaluates.
  • Concept — least-privileged access / governed agent data access: the access-scoping the sandbox + policy engine enforce by context.
  • Concept — defense-in-depth: the overall multi-layer stance (system + model + user + multi-agent + ecosystem).
  • System — out-of-band policy engine (cross-vendor sibling): the report's "contextual policy engine / supervisor layer" is the Google-Research research-agenda statement of the same pre-execution, outside-the-agent enforcement boundary.
  • Pattern — specialized agent decomposition: multi-agent collaboration, here with the new concern that agents may collude to violate contextual norms.
  • Pattern — central-proxy choke point: the supervisor layer sits in the path of every action/data flow.

Operational numbers / specifics

  • 50+ academic and industry co-authors / participants from the CAPS Workshop (New York, late 2025).
  • CI norm decomposition: 3 components — actors, information types, transmission principles.
  • Proposed defense decomposition: 5 layers — system sandboxing, model reasoning, user controls, multi-agent interactions, ecosystem governance.
  • Agent difference axes: 3 — unstructured input/ambiguity, probabilistic control flow, autonomy/delegation.
  • No performance, latency, throughput, or benchmark-score numbers — this is a framing/agenda report, not a measured system.

Caveats

  • Research agenda, not a system. No shipped architecture, no mechanism depth on how the dynamic policy-generation loop computes policies, composes overlapping norms, or bounds latency; no evaluation numbers. Treat all "contextual policy engine / supervisor layer" claims as proposed directions.
  • CI is a social-science theory operationalized by LLMs. The core bet — that models can reliably understand contextual norms well enough to generate and enforce correct policy — is unproven at scale and is itself a probabilistic component, which (per the wiki's out-of-band policy engine sources) is the exact property that argues enforcement should be deterministic and out of band. The report acknowledges the model layer must be complemented, not trusted alone.
  • "Agent Gym" is aspirational. The multi-agent benchmark environments are called for, not delivered.
  • Scope note. Included (Tier-1 Google Research) because the body is

    20% real system-design content — a layered enforcement architecture (supervisor layer, policy engine, dynamic sandboxing, multi-agent guardrails, pre-execution data-flow gating) — not pure ML research. The CI theory framing is the organizing lens, but the substance is agentic security architecture.

Source

Last updated · 854 distilled / 2,448 read