Skip to content

Toward provably private insights into AI use (Google Research, 2025-10-30)

Summary

Google Research introduces Provably Private Insights (PPI): a production serving-infrastructure pattern for answering aggregate analytical questions ("what topics do Recorder users record?", "is the user frustrated?", "how accurate are on-device summaries?") about real-world use of on-device GenAI features without the server operator ever seeing raw per-user data. The system composes four primitives into one pipeline — an LLM running inside a TEE performs structured summarization on encrypted user uploads; outputs are aggregated by a differentially-private histogram algorithm; the decryption key is released only to pre-approved, attested, reproducibly-buildable processing steps; and every piece of server-side code is published to a tamper-evident transparency log so third parties can verify the pipeline end-to-end. Deployed in the Recorder app on Pixel on an open-source Gemma 3 4B model hosted in an AMD SEV-SNP TEE, attested via Google's Project Oak stack, with code + workflow signatures published under Google Parfait. Extends the confidential federated analytics (CFA) shape originally shipped in Gboard to arbitrary LLM-over-unstructured-data analyses. User-level DP guarantee bound at ε = 1 for the published Recorder histogram.

Key takeaways

  1. LLMs are a first-class analysis primitive inside the confidential-computing boundary. The novelty is not TEEs and not DP — both predate this post. The novelty is "the analyst is a Gemma 3 4B instance running in an AMD SEV-SNP TEE" that replaces hand-written classifiers / regex / count aggregations over unstructured data (transcripts, free-form prompts, summaries). Structured summarization lets one pipeline answer many questions by just changing the LLM's prompt.

  2. The DP guarantee is decoupled from the LLM prompt. "The LLM's prompt can be changed frequently, because the DP guarantee applies to the aggregation algorithm regardless of the LLM prompt. Even if the developer asked a question designed to single out one user, the differentially private statistics would not reveal it." Architecturally this is per-user structured summarization → per-user sensitivity bound = 1 → differentially-private histogram with calibrated noise. The LLM is an oracle producing bounded outputs; the DP algorithm upper-bounds how much any one oracle answer can perturb the published result. The operator doesn't have to re-prove DP every time the prompt changes.

  3. Attested keystore release is the load-bearing primitive bridging per-device encryption to per-query processing. Each device encrypts its upload + the processing steps it authorises. The key management service (itself a TEE running published code) decrypts only for workflows whose workflow signatures match device-authorised ones and whose code is attested to be running in a properly configured TEE. "No other analyses can be performed on the data and no human can access data from individual devices." This is attested keystore release: the key never exists outside a TEE and never flows to code the device hasn't pre-approved.

  4. End-to-end verifiability rests on a chain of four attestations: (1) the key management service code matches a published open-source build in Rekor; (2) the KMS is running in a properly configured TEE inaccessible to Google; (3) the data-processing steps (LLM + DP aggregator

  5. TEE stack) are themselves reproducibly built and running in TEEs; (4) the workflow signatures authorising decryption are public. "Every part of the data processing pipeline can be verifiably linked to published code." (Source: sources/2025-10-30-google-toward-provably-private-insights-into-ai-use).

  6. Confidential Federated Analytics (CFA) is the substrate, not an analytics product. The post frames PPI as a new application on top of CFA, which itself first shipped in Gboard for discovering new words. CFA runs open-source analysis software in TEEs so that the mechanisms + privacy properties of server-side data processing can be externally inspected. PPI is the LLM-powered structured- summarization workflow on CFA. Google names differentially private clustering and synthetic data generation as follow-on workflows targeting the same substrate.

  7. Reproducible builds are a precondition for attestation to mean anything. The post names the requirement explicitly: "All privacy-relevant parts of our system are open source and reproducibly buildable — from the private aggregation algorithm to the TEE stack." Without reproducibility, the TEE-attested hash could match the published code's hash only if one party controls both the build and the TEE deployment — which is precisely the trust assumption PPI is trying to remove. Reproducibility is what lets external third parties rebuild from source, get the same hash, and verify the attestation chain themselves.

  8. Deployment profile in Recorder pins concrete primitives. Opt-in via Recorder's "Improve for everyone" setting → device encrypts transcript with a public key managed by a central TEE-hosted keystore → keystore is protected by Project Oak attestation stack running on AMD SEV-SNP CPUs → a Gemma 3 4B model in the same TEE classifies the transcript into topics → outputs are aggregated with differential privacy → only DP sums of output categories are released to Google. Example topics categorised in the published figure: "Notes to self", "Reminders", "Business meetings". Published ε = 1 (user-level).

  9. Evaluation of on-device models becomes possible without sacrificing privacy. A second workflow runs an LLM auto-rater inside the TEE that scores the on-device model's outputs (e.g. summary accuracy) against real user data — auto-rater outputs themselves get DP aggregated. This closes the can't-eval-on-real-data gap that normally forces on-device ML teams onto synthetic data. The auto-rater is a different prompt on the same substrate — one CFA substrate, N workflows via prompt change.

  10. Known weaknesses are acknowledged, not hidden. Post links to arXiv:2506.15924 for "known weaknesses in current-generation TEEs." The security argument is "subject to TEE limitations", not "TEEs are perfect." This is the honest posture required when the trust chain terminates at hardware whose side-channel / cold-boot / fault-injection attack surface is still an active research area.

  11. Future work extends the substrate, not the application. Google flags TPU confidential computing as the next expansion axis: higher-throughput accelerators inside the TEE enable "richer analyses… including detailed transcript analysis and auto-rating." Today AMD SEV-SNP hosts a 4B model on CPU-ish compute; tomorrow confidential TPUs would host much larger models inside the same trust boundary. The LLM + DP + attestation architectural shape doesn't change; the capability ceiling inside the boundary rises.

Systems / concepts / patterns introduced

Systems: systems/gemma-3, systems/google-parfait, systems/project-oak, systems/amd-sev-snp, systems/google-recorder, systems/gboard, systems/rekor (Sigstore transparency log reused outside software-release context), systems/tensorflow-federated (the private aggregation algorithm lives in tensorflow-federated C++ aggregation code).

Concepts: concepts/confidential-computing, concepts/trusted-execution-environment, concepts/differential-privacy, concepts/federated-analytics, concepts/remote-attestation, concepts/reproducible-builds, concepts/structured-summarization, user-level DP (ε = 1 as the published Recorder operating point), privacy budget (amortised across the DP histogram's queries).

Patterns: patterns/llm-in-tee-summarization, patterns/dp-aggregation-histogram, patterns/end-to-end-verifiable-pipeline, patterns/attested-keystore-release.

Operational numbers

Caveats

  • Raw file is near-empty (13 lines). The scraped raw markdown captured only the frontmatter + Acknowledgements. Body content summarised on this page was pulled from the article URL per AGENTS.md URL-verbatim rules. No other content (per-operation QPS, TEE cold-start cost, Gemma-3-4B inference latency inside SEV-SNP, DP noise-scale calibration details, per-region deployment topology, KMS failover) is on the page — it is described at what level, not how fast / how much level.
  • Whitepaper (arXiv:2510.21684) is the source for every quantitative claim beyond what's on the blog — DP parameter selection, TEE cold-start, prompt-to- histogram latency, KMS decryption throughput. Wiki pages are scoped to what the blog + linked open-source artifacts verifiably state.
  • User-level ε = 1 is the one number given. The post does not quote per-topic noise scales, the number of users contributing to the published figure, the sensitivity bound on LLM outputs, or the privacy-budget amortisation across multiple queries.
  • TEE weaknesses are acknowledged but not decomposed. The arXiv link covers them; the blog post itself only says "subject to known weaknesses in current-generation TEEs."
  • Confidential TPUs are future work, not current state. The "richer analyses" claim is aspirational; current deployment runs on AMD SEV-SNP CPU TEEs.
  • LLM auto-rater is mentioned as a second workflow but not quantified. No accuracy numbers, no comparison to synthetic- data-only evaluation, no labelled-data-gap quantification.

Source

Last updated · 178 distilled / 1,178 read