Skip to content

CONCEPT Cited by 1 source

Audit trail in OpenSearch

Keeping a permanently-retained audit log of security- sensitive operations in a long-lived search cluster (Fly.io uses OpenSearch) as a load-bearing architectural layer — not just a compliance checkbox.

Fly.io's framing:

"We also have a complete audit trail for token operations, in a permanently retained OpenSearch cluster index. Since virtually all the operations that happen on our platform are mediated by Macaroons, this audit trail is itself pretty powerful." (Source: sources/2025-03-27-flyio-operationalizing-macaroons.)

Why permanent retention matters

Token operations are the chokepoint through which almost every action on the Fly.io platform passes. A searchable, permanent index of those operations answers forensic questions that logs (often rolled or sampled) and metrics (aggregated, detail- lossy) cannot:

  • "When was this token first used, and from where?"
  • "What other tokens has this user ever been issued?"
  • "What operations did a compromised service token perform before we revoked it?"

The "virtually all operations are mediated by Macaroons" property is what makes this audit trail general-purpose — it's effectively an audit log for the whole platform, not just for the token system.

Design choices implied

  • Retention policy: permanent (not time-windowed).
  • Storage backend: OpenSearch — a searchable index, not just append-only logs. Forensic questions need arbitrary query shapes.
  • Write path: every token operation emits an audit record; write-path failures need to be treated as security incidents.

Contrast with aggregated metrics

Metrics tell you how many logins failed last week. Audit trail tells you which user, from which IP, with which client, against which token, at which timestamp. Different question, different storage primitive.

Seen in

Last updated · 200 distilled / 1,178 read