Skip to content

SYSTEM Cited by 3 sources

Backstage

Backstage is an open-source Internal Developer Portal (IDP) framework originally built by Spotify and now a CNCF incubating project. It gives engineering organisations a single web surface to catalogue services, maintain software templates, wire in CI/CD and observability, and navigate ownership / dependency graphs — the "single pane of glass" shape that platform-engineering teams are increasingly expected to produce.

Minimum viable framing on this wiki

Backstage is state-heavy. The Software Catalog alone holds every registered service, API, resource, system, domain, and group plus the edges between them; the TechDocs surface indexes per-service documentation; the Search surface indexes every catalog entity; Scaffolder holds software templates; add-ons (Kubernetes plugin, ArgoCD plugin, PagerDuty plugin, cost plugin) each add their own state. Out of the box Backstage persists most of this in Postgres (the default is standard Postgres / RDS) using Knex migrations. Its default search is

Portal by Spotify is the commercial, hosted evolution of Backstage. Its AiKA Modes feature — declarative agents on an ephemeral runtime — is the substrate behind Spotify's coding-agent token-routing story (systems/portal-by-spotify, systems/shunt-claude-code-plugin, sources/2026-09-03-spotify-portal-by-spotify-cut-my-claude-code-token-usage-by-90). in-memory; for anything real, teams swap in PgSearchEngine so search hits the same Postgres database.

This combination — notoriously fragile schema migrations on top of meaningful state — is why Thoughtworks picked Backstage as the migration-stress-test for Lakebase in the 2026-04-30 POC. Verbatim: "we chose Backstage for this POC because its schema migrations are notoriously fragile and it seemed like a perfect opportunity to test out a Lakebase integration."

Postgres-compatibility as integration property

Swapping Backstage's database backend for Lakebase (Neon-lineage serverless Postgres) required:

  1. Pointing app-config.yaml at Lakebase's endpoint (hostname + database + credentials).
  2. Swapping Backstage's default in-memory search for PgSearchEngine.
  3. Working around Lakebase's rejection of classic Databricks PATs by wrapping databricks postgres generate-database-credential in a 50-minute-cadence cron that rewrites DATABRICKS_TOKEN in .env (see credential-refresh-cron-as-auth-compat-shim).

Knex migrations ran cleanly against Lakebase. The integration surfaced no Postgres-compatibility gaps at application altitude — the entire friction was at the auth tier (short-lived JWT vs Backstage's expectation of long-lived credentials).

Catalog size at POC: ~63 MB (branchable in 1.09 s — see sources/2026-04-30-databricks-backstage-with-lakebase).

"IDP one layer up" — the AI-tool-registry framing (MongoDB, 2026-05-11)

MongoDB Engineering's 2026-05-11 Fighting Tool Sprawl position post (sources/2026-05-11-mongodb-fighting-tool-sprawl-the-case-for-ai-tool-registries) explicitly frames the enterprise AI tool registry as "an Internal Developer Portal (IDP) built for the agent era, solving the same coordination problem that IDPs solved for service teams, but one layer up." The structural carryover from Backstage's pattern: same four functions (discovery + versioning + ownership / certification metadata + access control); same dual-surface property (web UI for humans + API for machine consumers); same federation property (heterogeneous entity types in one catalog). The differences are at the consumer + entity layers: Backstage catalogs services humans operate, with deployment / observability tooling as machine consumers; an AI-tool-registry catalogs tools agents call, with the agents themselves as machine consumers querying at request-time speed.

This makes Backstage the canonical wiki IDP-shape ancestor for the patterns/on-behalf-of-agent-authorization pattern — and the post's argument that "platform teams already understand why IDPs exist" is precisely a claim that Backstage's empirical validation transfers to the agent-era extension. See tool-sprawl for the failure mode the extended pattern addresses, and enterprise-ai-tool-registry for the operational shape.

Backstage as context for agents (Spotify, 2026-06-03)

Spotify — Backstage's original creator — describes using Backstage as context for both humans and agents. Before Backstage, Spotify had roughly a hundred fragmented internal tools (one for deployments, one for CI, one for A/B tests); Backstage consolidated them into a single Software- Catalog-centered pane of glass. In the AI-coding transition Spotify now exposes Backstage's capabilities as MCPs and command-line tools so Claude can look up who owns a component, read its documentation, or ping the responsible team on Slack — i.e. the Software Catalog becomes an agent-queryable context/tool surface, not just a human UI. This is the canonical Spotify instance of patterns/on-behalf-of-agent-authorization and a concrete supply of the consistent context behind codebase-consistency-improves-agent-performance. Spotify also layers Soundcheck / golden state on Backstage to drive standardization, which — with static analysis + linting — becomes an active guardrail correcting agents as well as humans (see lint-feedback-loop-corrects-agents). Related Spotify systems distributed via Spotify Portal for Backstage include Fleetshift and Honk. (Source: sources/2026-06-03-spotify-coding-is-no-longer-the-constraint-scaling-developer-experie-35c3855d)

Thoughtworks Tech Radar endorsement

Thoughtworks's Technology Radar has consistently advocated Backstage as the IDP foundation. The 2026 Backstage-on-Lakebase POC is in part an exploration of whether a serverless-Postgres substrate removes the operational pain points that keep some Backstage adopters on shared staging environments and mocked tests rather than per-developer isolated instances.

Seen in

  • sources/2026-06-03-spotify-coding-is-no-longer-the-constraint-scaling-developer-experie-35c3855d — Backstage from its creator, as agent context. Spotify frames Backstage as the consolidation of ~100 fragmented internal tools into a single Software-Catalog pane of glass, then exposes its capabilities as MCPs + CLI tools so coding agents (Claude) can resolve ownership, read docs, and ping teams — the patterns/on-behalf-of-agent-authorization shape. Also layers Soundcheck / golden state as a standardization guardrail, and distributes Fleetshift + Honk via Spotify Portal for Backstage.

  • sources/2026-05-15-databricks-backstage-with-lakebase-part-2 — Second canonical Backstage face: governance-substrate- unified IDP. Part 2 of the Thoughtworks series. The Backstage catalog (Postgres on Lakebase) is exposed as a foreign catalog lakebase_bs in Unity Catalog via Lakehouse Federation. Standard UC GRANTs control access (no Postgres-level role management); every control-plane action against the Backstage database lands in system.access.audit; cost attribution against UC system billing tables by (project_id, branch_id, endpoint_id); UC attribute-level masking policies propagate to every branch at creation time. The architectural payoff for state-heavy IDPs: the operational tier (Backstage's catalog DB) and the analytical tier no longer live in two security paradigms — "the compliance side-channel" of operational DBs collapses into a single substrate. Worked example: branch-lifecycle audit query from the Part 1 disaster-recovery experiment as one SQL statement against system.access.audit. Canonical instance of operational-analytical-governance-unification.

  • sources/2026-04-30-databricks-backstage-with-lakebase — canonical wiki ingest of Backstage. Thoughtworks migrates a POC Backstage install off standard Postgres / RDS onto Lakebase to demonstrate the effect of cheap branching + PITR on state-heavy-application developer cycles. The ~63 MB Backstage catalog becomes the dataset whose branch creation lands in 1.09 s data plane; Backstage's final_entities table becomes the demonstration target for PITR (32 rows → 0 → recovered in 3.78 s). Canonicalises Backstage as the representative state-heavy IDP application — why this POC is architecturally interesting is precisely that Backstage is not a toy workload.

Last updated · 766 distilled / 2,225 read