Skip to content

SYSTEM Cited by 2 sources

Backstage

Backstage is an open-source Internal Developer Portal (IDP) framework originally built by Spotify and now a CNCF incubating project. It gives engineering organisations a single web surface to catalogue services, maintain software templates, wire in CI/CD and observability, and navigate ownership / dependency graphs — the "single pane of glass" shape that platform-engineering teams are increasingly expected to produce.

Minimum viable framing on this wiki

Backstage is state-heavy. The Software Catalog alone holds every registered service, API, resource, system, domain, and group plus the edges between them; the TechDocs surface indexes per-service documentation; the Search surface indexes every catalog entity; Scaffolder holds software templates; add-ons (Kubernetes plugin, ArgoCD plugin, PagerDuty plugin, cost plugin) each add their own state. Out of the box Backstage persists most of this in Postgres (the default is standard Postgres / RDS) using Knex migrations. Its default search is in-memory; for anything real, teams swap in PgSearchEngine so search hits the same Postgres database.

This combination — notoriously fragile schema migrations on top of meaningful state — is why Thoughtworks picked Backstage as the migration-stress-test for Lakebase in the 2026-04-30 POC. Verbatim: "we chose Backstage for this POC because its schema migrations are notoriously fragile and it seemed like a perfect opportunity to test out a Lakebase integration."

Postgres-compatibility as integration property

Swapping Backstage's database backend for Lakebase (Neon-lineage serverless Postgres) required:

  1. Pointing app-config.yaml at Lakebase's endpoint (hostname + database + credentials).
  2. Swapping Backstage's default in-memory search for PgSearchEngine.
  3. Working around Lakebase's rejection of classic Databricks PATs by wrapping databricks postgres generate-database-credential in a 50-minute-cadence cron that rewrites DATABRICKS_TOKEN in .env (see patterns/credential-refresh-cron-as-auth-compat-shim).

Knex migrations ran cleanly against Lakebase. The integration surfaced no Postgres-compatibility gaps at application altitude — the entire friction was at the auth tier (short-lived JWT vs Backstage's expectation of long-lived credentials).

Catalog size at POC: ~63 MB (branchable in 1.09 s — see sources/2026-04-30-databricks-backstage-with-lakebase).

"IDP one layer up" — the AI-tool-registry framing (MongoDB, 2026-05-11)

MongoDB Engineering's 2026-05-11 Fighting Tool Sprawl position post (sources/2026-05-11-mongodb-fighting-tool-sprawl-the-case-for-ai-tool-registries) explicitly frames the enterprise AI tool registry as "an Internal Developer Portal (IDP) built for the agent era, solving the same coordination problem that IDPs solved for service teams, but one layer up." The structural carryover from Backstage's pattern: same four functions (discovery + versioning + ownership / certification metadata + access control); same dual-surface property (web UI for humans + API for machine consumers); same federation property (heterogeneous entity types in one catalog). The differences are at the consumer + entity layers: Backstage catalogs services humans operate, with deployment / observability tooling as machine consumers; an AI-tool-registry catalogs tools agents call, with the agents themselves as machine consumers querying at request-time speed.

This makes Backstage the canonical wiki IDP-shape ancestor for the patterns/idp-extended-to-ai-agent-tools pattern — and the post's argument that "platform teams already understand why IDPs exist" is precisely a claim that Backstage's empirical validation transfers to the agent-era extension. See concepts/tool-sprawl for the failure mode the extended pattern addresses, and patterns/enterprise-ai-tool-registry for the operational shape.

Thoughtworks Tech Radar endorsement

Thoughtworks's Technology Radar has consistently advocated Backstage as the IDP foundation. The 2026 Backstage-on-Lakebase POC is in part an exploration of whether a serverless-Postgres substrate removes the operational pain points that keep some Backstage adopters on shared staging environments and mocked tests rather than per-developer isolated instances.

Seen in

  • sources/2026-05-15-databricks-backstage-with-lakebase-part-2 — Second canonical Backstage face: governance-substrate- unified IDP. Part 2 of the Thoughtworks series. The Backstage catalog (Postgres on Lakebase) is exposed as a foreign catalog lakebase_bs in Unity Catalog via Lakehouse Federation. Standard UC GRANTs control access (no Postgres-level role management); every control-plane action against the Backstage database lands in system.access.audit; cost attribution against UC system billing tables by (project_id, branch_id, endpoint_id); UC attribute-level masking policies propagate to every branch at creation time. The architectural payoff for state-heavy IDPs: the operational tier (Backstage's catalog DB) and the analytical tier no longer live in two security paradigms — "the compliance side-channel" of operational DBs collapses into a single substrate. Worked example: branch-lifecycle audit query from the Part 1 disaster-recovery experiment as one SQL statement against system.access.audit. Canonical instance of concepts/operational-analytical-governance-unification.

  • sources/2026-04-30-databricks-backstage-with-lakebase — canonical wiki ingest of Backstage. Thoughtworks migrates a POC Backstage install off standard Postgres / RDS onto Lakebase to demonstrate the effect of cheap branching + PITR on state-heavy-application developer cycles. The ~63 MB Backstage catalog becomes the dataset whose branch creation lands in 1.09 s data plane; Backstage's final_entities table becomes the demonstration target for PITR (32 rows → 0 → recovered in 3.78 s). Canonicalises Backstage as the representative state-heavy IDP application — why this POC is architecturally interesting is precisely that Backstage is not a toy workload.

Last updated · 542 distilled / 1,571 read