Skip to content

DATABRICKS 2026-10-29

Read original ↗

How Procore is powering construction analytics with OpenSharing

Summary

A Databricks customer story about Procore Analytics 2.0, the product that gives Procore's construction customers programmatic access to their own Procore data inside their tools (Power BI, Tableau, Snowflake, BigQuery, Databricks, on-prem SQL Server). Generation 1 extracted data from Procore's public APIs with deep recursive calls into an Azure SQL Server database. It refreshed at most once a day (three times a day for customers who paid more), and the API extraction was expensive, brittle, and hit timeouts and limits at tens of thousands of customers. Generation 2 runs a multi-region medallion pipeline on Databricks, pulls the gold tables of each region into one primary analytics region, and fans them out to customers with OpenSharing (the current Databricks name for Delta Sharing). A scheduled job reconciles share entitlements from the permissions in the Procore application. Shares go down to partition level, and Change Data Feed lets recipients pull only the changes. One gold layer feeds both in-app reporting and the customer extract. Breaking schema changes ship on a fixed release calendar. This is a Tier-3 vendor case study. It has no latency, size, or cost numbers, but the architecture section is real and well over the 20% bar.

Key takeaways

  1. API-driven extraction does not scale as a B2B data product. Gen 1 pulled data for tens of thousands of customers through public APIs with deep recursive calls. This was expensive, brittle, and prone to timeouts and rate limits, and it capped refresh at 1×/day (3×/day paid). Gen 2 shares curated files in place instead of re-deriving them per customer through the API. (Source: this page, "Evolving beyond the first generation")
  2. Process regionally, share centrally. Bronze, silver, and gold run in multiple regions, close to the source data. Gold tables from each region then go to one primary analytics region, which is the only place OpenSharing fans data out. This puts compute next to the data, and keeps one sharing surface and one governance plane. (Source: "The implementation")
  3. Massive share count with per-customer partitioning. Procore shares hundreds of curated tables to each customer and keeps partitioned access per customer. This comes to hundreds of thousands of distinct data shares. Some customers run hundreds or thousands of concurrent projects. This is tenant isolation at the data-sharing layer, not at the storage layer. (Source: "Why OpenSharing")
  4. Entitlements are reconciled, not hand-provisioned. A scheduled job watches for new shares and permission changes, reads each user's permissions from the Procore application (the source of truth), and reconciles the OpenSharing entitlements to match. The result is per-user, per-project enforcement through Unity Catalog with no manual provisioning. For example, a general contractor can share project progress with the owner but keep its own financials private (concepts/fine-grained-authorization). This is a desired-state reconciliation loop: the app's ACL model is authoritative and the sharing layer converges to it. (Source: "Automated entitlements")
  5. Two recipient paths on one protocol. Customers without Databricks generate an OpenSharing token inside the Procore app and connect with the open protocol (BI tools work out of the box; other warehouses use the open client libraries). Customers with Databricks give a workspace identifier and get native Databricks-to-Databricks sharing with no data movement. (Source: "Two recipient paths")
  6. Granularity plus incrementality. Shares can be at the schema, view, table, or partition level. Tables have Change Data Feed enabled, so a recipient can take the full dataset or only the changes since the last sync. This matters when single tables hold millions of records (concepts/change-data-capture). (Source: "Granularity and incrementality")
  7. One gold layer is the semantic layer for both products. The same gold tables power Procore's in-app reporting and the customer extract. Schemas and names are the same in both, so a value in the Procore UI has the same value and name in the extract. Before, there were two parallel pipelines (analytics and in-app reporting), each with its own ingestion, modeling, and governance. These became one. (Source: "A common semantic layer")
  8. Breaking-change cadence as a published contract. Procore ships data into thousands of downstream customer pipelines. A breaking change is any change that can disrupt a downstream pipeline: a field rename, a column type change, or a table restructure. Procore allows 4 breaking-change releases/year with 6 weeks' notice and 12 non-breaking releases/year with 3 weeks' notice, on the company release calendar. This is the change-process clause of a data contract applied to external consumers (concepts/schema-evolution, concepts/backward-compatibility). (Source: "A predictable change cadence")
  9. Org follows the pipeline. One combined pipeline does ingestion, landing, modeling, and governance. It then fans out to data-domain teams aligned to business areas (for example, finance). Each domain team owns its data end to end with the application teams. This replaces separate analytics, reporting, and intelligence teams that duplicated the same knowledge. The shape is close to data mesh domain ownership on a shared platform. (Source: "A reorganized data team")
  10. Outcome: catalog breadth and refresh freedom. The 2.0 catalog grew to hundreds of tables in less than half the time that 1.0 needed to build its baseline catalog ("effectively doubling the surface area"). The 1×/day refresh ceiling is gone, and customers refresh on their own cadence, often several times a day. New tool features become extractable almost as soon as they ship. Before, teams waited months for a parallel analytics pipeline. (Source: "Measurable impact")

Architecture

 Procore app DB + integration events
            │   (per region)
            ▼
   Bronze (raw, untouched) ─► Silver (clean, validate, DQ) ─► Gold (aggregate, model)
            │                     region A … region N
            ▼
   Gold tables pulled into ONE primary analytics region
            │
            ├─► Procore in-app reporting  (same gold = same names/values)
            │
            └─► OpenSharing / Delta Sharing  (Unity Catalog governed)
                   ├─ open-protocol token  → Power BI / Tableau / Snowflake / BigQuery / SQL Server
                   └─ D2D sharing (workspace id) → customer Databricks workspace
                   share scope: schema | view | table | partition ; CDF for incremental pulls

   Scheduled entitlement job: new shares + permission changes
      → read user permissions from Procore app → reconcile share grants

Operational numbers

  • Customers: tens of thousands (U.S., Europe, Middle East, Asia, Australia).
  • Gen 1 refresh: 1×/day, at most 3×/day for paying customers.
  • Gen 2 shares: hundreds of curated tables per customer; hundreds of thousands of distinct shares.
  • Some tables hold millions of records.
  • Release cadence: 4 breaking/yr (6-week notice) + 12 non-breaking/yr (3-week notice).
  • Catalog: hundreds of tables in less than half the time 1.0 took.

Systems / concepts / patterns extracted

Caveats

  • Tier-3 vendor customer story that ends in product CTAs. It gives no latency, data volume, cost, or share-count-to-performance numbers.
  • "OpenSharing" is the current product name; the CTA links to the Delta Sharing product page. This wiki keeps the canonical page at systems/delta-sharing.
  • The article does not say how gold tables move between regions (Deep Clone, replication, or a cross-region share), how often the entitlement job runs, or how it handles revocation lag.
  • "Doubling the surface area" is relative and self-reported.

Source

Last updated · 854 distilled / 2,448 read