Skip to content

Databricks — Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries

Databricks Blog (Databricks-for-Good arm) co-marketing post (2026-05-20) documenting the production-grade rebuild of Virtue Foundation's VF Match platform — the volunteer-matching substrate that connects medical professionals to opportunities in 72 low and low-middle income countries — on Databricks. Tier-3 vendor-blog source. Borderline-include scope decision: heavy Databricks-for-Good co-marketing framing in the bookend paragraphs (~40%), but the Building the Foundation / Entity Resolution at Scale / VF Agent sections (~60%) name a specific architectural shape worth canonicalising — a multi-step LLM extraction pipeline (verbatim "rather than attempting one-shot extraction, our pipeline breaks the task into targeted steps") over 25 million web pages orchestrated by Lakeflow Jobs across 15+ interdependent tasks, with a star schema state model and status-based checkpointing; an Entity Resolution stage built on the open-source Splink probabilistic record-linkage framework with a quantified curse-of-the-last-reducer straggler observation (one Spark partition running 30 minutes vs 52-second median) reduced 15× to ~2 minutes by enabling Photon; and a prototype VF Agent multi-agent architecture in LangGraph routing user queries to Vector Search or Genie sub-agents via a Multi-Agent Supervisor.

One-paragraph summary

Virtue Foundation's VF Match platform connects medical professionals to volunteer opportunities across 72 low and low-middle income countries. The marketplace's value depends on a comprehensive catalog of healthcare facilities and NGOs — a dataset that does not exist as a public source. Databricks-for-Good and Virtue Foundation built the Foundational Data Refresh (FDR): a production pipeline that ingests web-scale data from Overture Maps (Meta + Microsoft open-source geospatial data — authoritative facility locations) and Bright Data (industrial web scraping — real-time facility / NGO web pages), then runs 25 million+ web pages through OpenAI's GPT models to extract structured records. Rather than attempt one-shot extraction, the pipeline decomposes into targeted steps — classify medical relevance, identify organisation type (facility vs NGO), then extract specialties / equipment / procedures — with each step's prompt isolated to a narrow, high-precision task. Each record's progress is tracked by status in a star schema, so re-runs resume from the failure point without paying the LLM cost of already-processed rows. Extraction methods are registered as structured configuration objects (system prompt + extraction schema) — adding a new extraction is configuration, not code. Lakeflow Jobs orchestrates 15+ interdependent tasks with conditional branching, parallel execution, and intelligent retry. After extraction, the pipeline must solve entity resolution: the same facility shows up in multiple sources with name variations, inconsistent addresses, and missing fields. The team uses Splink, an open-source probabilistic record linkage framework, with weighted comparisons across phone / address / name fields producing a unified key per facility. Splink's pairwise comparison created the classic straggler partition problem: "one Spark partition running for 30 minutes while the median completed in 52 seconds — a textbook case of stragglers (the 'curse of the last reducer')." Enabling Photon (Databricks' vectorised query engine) cut worst-case partitions from 30 minutes to ~2 minutes — a 15× improvement. Forward roadmap: VF Agent — a prototype multi-agent system in LangGraph composed of a Medical Specialty Extractor (converts user free-text into standardised medical terminology), a Multi-Agent Supervisor (routes to specialised sub-agents based on query intent and complexity), a Vector Search Agent (facility discovery and search via Mosaic AI Vector Search), and a Genie Agent (analytical queries against the structured FDR data via AI/BI Genie). The architecture lets healthcare professionals query the FDR data in natural language to find matches by specialty.

Key takeaways

  1. Multi-step LLM extraction beats one-shot at production scale. "Rather than attempting one-shot extraction, our pipeline breaks the task into targeted steps: classifying medical relevance, identifying organization type (either a medical facility or NGO), and extracting specialties, equipment, and procedures. This approach dramatically reduces token consumption while focusing each model invocation on a narrow, high-precision task." Canonicalised in concepts/multi-step-llm-extraction + patterns/multi-step-llm-extraction-pipeline.

  2. Status-based checkpointing makes 25M-page LLM pipelines resumable. "Every record tracks its processing state, enabling pipelines to resume from any point without reprocessing rows with expensive LLM calls." The state column is the idempotency primitive for pipelines whose dominant cost is per-record LLM invocation. Canonicalised in concepts/status-based-llm-pipeline-checkpointing.

  3. Configurable extraction registry decouples prompts from code. "Each extraction method is controlled by a structured object specifying the system prompt, making extraction logic modular, reproducible, and extensible." Adding a new extraction (e.g., "extract NGO funding sources") is a configuration change, not a code deploy. Threaded through patterns/multi-step-llm-extraction-pipeline as a sub-property.

  4. Star schema is the right state model for LLM extraction pipelines. "Data at each step is stored in a star schema, simplifying downstream analytics and improving query performance." Pinned in concepts/star-schema — the central fact table tracks per-record extraction state, dimension tables hold reference metadata. Star schema's column-store-friendliness composes with Photon vectorisation. (Source: sources/2026-05-20-databricks-virtue-foundation-medical-volunteers-72-countries)

  5. Splink for probabilistic record linkage at terabyte scale. "The same facility may appear across multiple data sources with name variations, inconsistent addresses, or missing contact details. Traditional deduplication breaks down in these scenarios due to messy data, so we use Splink, an open source probabilistic record linkage framework. Using the information sourced in our IE step, Splink evaluates match pairs via weighted comparisons across fields like phone number, street address, and more. The result is a unified key per facility." Splink is the wiki's first canonical instance of an open-source probabilistic record linkage framework (systems/splink).

  6. The curse of the last reducer at LLM-pipeline altitude. "The core of record linkage is pairwise comparison, which creates inherently skewed workloads: common comparisons produce massive partitions while most others remain much smaller. Early runs made this painfully clear, with one Spark partition running for 30 minutes while the median completed in 52 seconds – a textbook case of stragglers (the 'curse of the last reducer') degrading job performance." The 30 min vs 52 s ratio is ~35× tail amplification at the partition level; at the job level, the slowest partition is the wall-clock floor. concepts/curse-of-the-last-reducer is the canonical name.

  7. Photon collapsed the straggler from 30 min to ~2 min — 15×. "Enabling Photon, Databricks' vectorized query engine, reduced worst-case data partitions from 30 minutes to approximately 2 minutes: a 15x improvement." Two architectural reasons Photon helps a record-linkage workload specifically: (a) pairwise comparison is dominated by string and numeric similarity functions amenable to SIMD vectorisation; (b) Splink's weight-aggregation step is a column-major reduce that benefits from column-store memory layout. The 15× number is the wiki's first-quantified instance of systems/photon applied to ER-shape skew, distinct from prior Photon mentions at OLAP-query altitude. concepts/vectorized-query-engine canonicalises the engine class.

  8. Lakeflow Jobs orchestrates 15+ interdependent FDR tasks. "These guarantees are enforced through Lakeflow Jobs, which orchestrate more than 15 interdependent tasks with conditional branching, parallel execution, and intelligent retry policies." Third canonical Lakeflow-Jobs face on the wiki after MapAid groundwater and Claroty CSAF ETL — the third independent customer using Lakeflow Jobs to compose multi-step LLM-driven pipelines with conditional branching + retry policies. The shape is converging.

  9. VF Agent: multi-agent supervisor routing in LangGraph. The prototype query-handling system has four named agents: Medical Specialty Extractor normalises user free-text into standardised medical terminology; Multi-Agent Supervisor classifies the normalised query's intent and complexity, routing to the right downstream agent; Vector Search Agent handles facility-discovery / search queries against the embedded FDR data; Genie Agent handles structured-analytical queries via AI/BI Genie over the FDR Delta tables. Canonicalised in patterns/multi-agent-supervisor-routing — a specialisation of supervised multi-agent architectures focused on query-shape routing rather than entity-resolution role decomposition. Distinct from Claroty's role-decomposed ER multi-agent (parse vs reason vs review collaborate on one canonicalisation task) — VF Agent's agents are alternatives selected per query, not collaborators on one task.

  10. Two complementary data sources resolve healthcare-facility ground truth. Overture Maps (Meta

    • Microsoft) provides authoritative locations; Bright Data provides real-time facility web pages. Joining the two via Splink ER produces facility records with both authoritative geospatial anchors and current contextual web evidence.

Architectural numbers

  • 72 low and low-middle income countries in scope.
  • 50,000+ patients delivered care to date by Virtue Foundation (special focus on Ghana and Mongolia).
  • 25M+ web pages processed through LLMs in the FDR pipeline.
  • 15+ interdependent tasks orchestrated by Lakeflow Jobs.
  • 30 minutes worst-case Spark partition duration before Photon (one straggler).
  • 52 seconds median Spark partition duration before Photon.
  • ~2 minutes worst-case Spark partition duration after Photon.
  • 15× improvement on worst-case partition latency from enabling Photon.
  • ~35× ratio between straggler (30 min) and median (52 s) partition durations before Photon — the magnitude of skew the vectorised engine had to absorb.
  • 2 complementary data sources (Overture Maps + Bright Data).
  • 4 named sub-agents in the VF Agent prototype (Medical Specialty Extractor + Multi-Agent Supervisor + Vector Search Agent + Genie Agent).
  • 0 specific FDR-pipeline cost / latency / accuracy numbers disclosed outside the Photon comparison — this is a reference- architecture post, not a benchmark.

Systems / concepts / patterns surfaced

Systems

  • systems/vf-match — Virtue Foundation's volunteer-matching platform (the post's user-facing system).
  • systems/vf-agent — Virtue Foundation's prototype multi-agent query system (LangGraph-based).
  • systems/splink — open-source probabilistic record-linkage framework (UK Ministry of Justice origin, Apache 2.0).
  • systems/photon — Databricks' vectorised query engine; first dedicated wiki page (previously only mentioned in the Databricks company-page tag list and Superhuman post).
  • systems/overture-maps — Meta + Microsoft open-source geospatial dataset (cross-citation in the source).
  • systems/bright-data — industrial web-scraping infrastructure (cross-citation in the source).
  • systems/lakeflow-jobs — Databricks orchestration product (third canonical face: FDR multi-step LLM pipeline).
  • systems/databricks-genie — Databricks AI/BI Genie (Genie Agent face in VF Agent).
  • systems/mosaic-ai-vector-search — Databricks managed vector search (Vector Search Agent face in VF Agent).
  • systems/langgraph — LangChain's agent-orchestration graph framework (cross-citation as VF Agent's substrate).

Concepts

Patterns

Caveats

  • Vendor co-marketing post. Databricks-for-Good content with Virtue Foundation as customer — architectural sections are substantive but selectively presented. Detailed numbers on end-to-end latency, accuracy, cost, or throughput are largely absent (only the Photon 30 min → 2 min comparison is quantified).
  • VF Agent is a prototype. The Multi-Agent Supervisor + Medical Specialty Extractor + Vector Search Agent + Genie Agent composition is described as "a prototype of an agent that enables experts to analyze data using natural language" — production deployment, accuracy, latency are not disclosed.
  • No detailed FDR-task DAG. The 15+ interdependent tasks are named at the level of "more than 15" — the actual DAG, the branching conditions, retry policies, and per-task latencies are not disclosed.
  • No Splink configuration disclosed. Match weight values, blocking rules, and the specific match-pair-comparison scoring algorithm Splink uses for this domain are not disclosed; only the figure-2 ruleset image is referenced.
  • No GPT model version / prompt content / token-cost data. "OpenAI's GPT models" is named at the family level. Per-step prompts are not in the post.
  • No accuracy / recall / precision numbers for any of the three extraction steps (medical relevance / org-type / specialties
  • equipment + procedures).
  • No cost-vs-quality tradeoff disclosure for the one-shot-vs-multi-step decision; the claim "dramatically reduces token consumption" is qualitative.
  • The 35× skew ratio is a single-job observation (early FDR runs). Post-Photon distribution shape (whether stragglers fully collapsed or just shrank from 30 to 2 minutes) is not disclosed in detail.

Source

Last updated · 542 distilled / 1,571 read