Skip to content

DATABRICKS 2026-10-08

Read original ↗

Biomedical Imaging's Real Bottleneck Is the Data, Not the Model

Summary

A Databricks Healthcare & Life Sciences position piece arguing that the hard part of medical-imaging AI is not the model but the data foundation: making petabyte-scale, gigapixel, multi-dimensional images governed, queryable, de-identified, linkable, and shareable. Part I frames the sector problem (hospitals, academia, medtech, pharma all hit the same wall: "the pixels are everywhere and queryable nowhere"). Part II — the architecturally substantive half — maps the requirements onto a concepts/data-lakehouse built on Databricks: a concepts/medallion-architecture adapted for binary imaging files, distributed single-core DICOM decoders fanned across a Spark cluster, systems/unity-catalog governing files and extracted metadata under one lineage model, de-identification of header and burned-in pixel PHI at scale, and two complementary cross-institution collaboration models — concepts/federated-learning (move weights, not data) and systems/delta-sharing + Clean Rooms (share governed views via credential vending, not export). The through-line: imaging is just the sharpest edge of a general R&D data-foundation problem that also covers omics, EHR, waveforms, and literature.

Key takeaways

  1. The model is the easy 10%. Of ~950 FDA-authorized AI/ML medical devices by mid-2024, ~723 (≈76%) were radiology tools — almost all narrow and human-supervised. The difficulty is that data sits locked inside PACS/VNAs built to serve a viewer, not answer research questions. (Source: sources/2026-10-08-databricks-biomedical-imaging-data-foundation)
  2. "A few million is a distributed-systems problem." DICOM decoders (pydicom, GDCM, pylibjpeg) are single-threaded; a few studies is trivial, millions is not. The fix is to distribute single-core parsing across a cluster with pandas UDFs / mapInPandas or a custom Spark data source. The Pixels team's zipdcm source read metadata straight out of zip archives in memory and cataloged >107,000 DICOMs in ~3.5 min on two 8-core workers — ~7× faster than prior approaches, shifting the bottleneck from I/O to pure CPU parsing.
  3. Medallion architecture, adapted for binary files. Bronze = restricted object-storage / UC Volumes landing zone where de-identification runs on headers and pixels before anything else, one catalog row per file, with Auto Loader for incremental arrival; Silver = text DICOM tags extracted into Delta tables (pixel blobs stay in the file) so metadata becomes SQL-queryable; Gold = curated cohorts joined to clinical and omics tables. The Pixels accelerator does the Bronze→Silver cataloging + tag extraction in parallel.
  4. Governance that survives an audit. UC Volumes govern raw files in S3/ADLS/GCS under a three-level namespace; extracted metadata lands in Delta; both sit under one access-control + lineage model that reaches the models trained on the data ("who touched this cohort, and what was built from it"). Row/column-level security and ABAC let you expose de-identified metadata broadly while keeping pixels locked down.
  5. De-identification is the quiet failure mode most platforms skip. Header PHI is handled with format-preserving encryption so the same patient identifier always maps to the same pseudonym — enabling cross-dataset linkage without ever exposing the real identifier. The harder problem is PHI burned into pixels; recent work shows vision-language models beat OCR-only pipelines. On MIDI-B, GPT-4o and Claude 3.7 Sonnet hit ~100% recall/precision, open Llama 4 Maverick hit 100% recall at a fraction of the cost, and Spark cut de-identification of 1,000 frames from 105 min to 6 min (VLM pipeline distributed with pandas UDFs).
  6. Two ways to collaborate without moving data. concepts/federated-learning keeps data physically in place and moves only weights (federated averaging + secure aggregation + concepts/differential-privacy) — the EXAM study (Nature Medicine 2021, 20 institutions) gained on average +16% AUC and +38% generalizability over single-site models without moving patient data. The lakehouse adds a lighter second option: systems/delta-sharing + Clean Rooms share governed tables/files or run joint analysis on de-identified cohorts via credential vending, not data export — often a far smaller lift than standing up a federated training ring, and the two can be mixed.
  7. Governed, queryable data makes the ML/serving layer the easy part. Segmentation/classification models register in the same catalog as MLflow models with lineage back to the exact cohort; NVIDIA MONAI / VISTA-3D (a foundation segmentation model covering 127 anatomical classes) plug in for auto-segmentation and fine-tuning, with an OHIF viewer embedded in a Databricks App. Inference runs either as real-time GPU model serving (concepts/scale-to-zero so a rarely used model isn't burning a GPU) or as distributed batch inference across millions of studies. Newer Pixels work ships a DICOMweb service (QIDO-RS/WADO-RS/STOW-RS) so the lakehouse can sit in a clinical/PACS workflow.
  8. Reproducibility, not accuracy, is what fails. A BMJ review (Nagendran et al., 2020) of 81 deep-learning-vs-clinician imaging studies found code unavailable in 93% and data in 95% — a debt that traces straight back to how hard imaging data is to share and re-run. Variability across scanners, protocols, and sites (non-IID / acquisition-skewed data) is the enemy of both federated training and trial endpoints.
  9. Multimodal is the payoff. Because imaging metadata is just Delta, it joins to FHIR/HL7 EHR feeds, genomic variant tables (VCF/BAM/CRAM), waveforms (WFDB high-frequency time series needing timestamp alignment), and clinical text — patient identity resolved through the pseudonymization layer, OMOP as the common schema. Vector Search indexes embeddings for semantic retrieval; Genie lets a scientist build a cohort in natural language; and RAG over images + reports (BiomedCLIP / CheXzero line) grounds report drafting in a patient's actual images and priors. Foundation models like Prov-GigaPath (Nature 2024, pretrained on 1.3B tiles from 171,189 whole-slide images) only work with data that is large, diverse, governed, and multimodal.

Architecture notes

The format "zoo" that breaks general-purpose tooling

Modality Formats Scale Why generic tooling buckles
Radiology (X-ray/CT/MRI) DICOM, ~a dozen transfer syntaxes (uncompressed → JPEG 2000 → HTJ2K) 2D → 3D volumes → 4D dynamic Built to move studies between machines, not analyze in bulk; decoders single-threaded
Digital pathology (WSI) Vendor formats: Aperio .svs, Hamamatsu .ndpi, Philips iSyntax (via OpenSlide); almost no DICOM Gigapixel, multi-GB, tens of thousands of tiles, read as a pyramid Too large for memory; stain/scanner color varies lab-to-lab, becomes its own model error
Ophthalmology / derm / other OCT volumes, clinical photography 2D & 3D, modality-specific One more set of conventions a folder-of-JPEGs tool was never built for

What the data foundation must do (the five requirements)

  1. Govern unstructured imaging files + extracted metadata in one place with access control, lineage, and auditability.
  2. Process petabyte-scale, gigapixel, multi-dimensional data without falling over.
  3. De-identify at scale — headers and pixels.
  4. Link imaging to EHR, genomics, waveforms, and text for one research question.
  5. Make cross-institution sharing a configuration step, not a six-month negotiation.

Distributed-parsing numbers

  • zipdcm Spark data source: >107,000 DICOMs cataloged in ~3.5 min on two 8-core workers, ~7× faster — reads metadata out of zip archives in memory, leaves files compressed in place; bottleneck shifts from disk/network I/O to CPU parsing.
  • VLM de-identification distributed with pandas UDFs: 1,000 frames 105 min → 6 min on Spark.

Operational numbers (reference)

  • FDA AI/ML devices by mid-2024: ~950 total, ~723 (≈76%) radiology.
  • MIMIC-CXR: 377,110 chest X-rays.
  • EXAM federated study (Nature Medicine 2021): 20 institutions, +16% AUC, +38% generalizability vs single-site, no patient data moved.
  • Nagendran BMJ 2020: 81 studies; code unavailable 93%, data 95%.
  • MONAI VISTA-3D: foundation segmentation over 127 anatomical classes.
  • Prov-GigaPath (Nature 2024): 1.3B tiles from 171,189 whole-slide images.
  • DICOM study sizes: CT 300–2,000 images, MRI 500–3,000 slices, mammography 8–12.

Caveats / notes

  • Vendor-authored position piece. Databricks is advocating for its own lakehouse stack (Unity Catalog, Delta, Spark, Pixels, MLflow, Delta Sharing, Genie, Model Serving). The architectural patterns (distribute single-core parsers, govern files + metadata under one catalog, FPE for pseudonymous linkage, move-weights-vs-move-data) generalize beyond Databricks; the product names are vendor-specific.
  • Clean Rooms / de-identification / format-preserving-encryption are recorded here as prose + tags rather than dedicated pages: each appears in a single source so far and is not yet in the controlled vocabulary (_taxonomy.md). They are candidates for promotion once a second source cites them.
  • Burned-in-pixel PHI removal is explicitly called an "active research problem"; the ~100% VLM recall/precision figures are on a balanced MIDI-B sample, not a production guarantee.
  • MONAI is NVIDIA-origin open tooling, not a Databricks system; it plugs into the lakehouse via Pixels 2.0 integration.

Source

Last updated · 854 distilled / 2,448 read