Skip to content

SYSTEM Cited by 1 source

Databricks Pixels

Databricks Pixels is an open-source Databricks Solution Accelerator (databricks-industry-solutions/pixels) that turns a folder of medical-imaging files (primarily DICOM) into governed, queryable Delta tables — the Bronze→Silver steps of a concepts/medallion-architecture adapted for binary files. It catalogs imaging files in parallel and extracts their text-valued metadata tags at scale, so data that was "a viewer can open it" becomes "an analyst can query it with SQL." (Source: sources/2026-10-08-databricks-biomedical-imaging-data-foundation)

The problem it solves

DICOM decoding libraries (pydicom, GDCM, pylibjpeg) are single-threaded. A few studies is trivial; a few million is a distributed-systems problem. The core trick Pixels implements is to distribute single-core parsers across a cluster — using pandas UDFs / mapInPandas to fan parsing out, or a custom Spark data source. The pixel blobs stay inside the original files; only the queryable metadata is lifted into Delta.

The zipdcm data source

Pixels ships a zipdcm Spark data source that reads DICOM metadata straight out of zip archives in memory, leaving the original files compressed in place. Published numbers:

  • Cataloged >107,000 DICOMs in ~3.5 minutes on two 8-core workers, ~7× faster than prior approaches (write-up).
  • The bottleneck shifts from disk/network I/O to pure CPU parsing — exactly the kind of embarrassingly-parallel work a cluster is good at.

See concepts/network-bound-vs-compute-bound for the general framing of this I/O→CPU bottleneck shift.

Role in the medallion flow

  • Bronze — raw studies land in a restricted UC Volume; a de-identification job runs on headers and pixels before anything else; one catalog row per file; Auto Loader handles incremental arrival.
  • Silver — Pixels extracts the text DICOM tags into Delta tables (binary pixel data stays in the file), making the trapped metadata SQL-queryable.
  • Gold — curated cohorts joined to clinical/omics tables for training, BI, or regulator-facing studies.

De-identification pipeline

The Pixels team published a vision-language-model de-identification pipeline that combines OCR with a VLM to catch PHI burned into pixels (patient names baked into ultrasounds or scanned forms — which OCR-only tools like Presidio miss). It runs distributed with pandas UDFs. On a balanced MIDI-B sample: GPT-4o and Claude 3.7 Sonnet reached ~100% recall/precision; open Llama 4 Maverick hit 100% recall at a fraction of the cost; and Spark cut de-identification of 1,000 frames from 105 min to 6 min. Header PHI is handled separately with format-preserving encryption (same identifier → same pseudonym) per the DICOM confidentiality profiles.

DICOMweb service

Newer Pixels work ships a DICOMweb service (QIDO-RS / WADO-RS / STOW-RS), so the lakehouse can sit directly inside a clinical or PACS/VNA workflow rather than off to the side. A Pixels 2.0 integration with NVIDIA MONAI / VISTA-3D adds auto-segmentation and fine-tuning with an OHIF viewer embedded in a Databricks App.

Seen in

Last updated · 854 distilled / 2,448 read