SYSTEM Cited by 2 sources
Databricks Runtime¶
The Databricks Runtime is Databricks' distributed, fault-tolerant, elastic execution engine, built on Apache Spark plus Photon (a vectorized native C++ query engine). It is the substrate that gives engine-native features their non-functional properties for free: tasks retry automatically, memory pressure spills to disk, and compute scales out for a run then scales down to zero after.
Those properties are precisely why Databricks built batch vector search (NEAREST BY) as a runtime-native feature rather than federating to an external real-time serving endpoint — the requirements (elastic parallelism across hundreds-to-thousands of cores, peak arithmetic per core via Photon SIMD kernels, and fault tolerance across hours-long jobs) are "properties of an execution engine, not features we can wrap around a real-time serving endpoint." (Source: sources/2026-10-05-databricks-nearest-by-join-scaling-vector-search-in-databricks-runtime)
Stub page — expand as more runtime-internals sources are ingested.
Seen in¶
- sources/2026-10-29-zalando-upgrading-mlflow-pipelines-across-a-python-version-boundary — an LTS upgrade is also a Python-version upgrade. Zalando's move from DBR LTS 14.3 (Python 3.10) to LTS 17.3 (Python 3.12) broke loading of every cloudpickled MLflow model; the team ran a one-time migration job on a DBR 14.3-matched environment to read the old artifacts and relog them.
- sources/2026-10-05-databricks-nearest-by-join-scaling-vector-search-in-databricks-runtime — the engine NEAREST BY is built into; named as the reason batch vector search is engine-native (distribute/spill/retry/scale-to-zero come from the runtime).
Related¶
- systems/mlflow — model artifacts are tied to the runtime's Python version when pickled.
- systems/apache-spark — the distributed execution core.
- systems/photon — the vectorized native C++ query engine.
- systems/nearest-by — a runtime-native batch vector-search feature.