SYSTEM Cited by 1 source

Apache Livy

Apache Livy is the open-source REST service for Apache Spark that exposes Spark job submission, interactive shell sessions, and batch-job lifecycle management as an HTTP API. It is the canonical REST front-end for Spark on YARN / EMR before custom gateways take over.

This is a stub page; Livy appears in the wiki as a prior art data point — it's the framework-typed REST submission path for Spark that already worked when Slack faced the SSH-deprecation problem. The hard part of Slack's migration was the non-Spark, non-Hive jobs (300+ CLI-based shell commands) that had no equivalent of Livy and were eventually unblocked by YARN Distributed Shell.

Seen in

• sources/2026-05-05-slack-from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines — referenced as Spark's pre-existing REST submission path, alongside HiveServer2 for Hive. Slack used Livy semantics inside Quarry's Spark adapter; the novel architectural work was the YARN Distributed Shell adapter for the shell-class job tail.

Related

• systems/apache-spark — what Livy fronts.
• systems/apache-yarn — YARN is Livy's typical resource manager.
• systems/slack-quarry — the gateway that wraps Livy + HiveServer2 + DistShell + Trino + Snowflake under a single REST surface.
• rest-based-job-submission — the paradigm Livy embodies for Spark.
• rest-gateway-for-compute-engine-job-submission — a higher-level pattern that uses Livy as a backend.