Apache Livy server for Docker with Spark dependencies.
Tested on Spark Standalone version 2.4.6 and Livy version 0.7.0.

1. Make sure you have Spark and Livy archives in this directory, you can download the latest supported versions in this repo by running:

./download_archives.sh

1. Build Livy image:

docker build . -t livy-ofekhod

1. Run a container:

docker run -d \
--name livy \
-e SPARK_MASTER_ENDPOINT=<spark-master-url> \
-e SPARK_MASTER_PORT=<spark-master-port> \
-p 8998:8998 \
-p 7077:7077 \
livy-ofekhod

For example:

docker run -d \
--name livy \
-e SPARK_MASTER_ENDPOINT=spark \
-e SPARK_MASTER_PORT=7077 \
-p 8998:8998 \
-p 7077:7077 \
livy-ofekhod

1. Access Livy's UI via http://localhost:8998

If you need to support batch jobs, add a Docker volume for the jars path and a corresponding local directory whitelist path for Livy (livy.file.local-dir-whitelist):

docker run -d \
--name livy \
...
...
-e LIVY_FILE_LOCAL_DIR_WHITELIST=/opt/jars \
-v <jars-path>:/opt/jars \
...
livy-ofekhod

For example:

docker run -d \
--name livy \
-e SPARK_MASTER_ENDPOINT=spark \
-e SPARK_MASTER_PORT=7077 \
-e LIVY_FILE_LOCAL_DIR_WHITELIST=/opt/jars \
-v $PWD/jars:/opt/jars \
-p 8998:8998 \
-p 7077:7077 \
livy-ofekhod

If you need to add extra parameters to livy.conf, please add them to livy.conf.extra and make this file accessible via Docker volume with:

docker run -d \
--name livy \
...
...
-v $PWD/livy.conf.extra:/opt/livy.conf.extra:ro \
...
livy-ofekhod

For example:

docker run -d \
--name livy \
-e SPARK_MASTER_ENDPOINT=spark \
-e SPARK_MASTER_PORT=7077 \
-p 8998:8998 \
-p 7077:7077 \
-v $PWD/livy.conf.extra:/opt/livy.conf.extra \
livy-ofekhod

Note that parameters from env (-e in docker run) overrides parameters from livy.conf.extra.
Supported parameters from env are:

SPARK_MASTER_ENDPOINT
SPARK_MASTER_PORT
LIVY_FILE_LOCAL_DIR_WHITELIST

Both SPARK_MASTER_ENDPOINT and SPARK_MASTER_PORT should be provided in order to compound livy.spark.master, otherwise it will be taken from livy.conf.extra.

Tested with bitnami/Spark version 2.4.6

1. Follow the stages 1,2 of Build & Run in order to build livy image on your computer (not available on DockerHub yet).

Use docker-compose.spark_bitnami.yml in order to run Livy with Spark Standalone on your computer:

docker-compose --file docker-compose.spark_bitnami.yml up -d

In order to use another Spark version:

1. Download relevant Spark archive (change download_archives.sh).
2. Edit Dockerfile's SPARK_VERSION (possibly HADOOP_VERSION as well).
3. (If Spark Standalone): update spark-standalone-support/bitnami/docker-compose.yml to use the correct spark/bitnami images tags for the desired Spark version.

In order to use another Livy version:

1. Download Livy archive (change download_archives.sh).
2. Edit Dockerfile's LIVY_VERSION.