spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Antal (Jira)" <>
Subject [jira] [Created] (SPARK-29474) CLI support for Spark-on-Docker-on-Yarn
Date Tue, 15 Oct 2019 10:16:00 GMT
Adam Antal created SPARK-29474:

             Summary: CLI support for Spark-on-Docker-on-Yarn
                 Key: SPARK-29474
             Project: Spark
          Issue Type: Improvement
          Components: Spark Shell, YARN
    Affects Versions: 2.4.4
            Reporter: Adam Antal

The Docker-on-Yarn feature is stable for a while now in Hadoop.
One can run Spark on Docker using the Docker-on-Yarn feature by providing runtime environments
to the Spark AM and Executor containers similar to this:
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=repo/image:tag
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro"
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=repo/image:tag
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro"

This is not very user friendly. I suggest to add CLI options to specify:
- whether docker image should be used ({{--docker}})
- which docker image should be used ({{--docker-image}})
- what docker mounts should be used ({{--docker-mounts}})
for the AM and executor containers separately.

Let's discuss!

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message