spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-23153) Support application dependencies in submission client's local file system
Date Mon, 01 Jul 2019 13:42:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876192#comment-16876192
] 

Stavros Kontopoulos edited comment on SPARK-23153 at 7/1/19 1:41 PM:
---------------------------------------------------------------------

[~ejblanco] nope. [~cloud_fan] is there going to be another 2.4.x release? Does it make sense
to backport?


was (Author: skonto):
[~cloud_fan] is there going to be another 2.4.x release? Does it make sense to backport?

> Support application dependencies in submission client's local file system
> -------------------------------------------------------------------------
>
>                 Key: SPARK-23153
>                 URL: https://issues.apache.org/jira/browse/SPARK-23153
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Yinan Li
>            Assignee: Stavros Kontopoulos
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Currently local dependencies are not supported with Spark on K8S i.e. if the user has
code or dependencies only on the client where they run {{spark-submit}} then the current implementation
has no way to make those visible to the Spark application running inside the K8S pods that
get launched.  This limits users to only running applications where the code and dependencies
are either baked into the Docker images used or where those are available via some external
and globally accessible file system e.g. HDFS which are not viable options for many users
and environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message