beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía (JIRA) <j...@apache.org>
Subject [jira] [Commented] (BEAM-470) Spark Runner does not send the job execution information into the Spark History Server
Date Wed, 23 Nov 2016 10:04:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689610#comment-15689610
] 

Ismaël Mejía commented on BEAM-470:
-----------------------------------

Just for reference this was solved long time ago, because the runner was not correctly passing
the variables 'spark.*' to the spark context. It is working ok right now.

> Spark Runner does not send the job execution information into the Spark History Server
> --------------------------------------------------------------------------------------
>
>                 Key: BEAM-470
>                 URL: https://issues.apache.org/jira/browse/BEAM-470
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>    Affects Versions: 0.2.0-incubating
>            Reporter: Ismaël Mejía
>            Priority: Minor
>             Fix For: Not applicable
>
>
> If you run a Beam pipeline using the spark runner from spark (via spark-submit), the
execution is registered in the spark-history-server if it is active and configured.
> if you do this directly from a main method with --runner=SparkRunner (the beam way) the
Beam runner does not report the execution to the history server, it seems the issue is the
runner does not take into account an existing spark configuration file SPARK_HOME/conf/spark-defaults.conf
(or there is not a way to tell the runner to take such conf into account).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message