spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2780) Create a StreamingContext.setLocalProperty for setting local property of jobs launched by streaming
Date Fri, 01 Aug 2014 10:29:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082118#comment-14082118
] 

Tathagata Das commented on SPARK-2780:
--------------------------------------

Yeah, this isnt very intuitive. Two possible solutions

1. Explicitly forward the local properties to the job executor thread.
That is, when streamingContext.setLocalProperty is set, all the
properties are explicitly picked up and set in the job executor
thread(s).
2. Leave it as it is, and document this better!

TD





> Create a StreamingContext.setLocalProperty for setting local property of jobs launched
by streaming
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-2780
>                 URL: https://issues.apache.org/jira/browse/SPARK-2780
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.0.0, 1.1.0
>            Reporter: Tathagata Das
>            Priority: Minor
>
> SparkContext.setLocalProperty makes all Spark jobs submitted using
> the current thread belong to the set pool. However, in Spark
> Streaming, all the jobs are actually launched in the background from a
> different thread. So this setting does not work. 
> Currently, there is a
> work around. If you are doing any kind of output operations on
> DStreams, like DStream.foreachRDD(), you can set the property inside
> that
> dstream.foreachRDD(rdd =>
>    rdd.sparkContext.setLocalProperty(...)
> )



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message