spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olivier Armand (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor
Date Sat, 15 Oct 2016 17:46:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578487#comment-15578487
] 

Olivier Armand commented on SPARK-650:
--------------------------------------

Sean, a singleton is not the best option in our case. The Spark Streaming executors are writing
to HBase, we need to initialize the HBase connection. The singleton seems (or seemed when
we tested it for our customer a few months after this issue was raised) to be created when
the first RDD is processed by the executor, and not when the driver starts. This imposes very
high processing time for the first events.

> Add a "setup hook" API for running initialization code on each executor
> -----------------------------------------------------------------------
>
>                 Key: SPARK-650
>                 URL: https://issues.apache.org/jira/browse/SPARK-650
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Matei Zaharia
>            Priority: Minor
>
> Would be useful to configure things like reporting libraries



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message