spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Schmidt <...@datapath.io>
Subject Re: Where to set properties for the retainedJobs/Stages?
Date Mon, 04 Apr 2016 07:46:21 GMT
Okay I put the props in the spark-defaults, but they are not recognized,
as they don't appear in the 'Environment' tab during a application
execution.

spark.eventLog.enabled for example.

Am 01.04.2016 um 21:22 schrieb Ted Yu:
> Please
> read https://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties
> w.r.t. spark-defaults.conf
>
> On Fri, Apr 1, 2016 at 12:06 PM, Max Schmidt <max@datapath.io
> <mailto:max@datapath.io>> wrote:
>
>     Yes but doc doesn't say any word for which variable the configs
>     are valid, so do I have to set them for the history-server? The
>     daemon? The workers?
>
>     And what if I use the java API instead of spark-submit for the jobs?
>
>     I guess that the spark-defaults.conf are obsolete for the java API?
>
>
>     Am 2016-04-01 18:58, schrieb Ted Yu:
>
>         You can set them in spark-defaults.conf
>
>         See
>         also https://spark.apache.org/docs/latest/configuration.html#spark-ui
>         [1]
>
>         On Fri, Apr 1, 2016 at 8:26 AM, Max Schmidt <max@datapath.io
>         <mailto:max@datapath.io>> wrote:
>
>             Can somebody tell me the interaction between the properties:
>
>             spark.ui.retainedJobs
>             spark.ui.retainedStages
>             spark.history.retainedApplications
>
>             I know from the bugtracker, that the last one describes
>             the number of
>             applications the history-server holds in memory.
>
>             Can I set the properties in the spark-env.sh? And where?
>
>
>             ---------------------------------------------------------------------
>             To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>             <mailto:user-unsubscribe@spark.apache.org>
>             For additional commands, e-mail:
>             user-help@spark.apache.org <mailto:user-help@spark.apache.org>
>
>
>
>
>         Links:
>         ------
>         [1]
>         https://spark.apache.org/docs/latest/configuration.html#spark-ui
>
>
>
>
>

-- 
*Max Schmidt, Senior Java Developer* | max@datapath.io
<mailto:max@datapath.io> | LinkedIn
<https://www.linkedin.com/in/maximilian-schmidt-9893b7bb/>
Datapath.io
 
Decreasing AWS latency.
Your traffic optimized.

Datapath.io GmbH
Mainz | HRB Nr. 46222
Sebastian Spies, CEO


Mime
View raw message