spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <>
Subject Re: spark config params conventions
Date Wed, 12 Mar 2014 21:50:15 GMT
One solution for typesafe config is to use
"spark.speculation" = true

Typesafe will recognize the key as a string rather than a path, so the name
will actually be "\"spark.speculation\"", so you need to handle this
contingency when passing the config operations to spark (stripping the
quotes from the key).

Solving this in Spark itself is a little tricky because there are ~5 such
conflicts (spark.serializer, spark.speculation, spark.locality.wait,
spark.shuffle.spill, and spark.cleaner.ttl), some of which are used pretty
frequently. We could provide aliases for all of these in Spark, but
actually deprecating the old ones would affect many users, so we could only
do that if enough users would benefit from fully hierarchical config

On Wed, Mar 12, 2014 at 9:24 AM, Mark Hamstra <>wrote:

> That's the whole reason why some of the intended configuration changes
> were backed out just before the 0.9.0 release.  It's a well-known issue,
> even if a completely satisfactory solution isn't as well-known and is
> probably something which should do another iteration on.
> On Wed, Mar 12, 2014 at 9:10 AM, Koert Kuipers <> wrote:
>> i am reading the spark configuration params from another configuration
>> object (typesafe config) before setting them as system properties.
>> i noticed typesafe config has trouble with settings like:
>> spark.speculation=true
>> spark.speculation.interval=0.5
>> the issue seems to be that if spark.speculation is a "container" that has
>> more values inside then it cannot be also a value itself, i think. so this
>> would work fine:
>> spark.speculation.enabled=true
>> spark.speculation.interval=0.5
>> just a heads up. i would probably suggest we avoid this situation.

View raw message