spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Config properties broken in master
Date Sun, 19 Jan 2014 17:05:04 GMT
Hey Mridul this was patched and we cut a new release candidate. There
were several different config options which had a.b and a.b.c... they
should all work in the new RC.

On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mridul@gmail.com> wrote:
> Chanced upon spill related config which exhibit same pattern ...
>
> - Mridul
>
> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <rxin@databricks.com> wrote:
>> I also just went over the config options to see how pervasive this is. In
>> addition to speculation, there is one more "conflict" of this kind:
>>
>> spark.locality.wait
>> spark.locality.wait.node
>> spark.locality.wait.process
>> spark.locality.wait.rack
>>
>>
>> spark.speculation
>> spark.speculation.interval
>> spark.speculation.multiplier
>> spark.speculation.quantile
>>
>>
>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>>
>>> This is definitely an important issue to fix. Instead of renaming
>>> properties, one solution would be to replace Typesafe Config with just
>>> reading Java system properties, and disable config files for this release.
>>> I kind of like that over renaming.
>>>
>>> Matei
>>>
>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mridul@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> >  Speculation was an example, there are others in spark which are
>>> > affected by this ...
>>> > Some of them have been around for a while, so will break existing
>>> code/scripts.
>>> >
>>> > Regards,
>>> > Mridul
>>> >
>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcgill@gmail.com>
>>> wrote:
>>> >> change spark.speculation to spark.speculation.switch?
>>> >>
>>> >> maybe we can restrict that all properties in Spark should be "three
>>> levels"
>>> >>
>>> >>
>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mridul@gmail.com
>>> >wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>>  Unless I am mistaken, the change to using typesafe ConfigFactory
has
>>> >>> broken some of the system properties we use in spark.
>>> >>>
>>> >>> For example: if we have both
>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95
>>> >>> set, then the spark.speculation property is dropped.
>>> >>>
>>> >>> The rules of parseProperty actually document this clearly [1]
>>> >>>
>>> >>>
>>> >>> I am not sure what the right fix here would be (other than replacing
>>> >>> use of config that is).
>>> >>>
>>> >>> Any thoughts ?
>>> >>> I would vote -1 for 0.9 to be released before this is fixed.
>>> >>>
>>> >>>
>>> >>> Regards,
>>> >>> Mridul
>>> >>>
>>> >>>
>>> >>> [1]
>>> >>>
>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29
>>> >>>
>>>
>>>

Mime
View raw message