spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mridul Muralidharan <mri...@gmail.com>
Subject Re: Config properties broken in master
Date Sun, 19 Jan 2014 18:33:03 GMT
Oh great, just saw the PR from Matei ... for some odd reason, the dev
mails are coming to be horribly delayed.


Thanks,
Mridul

On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwendell@gmail.com> wrote:
> Hey Mridul this was patched and we cut a new release candidate. There
> were several different config options which had a.b and a.b.c... they
> should all work in the new RC.
>
> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mridul@gmail.com> wrote:
>> Chanced upon spill related config which exhibit same pattern ...
>>
>> - Mridul
>>
>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <rxin@databricks.com> wrote:
>>> I also just went over the config options to see how pervasive this is. In
>>> addition to speculation, there is one more "conflict" of this kind:
>>>
>>> spark.locality.wait
>>> spark.locality.wait.node
>>> spark.locality.wait.process
>>> spark.locality.wait.rack
>>>
>>>
>>> spark.speculation
>>> spark.speculation.interval
>>> spark.speculation.multiplier
>>> spark.speculation.quantile
>>>
>>>
>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>>>
>>>> This is definitely an important issue to fix. Instead of renaming
>>>> properties, one solution would be to replace Typesafe Config with just
>>>> reading Java system properties, and disable config files for this release.
>>>> I kind of like that over renaming.
>>>>
>>>> Matei
>>>>
>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mridul@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> >  Speculation was an example, there are others in spark which are
>>>> > affected by this ...
>>>> > Some of them have been around for a while, so will break existing
>>>> code/scripts.
>>>> >
>>>> > Regards,
>>>> > Mridul
>>>> >
>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcgill@gmail.com>
>>>> wrote:
>>>> >> change spark.speculation to spark.speculation.switch?
>>>> >>
>>>> >> maybe we can restrict that all properties in Spark should be "three
>>>> levels"
>>>> >>
>>>> >>
>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mridul@gmail.com
>>>> >wrote:
>>>> >>
>>>> >>> Hi,
>>>> >>>
>>>> >>>  Unless I am mistaken, the change to using typesafe ConfigFactory
has
>>>> >>> broken some of the system properties we use in spark.
>>>> >>>
>>>> >>> For example: if we have both
>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95
>>>> >>> set, then the spark.speculation property is dropped.
>>>> >>>
>>>> >>> The rules of parseProperty actually document this clearly [1]
>>>> >>>
>>>> >>>
>>>> >>> I am not sure what the right fix here would be (other than replacing
>>>> >>> use of config that is).
>>>> >>>
>>>> >>> Any thoughts ?
>>>> >>> I would vote -1 for 0.9 to be released before this is fixed.
>>>> >>>
>>>> >>>
>>>> >>> Regards,
>>>> >>> Mridul
>>>> >>>
>>>> >>>
>>>> >>> [1]
>>>> >>>
>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29
>>>> >>>
>>>>
>>>>

Mime
View raw message