spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject Re: Config properties broken in master
Date Thu, 23 Jan 2014 16:46:00 GMT
Hi Mridul,

Would you mind help to vet the latest 0.9 RC 4 candidate to make sure
the fixes are included since you raised the issues in prev RC?

Thanks,

Henry

On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan <mridul@gmail.com> wrote:
> Oh great, just saw the PR from Matei ... for some odd reason, the dev
> mails are coming to be horribly delayed.
>
>
> Thanks,
> Mridul
>
> On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwendell@gmail.com> wrote:
>> Hey Mridul this was patched and we cut a new release candidate. There
>> were several different config options which had a.b and a.b.c... they
>> should all work in the new RC.
>>
>> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mridul@gmail.com> wrote:
>>> Chanced upon spill related config which exhibit same pattern ...
>>>
>>> - Mridul
>>>
>>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <rxin@databricks.com> wrote:
>>>> I also just went over the config options to see how pervasive this is. In
>>>> addition to speculation, there is one more "conflict" of this kind:
>>>>
>>>> spark.locality.wait
>>>> spark.locality.wait.node
>>>> spark.locality.wait.process
>>>> spark.locality.wait.rack
>>>>
>>>>
>>>> spark.speculation
>>>> spark.speculation.interval
>>>> spark.speculation.multiplier
>>>> spark.speculation.quantile
>>>>
>>>>
>>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>>>>
>>>>> This is definitely an important issue to fix. Instead of renaming
>>>>> properties, one solution would be to replace Typesafe Config with just
>>>>> reading Java system properties, and disable config files for this release.
>>>>> I kind of like that over renaming.
>>>>>
>>>>> Matei
>>>>>
>>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mridul@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> >  Speculation was an example, there are others in spark which are
>>>>> > affected by this ...
>>>>> > Some of them have been around for a while, so will break existing
>>>>> code/scripts.
>>>>> >
>>>>> > Regards,
>>>>> > Mridul
>>>>> >
>>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcgill@gmail.com>
>>>>> wrote:
>>>>> >> change spark.speculation to spark.speculation.switch?
>>>>> >>
>>>>> >> maybe we can restrict that all properties in Spark should be
"three
>>>>> levels"
>>>>> >>
>>>>> >>
>>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mridul@gmail.com
>>>>> >wrote:
>>>>> >>
>>>>> >>> Hi,
>>>>> >>>
>>>>> >>>  Unless I am mistaken, the change to using typesafe ConfigFactory
has
>>>>> >>> broken some of the system properties we use in spark.
>>>>> >>>
>>>>> >>> For example: if we have both
>>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95
>>>>> >>> set, then the spark.speculation property is dropped.
>>>>> >>>
>>>>> >>> The rules of parseProperty actually document this clearly
[1]
>>>>> >>>
>>>>> >>>
>>>>> >>> I am not sure what the right fix here would be (other than
replacing
>>>>> >>> use of config that is).
>>>>> >>>
>>>>> >>> Any thoughts ?
>>>>> >>> I would vote -1 for 0.9 to be released before this is fixed.
>>>>> >>>
>>>>> >>>
>>>>> >>> Regards,
>>>>> >>> Mridul
>>>>> >>>
>>>>> >>>
>>>>> >>> [1]
>>>>> >>>
>>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29
>>>>> >>>
>>>>>
>>>>>

Mime
View raw message