spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Config properties broken in master
Date Sat, 18 Jan 2014 20:27:33 GMT
Hey Mark - ya if we did add this I think it would be in the next major release.

On Sat, Jan 18, 2014 at 12:17 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
> That later release should be at least 0.10.0, then, since use of config
> files won't be backward compatible with 0.9.0.
>
>
> On Sat, Jan 18, 2014 at 12:11 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>
>> We can add config files in a later release. They were never officially
>> released, and were only in master for about a month.
>>
>> One other thing to note is that the config file feature is kind of limited
>> anyway. Users will want to have a separate config file with each app, which
>> they have to ship with its classpath, and there’s no great way of merging
>> them in the current setup. I’m not actually sure it’s a feature we want to
>> support, compared to say just a SparkConf.fromFile method that reads a Java
>> Properties file.
>>
>> Matei
>>
>> On Jan 18, 2014, at 12:01 PM, Mark Hamstra <mark@clearstorydata.com>
>> wrote:
>>
>> > Really?  Disabling config files seems to me to be a bigger/more onerous
>> > change for users than spark.speculation=true|false =>
>> > spark.speculation.enabled=true|false and spark.locality.wait =>
>> > spark.locality.wait.default.
>> >
>> >
>> > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaharia@gmail.com
>> >wrote:
>> >
>> >> This is definitely an important issue to fix. Instead of renaming
>> >> properties, one solution would be to replace Typesafe Config with just
>> >> reading Java system properties, and disable config files for this
>> release.
>> >> I kind of like that over renaming.
>> >>
>> >> Matei
>> >>
>> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mridul@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Speculation was an example, there are others in spark which are
>> >>> affected by this ...
>> >>> Some of them have been around for a while, so will break existing
>> >> code/scripts.
>> >>>
>> >>> Regards,
>> >>> Mridul
>> >>>
>> >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcgill@gmail.com>
>> >> wrote:
>> >>>> change spark.speculation to spark.speculation.switch?
>> >>>>
>> >>>> maybe we can restrict that all properties in Spark should be "three
>> >> levels"
>> >>>>
>> >>>>
>> >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <
>> mridul@gmail.com
>> >>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory
has
>> >>>>> broken some of the system properties we use in spark.
>> >>>>>
>> >>>>> For example: if we have both
>> >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95
>> >>>>> set, then the spark.speculation property is dropped.
>> >>>>>
>> >>>>> The rules of parseProperty actually document this clearly [1]
>> >>>>>
>> >>>>>
>> >>>>> I am not sure what the right fix here would be (other than replacing
>> >>>>> use of config that is).
>> >>>>>
>> >>>>> Any thoughts ?
>> >>>>> I would vote -1 for 0.9 to be released before this is fixed.
>> >>>>>
>> >>>>>
>> >>>>> Regards,
>> >>>>> Mridul
>> >>>>>
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> >>
>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29
>> >>>>>
>> >>
>> >>
>>
>>

Mime
View raw message