spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: Config properties broken in master
Date Sat, 18 Jan 2014 19:40:23 GMT
I also just went over the config options to see how pervasive this is. In
addition to speculation, there is one more "conflict" of this kind:

spark.locality.wait
spark.locality.wait.node
spark.locality.wait.process
spark.locality.wait.rack


spark.speculation
spark.speculation.interval
spark.speculation.multiplier
spark.speculation.quantile


On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> This is definitely an important issue to fix. Instead of renaming
> properties, one solution would be to replace Typesafe Config with just
> reading Java system properties, and disable config files for this release.
> I kind of like that over renaming.
>
> Matei
>
> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mridul@gmail.com>
> wrote:
>
> > Hi,
> >
> >  Speculation was an example, there are others in spark which are
> > affected by this ...
> > Some of them have been around for a while, so will break existing
> code/scripts.
> >
> > Regards,
> > Mridul
> >
> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcgill@gmail.com>
> wrote:
> >> change spark.speculation to spark.speculation.switch?
> >>
> >> maybe we can restrict that all properties in Spark should be "three
> levels"
> >>
> >>
> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mridul@gmail.com
> >wrote:
> >>
> >>> Hi,
> >>>
> >>>  Unless I am mistaken, the change to using typesafe ConfigFactory has
> >>> broken some of the system properties we use in spark.
> >>>
> >>> For example: if we have both
> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95
> >>> set, then the spark.speculation property is dropped.
> >>>
> >>> The rules of parseProperty actually document this clearly [1]
> >>>
> >>>
> >>> I am not sure what the right fix here would be (other than replacing
> >>> use of config that is).
> >>>
> >>> Any thoughts ?
> >>> I would vote -1 for 0.9 to be released before this is fixed.
> >>>
> >>>
> >>> Regards,
> >>> Mridul
> >>>
> >>>
> >>> [1]
> >>>
> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29
> >>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message