spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Moving to Typesafe Config?
Date Mon, 27 Jan 2014 18:05:12 GMT
Hey Heiko,

Spark 0.9 introduced a common config class for Spark applications. It
also (initially) supported loading config files in the nested typesafe
format, but this was removed last minute due to a bug. In 1.0 we'll
probably add support for config files, though it may not support
typesafe's tree-style config files because that conflicts with the
naming style of several spark options (we have options where x.y and
x.y.z are both named keys, and the typesafe parser doesn't allow
that).

- Patrick

On Mon, Jan 27, 2014 at 8:59 AM, Heiko Braun <ike.braun@googlemail.com> wrote:
> Thanks. I found the discussion myself ;)
>
> /heiko
>
>> Am 27.01.2014 um 17:34 schrieb Mark Hamstra <mark@clearstorydata.com>:
>>
>> And it would be more helpful if I gave you a usable link http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html
>>
>> Sent from my iPhone
>>
>>> On Jan 27, 2014, at 8:13 AM, Heiko Braun <ike.braun@googlemail.com> wrote:
>>>
>>> Thanks Mark.
>>>
>>>> On 27 Jan 2014, at 17:05, Mark Hamstra <mark@clearstorydata.com> wrote:
>>>>
>>>> Been done and undone, and will probably be redone for 1.0.  See
>>>> https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882
>>>>
>>>>
>>>> On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun <ike.braun@googlemail.com>wrote:
>>>>
>>>>>
>>>>> Is there any interest in moving to a more structured approach for
>>>>> configuring spark components? I.e. moving to the typesafe config [1].
Since
>>>>> spark already leverages akka, this seems to be a reasonable choice IMO.
>>>>>
>>>>> [1] https://github.com/typesafehub/config
>>>>>
>>>>> Regards, Heiko
>>>

Mime
View raw message