spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: RFC: [SPARK-529] Create constants for known config variables.
Date Tue, 24 Jun 2014 02:58:59 GMT
Hey Marcelo,

When we did the configuration pull request, we actually avoided having a big list of defaults
in one class file, because this creates a file that all the components in the project depend
on. For example, since we have some settings specific to streaming and the REPL, do we want
those settings to appear in a file that’s in “core”? It might be better to just make
sure we use constants for defaults in the code, or maybe have a separate class per project.

The other problem with this kind of change is that it’s disruptive to all the other ongoing
patches, so I wouldn’t consider it high-priority right now. We haven’t had a ton of problems
with settings being mistyped.

If you do want to do something like this though, apart from the comment above about modules,
please make sure this is not a public API. As soon as we add it to the API, it means we can’t
change or remove those config settings. I’d also suggest giving each config setting a single
name instead of having “ui”, “shuffle”, etc objects, since the chained calls to conf.ui.port.value
look somewhat confusing.

Matei

On Jun 23, 2014, at 3:57 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> I started with some code to implement an idea I had for SPARK-529, and
> before going much further (since it's a large and kinda boring change)
> I'd like to get some feedback from people.
> 
> Current code it at:
> https://github.com/vanzin/spark/tree/SPARK-529
> 
> There are still some parts I haven't fully fleshed out yet (see TODO
> list in the commit message), but that's the basic idea. Let me know if
> you have any feedback or different ideas.
> 
> Thanks!
> 
> 
> -- 
> Marcelo


Mime
View raw message