spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <>
Subject Re: Providing a namespace for third-party configurations
Date Fri, 30 Aug 2019 15:38:22 GMT
My 2 cents, I second with Sean, as it's not necessary for downstream
projects to have a rule for config name starting with "spark".

I guess I know why they want it, but then it should be clear what's the
goal. To differentiate with existing spark configs? Using another prefix
would help. To smoothly integrated with existing Spark configurations but
avoid the chance on collision? That makes sense, but I'd like to see
upstream projects supporting such case as I haven't seen such request .

On Sat, Aug 31, 2019 at 12:25 AM Sean Owen <> wrote:

> It's possible, but pretty unlikely to have an exact namespace
> collision. It's probably a best practice to clearly separates
> settings, etc that are downstream add-ons into a separate namespace,
> and I don't mind a sentence in a doc somewhere suggesting a
> convention, but I really think it's up to downstream projects to
> decide how they want to play it. If Spark did add a feature with this
> exact key, that'd be up to downstream projects to rationalize.
> On Fri, Aug 30, 2019 at 9:36 AM Nicholas Chammas
> <> wrote:
> >
> > I discovered today that EMR provides its own optimizations for Spark.
> Some of these optimizations are controlled by configuration settings with
> names like `spark.sql.dynamicPartitionPruning.enabled` or
> `spark.sql.optimizer.flattenScalarSubqueriesWithAggregates.enabled`. As far
> as I can tell, these are EMR-specific configurations.
> >
> > Does this create a potential problem, since it's possible that future
> Apache Spark configuration settings may end up colliding with these names
> selected by EMR?
> >
> > Should we document some sort of third-party configuration namespace
> pattern and encourage third parties to scope their custom configurations to
> that area? e.g. Something like `spark.external.[vendor].[whatever]`.
> >
> > Nick
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

Name : Jungtaek Lim
Blog :
Twitter :
LinkedIn :

View raw message