spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Using SHUFFLE_SERVICE_ENABLED for MesosCoarseGrainedSchedulerBackend, BlockManager, and Utils?
Date Mon, 27 Jun 2016 17:14:01 GMT
Thanks Sean. I'm going to create a JIRA for it and start the work under it.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Jun 27, 2016 at 9:19 AM, Sean Owen <sowen@cloudera.com> wrote:
> That seems OK. If it introduces another module dependency we'd have to
> think about it. I assume these constants should really be used
> consistently everywhere if possible, just because it otherwise means
> duplicating the defaults and possibly incorrectly. I think you could
> have a look at that more broadly too.
>
> On Sun, Jun 26, 2016 at 1:58 PM, Jacek Laskowski <jacek@japila.pl> wrote:
>> Hi,
>>
>> I've just noticed that there is the private[spark] val
>> SHUFFLE_SERVICE_ENABLED in package object config [1]
>>
>> [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L74-L75
>>
>> However MesosCoarseGrainedSchedulerBackend [2], BlockManager [3] and
>> Utils [4] are all using their own copies.
>>
>> Would that be acceptable* to send a pull request to get rid of this redundancy?
>>
>> [*] I'm staring at @srowen for his nodding in agreement :-)
>>
>> [2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L71
>> [3] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L73-L74
>> [4] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L748
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message