spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: getting a list of executors for use in getPreferredLocations
Date Fri, 04 Mar 2016 01:51:35 GMT
Thanks.  That looks pretty similar to what I'm doing, with the
difference being getPeers vs getMemoryStatus.  Seems like they're both
backed by the same blockManagerInfo, but getPeers is filtering in a
way that looks close to what I need.  Is there a reason to prefer
getMemoryStatus?

On Thu, Mar 3, 2016 at 5:15 PM, Shixiong(Ryan) Zhu
<shixiong@databricks.com> wrote:
> You can take a look at
> "org.apache.spark.streaming.scheduler.ReceiverTracker#getExecutors"
>
> On Thu, Mar 3, 2016 at 3:10 PM, Reynold Xin <rxin@databricks.com> wrote:
>>
>> What do you mean by consistent? Throughout the life cycle of an app, the
>> executors can come and go and as a result really has no consistency. Do you
>> just need it for a specific job?
>>
>>
>>
>> On Thu, Mar 3, 2016 at 3:08 PM, Cody Koeninger <cody@koeninger.org> wrote:
>>>
>>> I need getPreferredLocations to choose a consistent executor for a
>>> given partition in a stream.  In order to do that, I need to know what
>>> the current executors are.
>>>
>>> I'm currently grabbing them from the block manager master .getPeers(),
>>> which works, but I don't know if that's the most reasonable way to do
>>> it.
>>>
>>> Relevant code:
>>>
>>>
>>> https://github.com/koeninger/spark-1/blob/aaef0fc6e7e3aae18e4e03271bc0707d09d243e4/external/kafka-beta/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L107
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message