spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Best way to determine # of workers
Date Fri, 25 Mar 2016 15:54:42 GMT
Hi,

You may want to use SparkListener [1] (as webui) and listens to
SparkListenerExecutorAdded and SparkListenerExecutorRemoved.

[1] http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.scheduler.SparkListener

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Mar 24, 2016 at 3:24 PM, Aaron Jackson <ajackson@pobox.com> wrote:
> Well thats unfortunate, just means I have to scrape the webui for that
> information.  As to why, I have a cluster that is being increased in size to
> accommodate the processing requirements of a large set of jobs.  Its useful
> to know when the new workers have joined the spark cluster.  In my specific
> case, I may be growing the cluster size by a hundred nodes and if I fail to
> wait for that initialization to complete the job will not have enough memory
> to run my jobs.
>
> Aaron
>
> On Thu, Mar 24, 2016 at 3:07 AM, Takeshi Yamamuro <linguin.m.s@gmail.com>
> wrote:
>>
>> Hi,
>>
>> There is no way to get such information from your app.
>> Why do you need that?
>>
>> thanks,
>> maropu
>>
>> On Thu, Mar 24, 2016 at 8:23 AM, Ajaxx <ajackson@pobox.com> wrote:
>>>
>>> I'm building some elasticity into my model and I'd like to know when my
>>> workers have come online.  It appears at present that the API only
>>> supports
>>> getting information about applications.  Is there a good way to determine
>>> how many workers are available?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Best-way-to-determine-of-workers-tp26586.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>
>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message