spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Jackson <ajack...@pobox.com>
Subject Re: Best way to determine # of workers
Date Thu, 24 Mar 2016 14:24:07 GMT
Well thats unfortunate, just means I have to scrape the webui for that
information.  As to why, I have a cluster that is being increased in size
to accommodate the processing requirements of a large set of jobs.  Its
useful to know when the new workers have joined the spark cluster.  In my
specific case, I may be growing the cluster size by a hundred nodes and if
I fail to wait for that initialization to complete the job will not have
enough memory to run my jobs.

Aaron

On Thu, Mar 24, 2016 at 3:07 AM, Takeshi Yamamuro <linguin.m.s@gmail.com>
wrote:

> Hi,
>
> There is no way to get such information from your app.
> Why do you need that?
>
> thanks,
> maropu
>
> On Thu, Mar 24, 2016 at 8:23 AM, Ajaxx <ajackson@pobox.com> wrote:
>
>> I'm building some elasticity into my model and I'd like to know when my
>> workers have come online.  It appears at present that the API only
>> supports
>> getting information about applications.  Is there a good way to determine
>> how many workers are available?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Best-way-to-determine-of-workers-tp26586.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
> ---
> Takeshi Yamamuro
>

Mime
View raw message