storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junguk Cho <jman...@gmail.com>
Subject Re: STORM REBALACING
Date Thu, 11 Aug 2016 21:53:19 GMT
I have one more question.

Based on some tests,  some workers totally shut down and re-run, which
means each task lost all the data.
Some workers do not shut down and are resumed after rebalance, which means
each task keeps the data.

So, I assume Storm has some capabilities to distinguish them based on input
parameter for rebalancing.
What is the condition to decide this?

Thanks,
Junguk

2016-08-11 17:33 GMT-04:00 Junguk Cho <jmanbal@gmail.com>:

> Hi.
>
> Thank you for replies.
> Now, it worked.
>
> 2. second, you should make sure your topology has been assigned more than
> 10 task. because the rebalacing is based on the task to do more workers. If
> you did not assigne 10 tasks(at least) to the topology explicitly, the
> rebalancing can not work out as you want.
> -> Yes. I re-interpret that "rebalance function" is redistributing tasks
> into executors.
>
> I have one following question.
> I don't see some benefits to have multiple tasks in one executor since one
> executor is a java thread and multiple tasks run in sequence based on a
> grouping.
>
> Does anyone explain when we can get benefit from multiple tasks?
>
>
> Thanks,
> Junguk
>
> 2016-08-11 1:13 GMT-04:00 fanxinpu@travelsky.com <fanxinpu@travelsky.com>:
>
>> hi Junquk:
>>
>> 1. first, you should make sure your cluster has more than 10 slots.
>> 2. second, you should make sure your topology has been assigned more than
>> 10 task. because the rebalacing is based on the task to do more workers. If
>> you did not assigne 10 tasks(at least) to the topology explicitly, the
>> rebalancing can not work out as you want.
>>
>>
>> ------------------------------
>>
>>
>>
>>
>> *From:* Junguk Cho <jmanbal@gmail.com>
>> *Date:* 2016-08-11 11:09
>> *To:* user <user@storm.apache.org>
>> *Subject:* STORM REBALACING
>> Hi, All.
>>
>> I tried to use STORM rebalance (http://storm.apache.org/relea
>> ses/1.0.1/Command-line-client.html).
>>
>> I used Storm 1.0.1 version.
>> I first ran 3 workers (one spout, one split and one count.) for WordCount
>> example.
>>
>> After running it, In commandline, I typed
>> ./storm rebalance WordCount -w 10 -n 6 -e spout=1 -e split=2 -e count=2
>>
>> I saw something in nimbus.log and UI like
>>  Delaying event :do-rebalance for 10 secs for WordCount-1-1470884051.
>> However, when I checked UI, it was still 3 workers.
>>
>>
>> Are there some requirements to use rebalance feature?
>> Also, in UI, there are no input interfaces to specify # of workers and #
>> of components except for waiting time for rebalace.
>>
>>
>> Thanks in advance.
>> - Junguk
>>
>>
>

Mime
View raw message