storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Leung <ncle...@gmail.com>
Subject Re: Discrepancy in Storm while Job Distribution on available nodes
Date Wed, 25 Feb 2015 22:16:17 GMT
If you have only one node on the cluster then yes I would recommend to add
more nodes. If you have more than one I would recommend you increase the
worker processes in the topology configuration to use the other hardware.
On Feb 25, 2015 10:38 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:

> If I am taking you right, by adding more workers you mean adding more
> nodes to the existing cluster or/and enhancing the existing configuration
> with perhaps 8 or 16 core cpu?
>
> On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <ncleung@gmail.com> wrote:
>
>> In practice it's very difficult to fully maximize cpu usage, especially
>> when each thread has to do things like wait for i/o in any way (network,
>> disk, etc).  Since you're cpu limited, or very close to it, you should try
>> adding more workers.
>> On Feb 25, 2015 7:48 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
>>
>>> Hi Nathan,
>>>
>>> You are right, I guess that was the case.
>>>
>>> But I am not sure why even after increasing the spouts to 20 I couldn't
>>> end up with max throughput, I hold 4 core Xeon processor Intel(R) Xeon(R)
>>> CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU utilization
>>> reaching more than 350% but is it the only reason behind the less
>>> performance gain than expected or do I need to add some configuration
>>> tuning.
>>>
>>> Thanks!
>>>
>>> On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <ncleung@gmail.com> wrote:
>>>
>>>> It looks like you configured your topology to only use one worker
>>>> process.
>>>>
>>>> On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra <clearmidoubt@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am running a 3 node storm cluster with a Topology which is
>>>>> subscribed to Kafka using Kafka-Storm Spout.
>>>>>
>>>>> The Cluster is hosting Nimbus on a node and left with 2 other nodes
>>>>> for the workers and supervisor to function.
>>>>>
>>>>> While running the Topology I could see that all the spouts(20) and
>>>>> bolts(5) tasks are running on the same machine leaving the other node
>>>>> instance without any CPU utilization.
>>>>>
>>>>>   [1-1] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2264180 2264180
>>>>> 11.519 2262620 0  [2-2] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 841540 841540
>>>>> 19.717 840940 0  [3-3] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1080080 1080080
>>>>> 18.105 1079240 0  [4-4] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1113900 1113900
>>>>> 17.966 1105320 0  [5-5] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854220 854220
>>>>> 19.342 856240 0  [6-6] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1249640 1249640
>>>>> 17.071 1251320 0  [7-7] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2266520 2266520
>>>>> 11.63 2267260 0  [8-8] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 629760 629760
20.8
>>>>> 629120 0  [9-9] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854360 854360
>>>>> 19.513 851800 0  [10-10] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1417380 1417380
>>>>> 15.78 1418200 0  [11-11] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 858620 858620
>>>>> 20.897 858180 0  [12-12] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1242980 1242980
>>>>> 16.766 1241100 0  [13-13] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 352300 352300
>>>>> 19.975 349420 0  [14-14] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 627920 627920
>>>>> 20.329 625440 0  [15-15] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1061880 1061880
>>>>> 18.263 1061300 0  [16-16] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 706480 706480
>>>>> 20.297 702720 0  [17-17] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1518800 1518800
>>>>> 15.154 1514640 0  [18-18] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 622640 622640
>>>>> 20.743 626240 0  [19-19] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 3376960 3376960
>>>>> 8.361 3377060 0  [20-20] 31m 7s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1044940 1044940
>>>>> 18.569 1043760 0
>>>>>
>>>>> Same is the case for bolts,
>>>>>
>>>>>   [22-22]} 32m 13s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.002 0.05
>>>>> 1376560 0.042 1376540 0  [23-23]} 32m 13s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.001 0.021
>>>>> 347180 0.02 347180 0  [24-24]} 32m 13s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.023
34600
>>>>> 0.023 34600 0  [25-25]} 32m 13s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.017
8380
>>>>> 0.014 8380 0  [26-26]} 32m 13s ip-20-0-0-75 6703
>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.082 0.039
>>>>> 22217940 0.041 22217920 0
>>>>>
>>>>> ip-20-0-0-75 is only utilized while their is another host which is
>>>>> not utilized at all. Please find the supervisor summary below
>>>>>
>>>>>   8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m 32s 4 0
>>>>> 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m 53s 4 1
>>>>> This is not the regular behavior of Storm as I have seen both the
>>>>> nodes operating while running the jobs, but how we can comment on the
>>>>> stability of the storm with this kind off behavior.
>>>>>
>>>>> Urgent Issue, Request for expert intervention.
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>

Mime
View raw message