storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Mishra <clearmido...@gmail.com>
Subject Re: Discrepancy in Storm while Job Distribution on available nodes
Date Thu, 26 Feb 2015 12:51:37 GMT
Hi Andrew,

Thanks for your response, well I am already keeping the number of workers
equal to the number of worker nodes as I have already set that property,

        Config config=new Config();
        config.put(Config.TOPOLOGY_WORKERS, 2);
        config.setNumAckers(2);
        config.setNumWorkers(2);
        config.setMaxSpoutPending(20);
        config.setMaxTaskParallelism(20);

What I am achieving right now with two nodes(worker) cluster is around ~1
Million msg/min which is around 16k msg/sec, which seems to be a good
number with this k/o architecture.

I was wondering to go par the existing performance(since the CPU is highly
utilized ~> 320% on 4 core) would it be better to add more nodes or still
some config. tweaking can enhance the existing performance?

Thanks!


On Thu, Feb 26, 2015 at 1:24 PM, Andrew Neilson <arsneilson@gmail.com>
wrote:

> There is a configuration setting that you should be submitting to your
> topology, "topology.workers". You can see how many workers each topology is
> configured to use by running "storm list" under the "num workers" column.
> To confirm what Nathan suspects, you might see that this is set to 1 for
> your topology. You can also see how many workers your topology is
> configured to use from the Storm UI if you navigate to your running
> topology and just search the page for "topology.workers".
>
> Changing the number of workers can be done through the configuration that
> you pass to the topology like (in Java):
>
> Config conf = new Config();
> conf.setNumWorkers(2);
>
> By the way, you can find this info in the Storm tutorial:
> http://storm.apache.org/documentation/Tutorial.html. There is also some
> great documentation on understanding parallelism in Storm here:
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>
> On Wed, Feb 25, 2015 at 9:39 PM, Vineet Mishra <clearmidoubt@gmail.com>
> wrote:
>
>> Hi Nathan,
>>
>> I guess you missed the first mail of this thread.
>>
>> As mentioned before I m having 3 node cluster out of which node 1 is for
>> nimbus and ui while other 2 nodes are for worker and supervisor.
>>
>> The specification of machine is 4 core intel xeon processor which
>> currently is running with more than 300% cpu utilization.
>>
>> At this case how can I Increase number of workers
>>
>> Thanks!
>> On Feb 26, 2015 3:48 AM, "Nathan Leung" <ncleung@gmail.com> wrote:
>>
>>> If you have only one node on the cluster then yes I would recommend to
>>> add more nodes. If you have more than one I would recommend you increase
>>> the worker processes in the topology configuration to use the other
>>> hardware.
>>> On Feb 25, 2015 10:38 AM, "Vineet Mishra" <clearmidoubt@gmail.com>
>>> wrote:
>>>
>>>> If I am taking you right, by adding more workers you mean adding more
>>>> nodes to the existing cluster or/and enhancing the existing configuration
>>>> with perhaps 8 or 16 core cpu?
>>>>
>>>> On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <ncleung@gmail.com>
>>>> wrote:
>>>>
>>>>> In practice it's very difficult to fully maximize cpu usage,
>>>>> especially when each thread has to do things like wait for i/o in any
way
>>>>> (network, disk, etc).  Since you're cpu limited, or very close to it,
you
>>>>> should try adding more workers.
>>>>> On Feb 25, 2015 7:48 AM, "Vineet Mishra" <clearmidoubt@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nathan,
>>>>>>
>>>>>> You are right, I guess that was the case.
>>>>>>
>>>>>> But I am not sure why even after increasing the spouts to 20 I
>>>>>> couldn't end up with max throughput, I hold 4 core Xeon processor
Intel(R)
>>>>>> Xeon(R) CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU
>>>>>> utilization reaching more than 350% but is it the only reason behind
the
>>>>>> less performance gain than expected or do I need to add some configuration
>>>>>> tuning.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <ncleung@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It looks like you configured your topology to only use one worker
>>>>>>> process.
>>>>>>>
>>>>>>> On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra <
>>>>>>> clearmidoubt@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I am running a 3 node storm cluster with a Topology which
is
>>>>>>>> subscribed to Kafka using Kafka-Storm Spout.
>>>>>>>>
>>>>>>>> The Cluster is hosting Nimbus on a node and left with 2 other
nodes
>>>>>>>> for the workers and supervisor to function.
>>>>>>>>
>>>>>>>> While running the Topology I could see that all the spouts(20)
and
>>>>>>>> bolts(5) tasks are running on the same machine leaving the
other node
>>>>>>>> instance without any CPU utilization.
>>>>>>>>
>>>>>>>>   [1-1] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
2264180 2264180
>>>>>>>> 11.519 2262620 0  [2-2] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
841540 841540
>>>>>>>> 19.717 840940 0  [3-3] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1080080 1080080
>>>>>>>> 18.105 1079240 0  [4-4] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1113900 1113900
>>>>>>>> 17.966 1105320 0  [5-5] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
854220 854220
>>>>>>>> 19.342 856240 0  [6-6] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1249640 1249640
>>>>>>>> 17.071 1251320 0  [7-7] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
2266520 2266520
>>>>>>>> 11.63 2267260 0  [8-8] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
629760 629760
>>>>>>>> 20.8 629120 0  [9-9] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
854360 854360
>>>>>>>> 19.513 851800 0  [10-10] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1417380 1417380
>>>>>>>> 15.78 1418200 0  [11-11] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
858620 858620
>>>>>>>> 20.897 858180 0  [12-12] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1242980 1242980
>>>>>>>> 16.766 1241100 0  [13-13] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
352300 352300
>>>>>>>> 19.975 349420 0  [14-14] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
627920 627920
>>>>>>>> 20.329 625440 0  [15-15] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1061880 1061880
>>>>>>>> 18.263 1061300 0  [16-16] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
706480 706480
>>>>>>>> 20.297 702720 0  [17-17] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1518800 1518800
>>>>>>>> 15.154 1514640 0  [18-18] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
622640 622640
>>>>>>>> 20.743 626240 0  [19-19] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
3376960 3376960
>>>>>>>> 8.361 3377060 0  [20-20] 31m 7s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
1044940 1044940
>>>>>>>> 18.569 1043760 0
>>>>>>>>
>>>>>>>> Same is the case for bolts,
>>>>>>>>
>>>>>>>>   [22-22]} 32m 13s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
0 0 0.002 0.05
>>>>>>>> 1376560 0.042 1376540 0  [23-23]} 32m 13s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
0 0 0.001 0.021
>>>>>>>> 347180 0.02 347180 0  [24-24]} 32m 13s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
0 0 0 0.023
>>>>>>>> 34600 0.023 34600 0  [25-25]} 32m 13s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
0 0 0 0.017
>>>>>>>> 8380 0.014 8380 0  [26-26]} 32m 13s ip-20-0-0-75 6703
>>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log>
0 0 0.082 0.039
>>>>>>>> 22217940 0.041 22217920 0
>>>>>>>>
>>>>>>>> ip-20-0-0-75 is only utilized while their is another host
which is
>>>>>>>> not utilized at all. Please find the supervisor summary below
>>>>>>>>
>>>>>>>>   8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h
7m 32s 4
>>>>>>>> 0  9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h
57m 53s
>>>>>>>> 4 1
>>>>>>>> This is not the regular behavior of Storm as I have seen
both the
>>>>>>>> nodes operating while running the jobs, but how we can comment
on the
>>>>>>>> stability of the storm with this kind off behavior.
>>>>>>>>
>>>>>>>> Urgent Issue, Request for expert intervention.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>

Mime
View raw message