Hi Andrew,

Thanks for your response, well I am already keeping the number of workers equal to the number of worker nodes as I have already set that property,

        Config config=new Config();
        config.put(Config.TOPOLOGY_WORKERS, 2);
        config.setNumAckers(2);
        config.setNumWorkers(2);
        config.setMaxSpoutPending(20);
        config.setMaxTaskParallelism(20);

What I am achieving right now with two nodes(worker) cluster is around ~1 Million msg/min which is around 16k msg/sec, which seems to be a good number with this k/o architecture.

I was wondering to go par the existing performance(since the CPU is highly utilized ~> 320% on 4 core) would it be better to add more nodes or still some config. tweaking can enhance the existing performance?

Thanks!


On Thu, Feb 26, 2015 at 1:24 PM, Andrew Neilson <arsneilson@gmail.com> wrote:
There is a configuration setting that you should be submitting to your topology, "topology.workers". You can see how many workers each topology is configured to use by running "storm list" under the "num workers" column. To confirm what Nathan suspects, you might see that this is set to 1 for your topology. You can also see how many workers your topology is configured to use from the Storm UI if you navigate to your running topology and just search the page for "topology.workers".

Changing the number of workers can be done through the configuration that you pass to the topology like (in Java):

Config conf = new Config();
conf.setNumWorkers(2);

By the way, you can find this info in the Storm tutorial: http://storm.apache.org/documentation/Tutorial.html. There is also some great documentation on understanding parallelism in Storm here: http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html

On Wed, Feb 25, 2015 at 9:39 PM, Vineet Mishra <clearmidoubt@gmail.com> wrote:

Hi Nathan,

I guess you missed the first mail of this thread.

As mentioned before I m having 3 node cluster out of which node 1 is for nimbus and ui while other 2 nodes are for worker and supervisor.

The specification of machine is 4 core intel xeon processor which currently is running with more than 300% cpu utilization.

At this case how can I Increase number of workers

Thanks!

On Feb 26, 2015 3:48 AM, "Nathan Leung" <ncleung@gmail.com> wrote:

If you have only one node on the cluster then yes I would recommend to add more nodes. If you have more than one I would recommend you increase the worker processes in the topology configuration to use the other hardware.

On Feb 25, 2015 10:38 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
If I am taking you right, by adding more workers you mean adding more nodes to the existing cluster or/and enhancing the existing configuration with perhaps 8 or 16 core cpu?

On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <ncleung@gmail.com> wrote:

In practice it's very difficult to fully maximize cpu usage, especially when each thread has to do things like wait for i/o in any way (network, disk, etc).  Since you're cpu limited, or very close to it, you should try adding more workers.

On Feb 25, 2015 7:48 AM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
Hi Nathan,

You are right, I guess that was the case.

But I am not sure why even after increasing the spouts to 20 I couldn't end up with max throughput, I hold 4 core Xeon processor Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU utilization reaching more than 350% but is it the only reason behind the less performance gain than expected or do I need to add some configuration tuning.

Thanks!

On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <ncleung@gmail.com> wrote:
It looks like you configured your topology to only use one worker process.

On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra <clearmidoubt@gmail.com> wrote:
Hi All,

I am running a 3 node storm cluster with a Topology which is subscribed to Kafka using Kafka-Storm Spout.

The Cluster is hosting Nimbus on a node and left with 2 other nodes for the workers and supervisor to function.

While running the Topology I could see that all the spouts(20) and bolts(5) tasks are running on the same machine leaving the other node instance without any CPU utilization.

[1-1] 31m 7s ip-20-0-0-75 6703 2264180 2264180 11.519 2262620 0
[2-2] 31m 7s ip-20-0-0-75 6703 841540 841540 19.717 840940 0
[3-3] 31m 7s ip-20-0-0-75 6703 1080080 1080080 18.105 1079240 0
[4-4] 31m 7s ip-20-0-0-75 6703 1113900 1113900 17.966 1105320 0
[5-5] 31m 7s ip-20-0-0-75 6703 854220 854220 19.342 856240 0
[6-6] 31m 7s ip-20-0-0-75 6703 1249640 1249640 17.071 1251320 0
[7-7] 31m 7s ip-20-0-0-75 6703 2266520 2266520 11.63 2267260 0
[8-8] 31m 7s ip-20-0-0-75 6703 629760 629760 20.8 629120 0
[9-9] 31m 7s ip-20-0-0-75 6703 854360 854360 19.513 851800 0
[10-10] 31m 7s ip-20-0-0-75 6703 1417380 1417380 15.78 1418200 0
[11-11] 31m 7s ip-20-0-0-75 6703 858620 858620 20.897 858180 0
[12-12] 31m 7s ip-20-0-0-75 6703 1242980 1242980 16.766 1241100 0
[13-13] 31m 7s ip-20-0-0-75 6703 352300 352300 19.975 349420 0
[14-14] 31m 7s ip-20-0-0-75 6703 627920 627920 20.329 625440 0
[15-15] 31m 7s ip-20-0-0-75 6703 1061880 1061880 18.263 1061300 0
[16-16] 31m 7s ip-20-0-0-75 6703 706480 706480 20.297 702720 0
[17-17] 31m 7s ip-20-0-0-75 6703 1518800 1518800 15.154 1514640 0
[18-18] 31m 7s ip-20-0-0-75 6703 622640 622640 20.743 626240 0
[19-19] 31m 7s ip-20-0-0-75 6703 3376960 3376960 8.361 3377060 0
[20-20] 31m 7s ip-20-0-0-75 6703 1044940 1044940 18.569 1043760 0

Same is the case for bolts,

[22-22]} 32m 13s ip-20-0-0-75 6703 0 0 0.002 0.05 1376560 0.042 1376540 0
[23-23]} 32m 13s ip-20-0-0-75 6703 0 0 0.001 0.021 347180 0.02 347180 0
[24-24]} 32m 13s ip-20-0-0-75 6703 0 0 0 0.023 34600 0.023 34600 0
[25-25]} 32m 13s ip-20-0-0-75 6703 0 0 0 0.017 8380 0.014 8380 0
[26-26]} 32m 13s ip-20-0-0-75 6703 0 0 0.082 0.039 22217940 0.041 22217920 0

ip-20-0-0-75 is only utilized while their is another host which is not utilized at all. Please find the supervisor summary below

8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m 32s 4 0
9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m 53s 4 1
This is not the regular behavior of Storm as I have seen both the nodes operating while running the jobs, but how we can comment on the stability of the storm with this kind off behavior.

Urgent Issue, Request for expert intervention.

Thanks!