storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazansky, Michael" <michael.kazan...@jpmchase.com>
Subject RE: not all the servers in storm cluster processing messages
Date Thu, 09 Jun 2016 19:41:23 GMT
Thanks I will try it out.

Thanks,
Michael Kazansky

From: Nathan Leung [mailto:ncleung@gmail.com]
Sent: Thursday, June 09, 2016 3:07 PM
To: user
Subject: Re: not all the servers in storm cluster processing messages

If you use localOrShuffleGrouping, the tuple will stay in process if possible.  The solution,
assuming your bolts are distributed, is to use shuffleGrouping to send tuples from spout->bolt
so that the tuples are distributed to all of your workers, then use localOrShuffleGrouping
from bolt->bolt to reduce inter-worker communication overhead.

On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael <michael.kazansky@jpmchase.com<mailto:michael.kazansky@jpmchase.com>>
wrote:
We are using localOrShuffleGrouping

Thanks,
Michael Kazansky

From: Satish Duggana [mailto:satish.duggana@gmail.com<mailto:satish.duggana@gmail.com>]
Sent: Thursday, June 09, 2016 1:54 PM
To: user@storm.apache.org<mailto:user@storm.apache.org>
Subject: Re: not all the servers in storm cluster processing messages

Hi Michael,
Kafka spout is given parallelism-hint as 1 and the topic has only one partition. So, only
one kafka spout task is run in one of the workers of the topology. Are you asking that bolts
connected directly/indirectly from that kafka spout are all executed in the same worker as
kafka spout? If so, What are the groupings and parallellism-hint that you have configured
on those bolts?

Thanks,
Satish.


On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <michael.kazansky@jpmchase.com<mailto:michael.kazansky@jpmchase.com>>
wrote:
Hi All,
We have 3 servers in storm cluster. To prevent the order of the messages which come to Kafka
the Kafka partition was set to 1. KafkaSpout parallelism hint is set to 1 as well. When messages
flow through the Storm we noticed that only one server in the cluster is busy processing messages
other 2 are idle although the number of workers  is set to 3 in Config. Is it excepted behavior
or something is missing?
If it is expected behavior is there any way to have all 3 servers in the cluster processing
messages?
Storm version is 0.9.5
Thanks,
Michael Kazansky


This communication is for informational purposes only. It is not intended as an offer or solicitation
for the purchase or sale of any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not warranted as to completeness
or accuracy and are subject to change without notice. Any comments or statements made herein
do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates
(collectively, "JPMC"). This transmission may contain information that is proprietary, privileged,
confidential and/or exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any disclosure, copying, distribution, or use of the
information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you
received this transmission in error, please immediately contact the sender and destroy the
material in its entirety, whether in electronic or hard copy format. Although this transmission
and any attachments are believed to be free of any virus or other defect that might affect
any computer system into which it is received and opened, it is the responsibility of the
recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any
loss or damage arising in any way from its use. Please note that any electronic communication
that is conducted within or through JPMC's systems is subject to interception, monitoring,
review, retention and external production in accordance with JPMC's policy and local laws,
rules and regulations; may be stored or otherwise processed in countries other than the country
in which you are located; and will be treated in accordance with JPMC policies and applicable
laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures
relating to European legal entities.


This communication is for informational purposes only. It is not intended as an offer or solicitation
for the purchase or sale of any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not warranted as to completeness
or accuracy and are subject to change without notice. Any comments or statements made herein
do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates
(collectively, "JPMC"). This transmission may contain information that is proprietary, privileged,
confidential and/or exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any disclosure, copying, distribution, or use of the
information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you
received this transmission in error, please immediately contact the sender and destroy the
material in its entirety, whether in electronic or hard copy format. Although this transmission
and any attachments are believed to be free of any virus or other defect that might affect
any computer system into which it is received and opened, it is the responsibility of the
recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any
loss or damage arising in any way from its use. Please note that any electronic communication
that is conducted within or through JPMC's systems is subject to interception, monitoring,
review, retention and external production in accordance with JPMC's policy and local laws,
rules and regulations; may be stored or otherwise processed in countries other than the country
in which you are located; and will be treated in accordance with JPMC policies and applicable
laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures
relating to European legal entities.


This communication is for informational purposes only.  It is not intended as an offer or
solicitation for the purchase or sale of any financial instrument or as an official confirmation
of any transaction.  All market prices, data and other information are not warranted as to
completeness or accuracy and are subject to change without notice.  Any comments or statements
made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates (collectively, "JPMC").

This transmission may contain information that is proprietary, privileged, confidential and/or
exempt from disclosure under applicable law.  If you are not the intended recipient, you are
hereby notified that any disclosure, copying, distribution, or use of the information contained
herein (including any reliance thereon) is STRICTLY PROHIBITED.  If you received this transmission
in error, please immediately contact the sender and destroy the material in its entirety,
whether in electronic or hard copy format.  Although this transmission and any attachments
are believed to be free of any virus or other defect that might affect any computer system
into which it is received and opened, it is the responsibility of the recipient to ensure
that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising
in any way from its use.  Please note that any electronic communication that is conducted
within or through JPMC's systems is subject to interception, monitoring, review, retention
and external production in accordance with JPMC's policy and local laws, rules and regulations;
may be stored or otherwise processed in countries other than the country in which you are
located; and will be treated in accordance with JPMC policies and applicable laws and regulations.

Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European
legal entities.
Mime
View raw message