storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Leung <ncle...@gmail.com>
Subject Re: not all the servers in storm cluster processing messages
Date Thu, 09 Jun 2016 19:06:50 GMT
If you use localOrShuffleGrouping, the tuple will stay in process if
possible.  The solution, assuming your bolts are distributed, is to use
shuffleGrouping to send tuples from spout->bolt so that the tuples are
distributed to all of your workers, then use localOrShuffleGrouping from
bolt->bolt to reduce inter-worker communication overhead.

On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael <
michael.kazansky@jpmchase.com> wrote:

> We are using localOrShuffleGrouping
>
>
>
> Thanks,
>
> Michael Kazansky
>
>
>
> *From:* Satish Duggana [mailto:satish.duggana@gmail.com]
> *Sent:* Thursday, June 09, 2016 1:54 PM
> *To:* user@storm.apache.org
> *Subject:* Re: not all the servers in storm cluster processing messages
>
>
>
> Hi Michael,
>
> Kafka spout is given parallelism-hint as 1 and the topic has only one
> partition. So, only one kafka spout task is run in one of the workers of
> the topology. Are you asking that bolts connected directly/indirectly from
> that kafka spout are all executed in the same worker as kafka spout? If so,
> What are the groupings and parallellism-hint that you have configured on
> those bolts?
>
>
>
> Thanks,
>
> Satish.
>
>
>
>
>
> On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <
> michael.kazansky@jpmchase.com> wrote:
>
> Hi All,
>
> We have 3 servers in storm cluster. To prevent the order of the messages
> which come to Kafka the Kafka partition was set to 1. KafkaSpout
> parallelism hint is set to 1 as well. When messages flow through the Storm
> we noticed that only one server in the cluster is busy processing messages
> other 2 are idle although the number of workers  is set to 3 in Config. Is
> it excepted behavior or something is missing?
>
> If it is expected behavior is there any way to have all 3 servers in the
> cluster processing messages?
>
> Storm version is 0.9.5
>
> Thanks,
>
> Michael Kazansky
>
>
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
> transmission may contain information that is proprietary, privileged,
> confidential and/or exempt from disclosure under applicable law. If you are
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
> this transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard copy
> format. Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer system
> into which it is received and opened, it is the responsibility of the
> recipient to ensure that it is virus free and no responsibility is accepted
> by JPMC for any loss or damage arising in any way from its use. Please note
> that any electronic communication that is conducted within or through
> JPMC's systems is subject to interception, monitoring, review, retention
> and external production in accordance with JPMC's policy and local laws,
> rules and regulations; may be stored or otherwise processed in countries
> other than the country in which you are located; and will be treated in
> accordance with JPMC policies and applicable laws and regulations. Please
> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
> relating to European legal entities.
>
>
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
> transmission may contain information that is proprietary, privileged,
> confidential and/or exempt from disclosure under applicable law. If you are
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
> this transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard copy
> format. Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer system
> into which it is received and opened, it is the responsibility of the
> recipient to ensure that it is virus free and no responsibility is accepted
> by JPMC for any loss or damage arising in any way from its use. Please note
> that any electronic communication that is conducted within or through
> JPMC's systems is subject to interception, monitoring, review, retention
> and external production in accordance with JPMC's policy and local laws,
> rules and regulations; may be stored or otherwise processed in countries
> other than the country in which you are located; and will be treated in
> accordance with JPMC policies and applicable laws and regulations. Please
> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
> relating to European legal entities.
>

Mime
View raw message