storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Neilson <arsneil...@gmail.com>
Subject Re: Issues With Parallelism In Kafka Spout
Date Tue, 23 Dec 2014 19:21:36 GMT
Harsha asked a similar question, but I would definitely make certain that
messages being written to kafka are being partitioned the way you are
expecting. If you are trying to consume live events coming from a lone
kafka producer using default configuration, then messages are only going to
appear in one partition at a time.

See the Kafka FAQ for details
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

On Tue, Dec 23, 2014 at 10:22 AM, Nilesh Chhapru <
nilesh.chhapru@ugamsolutions.com> wrote:

>  Hi Harsha,
>
>
>
> PFB the link for git.
>
>
>
>
> https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
>
>
>
> Also  there is the caller program in with method : setHighLevelKafkaSpout
>
>
> https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
>
>
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Harsha [mailto:storm@harsha.io <storm@harsha.io>]
> *Sent:* 18 December 2014 10:34 PM
>
> *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
>
>
> Its going to be difficult without looking your code. If its on github do
> give us a link to it.
>
> -Harsha
>
>
>
>
>
> On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
>
>  Hi Harsha,
>
>
>
> The partitions and topics are having the data and the zookeeper also shows
> the group id of all the consumer in its logs.
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Harsha [mailto:storm@harsha.io <storm@harsha.io>]
> *Sent:* 18 December 2014 09:13 PM
> *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
>
>
> Hi Nilesh,
>
>             Did you check if your kafka topic partitions all being
> writtten to.
>
> -Harsha
>
>
>
>
>
> On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
>
>  Hi All,
>
>
>
> Please give some inputs as this is pending since long and need to meet the
> deadlines
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com
> <nilesh.chhapru@ugamsolutions.com>]
> *Sent:* 18 December 2014 01:24 PM
> *To:* user@storm.apache.org; users@kafka.apache.org
> *Subject:* Issues With Parallelism In Kafka Spout
>
>
>
> Hi All,
>
>
>
> I have implemented a high level Kafka consumer in Storm but looks like the
> parallelism isn’t getting achieved as I have 3 partitions and 2 task for
> the spout, but only one of it is emitting the data.
>
> PFB the screen grab for number of task of spout and data emitted by only
> one of them.
>
>
>
> Please assist on how to achieve parallelism using high level Kafka spout.
>
>
>
>
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
>
>  ------------------------------
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
>   ------------------------------
>
>
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
>
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
>
>
>   ------------------------------
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
>
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
>
> ------------------------------
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>

Mime
View raw message