storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <st...@harsha.io>
Subject Re: Issues With Parallelism In Kafka Spout
Date Thu, 18 Dec 2014 16:20:13 GMT

Nilesh, storm-kafka in the storm package stores the offsets in zookeeper
not in kafka.

"Moreover the partition with standard one isn’t working as desired as it
stops the process in middle for some time if the partition is empty(eg
partitions one stops even if it has data if partitions 2 is empty)"

This seems suspicious to me. Have you given kafak zookeeper hosts to the
KafkaConfig ?

-Harsha


On Thu, Dec 18, 2014, at 01:52 AM, Nilesh Chhapru wrote:
> Hi Kobi,


>


> In standard implementation we have the offsets maintained by kafka
> which causes issues, need to implement high level consumer which
> maintains in the zookeeper.


>


> Moreover the partition with standard one isn’t working as desired as
> it stops the process in middle for some time if the partition is
> empty(eg partitions one
 stops even if it has data if partitions 2 is empty)


>


> *Regards*,


> *Nilesh Chhapru.*


>


> *From:* Kobi Salant [mailto:kobis@liveperson.com]
>
> *Sent:* 18 December 2014 02:02 PM *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout

>


> Hi Nilesh,


>


> Why don't you use the standard storm-contrib kafka spout?


> It has built in mechanism for parallelism that split the partitions
> according to the task number of the spout,


>


> Kobi


>


>


>


> On Thu, Dec 18, 2014 at 8:57 AM, Nilesh Chhapru
> <nilesh.chhapru@ugamsolutions.com> wrote:


> Hi All,


>


> I have implemented a high level Kafka consumer in Storm but looks like
> the parallelism isn’t getting achieved as I have 3 partitions and 2
> task for the spout, but only one of it
 is emitting the data.


> PFB the screen grab for number of task of spout and data emitted by
> only one of them.


>


> Please assist on how to achieve parallelism using high level
> Kafka spout.


>


>


>


> *Regards*,


> *Nilesh Chhapru.*


>


>


>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
 ****

>


> This message may contain confidential and/or privileged information.


> If you are not the addressee or authorized to receive this on behalf
> of the addressee you must not use, copy, disclose or take action based
> on this message or any information herein.


> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.


>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
 ****
> Email had 1 attachment:


>  * image001.png 68k (image/png)


Mime
View raw message