kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal C <kamaltar...@gmail.com>
Subject Re: Kafka consumers in cluster
Date Fri, 05 Aug 2016 04:05:07 GMT
Yes, it gets called on every re-balance.

-- Kamal

On Thu, Aug 4, 2016 at 11:24 PM, sat <sathish.alwar@gmail.com> wrote:

> Hi Kamal,
>
> Thanks for your prompt response. Does our custom partition assignor gets
> called during every rebalancing.
>
> Thanks and Regards
> A.SathishKumar
>
>
>
> >Implement your own custom
> >`org.apache.kafka.clients.consumer.internals.PartitionAssignor`
> >and assign all the subscribed partitions to the first consumer instance in
> >the group.
>
> >See 'partition.assignment.strategy' config in the consumer configs [1]
>
> >[1]: http://kafka.apache.org/documentation.html#newconsumerconfigs
>
>
>
> >On Thu, Aug 4, 2016 at 8:54 AM, sat <sathish.alwar@gmail.com> wrote:
>
> > Hi,
> >
> > We have Kafka server/broker running in a seperate machine (say machine
> A),
> > for now we are planning to have in one node. We have multiple topics and
> > all topics have only 1 partition for now.
> >
> > We have our application which includes Kafka consumers installed in
> machine
> > B and machine C. Our application in machine B and C are in cluster, hence
> > Kafka Consumers will also be in cluster. Both our consumers will have
> same
> > group id. We want all the messages to be consumed by consumer in machine
> B
> > and only when machine B is down consumer in machine C should pull
> messages.
> >
> > Since consumer in machine B and C have same group id, we came to know
> > consumer in machine B will get message for some time duration (10mins)
> and
> > then consumer in machine C will get message for some time duration. Since
> > our consumers are in cluster, we want only consumer to be active or
> receive
> > all the messages as long as it is alive.
> >
> > Please let us know how to achieve this.
> >
> >
> > Thanks and Regards
> > A.SathishKumar
> > 044-24735023
> >
>
>
>
> On Wed, Aug 3, 2016 at 8:24 PM, sat <sathish.alwar@gmail.com> wrote:
>
> > Hi,
> >
> > We have Kafka server/broker running in a seperate machine (say machine
> A),
> > for now we are planning to have in one node. We have multiple topics and
> > all topics have only 1 partition for now.
> >
> > We have our application which includes Kafka consumers installed in
> > machine B and machine C. Our application in machine B and C are in
> cluster,
> > hence Kafka Consumers will also be in cluster. Both our consumers will
> have
> > same group id. We want all the messages to be consumed by consumer in
> > machine B and only when machine B is down consumer in machine C should
> pull
> > messages.
> >
> > Since consumer in machine B and C have same group id, we came to know
> > consumer in machine B will get message for some time duration (10mins)
> and
> > then consumer in machine C will get message for some time duration. Since
> > our consumers are in cluster, we want only consumer to be active or
> receive
> > all the messages as long as it is alive.
> >
> > Please let us know how to achieve this.
> >
> >
> > Thanks and Regards
> > A.SathishKumar
> > 044-24735023
> >
>
>
>
> --
> A.SathishKumar
> 044-24735023
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message