kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weide Zhang <weo...@gmail.com>
Subject Re: kafka consumer fail over
Date Sat, 02 Aug 2014 01:38:18 GMT
Hi Guozhang,

If I use high level consumer, how do I ensure all data goes to master even
if slave was up and running ? Is it just by forcing master to have enough
consumer thread to cover maximum number of partitions of a topic  since
high level consumer doesn't have assumption of consumers who are master and
consumers who are slave.

For example, master A initiate enough thread such that it can cover all the
partitions. slave B is standby with same consumer group and same number of
threads but since master A has enough thread to cover all the partitions.
Slave B won't get any data.

Suddenly master A goes down, slave B becomes new master, and it start to
get data based on high level consumer rebalance design.

After that old master A comes up and becomes slave, will A get data ?  Or A
will not get data because B has enough thread to cover all partitions in
the rebalancing logic.



On Fri, Aug 1, 2014 at 4:45 PM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hello Weide,
> That should be doable via high-level consumer, you can take a look at this
> page:
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> Guozhang
> On Fri, Aug 1, 2014 at 3:20 PM, Weide Zhang <weoccc@gmail.com> wrote:
> > Hi,
> >
> > I have a use case for a master slave  cluster where the logic inside
> master
> > need to consume data from kafka and publish some aggregated data to kafka
> > again. When master dies, slave need to take the latest committed offset
> > from master and continue consuming the data from kafka and doing the
> push.
> >
> > My questions is what will be easiest kafka consumer design for this
> > scenario to work ? I was thinking about using simpleconsumer and doing
> > manual consumer offset syncing between master and slave. That seems to
> > solve the problem but I was wondering if it can be achieved by using high
> > level consumer client ?
> >
> > Thanks,
> >
> > Weide
> >
> --
> -- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message