kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Questions about new consumer API
Date Tue, 02 Dec 2014 23:06:13 GMT
The offsets are keyed on <group, topic, partition> so if you have more than
one owner per partition, they will rewrite each other's offsets and lead to
incorrect state.

On Tue, Dec 2, 2014 at 2:32 PM, hsy541@gmail.com <hsy541@gmail.com> wrote:

> Thanks Neha, another question, so if offsets are stored under group.id,
> dose it mean in one group, there should be at most one subscriber for each
> topic partition?
>
> Best,
> Siyuan
>
> On Tue, Dec 2, 2014 at 12:55 PM, Neha Narkhede <neha.narkhede@gmail.com>
> wrote:
>
> > 1. In this doc it says kafka consumer will automatically do load balance.
> > Is it based on throughtput or same as what we have now balance the
> > cardinality among all consumers in same ConsumerGroup? In a real case
> > different partitions could have different peak time.
> >
> > Load balancing is still based on # of partitions for the subscribed
> topics
> > and
> > ensuring that each partition has exactly one consumer as the owner.
> >
> > 2. In the API, threre is subscribe(partition...) method saying not using
> > group management, does it mean the group.id property will be discarded
> and
> > developer has full control of distributing partitions to consumers?
> >
> > group.id is also required for offset management, if the user chooses to
> > use
> > Kafka based offset management. The user will have full control over
> > distribution
> > of partitions to consumers.
> >
> > 3. Is new API compatible with old broker?
> >
> > Yes, it will.
> >
> > 4. Will simple consumer api and high-level consumer api still be
> supported?
> >
> > Over time, we will phase out the current high-level and simple consumer
> > since the
> > 0.9 API supports both.
> >
> > Thanks,
> > Neha
> >
> > On Tue, Dec 2, 2014 at 12:07 PM, hsy541@gmail.com <hsy541@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I'm interested in the new Consumer API.
> > > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/
> > >
> > > I have couple of question.
> > > 1. In this doc it says kafka consumer will automatically do load
> balance.
> > > Is it based on throughtput or same as what we have now balance the
> > > cardinality among all consumers in same ConsumerGroup? In a real case
> > > different partitions could have different peak time.
> > > 2. In the API, threre is subscribe(partition...) method saying not
> using
> > > group management, does it mean the group.id property will be discarded
> > and
> > > developer has full control of distributing partitions to consumers?
> > > 3. Is new API compatible with old broker?
> > > 4. Will simple consumer api and high-level consumer api still be
> > supported?
> > >
> > > Thanks!
> > >
> > > Best,
> > > Siyuan
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message