kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: New consumer - consumer group init
Date Sun, 19 Jul 2015 21:13:32 GMT
Hello Guozhang,

It would be enough if consumer group could, besides at construction time,
be set once only after construction. Have to retest, but high level
consumer in 0.8.1 used to be very heavy weight object (lots of threads
started, and it would block and take time to construct it). It's
understandable, considering all of the high level features it has, and
since it's supposed to be long living object. What would improve with this
change is that construction penalty could be paid upfront, while topic
subscription and joining consumer group ensemble could be done on first
use, so that first use does not have to suffer from both init and
subscription penalties.

It would be nice also if consumer group just as subscription could be
changed later even, so multiple times throughout lifetime of high level
consumer instance, to avoid constructing new consumer when instance purpose

After looking more into the HLC API, thought maybe this is not needed,
since there is "public void subscribe(TopicPartition... partitions)" which
does not use consumer group management, but problem is that there is no
matching explicit commit where one could pass consumer group parameter as
well, to label for which consumer group should offset(s) be committed.

Seems like new HLC has split personality. Maybe (at least) two APIs could
have been provided instead of one with such differing behaviors.

Kind regards,
Stevo Slavic.

On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hi Stevo,
> Hmm this is interesting, do you have any use cases in mind that need
> dynamic group changing?
> Guozhang
> On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <sslavic@gmail.com> wrote:
> > Hello Apache Kafka community,
> >
> > In new KafkaConsumer API on trunk, it seems it's only possible to define
> > consumer group id at construction time of KafkaConsumer, through property
> > with group.id key.
> >
> > Would it make sense and be possible to support setting/changing consumer
> > group id after construction, but before it's actually used for the first
> > time, similar to how subscription is supported through "public void
> > subscribe(String... topics)"?
> >
> > Maybe this can be done through additional method "public void
> > subscribe(String consumerGroupId, String... topics)" which would first
> set
> > provided consumer group id in coordinator and then call "public void
> > subscribe(String... topics)".
> >
> > Kind regards,
> > Stevo Slavic.
> >
> --
> -- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message