kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: New consumer - consumer group init
Date Tue, 21 Jul 2015 09:13:54 GMT
Thanks all for fast feedback!

It's great news if that aspect is improved as well in new HLC. I will test
and get back with any related findings.

Kind regards,
Stevo Slavic.

On Mon, Jul 20, 2015 at 9:57 PM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hi Stevo,
>
> I am still not very clear on your point yet, I guess I was trying to figure
> out under which circumstances would users prefer to re-set the group id at
> an existing consumer rather than creating a new instance. As Jason
> mentioned, since the new consumer is single threaded it should usually be
> cheap to construct.
>
> Guozhang
>
> On Mon, Jul 20, 2015 at 11:06 AM, Jason Gustafson <jason@confluent.io>
> wrote:
>
> > Hey Stevo,
> >
> > The new consumer doesn't have any threads of its own, so I think
> > construction should be fairly cheap.
> >
> > -Jason
> >
> > On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić <sslavic@gmail.com> wrote:
> >
> > > Hello Guozhang,
> > >
> > > It would be enough if consumer group could, besides at construction
> time,
> > > be set once only after construction. Have to retest, but high level
> > > consumer in 0.8.1 used to be very heavy weight object (lots of threads
> > > started, and it would block and take time to construct it). It's
> > > understandable, considering all of the high level features it has, and
> > > since it's supposed to be long living object. What would improve with
> > this
> > > change is that construction penalty could be paid upfront, while topic
> > > subscription and joining consumer group ensemble could be done on first
> > > use, so that first use does not have to suffer from both init and
> > > subscription penalties.
> > >
> > > It would be nice also if consumer group just as subscription could be
> > > changed later even, so multiple times throughout lifetime of high level
> > > consumer instance, to avoid constructing new consumer when instance
> > purpose
> > > changes.
> > >
> > > After looking more into the HLC API, thought maybe this is not needed,
> > > since there is "public void subscribe(TopicPartition... partitions)"
> > which
> > > does not use consumer group management, but problem is that there is no
> > > matching explicit commit where one could pass consumer group parameter
> as
> > > well, to label for which consumer group should offset(s) be committed.
> > >
> > > Seems like new HLC has split personality. Maybe (at least) two APIs
> could
> > > have been provided instead of one with such differing behaviors.
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > >
> > > On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang <wangguoz@gmail.com>
> > > wrote:
> > >
> > > > Hi Stevo,
> > > >
> > > > Hmm this is interesting, do you have any use cases in mind that need
> > > > dynamic group changing?
> > > >
> > > > Guozhang
> > > >
> > > > On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić <sslavic@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Apache Kafka community,
> > > > >
> > > > > In new KafkaConsumer API on trunk, it seems it's only possible to
> > > define
> > > > > consumer group id at construction time of KafkaConsumer, through
> > > property
> > > > > with group.id key.
> > > > >
> > > > > Would it make sense and be possible to support setting/changing
> > > consumer
> > > > > group id after construction, but before it's actually used for the
> > > first
> > > > > time, similar to how subscription is supported through "public void
> > > > > subscribe(String... topics)"?
> > > > >
> > > > > Maybe this can be done through additional method "public void
> > > > > subscribe(String consumerGroupId, String... topics)" which would
> > first
> > > > set
> > > > > provided consumer group id in coordinator and then call "public
> void
> > > > > subscribe(String... topics)".
> > > > >
> > > > > Kind regards,
> > > > > Stevo Slavic.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message