kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Consumers in the same group but consuming different topics?
Date Mon, 19 Dec 2016 21:53:39 GMT
Hello Avi,

Similar questions have been discussed previously:

https://issues.apache.org/jira/browse/KAFKA-3775

One concern of doing this, though is that if you want to do the change
on-the-fly by rolling bounce (i.e. combining step 5 6 and 7) the instances
to switch back the assignor class, then there will be a period of time
while some instances are using the old assignor while some other are using
the new one, hence undefined behavior.

But since you are doing steps 5 6 7 consecutively I think it should be fine.


BTW regarding your first email, it actually is OK to have consumers in the
same group to be subscribing to different topics (e.g. consumerA
subscribing to topic 1 2 3, consumerB subscribing to topic 4 5 6), and the
topic partitions will then be assigned correspondingly if you are using the
consumer's default assignor, however in streams partition assignor we did
not do this yet.


Guozhang

On Thu, Dec 15, 2016 at 10:28 AM, Avi Flax <avi.flax@parkassist.com> wrote:

>
> > On Dec 15, 2016, at 11:33, Damian Guy <damian.guy@gmail.com> wrote:
> >
> > Technically you can, but not without writing some code. If you want to
> use
> > consumer groups then you would need to write a custom PartitionAssignor
> and
> > configure it in your Consumer Config, like so:
> > consumerProps.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> > YourPartitionAssignor.class.getName());
> >
> > Alternatively you could use manual partition assignment to temporarily
> work
> > around it.
>
> Thanks Damian! Very helpful!
>
> As an alternative, would there be any adverse effects of changing the
> config of a Kafka Streams app to temporarily (say, for 30 minutes) consume
> from a smaller set of topics and then change it right back?
>
> To be clear, the workflow I have in mind is:
>
> 1. Stop app (gracefully)
> 2. Change config to have the app consume from a subset of its current
> topics
> 3. Start app
> 4. Wait ~30 minutes
> 5. Stop app (gracefully)
> 6. Revert config to the full set of topics
> 6. Start app
>
> Would there be any adverse effects, like lost local state, lost offsets,
> etc?
>
> It may be relevant that my app currently creates an instance of
> KafkaStreams for each topic it consumes from.
>
> Thanks!
> Avi
>
> ————
> Software Architect @ Park Assist
> We’re hiring! http://tech.parkassist.com/jobs/
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message