kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avi Flax <avi.f...@parkassist.com>
Subject Re: Consumers in the same group but consuming different topics?
Date Thu, 15 Dec 2016 18:28:59 GMT

> On Dec 15, 2016, at 11:33, Damian Guy <damian.guy@gmail.com> wrote:
> Technically you can, but not without writing some code. If you want to use
> consumer groups then you would need to write a custom PartitionAssignor and
> configure it in your Consumer Config, like so:
> consumerProps.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> YourPartitionAssignor.class.getName());
> Alternatively you could use manual partition assignment to temporarily work
> around it.

Thanks Damian! Very helpful!

As an alternative, would there be any adverse effects of changing the config of a Kafka Streams
app to temporarily (say, for 30 minutes) consume from a smaller set of topics and then change
it right back?

To be clear, the workflow I have in mind is:

1. Stop app (gracefully)
2. Change config to have the app consume from a subset of its current topics
3. Start app
4. Wait ~30 minutes
5. Stop app (gracefully)
6. Revert config to the full set of topics
6. Start app

Would there be any adverse effects, like lost local state, lost offsets, etc?

It may be relevant that my app currently creates an instance of KafkaStreams for each topic
it consumes from.


Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/
View raw message