kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Widman <j...@jeffwidman.com>
Subject Re: Adding or removing input topics to a Kafka Consumer without downtime
Date Tue, 05 Sep 2017 21:01:25 GMT
I haven't played with the Streams app, but I know at the consumer group
level that consumers who are part of the same group can have different
subscriptions. So at the consumer group level at least this is
theoretically possible to have a rolling upgrade (and in fact was
purposefully designed to support this use case)

On Tue, Sep 5, 2017 at 7:00 AM, Håkon Åmdal <hakon.amdal2@gmail.com> wrote:

> Hi there,
>
> In our company, we are running multiple Kafka Streams app that are deployed
> in a red/black fashion. A deployment means starting a new set of hosts, run
> them in parallel with the old hosts until they pass the health check, and
> then scale down the old hosts. This approach works very well as we can
> deploy application changes without any downtime.
>
> However, as far as I understand, there is no way we can run a new and old
> version in parallel if they don’t consume from the same set of topics.
> Consumers will try to rebalance tasks between the two different application
> version, causing errors like the one below:
>
> Uncaught exception: Thread xyz-StreamThread-1 stopped unexpectedly after
> Assigned partition foo-1 for non-subscribed topic regex pattern;
> subscription pattern is bar
>
> To mitigate the problem, we have so far scaled down the old cluster to 0
> instances before deploying the new application if we’re adding or removing
> input topics. However, this causes service downtime which we no longer can
> accept.
>
> I’m curious to hear if anyone are experiencing the same issues, or if
> anyone have any thoughts or opinions? Are we doing something wrong, or is
> this something that can be solved by the Kafka Consumer client?
>
> Thanks,
>
> Håkon
>



-- 

*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message