kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Hauch <rha...@gmail.com>
Subject Re: Kafka Connect sink connector in distributed mode: how are records distributed to workers?
Date Mon, 11 Sep 2017 17:39:55 GMT
Connect uses Kafka's group coordination mechanism and normal consumers to
balance/assign to the various sink tasks the topic partitions for the
subscribed topics. A task will continue to receive the records from its
assigned topic partitions until a rebalance occurs, which can happen when
any connector config is added/changed/removed or when a worker adds/leaves
the cluster.

On Sun, Sep 10, 2017 at 10:40 PM, Behrang Saeedzadeh <behrangsa@gmail.com>

> Hi,
> How does Kafka Connect distribute records between workers for a sink
> connector when the connector is only configured to get data from one topic?
> * Does it ensure all records in a given partition are sent to the same
> worker instance?
> * When a new worker is added to the cluster, what steps are taken by
> Connect? In particular, if Connect unassigns some partitions from the given
> workers and assigns them to this new worker, would it wait for all
> in-flight records belonging to these partitions to be processed by the
> existing workers?
> Best regards,
> Behrang Saeedzadeh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message