kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka connect distributed mode not distributing the work
Date Sun, 18 Dec 2016 00:54:58 GMT
Hi Manjunath,

I think you're seeing a case of this issue: https://issues.apache.
org/jira/browse/KAFKA-4553 where the way round robin assignment works with
an even # of workers and connectors that generate only 1 task generates
uneven work assignments because connectors aren't really equivalent to
tasks in practice (all the connectors are getting assigned to 1 worker, all
the tasks to the other).

I've filed a PR to address the issue, though it probably won't be available
until the next release (but it'll work automatically once you upgrade). The
easiest short term solution is to simply increase to 3 worker processes
since that'll avoid assigning all the connectors to the first worker and
all tasks to the second.

-Ewen

On Wed, Dec 14, 2016 at 8:59 PM, Manjunath Shetty H <
manjunathshetty@live.com> wrote:

> Hi all,
>
>
> I am running kafka connect using 2 node cluster. I have 5 connectors
> running with maxtask 1 each. But all the tasks are running in same node,
> work is not distributed across 2 nodes.
>
>
> I am using a custom source connectors.
>
>
> Any help is appreciated
>
>
> Thanks
>
> Manjunath
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message