kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernal, Marcos" <marcos.ber...@sky.uk>
Subject Mitigating impact on connected producers and consumers when restarting broker node
Date Tue, 16 May 2017 11:19:13 GMT
Hello,

We've got a Kafka 0.9.0.1 cluster, with 4 brokers, a few hundred topics, 20-50 consumers and
10-15 producers.

Often we've got to do changes on the cluster config and we do rolling restarts, with controlled
shutdowns. However, our consumers and producers see a big impact when:

- The broker is shutting down
- The broker joins the cluster again
- The cluster does a rebalance

The type of exceptions the clients are producing are in the range of:

NotLeaderForPartitionException
NotEnoughReplicasException
NotEnoughReplicasAfterAppendException
TimeoutException
NetworkException

My question is, is there a way to mitigate the impact, or is it considered the normal way
kafka and the clients operate?

Is any change in the topology of the cluster (partition leader and ISRs) supposed to cause
these exceptions?

Thank you,

Information in this email including any attachments may be privileged, confidential and is
intended exclusively for the addressee. The views expressed may not be official policy, but
the personal views of the originator. If you have received it in error, please notify the
sender by return e-mail and delete it from your system. You should not reproduce, distribute,
store, retransmit, use or disclose its contents to anyone. Please note we reserve the right
to monitor all e-mail communication through our internal and external networks. SKY and the
SKY marks are trademarks of Sky plc and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075)
and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries
of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are
incorporated in England and Wales and share the same registered office at Grant Way, Isleworth,
Middlesex TW7 5QD.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message