kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhjot Bharaj <prabhbha...@gmail.com>
Subject producer-consumer issues during deployments
Date Thu, 26 Nov 2015 11:13:23 GMT

We arrange our kafka machines in groups and deploy these phases.

For kafka, we’ll have to map groups with phases. During each phase of the
release, all the machines in that group can go down.

When this happens, there are a couple of cases:-

   1. All replicas are residing in a group of machines which will all go
   down in this phase
      - Affect on Producer –
         - What happens to the produce requests (whether produce can
         dynamically keep writing to the remaining partitions now)
         - What happens to the already queued requests which were being
         sent to the earlier replicas – they will fail (we’ll have to
use producer
         callback feature to take care of retrying in case the above step
         works fine)
      - Affect on Consumer -
         - Can the consumers consume from a lesser number of partitions?
         - Does the consumer 'consume' api gives any callback/failure when
         all replicas of a partition go down?

If you have come across any of the above cases, please provide how you
solved the problem ? or whether everything works just well with Kafka
during deployments and my cases described above are all invalid or handled
by kafka and its clients internally ?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message