kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot West <tea...@gmail.com>
Subject Topic migration patterns
Date Thu, 11 Jul 2019 08:50:18 GMT
Hello,

I like to understand what strategies can be applied to migrate events,
producers, and consumers from one topic to another. Typically I'm thinking
of cases where:


   - we wish to migrate a topic from one cluster to another
   - high availability - minimise amount of time topic is unavailable to
   producers/consumers
   - no data loss (at least once)
   - avoid excessive latency (keep events flowing - no 'stopping the world')
   - preserve event ordering wrt to partitions
   - there is a significant retention interval (days) that we wish to move
   to the destination topic also


I can imagine that tools such as MM1+2, Replicator, etc. are useful in
these circumstances. However, I'm more interested in orchestration
patterns: what things need to be moved, in what order, and subject to what
conditions?

Thanks for your time,

Elliot.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message