kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greenhorn Techie <greenhorntec...@gmail.com>
Subject Re: MirrorMaker - Topics Identification and Replication
Date Thu, 05 Jan 2017 11:46:16 GMT
Thanks Ewen for your response.

Just to summarise, here is my understanding. Apologies if something is
mis-understood. I am new to Kafka and hence still short in knowledge.


   - MirrorMarker process automatically picks-up new topics added on the
   source cluster and hence no restart of the process is needed at regular
   intervals to update the list of topics
   - For existing topics, MirrorMaker will replicate messages to the target
   kafka cluster as and when it sees data in the source kafka cluster's topics
   - However for new topics, there might be a delay of up to (default) 5
   min i.e. metadata refresh interval to start replicating the data to the
   target kafka cluster

Please let me know if something is wrong in my understanding.

Thanks

On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <ewen@confluent.io> wrote:

> Yes, the consumer will pick up the new topics when it refreshes metadata
> (defaults to every 5 min) and start subscribing to the new topics.
>
> -Ewen
>
> On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie <
> greenhorntechie@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am new to Kafka and as well as MirrorMaker. So wondering whether MM
> would
> > pick-up new topics that are created on the source cluster automatically,
> > provided topic matches the while list pattern?
> >
> > For example, if I start MM as below, would it replicate any new topics
> that
> > are created after the MM process is launched?
> >
> > kafka-mirror-maker.sh --new.consumer --consumer.config /opt/kafka/config/
> > consumer.properties --producer.config /opt/kafka/config/producer.
> > properties
> > --whitelist ".*"
> >
> > Or is there a need to restart MM process when a new topic / pattern is
> > started on the source cluster?
> >
> > What should be the recommended approach in regards to this?
> >
> > Thanks
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message