kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: MirrorMaker - Topics Identification and Replication
Date Fri, 06 Jan 2017 04:23:12 GMT
That all sounds right!

Usually the delay for picking up the metadata update and starting to
replicate the topic won't be an issue since it's a one-time issue (around
topic creation) and the time window is pretty small for that. In steady
state, none of the mentioned delays would apply.

-Ewen

On Thu, Jan 5, 2017 at 3:46 AM, Greenhorn Techie <greenhorntechie@gmail.com>
wrote:

> Thanks Ewen for your response.
>
> Just to summarise, here is my understanding. Apologies if something is
> mis-understood. I am new to Kafka and hence still short in knowledge.
>
>
>    - MirrorMarker process automatically picks-up new topics added on the
>    source cluster and hence no restart of the process is needed at regular
>    intervals to update the list of topics
>    - For existing topics, MirrorMaker will replicate messages to the target
>    kafka cluster as and when it sees data in the source kafka cluster's
> topics
>    - However for new topics, there might be a delay of up to (default) 5
>    min i.e. metadata refresh interval to start replicating the data to the
>    target kafka cluster
>
> Please let me know if something is wrong in my understanding.
>
> Thanks
>
> On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
> > Yes, the consumer will pick up the new topics when it refreshes metadata
> > (defaults to every 5 min) and start subscribing to the new topics.
> >
> > -Ewen
> >
> > On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie <
> > greenhorntechie@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am new to Kafka and as well as MirrorMaker. So wondering whether MM
> > would
> > > pick-up new topics that are created on the source cluster
> automatically,
> > > provided topic matches the while list pattern?
> > >
> > > For example, if I start MM as below, would it replicate any new topics
> > that
> > > are created after the MM process is launched?
> > >
> > > kafka-mirror-maker.sh --new.consumer --consumer.config
> /opt/kafka/config/
> > > consumer.properties --producer.config /opt/kafka/config/producer.
> > > properties
> > > --whitelist ".*"
> > >
> > > Or is there a need to restart MM process when a new topic / pattern is
> > > started on the source cluster?
> > >
> > > What should be the recommended approach in regards to this?
> > >
> > > Thanks
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message