kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Mirror maker doesn't replicate new topics
Date Tue, 10 Sep 2013 17:18:30 GMT
Oh got it. Did you set auto.offset.reset = smallest or largest? If it is
largest it could be due to this bug:

https://issues.apache.org/jira/browse/KAFKA-1006

Guozhang



On Tue, Sep 10, 2013 at 10:09 AM, Rajasekar Elango
<relango@salesforce.com>wrote:

> Hi Guozhang ,
>
> 1) When I say "I send messages to new topic" -> yes I am sending new
> messages to source cluster via console producer.
> 2) The log message "Handling 0 events" doesn't output topic name. But I
> would believe its for both old and new topics, because no other app is
> sending messages to source cluster other than me trying to test using
> console producer.
>
> Thanks,
> Raja.
>
>
> On Tue, Sep 10, 2013 at 1:03 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > Hi Raja,
> >
> > When you say "I send messages to new topic" I guess you mean that you
> send
> > messages to the source cluster right? It may be due to the fact that
> > producers of mirror make have not catched up with the mirror maker
> > consumer.
> >
> > When you say "I always see Handling 0 events" do you mean that you see
> this
> > for both messages for the new topic and for the old topics, or it only
> > shows this log for new topic?
> >
> > Guozhang
> >
> >
> > On Tue, Sep 10, 2013 at 7:47 AM, Rajasekar Elango <
> relango@salesforce.com
> > >wrote:
> >
> > > Thanks Guozhang,
> > >
> > > 1, 2, 3 all are true. We are using default value 200 for
> > batch.num.messages
> > > and 5000ms queue.buffering.max.ms. I believe it should batch either if
> > > batch.num.messages is reached or queue.buffering.max.ms is reached.
> > >
> > > I see log message "5000ms elapsed , Queue time reached. Sending.  "  on
> > > regular interval. But when I send messages to new topic, I always see
> > > "Handling 0 events" and it doesn't produce to target cluster. But when
> I
> > > resend it second time, I see "Handling x events" and starts producing.
> > Any
> > > clues on how to debug further?
> > >
> > > Thanks,
> > >
> > > Raja.
> > >
> > >
> > > On Mon, Sep 9, 2013 at 6:02 PM, Guozhang Wang <wangguoz@gmail.com>
> > wrote:
> > >
> > > > Hi Raja,
> > > >
> > > > So just to summarize the scenario:
> > > >
> > > > 1) The consumer of mirror maker is successfully consuming all
> > partitions
> > > of
> > > > the newly created topic.
> > > > 2) The producer of mirror maker is not producing the new messages
> > > > immediately when the topic is created (observed from
> > ProducerSendThread's
> > > > log).
> > > > 3) The producer of mirror maker will start producing the new messages
> > > when
> > > > more messages are sent to the source cluster.
> > > >
> > > > If 1) is true then KAFKA-1030 is excluded, since the consumer
> > > successfully
> > > > recognize all the partitions and start consuming.
> > > >
> > > > If both 2) and 3) is true, I would wonder if the batch size of the
> > mirror
> > > > maker producer is large and hence will not send until enough messages
> > are
> > > > accumulated at the producer queue.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, Sep 9, 2013 at 2:36 PM, Rajasekar Elango <
> > relango@salesforce.com
> > > > >wrote:
> > > >
> > > > > yes, the data exists in source cluster, but not in target cluster.
> I
> > > > can't
> > > > > replicate this problem in dev environment and it happens only in
> prod
> > > > > environment. I turned on debug logging, but not able to identify
>  the
> > > > > problem. Basically, whenever I send data to new topic, I don't see
> > any
> > > > log
> > > > > messages from ProducerSendThread in mirrormaker log so they are not
> > > > > produced to target cluster. If I send more messages to same topic,
> > the
> > > > > producer send thread kicks off and replicates the messages. But
> > > whatever
> > > > > messages send first time gets lost. How can I trouble shoot this
> > > problem
> > > > > further? Even this could be due to know issue
> > > > > https://issues.apache.org/jira/browse/KAFKA-1030, how can I
> confirm
> > > > that?
> > > > > Is there config tweaking I can make to workaround this..?
> > > > > ConsumerOffsetChecks helps to track consumers. Its there any other
> > tool
> > > > we
> > > > > can use to track producers in mirrormaker. ?
> > > > >
> > > > > Thanks in advance for help.
> > > > >
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 6, 2013 at 3:50 AM, Swapnil Ghike <sghike@linkedin.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Rajasekar,
> > > > > >
> > > > > > You said that ConsumerOffsetChecker shows that new topics are
> > > > > successfully
> > > > > > consumed and the lag is 0. If that's the case, can you verify
> that
> > > > there
> > > > > > is data on the source cluster for these new topics? If there
is
> no
> > > data
> > > > > at
> > > > > > the source, MirrorMaker will only assign consumer streams to
the
> > new
> > > > > > topic, but the lag will be 0.
> > > > > >
> > > > > > This could otherwise be related to
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1030.
> > > > > >
> > > > > > Swapnil
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 9/5/13 8:38 PM, "Guozhang Wang" <wangguoz@gmail.com>
wrote:
> > > > > >
> > > > > > >Could you let me know the process of reproducing this issue?
> > > > > > >
> > > > > > >Guozhang
> > > > > > >
> > > > > > >
> > > > > > >On Thu, Sep 5, 2013 at 5:04 PM, Rajasekar Elango
> > > > > > ><relango@salesforce.com>wrote:
> > > > > > >
> > > > > > >> Yes guozhang
> > > > > > >>
> > > > > > >> Sent from my iPhone
> > > > > > >>
> > > > > > >> On Sep 5, 2013, at 7:53 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Rajasekar,
> > > > > > >> >
> > > > > > >> > Is auto.create.topics.enable set to true in your
target
> > cluster?
> > > > > > >> >
> > > > > > >> > Guozhang
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, Sep 5, 2013 at 4:39 PM, Rajasekar Elango
> > > > > > >><relango@salesforce.com
> > > > > > >> >wrote:
> > > > > > >> >
> > > > > > >> >> We having issues that mirormaker not longer
replicate newly
> > > > created
> > > > > > >> topics.
> > > > > > >> >> It continues to replicate data for existing
topics and but
> > new
> > > > > topics
> > > > > > >> >> doesn't get created on target cluster.
> ConsumerOffsetTracker
> > > > shows
> > > > > > >>that
> > > > > > >> new
> > > > > > >> >> topics are successfully consumed and Lag is
0. But those
> > topics
> > > > > > >>doesn't
> > > > > > >> get
> > > > > > >> >> created in target cluster. I also don't see
mbeans for this
> > new
> > > > > topic
> > > > > > >> under
> > > > > > >> >> kafka.producer.ProducerTopicMetrics.<topic
name>metric. In
> > > logs I
> > > > > see
> > > > > > >> >> warning for NotLeaderForPatition. but don't
see major
> error.
> > > What
> > > > > > >>else
> > > > > > >> can
> > > > > > >> >> we look to troubleshoot this further.
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Thanks,
> > > > > > >> >> Raja.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > -- Guozhang
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >--
> > > > > > >-- Guozhang
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Raja.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Raja.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Thanks,
> Raja.
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message