kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Swift <craig.sw...@returnpath.com.INVALID>
Subject Re: MirrorMaker - Not consuming from all partitions
Date Fri, 11 Sep 2015 21:30:10 GMT
Scary but thanks! :) We'll start digging into the network and see if we can
find a smoking gun. Appreciate the response, thanks again.

Craig J. Swift
Software Engineer - Data Pipeline
ReturnPath Inc.
Work: 303-999-3220 Cell: 720-560-7038

On Fri, Sep 11, 2015 at 11:29 AM, Steve Miller <steve@idrathernotsay.com>
wrote:

>    I have a vague feeling that I've seen stuff like this when the network
> on the broker that's disappearing is actually unreachable from time to time
> -- though I'd like to believe that's not such an issue when talking to AWS
> (though there could be a lot of screwed-up Internet between you and it,
> depending on exactly wht you're doing).
>
>    One thing you could consider doing would be to look at some of the
> traffic.  wireshark/tshark knows how to decode Kafka transactions, though
> digging through the output is... exciting, because there can be so very
> much of it.  I'd written up some notes on how to do that, which you can see
> at:
>
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201408.mbox/%3C20140812180358.GA24108@idrathernotsay.com%3E
>
> (though I expect in your case, you'd need to be doing a capture all the
> time, and then figure out when MirrorMaker stops fetching from that broker,
> and then stare at a ton of data to find whatever protocol thing has
> happened -- all of which is ugly enough that I'd been reluctant to mention
> it until now).
>
>         -Steve
>
> On Fri, Sep 11, 2015 at 08:52:21AM -0600, Craig Swift wrote:
> > Just wanted to bump this again and see if the community had any thoughts
> or
> > if we're just missing something stupid. For added context the topic we're
> > reading from has 24 partitions and we see roughly 15k messages per
> minute.
> > As I mentioned before the throughput seems fine, but I'm not entirely
> sure
> > how the MirrorMaker cycles through it's topics/partitions and why it
> would
> > read very slowly or bypass reading from certain partitions.
> >
> > Craig J. Swift
> > Software Engineer - Data Pipeline
> > ReturnPath Inc.
> > Work: 303-999-3220 Cell: 720-560-7038
> >
> > On Wed, Sep 9, 2015 at 9:29 AM, Craig Swift <craig.swift@returnpath.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Hope everyone is doing well. I was hoping to get some assistance with a
> > > strange issue we're experiencing while using the MirrorMaker to pull
> data
> > > down from an 8 node Kafka cluster in AWS into our data center. Both
> Kafka
> > > clusters and the mirror are using version 0.8.1.1 with dedicated
> Zookeeper
> > > clusters for each cluster respectively (running 3.4.5).
> > >
> > > The problem we're seeing is that the mirror starts up and begins
> consuming
> > > from the cluster on a specific topic. It correctly attaches to all 24
> > > partitions for that topic - but inevitably there are a series of
> partitions
> > > that either don't get read or are read at a very slow rate. Those
> > > partitions are always associated with the same brokers. For example,
> all
> > > partitions on broker 2 won't be read or all partitions on broker 2 and
> 4
> > > won't be read. On restarting the mirror, these 'stuck' partitions may
> stay
> > > the same or move. If they move the backlog is drained very quickly. If
> we
> > > add more mirrors for additional capacity the same situation happens
> except
> > > that each mirror has it's own set of stuck partitions. I've included
> the
> > > mirror's configurations below along with samples from the logs.
> > >
> > > 1) The partition issue seems to happen when the mirror first starts up.
> > > Once in a blue moon it reads from everything normally, but on restart
> it
> > > can easily get back into this state.
> > >
> > > 2) We're fairly sure it isn't a processing/throughput issue. We can
> turn
> > > the mirror off for a while, incur a large backlog of data, and when it
> is
> > > enabled it chews through the data very quickly minus the handful of
> stuck
> > > partitions.
> > >
> > > 3) We've looked at both the zookeeper and broker logs and there doesn't
> > > seem to be anything out of the normal. We see the mirror connecting,
> there
> > > are a few info messages about zookeeper nodes already existing, etc. No
> > > specific errors.
> > >
> > > 4) We've enabled debugging on the mirror and we've noticed that during
> the
> > > zk heartbeat/updates we're missing these messages for the 'stuck'
> > > partitions:
> > >
> > > [2015-09-08 18:38:12,157] DEBUG Reading reply
> sessionid:0x14f956bd57d21ee,
> > > packet:: clientPath:null serverPath:null finished:false header:: 357,5
> > >  replyHeader:: 357,8597251893,0  request::
> > >
> '/consumers/mirror-kafkablk-kafka-gold-east-to-kafkablk-den/offsets/MessageHeadersBody/5,#34303537353838,-1
> > >  response::
> > >
> s{4295371756,8597251893,1439969185754,1441759092134,19500,0,0,0,7,0,4295371756}
> > >  (org.apache.zookeeper.ClientCnxn)
> > >
> > > i.e. we see this message for all the processing partitions, but never
> for
> > > the stuck ones. There are no errors in the log prior to this though,
> and
> > > once in a great while we might see a log entry for one of the stuck
> > > partitions.
> > >
> > > 5) We've checked latency/response time with zookeeper from the brokers
> and
> > > the mirror and it appears fine.
> > >
> > > Mirror consumer config:
> > > group.id=mirror-kafkablk-kafka-gold-east-to-kafkablk-den
> > > consumer.id
> =mirror-kafkablk-mirror00-den-kafka-gold-east-to-kafkablk-den
> > > zookeeper.connect=zk.strange.dev.net:2181
> > > fetch.message.max.bytes=15728640
> > > socket.receive.buffer.bytes=64000000
> > > socket.timeout.ms=60000
> > > zookeeper.connection.timeout.ms=60000
> > > zookeeper.session.timeout.ms=30000
> > > zookeeper.sync.time.ms=4000
> > > auto.offset.reset=smallest
> > > auto.commit.interval.ms=20000
> > >
> > > Mirror producer config:
> > > client.id=mirror-kafkablk-mirror00-den-kafka-gold-east-to-kafkablk-den
> > > metadata.broker.list=kafka00.lan.strange.dev.net:9092,
> > > kafka01.lan.strange.dev.net:9092,kafka02.lan.strange.dev.net:9092,
> > > kafka03.lan.strange.dev.net:9092,kafka04.lan.strange.dev.net:9092
> > > request.required.acks=1
> > > producer.type=async
> > > request.timeout.ms=20000
> > > retry.backoff.ms=1000
> > > message.send.max.retries=6
> > > serializer.class=kafka.serializer.DefaultEncoder
> > > send.buffer.bytes=134217728
> > > compression.codec=gzip
> > >
> > > Mirror startup settings:
> > > --num.streams 2 --num.producers 4
> > >
> > > Any thoughts/suggestions would be very helpful. At this point we're
> > > running out of things to try.
> > >
> > >
> > > Craig J. Swift
> > > Software Engineer
> > >
> > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message