kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajiv Kurian <ra...@signalfuse.com>
Subject Re: kafka async producer takes a lot of cpu
Date Mon, 15 Dec 2014 05:31:11 GMT
I'll try it and let you guys know. Is there anything that comes to mind
from these log messages though? Why would there be so many log messages?
Would you suggest doing something else to find out why things are working
so poorly? I am worried about making the risky transition to the beta
producer client and finding out that things don't improve at all. Another
thing that was peculiar was that only one of my producers (out of 3, each
on a separate host) were dropping so many messages.

On Sun, Dec 14, 2014 at 4:58 PM, Neha Narkhede <neha@confluent.io> wrote:

> Thanks for reporting the issue, Rajiv. Since we are actively phasing out
> the old client, it will be very helpful to know what the behavior on the
> new client is.
>
>
> On Fri, Dec 12, 2014 at 8:12 PM, Rajiv Kurian <rajiv@signalfuse.com>
> wrote:
> >
> > I am using the kafka java api async client (the one that wraps the Scala
> > client). It's dropping quite a bit of data due to the queue being full,
> and
> > the process ends up taking a lot of cpu too.
> >
> > I am posting to a topic with 1024 partitions (across 3 brokers) - maybe
> the
> > high number of brokers is one of the issues. Profiling with YourKit
> showed
> > 37% of my CPU being spent on
> kafka.producer.async.ProducerSendThread.run().
> > So seems like the producer is not able to keep up with my application and
> > starts dropping. When I expand this waterfall in YourKit, I see that 23%
> > total (not out of the 37%) is being spent on logging! Something like
> this:
> >
> >
> >
> kafka.producer.BrokerPartitionInfo$$anonfun$getBrokerPartitionInfo$2.apply(PartitionMetadata)
> > ->kafka.producer.BrokerPartitionInfo.debug(Function0)
> >    ->kafka.utils.Logging$class.debug(Logging, Function0)
> >       ->org.apache.log4j.Category.isDebugEnabled()
> >          ->... (a bunch of other things that finally break down into)
> >              ->LoggerContext.java:252
> >
> >
> ch.qos.logback.classic.spi.TurboFilterList.getTurboFilterChainDecision(Marker,
> > Logger, Level, String, Object[], Throwable)
> >
> > I am not sure what's going on here. When I look at my process log, none
> of
> > these messages are actually logged (probably because of the log level).
> > Further I don't see anything very suspicious on the broker logs. They are
> > at 60-70% cpu.
> >
> > I am planning to try the new Java beta producer client, but I am afraid
> > something deeper is going on here, that might not be solved by switching
> to
> > the newer client.
> >
>
>
> --
> Thanks,
> Neha
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message