kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor L <vlyamt...@gmail.com>
Subject Re: cpu over-consumption in Kafka producer -?
Date Wed, 15 Apr 2015 17:33:07 GMT
Hi Evan,
It's hardcoded:
MaxBufferTime       = 1000
MaxBufferedBytes    = 1024

How's it versioned? Should i just download zip file from project github?

On Wed, Apr 15, 2015 at 12:48 PM, Evan Huus <evan.huus@shopify.com> wrote:

> Hi Victor, two points:
>
> - Based on the backtrace, you are using a very old version of Sarama. You
> might have better luck using a more recent stable version.
> - Are you setting `MaxBufferTime` in the configuration to 0 or a very small
> value? If so the loop will spin on that timer. Try making this value
> larger.
>
> Evan
>
> On Wed, Apr 15, 2015 at 12:35 PM, Victor L <vlyamtsev@gmail.com> wrote:
>
> > I am using sarama "golang" kafka 1.8.1 client (
> > https://github.com/Shopify/sarama)  to send messages to message queue
> once
> > in 3secs and this task drives cpu consumption to 130% on the quad-cpu
> > blade; The number stays this high regardless of number of
> > partitions/consumers.... According to results of profiling with 'pprof',
> > most of the cycles are spent in producer.go/NewBrokerProducer;
> > Total: 47297 samples
> >     5947  12.6%  12.6%    15013  31.7% selectgo
> >     3617   7.6%  20.2%     3617   7.6% runtime.xchg
> >     2819   6.0%  26.2%    28506  60.3%
> github.com/Shopify/sarama.funcĀ·008
> <http://github.com/Shopify/sarama.func%C2%B7008>
> > <http://github.com/Shopify/sarama.func%C2%B7008>
> > <http://github.com/Shopify/sarama.func%C2%B7008>
> > ----------------------------------------------------------------------
> > NewBrokerProducer
> > -----------------------------------------------------------
> > 229:     go func() {
> >      .      .  230:         timer :=
> time.NewTimer(p.config.MaxBufferTime)
> >      .      .  231:         var shutdownRequired bool
> >      .      .  232:         wg.Done()
> >      .      .  233:         for {
> >    208  17598  234:             select {
> >    153    749  235:             case <-bp.flushNow:
> >      .      .  236:                 if shutdownRequired = bp.flush(p);
> > shutdownRequired {
> >      .      .  237:                     goto shutdown
> >      .      .  238:                 }
> >   2230   3144  239:             case <-timer.C:
> >    144   3606  240:                 if shutdownRequired =
> > bp.flushIfAnyMessages(p); shutdownRequired {
> >      .      .  241:                     goto shutdown
> >
> >
> > wonder if there's some known issue with this method or if anyone already
> > seen it before...
> > Thank you,
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message