kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Issue with async producer
Date Fri, 07 Nov 2014 00:29:50 GMT
In 0.8.2 beta, we have a new java producer that's more performing. You can
give it a try.

Thanks,

Jun

On Thu, Nov 6, 2014 at 10:16 AM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:

> Hi,
>
> Which version of Kafka are you using?
> 0.8.0
>
> Is the broker I/O or network saturated? - typical writes are around 45 MB
> per broker & there are streaming as well as batch consumers.
>
> We are already sending across compressed packets & the batch size is 100.
>
> Also, we tried the sync producer but were not able to enforce the timeouts
> on it.Is there anything that needs to be taken care of in this case?
>
> Regards,
> Dev
> ________________________________
> From: Devendra Tagare
> Sent: Tuesday, November 04, 2014 10:02 AM
> To: users@kafka.apache.org
> Cc: Devendra Tagare
> Subject: Issue with async producer
>
> Hi,
>
> We are using an async producer to send data to kafka.
>
> The load the sender handles is around 250 rps ,the size of a message is
> around 25K.
>
> The configs used in the producer are :
>
> request.required.acks=0
> producer.type=async
> batch.num.messages=10
> topic.metadata.refresh.interval.ms=30000
> queue.buffering.max.ms=300
> queue.enqueue.timeout.ms=50
>
>
> While the asnc producer works  perfectly fine  at 150-175 rps,the invoking
> method returns under 10 ms.The invoking method takes around 20000ms to
> return when the load increases to 250rps.
>
> On investigation we noticed QueueFullExceptions in the logs.
>
> On the kafka side the memory utilization was high.Is is because the fsync
> to memory is not happening fast enough?
> The document says the OS level fsync interval should take care of the
> interval at which writes are happening.Should we expediate the write using
>
> log.flush.interval.messages
> and log.flush.interval.ms.
>
> Also, we tried the sync producer but were not able to enforce the timeouts
> on it.
>
> We have 45 producers & 11 Kafka servers which handle a load of around
> 500mn events per day.
>
> Some of the server side properties we are using:
>
> log.segment.bytes=536870912
> log.retention.check.interval.ms=60000
> zookeeper.connection.timeout.ms=1000000
>
> Regards,
> Dev
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message