kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: High Latency in Kafka
Date Sun, 08 Feb 2015 14:36:58 GMT
I'm wondering how much of the time is spent by Logstash reading and
processing the log vs. time spent sending data to Kafka. Also, I'm not
familiar with log.stash internals, perhaps it can be tuned to send the data
to Kafka in larger batches?

At the moment its difficult to tell where is the slowdown. More information
about the breakdown of time will help.

You can try Flume's SpoolingDirectory source with Kafka Channel or Sink and
see if you get improved performance out of other tools.


Gwen

On Sun, Feb 8, 2015 at 12:06 AM, Vineet Mishra <clearmidoubt@gmail.com>
wrote:

> Hi All,
>
> I am having some log files of around 30GB, I am trying to event process
> these logs by pushing them to Kafka. I could clearly see the throughput
> achieved while publishing these event to Kafka is quiet slow.
>
> So as mentioned for the single log file of 30GB, the Logstash is
> continuously emitting to Kafka and it is running from more than 2 days but
> still it has processed just 60% of the log data. I was looking out for a
> way to increase the efficiency of the publishing the event to kafka as with
> this rate of data ingestion I don't think it will be a good option to move
> ahead.
>
> Looking out for performance improvisation for the same.
>
> Experts advise required!
>
> Thanks!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message