kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Morales <dmora...@stratio.com>
Subject RE: How to use kafka for log aggregation
Date Fri, 31 Jul 2015 15:05:37 GMT
Take a look at the real-time aggregation engine that we offer at Stratio.

Fully open - source,  no coding, kafka integration out of the box, based on


El 31/07/2015 16:57, "Todd Snyder" <tsnyder@blackberry.com> escribió:

> Flume is one of the options to do this, yes.  There are a (growing) number
> of tools you could use to get logs in to Kafka, depending on your
> environment, needs, level of skill, deployment size, applications, log
> volume:
> (in no particular order)
>         Kafkacat : https://github.com/edenhill/kafkacat
>         Logstash: https://www.elastic.co/products/logstash
>         Klogger (from my team): https://github.com/blackberry/Klogger
>         Rsyslog:
> http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html
>         Log4j:
> http://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender
> Depending on your app, you may need to read the logs in from the
> filesystem to send to kafka.  Better though, would be not bothering with
> the filesystem read/write, and using something that will listen to
> syslog/use log4j, and write the log data directly to Kafka.  If you have
> control over the application code, you could also look at tacking in a
> producer library (kafkas provided ones, or others (
> https://github.com/blackberry/Krackle))   and writing directly to kafka
> from the app - no need for log4j, syslog, filesystem logs.
> If you're starting off, testing is nice and simple using logstash or
> klogger - you can take existing logs and just grab them from where they
> are, and put them in to kafka without changing any of your existing
> workflow/code/configuration.
> There are lots of solutions, you need to experiment to find the one that
> fits your environment best.
> Cheers,
> T.
> -----Original Message-----
> From: Snehalata Nagaje [mailto:snehalata.nagaje@harbingergroup.com]
> Sent: Friday, July 31, 2015 3:32
> To: users@kafka.apache.org
> Subject: How to use kafka for log aggregation
> Hi All,
> I want to use kafka for log aggregation, how can we use it?
> Do we need to manually read log files and post this logs into kafka using
> producer API?
> Can Apache Flume help in it?
> Thanks,
> Snehalata

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message