kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flinkster <flinks...@gmail.com>
Subject Re: Kafka syslog producer
Date Mon, 23 Apr 2012 20:01:17 GMT
How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to
application logs & metrics and syslog messages (e.g. network, OS logs)
is a separate flow and datastore?

On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <jjkoshy.w@gmail.com> wrote:
> This is an interesting use-case and similar to using the log4j appender for
> application logs. What you describe sounds reasonable: i.e., have a
> producer process on the syslog server to send syslog messages to your Kafka
> brokers.
>> 2) How about tailing a file to a central logging like you can do with
>> scribe/flume agents?
> You could use tail and pipe (or a named pipe) to a console-based producer.
> I don't know enough about syslog.d to tell if this would be too much of a
> hack or not, but I think it would be more reliable than using tail+pipes:
> you could configure your local machines to direct syslog messages to the
> remote server and you could write a simple component that listens on the
> syslog socket and redirects incoming messages to a Kafka producer.
> Joel

View raw message