chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <>
Subject [jira] Commented: (CHUKWA-389) Send chunks directly from a log4j appender to the collector without writing to local drive
Date Wed, 16 Sep 2009 18:55:57 GMT


Jerome Boulon commented on CHUKWA-389:

My proposal is to use Thrift for the communication layer between Log4J and the collector.

I want to use Thrift for 2 reasons, it's available and well debugged with good performance
and using Thrift will open chukwa to a large number of programming language.

The new Log4j Appender will store all logs to an in-memory buffering queue. One or more sender
will read from the in-memory queue and send the data to the collector using Thrift for data
serialization and transport. A sender will collect up to N messages at regular interval. If
the queue's size reach a threshold then the sender will be wakeup and the data will be send
A sender will stay with the same collector unless the connection is dropped. In that case
the sender will reconnect to another collector and send the same data.
Data from the same log4j appender could be sent to different collectors for load-balancing
and fail-over.

The new ThriftCollector class will act at the same level as the current JettyCollector.
The writer interface and all current writers would be reusable without any modification. 

> Send chunks directly from a log4j appender to the collector without writing to local
> ------------------------------------------------------------------------------------------
>                 Key: CHUKWA-389
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>            Reporter: Jerome Boulon
>            Assignee: Jerome Boulon
> Currently Chukwa required to first write the data to the local drive and run an agent
on every single box in order to be able to collect logs.
> This is a good solution if you cannot afford to loose any data but sometime you don't
want to run an agent on very single box.
> In that case, a new log4J appender could be used to send data directly from the application
to a collector.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message