chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <>
Subject [jira] [Created] (CHUKWA-678) Make use of ChukwaWriter in agent
Date Thu, 03 Jan 2013 03:30:12 GMT
shreyas subramanya created CHUKWA-678:

             Summary: Make use of ChukwaWriter in agent
                 Key: CHUKWA-678
             Project: Chukwa
          Issue Type: Task
          Components: Data Collection
         Environment: MacOSX, Java 6
            Reporter: shreyas subramanya

The chukwa agent sends out data chunks to various destinations through the combination of
Connector and ChukwaSender interfaces. For sending chunks to collector, we have http implementation
of these interfaces. The collector writes out the received chunks to various destinations
through classes implementing ChukwaWriter interface. Optionally, multiple destinations can
be chosen by specifying PipelineStageWriter.

The proposal is to:
1. Use ChukwaWriter to send out data chunks to multiple destinations from the agent. Further,
PipelinestageWriter can be made default and pipeline configuration specified in the agent
config file
2. Implement (or modify) Pipelineable writers for HBase, Http, Hdfs and WebHdfs
3. Do away with the Connector interface and have a single non configurable connector object
as part of the agent. This class initiates the configured writer, waits for data chunks and
passes the chunks to Writer.add()/send(). Connection protocol for each destination is handled
by the init() of the individual writers.

1. In case of Pipelineable writers, we need a way to merge the results of each pipeline stage
before the agent commits the chunk.
2. Handling pipeline failure

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message