chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: piping data into Cassandra
Date Wed, 26 Oct 2011 04:57:58 GMT
Hi AD,

Data is stored in demuxOutputDir_* by demux and there is a
postProcessorMananger (bin/chukwa dp) which monitors postProcess
directory and load data to MySQL.  For your use case, you will need to
modify PostProcessorManager.java to adopt to your use case.  Hope this
helps.

regards,
Eric

On Tue, Oct 25, 2011 at 6:34 PM, AD <straightflush@gmail.com> wrote:
> hello,
>  I currently push apache logs into Chukwa.  I am trying to figure out how to
> get all those logs into Cassandra and run mapreduce there.  Is the best
> place to do this in Demux (right my own version of TSProcessor?)
>  Also the data flow seems to miss a step.  The
> page http://incubator.apache.org/chukwa/docs/r0.4.0/dataflow.html says in
> 3.3 that
>    - demux moves complete files to: dataSinkArchives/[yyyyMMdd]/*/*.done
>  - the next step is to move files
> from postProcess/demuxOutputDir_*/[clusterName]/[dataType]/[dataType]_[yyyyMMdd]_[HH].R.evt
>   How do they get from dataSinkArchives to postProcess?  does this run
> inside of DemuxManager or a separate process (bin/chukwa demux) ?
>  Thanks
>  AD

Mime
View raw message