chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: piping data into Cassandra
Date Wed, 26 Oct 2011 16:34:01 GMT
"dp" serves as two functions, first it loads data to mysql, second, it runs SQL for aggregated
views.  demuxOutputDir_* is created if the demux mapreduce produces data.  Hence, make sure
that there is a demux processor mapped to your data type for the extracting process in chukwa-demux-conf.xml.


On Oct 26, 2011, at 5:15 AM, AD wrote:

> Hmm, i am running bin/chukwa demux and i dont have anything past dataSinkArchives, there
is no directory named demuxOutputDir_*.
> Also isnt dp an aggregate view?  I need to parse the apache logs to do custom reports
on things like remote_host , query strings, etc so i was hoping to parse the raw record and
load it into Cassandra and run M/R there to do the aggregate views.  I thought a new version
of TSProcessor was the right place here but i could be wrong.
> Thoughts?
> If not, how do you write a custom postProcessor?
> On Wed, Oct 26, 2011 at 12:57 AM, Eric Yang <> wrote:
> Hi AD,
> Data is stored in demuxOutputDir_* by demux and there is a
> postProcessorMananger (bin/chukwa dp) which monitors postProcess
> directory and load data to MySQL.  For your use case, you will need to
> modify to adopt to your use case.  Hope this
> helps.
> regards,
> Eric
> On Tue, Oct 25, 2011 at 6:34 PM, AD <> wrote:
> > hello,
> >  I currently push apache logs into Chukwa.  I am trying to figure out how to
> > get all those logs into Cassandra and run mapreduce there.  Is the best
> > place to do this in Demux (right my own version of TSProcessor?)
> >  Also the data flow seems to miss a step.  The
> > page says in
> > 3.3 that
> >    - demux moves complete files to: dataSinkArchives/[yyyyMMdd]/*/*.done
> >  - the next step is to move files
> > from postProcess/demuxOutputDir_*/[clusterName]/[dataType]/[dataType]_[yyyyMMdd]_[HH].R.evt
> >   How do they get from dataSinkArchives to postProcess?  does this run
> > inside of DemuxManager or a separate process (bin/chukwa demux) ?
> >  Thanks
> >  AD

View raw message