chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: hicc start problem:"Unable to load dashboard"
Date Tue, 19 Jun 2012 05:07:08 GMT
On Mon, Jun 18, 2012 at 7:37 PM, scott <> wrote:
> Thanks,Eric.
> I still have some doubts about the process of writing logs to hbase and
> hdfs.
> 1. In my project, i need to collect and record some metrics for near
> real-time monitoring, and also store some logs for later analysis. For
> monitoring, hbase can be used, and for later log analysis, logs should be
> stored into hdfs.  Chukwa has provided writting to both hbase and hdfs.
> which can be set in
> chukwa-collector-conf.xml  using chukwaCollector.pipeline
> <name>chukwaCollector.pipeline</name>
> <value>org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter,org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>
> ....
> However, in chukwaCollector.pipeline, we must write hbase.HBaseWriter ahead
> of writer.SeqFileWriter,  for in SeqFileWriter source code, i found
> that chunks will not pass to next writer. please verify that.

SeqFileWriter does not pass to next writer, this is the reason that it
has to be the last writer in the pipeline as a workaround.

> 2. For HDFS, I want to store the data categorized by
> [dataType]/[yyyyMMdd]/[HH]/[mm]/.  in your last letter, you said that it's
> old design to have PostProcessor to extract metrics to store in DB and it
> should not be used now, then, what will we do to achieve aggregating the
> data into such category. Is there any new code to check in to solve it ?

There is no new code written for inject MR data to DB or HBase,
patches are welcome.


View raw message