chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <>
Subject Re:
Date Thu, 28 Jan 2010 19:58:14 GMT
We don't use demux at my site, so I'd love to have Eric or Jerome jump
in here.  But that said:

I believe the typical way to set this up is to have conf/
define HADOOP_CONF_DIR; the filesystem is then specified via the
Hadoop configuration. (  You shouldn't need to change

In re processSinkFiles -- What version of Chukwa are you using?  In
Chukwa 0.3, the only formal release we've done so far, there's no, and the line in start-data-processors that
references it has been commented out.  You don't need it; references
to it are a historical artifact that should go away in the next


On Thu, Jan 28, 2010 at 11:15 AM, Corbin Hoenes <> wrote:
> I'm having some difficulty with the demux part of setting up chukwa.  I assume I am
supposed to run the script to startup all the map reduce jobs that
handle demux and archiving.
> My goal is to pull the logs we are collecting out of the sink files and into something
we can start to run our pig scripts on.
> When I run start-data-processors it gives me this though:
> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/chukwa/demuxProcessing/mrInput
>        at org.apache.hadoop.mapred.FileInputFormat.listStatus(
>        at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(
>        at org.apache.hadoop.mapred.FileInputFormat.getSplits(
>        at org.apache.hadoop.mapred.JobClient.writeOldSplits(
>        at org.apache.hadoop.mapred.JobClient.submitJobInternal(
>        at org.apache.hadoop.mapred.JobClient.submitJob(
>        at org.apache.hadoop.mapred.JobClient.runJob(
>        at
> Which seems like I need to configure it to try to connect to hdfs rather than file:/
> Only docs I've found are here:
> Is there a guide to configuring chukwa-demux-conf.xml?
> I also noticed tries to start which doesn't
exist for me--do I need to get this script?s

Ari Rabkin
UC Berkeley Computer Science Department

View raw message