chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken White <>
Subject Re: chukwa suitability for collecting data from multiple datanodes to HDFS
Date Tue, 30 Jun 2009 08:09:40 GMT
>Chukwa does indeed aim to solve the problem you have. Reliability is a
>goal for us, but not our highest priority.  The current implementation
>falls short in a few places. These are known bugs, and shouldn't be
>hard to fix -- they just haven't been priorities.

Ari, thank you for the answer, I appreciate it! 

Great, that takes a bit of coding from me, I was going to write the collectors 
myself. I will install Chukwa and try to set it up. 

>3) There are a few times when a collector can crash, and data hasn't
>yet been committed to stable storage. This is mostly a  consequence of
>not having flush() in HDFS yet.  It is possible to hack around that.
>Fixing this will require writing some code, but not any major
>architectural change.

Thank you for the pointers! I will take a look at the code and see if anything 
can be done to improve reliability. 

If nothing else, I could be running two instances of chukwa on different nodes 
and write the important data to both of them. 

Thank you again, kind regards!


View raw message