chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: Multiple agents and multiple clusters
Date Fri, 15 Jul 2011 16:22:30 GMT
The data is written locally on the source node.  Both agents will progress independently, and
sending data at their own pace to the collectors.  They shouldn't affect each other.  The
UTF8FileTailingAdaptor should track the timestamp of the log files and send it over in time
sequence order, hence one failed does not affect others.


On Jul 14, 2011, at 8:45 PM, highpointe wrote:

> Ahhhh. Good insight. I didn't think about that.
> Yes, I am aggregating the same data for two clusters.   
> So now my challenge is:  If one cluster is unreachable...  How do I that cluster's sender
fail without affecting the other sender or the connector. 
> Excellent insight Ari!  Thank you!
> Sent from my iPhone
> On Jul 14, 2011, at 11:54 AM, Ariel Rabkin <> wrote:
>> Just to be clear -- you want the same data to end up in each cluster?
>> You'll have to do some hacking. I think it won't be so very monstrous.
>> In the Agent, there's a notion of pluggable Connectors and Senders. I
>> think you want a new Connector with two senders.
>> One caveat -- you'll need to think through what happens if just one
>> cluster is down, and perhaps other partial failure scenarios too.
>> --Ari
>> On Thu, Jul 14, 2011 at 10:44 AM, high pointe <> wrote:
>>> I have a situation where I need to take the one data stream, send it through
>>> two agents and have the end point residing in separate clusters.
>>> Rough explanation....
>>> Streaming Data Box ---> Chukwa Box
>>> Agent one on Chukwa Box sends feed to cluster one.
>>> Agent two on Chukwa box sends feed to cluster two.
>>> Does anyone know of a way to do this without hacking this baby to shreds?
>>> Thanks!
>> -- 
>> Ari Rabkin
>> UC Berkeley Computer Science Department

View raw message