metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Elliston Ball <si...@simonellistonball.com>
Subject Re: When things change in hdfs, how do we know
Date Fri, 26 Jan 2018 13:30:27 GMT
Interesting, so you have an INotify listener to filter events, and then on given changes, propagate
a notification to zookeeper, which then triggers the reconfiguration event via the curator
client in Metron. I kinda like it given our existing zookeeper methods. 

Simon

> On 26 Jan 2018, at 13:27, Otto Fowler <ottobackwards@gmail.com> wrote:
> 
> https://github.com/ottobackwards/hdfs-inotify-zookeeper <https://github.com/ottobackwards/hdfs-inotify-zookeeper>
> 
> Working on a poc
> 
> 
> 
> On January 26, 2018 at 07:41:44, Simon Elliston Ball (simon@simonellistonball.com <mailto:simon@simonellistonball.com>)
wrote:
> 
>> Should we consider using the Inotify interface to trigger reconfiguration, in same
way we trigger config changes in curator? We also need to fix caching and lifecycle in the
Grok parser to make the zookeeper changes propagate pattern changes while we’re at it. 

>> 
>> Simon 
>> 
>> > On 26 Jan 2018, at 03:16, Casey Stella <cestella@gmail.com <mailto:cestella@gmail.com>>
wrote: 
>> >  
>> > Right now you have to restart the parser topology. 
>> >  
>> > On Thu, Jan 25, 2018 at 10:15 PM, Otto Fowler <ottobackwards@gmail.com <mailto:ottobackwards@gmail.com>>

>> > wrote: 
>> >  
>> >> At the moment, when a grok file or something changes in HDFS, how do we

>> >> know? Do we have to restart the parser topology to pick it up? 
>> >> Just trying to clarify for myself. 
>> >>  
>> >> ottO 
>> >> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message