metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <>
Subject Re: When things change in hdfs, how do we know
Date Fri, 26 Jan 2018 13:47:11 GMT
In the end, what I’m thinking is this:

We have an ambari service that runs the notification -> zookeeper
it reads the ‘registration area’ from zookeeper to get it’s state and what
to watch
post 777 when parsers are installed and registered it is trivial to have my
installer also register the files to watch

the notifications service also has a notification from zookeeper for new

On notify event, the ‘notification node’ has it’s content set to the event
details and time
which the parser would pick up…. causing the reload

This would work for the future script parser etc etc.

On January 26, 2018 at 08:30:32, Simon Elliston Ball ( wrote:

Interesting, so you have an INotify listener to filter events, and then on
given changes, propagate a notification to zookeeper, which then triggers
the reconfiguration event via the curator client in Metron. I kinda like it
given our existing zookeeper methods.


On 26 Jan 2018, at 13:27, Otto Fowler <> wrote:

Working on a poc

On January 26, 2018 at 07:41:44, Simon Elliston Ball ( wrote:

Should we consider using the Inotify interface to trigger reconfiguration,
in same way we trigger config changes in curator? We also need to fix
caching and lifecycle in the Grok parser to make the zookeeper changes
propagate pattern changes while we’re at it.


> On 26 Jan 2018, at 03:16, Casey Stella <> wrote:
> Right now you have to restart the parser topology.
> On Thu, Jan 25, 2018 at 10:15 PM, Otto Fowler <>
> wrote:
>> At the moment, when a grok file or something changes in HDFS, how do we
>> know? Do we have to restart the parser topology to pick it up?
>> Just trying to clarify for myself.
>> ottO

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message