nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: Data SLA
Date Tue, 06 Dec 2016 18:23:02 GMT
Richard,

In addition to the flow that loads those files and performs whatever follow-on transformation
and routing, you can have a parallel flow which monitors the directory and uses cron scheduling
to run at a specific time (0900 daily) and is configured to expect a threshold number of files
and in a specific naming pattern. This separate flow can route to email processors, etc. to
alert relevant teams. If ListHDFS does not meet your requirements for monitoring, ExecuteScript
is very versatile in performing non-standard actions within NiFi.

You should also take a look at the monitoring capabilities of the NiFi UI/API such as data
provenance, queue interaction, and statistics/monitoring. These features provide live monitoring
for your administrators/data flow managers. You can read from these resources/queries via
external monitoring tools if necessary as well.

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Dec 6, 2016, at 9:29 AM, Richard Duarte <richard.c.duarte@gmail.com> wrote:
> 
> Hello,
> 
> I'm working on a project that I'd like to use Nifi for, however I'm trying to figure
out if there's a way to alert on the availability of data being provided correctly to an agreement.
> 
> Example:
> 1. Team X has agreed to provide files in directory Y in HDFS
> 2. Team X has agreed to provide those files once per day by 9 AM PST
> 3. Team X has agreed to name those files following naming convention [x]-[date]-[abc]
> 
> 
> Is there any way for Nifi to alert on Team X violating any of those agreements?  I.E.
can we get Nifi to alert on the files not being there, the files not being there in time,
and/or the files not following the correct naming conventions?
> 
> If not, are there any other tools that tie into Nifi to provide help on the Data SLA?
> 
> Thanks,
> Richard


Mime
View raw message