nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Metrics on Lineage Duration
Date Tue, 07 Nov 2017 13:26:06 GMT
Peter,

In the UI you can right-click on a Processor and go to "View Status History" to see a chart
of different metrics
over time. One of those metrics is the Average Lineage Duration (averaged over a 5-minute
period).
Of course, you are asking for a reporting task/processor that does this. There's nothing that
provides this info
directly, but you could always use InvokeHTTP, or something like that to poll the REST API,
using the path
/nifi-api/flow/processors/<Processor ID>/status/history

Or, another approach that may be valuable would be to use the SiteToSiteProvenanceReportingTask
to send
Provenance Events via Site-to-Site back to the NiFi instance. For a given Provenance Event,
you can get Event Time
and Lineage Start Date, so to calculate the Lineage Duration / latency for the FlowFile, you
can just use
EventTime - LineageStartDate. This approach would be a little different, as it provides lineage
duration of each event.
So this could be used, for instance, to detect any particular FlowFiles that exceed some SLA
or calculate a 90th percentile
type of thing, rather than getting an average.

Does this help at all?

Thanks
-Mark


On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) <pwicks@micron.com<mailto:pwicks@micron.com>>
wrote:

Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a
point in time so I can monitor my flow using this metric?

Thanks,
  Peter

Mime
View raw message