While Pierre’s solution will probably be more helpful in the long run, this provides an easy way for me to examine the data by hand right now. I didn’t know that metric was on the Status History window, good to now!
In the UI you can right-click on a Processor and go to "View Status History" to see a chart of different metrics
over time. One of those metrics is the Average Lineage Duration (averaged over a 5-minute period).
Of course, you are asking for a reporting task/processor that does this. There's nothing that provides this info
directly, but you could always use InvokeHTTP, or something like that to poll the REST API, using the path
Or, another approach that may be valuable would be to use the SiteToSiteProvenanceReportingTask to send
Provenance Events via Site-to-Site back to the NiFi instance. For a given Provenance Event, you can get Event Time
and Lineage Start Date, so to calculate the Lineage Duration / latency for the FlowFile, you can just use
EventTime - LineageStartDate. This approach would be a little different, as it provides lineage duration of each event.
So this could be used, for instance, to detect any particular FlowFiles that exceed some SLA or calculate a 90th percentile
type of thing, rather than getting an average.
Does this help at all?
On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) <firstname.lastname@example.org> wrote:
Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can monitor my flow using this metric?