nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Yar <andyya...@gmail.com>
Subject Re: Reliable operation use case
Date Tue, 16 May 2017 18:05:50 GMT
Thank your for the info.

I guess I will try a small PoC and see how it goes.

On Mon, May 15, 2017 at 8:13 PM, Pierre Villard
<pierre.villard.fr@gmail.com> wrote:
> Hi Andy,
>
> Regarding bulletins, you can use the new SiteToSiteBulletinReportingTask [1]
> to achieve what you are expecting (in NiFi 1.2.0). For provenance, you can
> definitely use the REST API but could also be interested by the
> SiteToSiteProvenanceReportingTask [2] with the addition of NIFI-3859 [3].
> You may also be interested by this article [4].
>
> From a high-level perspective all what you described is perfectly doable.
> Obviously, it'd be necessary to go into the details of what are the
> monitoring tools you want to integrate with but, again, NiFi sounds totally
> suitable to your use case.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteBulletinReportingTask/index.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html
> [3] https://issues.apache.org/jira/browse/NIFI-3859
> [4]
> https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/
>
> Pierre
>
>
>
> 2017-05-15 19:54 GMT+02:00 Andy Yar <andyyar66@gmail.com>:
>>
>> Hello,
>> I'd like to know if NiFi is suitable for my slightly unusual use case.
>>
>> Basically, the system should route files/events/etc. from various
>> sources, validate, process them and route somewhere. However, I need
>> to do this in a reliable way. This means, the system needs to have
>> extensive monitoring/reprocessing capabilities. Various errors can
>> occur in the system - an outbound destination gets unavailable, a
>> malformed event doesn't match schema, an exception occurred during
>> processing, etc. In my case, the system has to allow discovering +
>> fixing + reprocessing of any failed event.
>>
>> I've figured that it is possible to route every processor's "failure"
>> state to a dead-letter-queue, fire up a notification (Slack, API call,
>> etc.), reroute the DLQ to the input again and so on. It seems to be a
>> bit complicated with adding a few retry steps before sending to the
>> DLQ everywhere but OK.
>>
>> However, I'm a bit worried about the monitoring side. For instance, is
>> NiFi, in its current state, able/intended to persist Bulletin Board
>> messages? I can imagine that after a operator sees an error on the
>> Bulletin Board he/she traces the flow file, fixes the issue and marks
>> the incident as solved. Currently the BB messages seems to be a bit
>> temporary. I guess the intended way is to create a custom Reporting
>> task in order to handle this.
>>
>> The Data Provenance itself is great, but still I can't imagine it
>> being usable without a major customization. In my case I would like to
>> monitor mostly a set of "inbound oriented" processors. So it would be
>> possible to list eg. all current events on inbound processors in a
>> (pseudo) realtime way. Again, I guess custom API/tasks could handle
>> this.
>>
>> To summarize: is NiFi usable for this particular use case?
>>
>> Thanks for any answers and also for your effort with this project!
>>
>> A.
>
>

Mime
View raw message