nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Wing <jvw...@gmail.com>
Subject Re: Logical view of flow for error handling
Date Sun, 25 Feb 2018 01:10:17 GMT
Georg,

NiFi is quite frequently used for ingesting data into Hadoop/HDFS, and NiFi
can absolutely retry on errors.  However, I think of this more of a flow
design and process modeling question than a retry or error handling
capability question.

Orchestration tools start with the concept of a job, where the files in the
job succeed, fail, and perhaps retry together.

In contrast, NiFi will work best when you model your data processing as a
continuous flow - files enter NiFi, get validated, reformatted, bundled,
and then written out to HDFS as soon as possible.  NiFi can and will retry
an individual failure, or route a file to a failure path as you configure
it.  But NiFi does not really have the concept of failing a batch job of
many files together.  You would have to organize your NiFi flow to manage
how records are kept together, and how they are to be routed in the case of
failures.  Some flows model a job as a single file of many records, where
the entire file can be routed to success or failure.  Other flows decompose
the inputs to granular records, and route them to outputs on an individual
basis.  It's up to you.

Last, Scoop has the advantage of being a more specialized tool for
importing data from an RDBMS to Hadoop.  NiFi allows for a wider range of
processing options, but will require more effort from you to specify the
process.

Thanks,

James




On Sat, Feb 24, 2018 at 9:51 AM, Georg Heiler <georg.kf.heiler@gmail.com>
wrote:

> Using NiFi for batch RDBMS imports into Hadoop  I wonder if it is the
> right tool and i.e. if a scoop job scheduled via Ozzie might be simpler
> regarding error handling.
>
> Ozzie simply will list all failed task and automatically retry a couple of
> times.
> Is something like this (logically see that some scheduled flow failed,
> retry) also available in NiFi?
>
> Best,
> Georg
>

Mime
View raw message