nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Loughran <a...@zrmt.com>
Subject Batch Processing
Date Wed, 16 Aug 2017 13:55:02 GMT
Hey everyone,

This is my first post.

I'm building out a pipeline with Nifi, but am stuck on an architectural
decision around some fairly basic design.  I think I'm stuck as I'm
operating on the wrong paradigm, but the application receiving my flow is
the limitation in this context.

I'm using ListS3 to poll for csv files.  There need to be 5 different types
of file uploaded with a unique batch identifier for them to be released.
I'm using UpdateAttribute to rip the type and batch from the filename, then
using wait to hold the batch.

At the moment though, I'm holding until a batch has 5 files, rather than 5
files with each attribute type matching the expected types.

Is this the wrong way to be thinking about this problem, or does this sound
like a good use case for Nifi - but using a better combination of
processors.  If anyone could give me guidance or point me toward an example
template for batch process I'd be grateful.

Look forward to helping out in the community where I can.

Thanks,

Andy

Mime
View raw message