nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: End of stream?
Date Fri, 06 Nov 2015 16:38:27 GMT
Matt,

For processors in the middle of the flow the null check is important
for race conditions where it is told it can run but by the time it
does there are no flowfiles left.  The framework though in general
will avoid this because it is checking if there is work to do.  So, in
short you can't use that mechanism to know there are no items left to
process.

The only way to know that a given flowfile was the last in a bunch
would be for that fact to be an attribute on a given flow file.

There is really no concept of an end of stream so to speak from a
processor perspective.  Processors are either running on not running.
You can, as i mentioned before though, use attributes of flowfiles to
annotate their relative position in a stream.

Does that help explain it at all or did I make it more confusing?

Thanks
Joe

On Fri, Nov 6, 2015 at 11:32 AM, Matthew Burgess <mattyb149@gmail.com> wrote:
> Does NiFi have the concept of an "end of stream" or is it designed to pretty
> much always be running? For example if I use a GetFile processor pointing at
> a single directory (with remove files = true), once all the files have been
> processed, can downstream processors know that?
>
> I'm working on a ReservoirSampling processor, and I have it successfully
> building the reservoir from all incoming FlowFiles. However it never gets to
> the logic that sends the sampled FlowFiles to the downstream processor (just
> a PutFile at this point). I have the logic in a block like:
>
> FlowFile flowFile = session.get();
> if(flowFile == null) {
>   // send reservoir
> }
> else {
>  // build reservoir
> }
>
> But the if-clause never gets entered.  Is there a different approach and/or
> am I misunderstanding how the data flow works?
>
> Thanks in advance,
> Matt
>
>

Mime
View raw message