nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: backpressure issue
Date Wed, 22 May 2019 18:37:20 GMT
Boris,

Yes, this is by design. The backpressure threshold is a "soft limit" meaning that it can be
exceeded.
Once you reach or exceed the threshold, it will stop triggering your processor to run until
the level
drops back down below the configured threshold. But you can certainly exceed the limit.

Thanks
-Mark

> On May 22, 2019, at 2:23 PM, Boris Tyukin <boris@boristyukin.com> wrote:
> 
> Hi guys,
> 
> we run on 1.9.2 and experience an interesting issue with one of the custom groovy processors.

> 
> It batches 2000 flowfiles (sessing.get(2000)) and routes some to a waiting relationship
for retries. that relationship has backpressure set to 5000 flowfiles.
> 
> What happens is we can see often 6000-8000 flowfiles there while according to backpressure
setting we should only see 5000 max.
> 
> Is this by design or flowfiles can overflow or something wrong with the way our custom
processor routes them? 
> 
> code is really easy and based on some conditional logic it would either 
> 
> let the flowfiles pass downstream with
> 
> session.transfer(flowFile, REL_SUCCESS)
> 
> 
> or route flowfile to a waiting relationship like so
> 
> session.penalize(flowFile)
> session.transfer(flowFile, REL_WAIT)
> 
> then session is commited for the entire batch 
> session.commit() 
> 
>  
> 


Mime
View raw message