nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: backpressure issue
Date Wed, 22 May 2019 19:41:36 GMT
this is good to know, thanks Andrew and Mark.

We really love this feature and along with a penalty duration/schedule and
concurrency, it makes NiFi extremely powerful without writing tons of
error-prone custom code.

As a matter of fact, I do not know what we would do if NiFi did not have it
(or we did not decide to have NiFi).

Thanks for all the great work you do!


On Wed, May 22, 2019 at 2:39 PM Andrew Grande <aperepel@gmail.com> wrote:

> Boris,
>
> In my experience, the backpressure is a threshold, not a hard cap.
> Depending on the nature of the transaction, the actual number of events may
> go slightly beyond the set value. Otherwise, how would you be able to split
> 1 unit of work?
>
> Andrew
>
> On Wed, May 22, 2019, 11:33 AM Boris Tyukin <boris@boristyukin.com> wrote:
>
>> Hi guys,
>>
>> we run on 1.9.2 and experience an interesting issue with one of the
>> custom groovy processors.
>>
>> It batches 2000 flowfiles (sessing.get(2000)) and routes some to a
>> waiting relationship for retries. that relationship has backpressure set to
>> 5000 flowfiles.
>>
>> What happens is we can see often 6000-8000 flowfiles there while
>> according to backpressure setting we should only see 5000 max.
>>
>> Is this by design or flowfiles can overflow or something wrong with the
>> way our custom processor routes them?
>>
>> code is really easy and based on some conditional logic it would either
>>
>> let the flowfiles pass downstream with
>>
>> session.transfer(flowFile, REL_SUCCESS)
>>
>>
>> or route flowfile to a waiting relationship like so
>>
>> session.penalize(flowFile)
>> session.transfer(flowFile, REL_WAIT)
>>
>> then session is commited for the entire batch
>> session.commit()
>>
>>
>>
>>

Mime
View raw message