nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Grande <apere...@gmail.com>
Subject Re: backpressure issue
Date Wed, 22 May 2019 18:39:23 GMT
Boris,

In my experience, the backpressure is a threshold, not a hard cap.
Depending on the nature of the transaction, the actual number of events may
go slightly beyond the set value. Otherwise, how would you be able to split
1 unit of work?

Andrew

On Wed, May 22, 2019, 11:33 AM Boris Tyukin <boris@boristyukin.com> wrote:

> Hi guys,
>
> we run on 1.9.2 and experience an interesting issue with one of the custom
> groovy processors.
>
> It batches 2000 flowfiles (sessing.get(2000)) and routes some to a waiting
> relationship for retries. that relationship has backpressure set to 5000
> flowfiles.
>
> What happens is we can see often 6000-8000 flowfiles there while according
> to backpressure setting we should only see 5000 max.
>
> Is this by design or flowfiles can overflow or something wrong with the
> way our custom processor routes them?
>
> code is really easy and based on some conditional logic it would either
>
> let the flowfiles pass downstream with
>
> session.transfer(flowFile, REL_SUCCESS)
>
>
> or route flowfile to a waiting relationship like so
>
> session.penalize(flowFile)
> session.transfer(flowFile, REL_WAIT)
>
> then session is commited for the entire batch
> session.commit()
>
>
>
>

Mime
View raw message