nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Penalization on retry, on failure and session commit clarificatoion
Date Tue, 07 May 2019 19:55:57 GMT
Hi Michal,

Regarding the session.commit... most processors extend from
AbstractProcessor which takes care of the commit [1]. If you extend
from AbstractSessionFactoryProcessor, or directly implement the
Processor interface with no parent class, then it is up to you to call
commit.

For the specific elastic search situation, I'm not sure if penalizing
makes sense in all those cases. The catch block for
IdentifierNotFoundException has a penalize, but this doesn't seem like
something that would resolve itself if tried again, similar for
MalformedRecordException in the catch block below.

[1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java#L28

On Tue, May 7, 2019 at 6:10 AM Michal Klempa <michal.klempa@gmail.com> wrote:
>
> Hi,
> I came across an issue, which looks similar to
> https://stackoverflow.com/questions/51891413/nifi-processor-task-count-is-very-high-what-must-be-the-reason
> Many tasks/time on downstream processor.
>
> In my case, we are using PutElasticsearchHttpRecord and the issue is,
> when RecordReader cannot parse input records (e.g. invalid JSON).
> We fall into catch block:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L455
>
> Where the FlowFile is penalized and transferred to FAILURE.
>
> Now the questions:
> 1. is penalization meant for retry-able errors? According to
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
> it seems like the answer is Yes, but the example given is also with
> FAILURE relationship (not RETRY relationship)
> 2. I can't find any session.commit in processor code. Is it OK? May it
> be, that the problem happens when session is not committed? Does NiFi
> any auto-commit then?
> 3. is there any design document for new processor regarding Penalize +
> RETRY/FAILURE relationship?
>
> Thanks.
> Michal

Mime
View raw message