nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Klempa <michal.kle...@gmail.com>
Subject Penalization on retry, on failure and session commit clarificatoion
Date Tue, 07 May 2019 10:10:04 GMT
Hi,
I came across an issue, which looks similar to
https://stackoverflow.com/questions/51891413/nifi-processor-task-count-is-very-high-what-must-be-the-reason
Many tasks/time on downstream processor.

In my case, we are using PutElasticsearchHttpRecord and the issue is,
when RecordReader cannot parse input records (e.g. invalid JSON).
We fall into catch block:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L455

Where the FlowFile is penalized and transferred to FAILURE.

Now the questions:
1. is penalization meant for retry-able errors? According to
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
it seems like the answer is Yes, but the example given is also with
FAILURE relationship (not RETRY relationship)
2. I can't find any session.commit in processor code. Is it OK? May it
be, that the problem happens when session is not committed? Does NiFi
any auto-commit then?
3. is there any design document for new processor regarding Penalize +
RETRY/FAILURE relationship?

Thanks.
Michal

Mime
View raw message