nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro D'Armiento" <alessandro.darmiento1...@gmail.com>
Subject PutDatabaseRecord after MergeRecord randomly hangs forcing to discard the whole queue
Date Mon, 11 Jan 2021 11:16:16 GMT
Good morning everyone,

I discovered the following issue which is bugging me lately:
If I connect the output of a MergeRecord processor to the input of a
PutDatabaseRecord processor, sometimes a FlowFile makes the latter fail
with FlowFileHandlingException: FlowFile already marked for transfer.
The issue is that when this happens, the FlowFile is not routed to the
failed relationship, but rather the whole processor remain stuck, the queue
must be manually emptied, losing all content.

I created this issue with further details:
https://issues.apache.org/jira/browse/NIFI-8130

I will start looking at the processor code, but I am not quite sure about
what a good solution would be:

   - A naive yet effective approach would be to just catch such exceptions
   and route the Flow File to the failed relationship. This would prevent the
   issue of having to manually empty the full queue, yet 1 FlowFile or 1000
   FlowFile, it's still data loss.
   - A more advanced approach would be to find out exactly what causes this
   issue to raise and prevent it, but the random nature of this error makes it
   very hard to investigate it.

I have no idea if this might be related, but here's an example of Processor
configuration which raises the problem:

The flow is the following: I consume a bunch of records (each record is a
json) from a Kafka queue, I merge those records in bins, and then I execute
a DB transaction for each bin (the DB I'm using is Vertica, which supports
a jdbc driver, therefore at NiFi eyes it's treated as a plain relational DB)

[image: Screenshot 2021-01-11 at 11.38.45.png]

Merge Record configuration:
[image: image.png]

PutDatabaseRecord configuration:
[image: image.png]

Any suggestion is appreciated, thanks.

Regards,
Alessandro D'Armiento

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message