nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Omer Hadari <hadari.o...@gmail.com>
Subject Drop event disordering
Date Wed, 22 Nov 2017 15:51:46 GMT
Hi!
We’ve been using NiFi for a while now, and we save all provenance events
for logging purposes and such. We encountered an issue while looking at
lineages of some flow files, which showed drop events as if they happened
before another event, that in fact preceded it (and indeed has a lower
event ordinal).

For example in a split json processor, the original FlowFile is dropped
after all splits happen and are assigned fragment counts, but still the
timestamp of the drop event is earlier than the timestamp of the attributes
modified event. That causes the graph to look as if the attributes modified
event comes out of the drop event, which doesn’t really make sense to us
(should it?). It’s probably worth noting that the drop event ordinal is
higher than the attributes modified event ordinal. Also we noticed that
1. This only happens every once per a few thousand events.
2. This does not reproduce by replaying.
3. The drop event’s timestamp is earlier by 1ms in the cases we
encountered, and the ordinal is always larger by one.

This might be an error with the split json processor or a more general one.
We’d love any clues or corrections to misconceptions we might have (maybe
this is not a problem and drop events can precede other events?)

Thank you!

Mime
View raw message