nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Drop event disordering
Date Wed, 22 Nov 2017 18:34:27 GMT
Omer,

Yes, I think that is sufficient. I think the issue is that the framework is creating both
the
ATTRIBUTES_MODIFIED and DROP events, and the generation of these objects is
very fast. But if the timestamp happens to 'rollover' from millisecond 1 to millisecond 2,
for example, those events get different timestamps. So I think it's just a timing thing that
will be somewhat difficult to reproduce reliably. But just a description of the behavior that
you're experiencing should be fine.

Thanks
-Mark

On Nov 22, 2017, at 1:04 PM, Omer Hadari <hadari.omer@gmail.com<mailto:hadari.omer@gmail.com>>
wrote:

I’ll be glad to open a jira, though the problem is hardly coherent imo, what would you like
to see there? Simply “Disordering of drop events” and the explanation I have here? Sadly
I cannot provide a concrete example since the problem does not reproduce.

On Wed, 22 Nov 2017 at 18:23 Joe Witt <joe.witt@gmail.com<mailto:joe.witt@gmail.com>>
wrote:
also - awesome find!  And glad you're at such a level with provenance
data to catch that.  Thanks Omer!

On Wed, Nov 22, 2017 at 11:21 AM, Mark Payne <markap14@hotmail.com<mailto:markap14@hotmail.com>>
wrote:
> Omer,
>
> This is likely an issue related to the order in which we generate those events in the
framework.
> Do you mind filing a JIRA?
>
> Thanks
> -Mark
>
>
>> On Nov 22, 2017, at 10:51 AM, Omer Hadari <hadari.omer@gmail.com<mailto:hadari.omer@gmail.com>>
wrote:
>>
>> Hi!
>> We’ve been using NiFi for a while now, and we save all provenance events for logging
purposes and such. We encountered an issue while looking at lineages of some flow files, which
showed drop events as if they happened before another event, that in fact preceded it (and
indeed has a lower event ordinal).
>>
>> For example in a split json processor, the original FlowFile is dropped after all
splits happen and are assigned fragment counts, but still the timestamp of the drop event
is earlier than the timestamp of the attributes modified event. That causes the graph to look
as if the attributes modified event comes out of the drop event, which doesn’t really make
sense to us (should it?). It’s probably worth noting that the drop event ordinal is higher
than the attributes modified event ordinal. Also we noticed that
>> 1. This only happens every once per a few thousand events.
>> 2. This does not reproduce by replaying.
>> 3. The drop event’s timestamp is earlier by 1ms in the cases we encountered, and
the ordinal is always larger by one.
>>
>> This might be an error with the split json processor or a more general one. We’d
love any clues or corrections to misconceptions we might have (maybe this is not a problem
and drop events can precede other events?)
>>
>> Thank you!
>

Mime
View raw message