nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Omer Hadari <hadari.o...@gmail.com>
Subject Re: Drop event disordering
Date Wed, 22 Nov 2017 18:04:06 GMT
I’ll be glad to open a jira, though the problem is hardly coherent imo,
what would you like to see there? Simply “Disordering of drop events” and
the explanation I have here? Sadly I cannot provide a concrete example
since the problem does not reproduce.

On Wed, 22 Nov 2017 at 18:23 Joe Witt <joe.witt@gmail.com> wrote:

> also - awesome find!  And glad you're at such a level with provenance
> data to catch that.  Thanks Omer!
>
> On Wed, Nov 22, 2017 at 11:21 AM, Mark Payne <markap14@hotmail.com> wrote:
> > Omer,
> >
> > This is likely an issue related to the order in which we generate those
> events in the framework.
> > Do you mind filing a JIRA?
> >
> > Thanks
> > -Mark
> >
> >
> >> On Nov 22, 2017, at 10:51 AM, Omer Hadari <hadari.omer@gmail.com>
> wrote:
> >>
> >> Hi!
> >> We’ve been using NiFi for a while now, and we save all provenance
> events for logging purposes and such. We encountered an issue while looking
> at lineages of some flow files, which showed drop events as if they
> happened before another event, that in fact preceded it (and indeed has a
> lower event ordinal).
> >>
> >> For example in a split json processor, the original FlowFile is dropped
> after all splits happen and are assigned fragment counts, but still the
> timestamp of the drop event is earlier than the timestamp of the attributes
> modified event. That causes the graph to look as if the attributes modified
> event comes out of the drop event, which doesn’t really make sense to us
> (should it?). It’s probably worth noting that the drop event ordinal is
> higher than the attributes modified event ordinal. Also we noticed that
> >> 1. This only happens every once per a few thousand events.
> >> 2. This does not reproduce by replaying.
> >> 3. The drop event’s timestamp is earlier by 1ms in the cases we
> encountered, and the ordinal is always larger by one.
> >>
> >> This might be an error with the split json processor or a more general
> one. We’d love any clues or corrections to misconceptions we might have
> (maybe this is not a problem and drop events can precede other events?)
> >>
> >> Thank you!
> >
>

Mime
View raw message