nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nissim Shiman <nshi...@yahoo.com.INVALID>
Subject Re: PULL ProvenanceEvent
Date Mon, 28 Oct 2019 22:46:37 GMT
 Adam,
I believe there is a need for more detailed ProvenanceEvents.
A use case would be a customer that is trying to track data passed between two nifi's and
trying to match up SENDs and RECEIVEs

So a flowfile that has a SEND event on the first nifi should have a RECEIVE event on the second
nifi.
But a flowfile that was PULLed by the second nifi (from the first nifi) will not necessarily
have any provenance event generated by the first nifi.

(I realize that FETCH is already a "reserved word" in the current ProvenanceEvents setup,
so I was hoping PULL could be used instead.)
There is another Provenance Event, ACKNOWLEDGE, which would also fit occasionally to this
model as well (an example would be HandleHttpResponse processor which could send this instead
of SEND when responding to a HTTP request)
This being said, you make an excellent point when you said
"However even more important to realize,
this change would affect many other downstream consumers of provenance data
which aren't necessarily in the stock NiFi distribution."
Thanks,
Nissim

    On Friday, October 11, 2019, 11:30:19 AM EDT, Nissim Shiman <nshiman@yahoo.com.invalid>
wrote:  
 
  Adam,
"Yes" to your first question and the four processor examples you listed.

I will need to get back to you regarding your other points.

Thanks,
Nissim

    On Thursday, October 10, 2019, 7:05:57 PM EDT, Adam Taft <adam@adamtaft.com> wrote: 

 
 Nissim,

Just to be clear, you are trying to distinguish between processors which
are actively "pulling" data (GetXYZ) vs. processors which just "listen" for
data (ListenXYZ)?  Is that your basic vision?

GetFile => PULL
GetHTTP => PULL
ListenHTTP => RECEIVE
ListenTCP => RECEIVE

Could you clarify what advantages this would have in terms of data
provenance?  What would you use this new event type for specifically?  What
are you missing now? Do you have a use case that needs this, or are you
just generally trying to round out the provenance event types for sake of
completeness?  I honestly don't know a use case where you care whether you
polled for the data or listened for it.  The provenance model today just
cares that you received the data, not so much how you received it.

You're right that this proposal will affect many processors and the
internal visualization tools, etc.  However even more important to realize,
this change would affect many other downstream consumers of provenance data
which aren't necessarily in the stock NiFi distribution.  For example, any
third-party/custom ReportingTask that handles provenance data would need to
be updated with this change.  There's probably need for a strong vision to
help demonstrate the value for this vs. the cost of the cascading effects
related to this change.

Thanks,
Adam


On Thu, Oct 10, 2019 at 4:02 PM Nissim Shiman <nshiman@yahoo.com.invalid>
wrote:

> Hello Team,
>
> The ProvenanceEventType class does a good job capturing possible events,
> but the PULL event doesn't seem to fall nicely into any of the existing
> types.
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
> RECEIVE is the closest, but RECEIVE is passive and doesn't capture the
> active action of a PULL
>
> And... maybe it would fall into FETCH, but FETCH is more focused on
> contents of an existing flow file being overwritten.
>
> What does the community think about a new PULL event type,
> or
>  using FETCH for PULL, and having what FETCH does now be a new event such
> as REUSE
>
> NOTE: a new PULL event would have a cascading effect of many processors
> that currently are emitting RECEIVE's being modified to be PULL
> (i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but
> would more accurately capture the event.
>
> Thanks,
> Nissim Shiman
>
>
    
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message