nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Watson <dcwatso...@gmail.com>
Subject Re: NiFi hangs at Computing Data Lineage... 100% for a specific flow
Date Fri, 17 Aug 2018 15:24:15 GMT
Mark,

I just read somewhere that the EnforceOrder processor is not meant to be
used when numbers will be "skipped". Since im using a timestamp it wont
work for me. So I'll need to find another way to do that ordering. I looked
at the PriorityAttributePrioritizer, but that doesn't seem to be able to
queue for some amount of time before proceeding. Is that correct?

Thanks,
Dan

On Fri, Aug 17, 2018 at 11:13 AM Daniel Watson <dcwatson84@gmail.com> wrote:

> Mark,
>
> Thanks! That worked in allowing the DP page to load. Now it exposes the
> underlying problem, which is that the EnforceOrder processor seems to be
> adding hundreds of "attributes modified" events, even when no attributes
> are being modified. Does this have something to do with the flow files in
> the "wait" state cycling through the processor over and over again? In the
> examples of that processor I see people looping the "wait" event back
> around to the processor so that the files are held in that processor until
> the batch is complete, so that's what I did. Is that an intended usage of
> it? If so, is there something that can be done about those DP events?
>
> Thanks,
> Dan
>
> On Fri, Aug 17, 2018 at 10:30 AM Mark Payne <markap14@hotmail.com> wrote:
>
>> OK, my recommendation would be to change to the
>> WriteAheadProvenanceRepository. You can do this by
>> changing the value of the "nifi.provenance.repository.implementation"
>> property in nifi.properties to
>> "org.apache.nifi.provenance.WriteAheadProvenanceRepository"
>>
>> The WriteAhead implementation was written several releases back, but we
>> left Persistent Provenance Repo
>> as the default just to ensure that the Write-Ahead implementation was
>> stable enough. At this point, it's pretty
>> clear that it is not only much faster than the Persistent Prov Repo but
>> also far more stable. There is a JIRA [1]
>> to make this the new default.
>>
>> The nice thing is that it is backward compatible, so if you change to the
>> WriteAhead impl and restart, you should
>> still be able to query all of your existing data without losing anything.
>>
>> Thanks
>> -Mark
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-5482
>>
>>
>> On Aug 17, 2018, at 10:26 AM, Daniel Watson <dcwatson84@gmail.com> wrote:
>>
>> Yes.
>>
>> On Fri, Aug 17, 2018 at 10:19 AM Mark Payne <markap14@hotmail.com> wrote:
>>
>>> OK, thanks. Are you using the default implementation of the Provenance
>>> Repository? I.e., the PersistentProvenanceRepository?
>>>
>>>
>>> On Aug 17, 2018, at 10:10 AM, Daniel Watson <dcwatson84@gmail.com>
>>> wrote:
>>>
>>> Mark,
>>>
>>> 1.7.0
>>> 06/19/2018 21:55:30 EDT
>>> Tagged nifi-1.7.0-RC1
>>>
>>> Thanks
>>>
>>> On Fri, Aug 17, 2018 at 10:09 AM Mark Payne <markap14@hotmail.com>
>>> wrote:
>>>
>>>> Hi Daniel,
>>>>
>>>> What version of NiFi are you running?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> > On Aug 17, 2018, at 10:07 AM, Daniel Watson <dcwatson84@gmail.com>
>>>> wrote:
>>>> >
>>>> >   Anyone have any issues with the data lineage screen? My NiFi
>>>> instance can't compute the data lineage for a specific flow. It worked
>>>> originally, then after running some files through, it no longer will. It
>>>> gets stuck at 100% and the UI shows nothing.
>>>> >
>>>> > The flow is...
>>>> >
>>>> > list files -> update att -> order -> fetch files -> update
att -> put
>>>> files
>>>> >
>>>> > I removed the provenance directory and restarted, and it did the same
>>>> thing.
>>>>
>>>>
>>>
>>

Mime
View raw message