nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyllys Ingersoll <wyllys.ingers...@keepertech.com>
Subject Re: Not Seeing Provenance data
Date Fri, 10 Apr 2020 15:17:42 GMT
I have a similar problem with viewing provenance.  I have a 3-node cluster
in a kubernetes environment, the provenance_repository directory for each
node is on a persistent data store so it is not deleted or lost between
container restarts (which are not very common).  My
nifi.provenance.repository.max.storage.time is 24 hours.

Whenever I try to view any provenance, nothing is ever shown.  If I
manually inspect the provenance_repository directory, there is a lucene
index and TOC being created.

I see log messages like these:

Submitting query +processorId:882133fe-b684-148b-ad88-7850437ca591 with
identifier 64a703fe-0171-1000-0000-000065abd91a against index directories
[./provenance_repository/lucene-8-index-1560864819888]
Returning the following list of index locations because they were finished
being written to before 1586531601311: []
Found no events in the Provenance Repository. In order to perform
maintenace of the indices, will assume that the first event time is now
(1586531601311)


Any suggestions?

-Wyllys Ingersoll



On Thu, Apr 9, 2020 at 11:25 AM Dobbernack, Harald (Key-Work) <
harald.dobbernack@key-work.de> wrote:

> Hey Mark,
>
>
>
> great news and thank you very much!
>
>
>
> Happy Holidays!
>
> Harald
>
>
>
> *Von:* Mark Payne <markap14@hotmail.com>
> *Gesendet:* Donnerstag, 9. April 2020 17:18
> *An:* users@nifi.apache.org
> *Betreff:* Re: Not Seeing Provenance data
>
>
>
> Thanks Harald,
>
>
>
> I have created a Jira [1] for this. There’s currently a PR up for it as
> well.
>
>
>
> Thanks
>
> -Mark
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-7346
>
>
>
> On Apr 9, 2020, at 11:14 AM, Dobbernack, Harald (Key-Work) <
> harald.dobbernack@key-work.de> wrote:
>
>
>
> Hi Mark,
>
>
>
> I can confirm after testing that if no provenance event has been generated
> in a time greater than the set nifi.provenance.repository.max.storage.time
> then as expected the last recorded provenance events don’t exist anymore
> but also from then on any new provenance events are also not searchable,
> the provenance Search remains completely empty regardless of how many flows
> are active.  As described also *.prov file is then missing in provenance
> repository. After restart of Nifi new prov File will be generated and
> provenance will work again, but only showing stuff generated since last
> NiFi Start.
>
>
>
> So yes, I’d say your Idea
>
>     ‘If so, then I think that would understand why it deleted the data.
> It’s trying to age off old data
>
>      but unfortunately it doesn’t perform a check to first determine
> whether or not the “old file”
>
>      that it’s about to delete is also the “active file”.’
>
> fits very nicely to my test.
>
>
>
> As a workaround we’re going to set a greater nifi.provenance.repository.max.storage.time
> until this can be resolved.
>
>
>
> Thanks again for looking into this.
>
> Harald
>
>
>
>
>
> *Von:* Dobbernack, Harald (Key-Work)
> *Gesendet:* Donnerstag, 9. April 2020 15:22
> *An:* users@nifi.apache.org
> *Betreff:* AW: Not Seeing Provenance data
>
>
>
> Hi Mark,
>
>
>
> thank you for looking into this.
>
>
>
> The nifi.provenance.repository.max.storage.time setting might explain why
> I haven’t been experiencing the effect so often since changing from the
> default to 120 hours a few months ago 😉
>
>
>
> But I believe provenance stopped working last time although there was an
> ‘active’ flows in wait Processor, expiring every hour, going on to ‘send a
> message’ before being rerouted to the same wait processor. I would have
> expected this generates provenance entries?  As I am not actually 100% sure
> if that wait processor was in use when last provenance got lost I will
> check with a testing system to see if I can reproduce provenance breakage
> when no active flows are around for a time greater
>  nifi.provenance.repository.max.storage.time and I will get back to you.
>
>
>
> Thank you!
>
> Harald
>
>
>
>
>
> *Von:* Mark Payne <markap14@hotmail.com>
> *Gesendet:* Donnerstag, 9. April 2020 14:41
> *An:* users@nifi.apache.org
> *Betreff:* Re: Not Seeing Provenance data
>
>
>
> Hey Daren, Herald,
>
>
>
> Thanks for the note. I have seen this once before but couldn’t figure out
> what caused it. Restarting addressed the issue.
>
>
>
> I think I may understand the problem, now, though, after looking at it
> again.
>
>
>
> In nifi.properties, there are a couple of property named
> “nifi.provenance.repository.max.storage.time” that defaults to “24 hours"
>
> Is it possible that you went 24 hours (or whatever value is set for that
> property) without generating any Provenance events?
>
>
>
> If so, then I think that would understand why it deleted the data. It’s
> trying to age off old data but unfortunately it doesn’t perform a check to
> first determine whether or not the “old file” that it’s about to delete is
> also the “active file”.
>
>
>
> Can you confirm whether or not you would expect to see 24 hours pass
> without any provenance data?
>
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
>
> On Apr 9, 2020, at 4:32 AM, Dobbernack, Harald (Key-Work) <
> harald.dobbernack@key-work.de> wrote:
>
>
>
> What I noticed is that as long as provenance is working there will be
> *.prov files in the directory. When Provenance isn’t working these files
> are not to be seen. Maybe some Cleaning Process deletes those files
> prematurely or the process building them doesn’t work any more?
>
>
>
> *Von:* Dobbernack, Harald (Key-Work) <harald.dobbernack@key-work.de>
> *Gesendet:* Donnerstag, 9. April 2020 10:27
> *An:* users@nifi.apache.org
> *Betreff:* AW: Not Seeing Provenance data
>
>
>
> This is something I experience too from time to time. My quick and dirty
> workaround is stop nifi, delete everything in the provenance directory,
> restart….  Then Provenance is usable again (of course only with data since
> the delete) . I’m hoping very much there is a better way, someone can show
> us better settings or a potential bug can be discovered…
>
>
>
> *Von:* Darren Govoni <darren@ontrenet.com>
> *Gesendet:* Mittwoch, 8. April 2020 20:31
> *An:* users@nifi.apache.org
> *Betreff:* Not Seeing Provenance data
>
>
>
> Hi,
>
>   When I go to "View data provenance" in Nifi, I never see any logs for my
> flow. Am I missing some configuration setting somewhere?
>
>
>
> thanks,
>
> Darren
>
>
>
>
>
> *Harald Dobbernack*
> Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany |
> https://www.key-work.de | Datenschutz
> <https://www.key-work.de/de/footer/datenschutz.html>
> Fon: +49-721-78203-264 | E-Mail: harald.dobbernack@key-work.de | Fax:
> +49-721-78203-10
>
> Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
> Geschäftsführer: Andreas Stappert, Tobin Wotring
>
>
>

Mime
View raw message