nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Moser <moser...@gmail.com>
Subject Re: After 1.7.1 upgrade, no Provenance data is visible
Date Fri, 10 Aug 2018 21:48:53 GMT
Hi Peter,

There was a change to provenance related access policies in 1.7.0.  Check
out the Migration Guide [1] for 1.7.0.  It talks about what you'll need to
do.

[1] - https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance

-- Mike


On Fri, Aug 10, 2018 at 5:39 PM Peter Wicks (pwicks) <pwicks@micron.com>
wrote:

> After upgrading our NiFi instances to 1.7.1 we are not able to see
> Provenance data anymore in the UI. We see this across about a dozen
> instances.
>
> In the UI it tells me provenance is available for about the last 24 hours,
> and I can see that files have moved in and out of the processor in the last
> 5 min. In the logs, I can see it query provenance, and that the query
> returns 0 results.
>
>
>
> Thoughts? I saw a few tickets related to Provenance in 1.7, but not sure
> if they have an impact.
>
>
>
> Here are our properties:
>
>
>
> # Provenance Repository Properties
>
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
>
> nifi.provenance.repository.debug.frequency=1_000_000
>
> nifi.provenance.repository.encryption.key.provider.implementation=
>
> nifi.provenance.repository.encryption.key.provider.location=
>
> nifi.provenance.repository.encryption.key.id=
>
> nifi.provenance.repository.encryption.key=
>
>
>
> # Persistent Provenance Repository Properties
>
>
> nifi.provenance.repository.directory.default=/data/nifi/repositories/provenance_repository
>
> nifi.provenance.repository.max.storage.time=24 hours
>
> nifi.provenance.repository.max.storage.size=1 GB
>
> nifi.provenance.repository.rollover.time=30 secs
>
> nifi.provenance.repository.rollover.size=100 MB
>
> nifi.provenance.repository.query.threads=2
>
> nifi.provenance.repository.index.threads=2
>
> nifi.provenance.repository.compress.on.rollover=true
>
> nifi.provenance.repository.always.sync=false
>
> nifi.provenance.repository.journal.count=16
>
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
>
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details
>
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID,
> Filename, ProcessorID, Relationship
>
> # FlowFile Attributes that should be indexed and made searchable.  Some
> examples to consider are filename, uuid, mime.type
>
> nifi.provenance.repository.indexed.attributes=
>
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
>
> # but should provide better performance
>
> nifi.provenance.repository.index.shard.size=500 MB
>
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
>
> # the repository. If the length of any attribute exceeds this value, it
> will be truncated when the event is retrieved.
>
> nifi.provenance.repository.max.attribute.length=65536
>
> nifi.provenance.repository.concurrent.merge.threads=2
>
> nifi.provenance.repository.warm.cache.frequency=1 hour
>
>
>
> Thanks,
>
>   Peter
>

Mime
View raw message