nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James McMahon <jsmcmah...@gmail.com>
Subject Re: How to recover from a full provenance repository?
Date Mon, 13 Mar 2017 21:39:15 GMT
Sent you that info Joe.

Just to be certain I understood clearly: I should go ahead and rm -rf
./provenance_repository? Won't the problem just reappear again as soon as
it recreates provenance_repository at start up?

On Mon, Mar 13, 2017 at 5:04 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Jim
>
> Blow away the prov repo and restart to get back going.  But to
> troubleshoot why it isn't cleaning up it seems as though there is some
> reason nifi is failing to rollover.
>
> What version precisely are you running?
>
> Thanks
> Joe
>
> On Mar 13, 2017 2:01 PM, "James McMahon" <jsmcmahon3@gmail.com> wrote:
>
>> Here is all the info Joe. I had to manually recreate this here. Tried to
>> avoid typos. Anything that looks bizarre or oddly misspelled is on me.
>>
>> Certainly Joe. I need to do whatever is humanly possible to get this up
>> and
>> rolling again. Here is what I can provide:
>>
>> (Background)
>> /mnt/provenance_repo 50G size, 47G used, 17M available, 100% capacity
>> under this is
>> database_repository
>> provenance_repository
>> du -h output:
>> 8.5M ./database_repository
>> 16K ./lost+found
>> 4.0K ./provenance_repository/index-1489...124
>> 92K ./provenance_repository/toc
>> 47G ./provenance_repository/journals
>> 47G ./provenance_repository
>> 47G .
>>
>> In my nifi.properties:
>> #H2 Settings
>> nifi.database.directory=/mnt/provenance_repo/database_repository
>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>> nifi.provenance.repository.directory.default=/mnt/provenance
>> _repo/provenance_repos
>> itory
>> nifi.provenance.repository.max.storage.time=24 hours
>> nifi.provenance.repository.max.storage.size=1 GB
>> nifi.provenance.repository.rollover.time=30 secs
>> nifi.provenance.repository.rollover.size=100 MB
>> nifi.provenance.repository.query.threads=2
>> nifi.provenance.repository.index.threads=1
>> nifi.provenance.repository.compress.on.rollover=true
>> nifi.provenance.repository.always.sync=false
>> nifi.provenance.repository.journal.count=16
>> nifi.provenance.repository.indexed.fields=EventType,FlofFile
>> UUID,Filename,ProcessorID,Relationship
>> nifi.provenance.repository.indexed.attributes=
>> nifi.provenance.repository.index.shard.size=500 MB
>> nifi.provenance.repository.max.attributes.length=65536
>> nifi.provenance.repository.buffer.size=100000
>> (Errors)
>> in nifi-bootstrap.log, java.io.IOException No space left on device
>> /mnt/provenance_repo/database_repository/nifi-users.h2.db
>> similar error reported in nifi-app.log, fails when trying to create
>> org.springframework.beans.factory.BeanCreationException
>>
>> I think both are just symptoms of the true problem, which is to say too
>> much stuff in
>> journals under provenance_repository
>> Jim
>>
>> On Mon, Mar 13, 2017 at 4:10 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>>> Jim,
>>>
>>> Processors create provenance events which get written to the
>>> provenenance repo but that is the end of their relationship.  The
>>> provenance repository is then self managing.  Can you share your
>>> nifi.properties provenance configuration?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mon, Mar 13, 2017 at 1:05 PM, James McMahon <jsmcmahon3@gmail.com>
>>> wrote:
>>> > Good evening. I have a number of process groups in my NiFi instance
>>> that run
>>> > concurrently. Evidently I have filled up my provenance repo. How do I
>>> > recover from this? NiFi will not start back up after shutdown.
>>> >
>>> > I am in a dev environment right now so I can afford to lose what had
>>> been in
>>> > my flows. Does that give me other options to recover?
>>> >
>>> > However I must determine what process group and processor are not
>>> freeing
>>> > resources from the provenance repo. I can't allow this workflow to
>>> promote
>>> > to production until I determine how to prevent this from reoccurring.
>>> How
>>> > can I do that?
>>> >
>>> > My provenance, content, and flow repos are all on separate disk
>>> devices,
>>> > each of which is 50GB in capacity.
>>> >
>>> > Thank you for your help. -Jim
>>>
>>
>>

Mime
View raw message