nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elli Schwarz <eliezer_schw...@yahoo.com.INVALID>
Subject Re: Content Repo Large.. Archive in there?
Date Fri, 23 Oct 2015 18:57:00 GMT
We had max storage size of 1GB, but that's for provenance repo and our problem was with content_repo.
Our disk was 60GB, all on one partition, and 55GB were taken up by content_repo. Now, it only
contains 233MB.
 


     On Friday, October 23, 2015 2:50 PM, Mark Payne <markap14@hotmail.com> wrote:
   
 

 OK, so this is interesting. Do you have your content repository and provenance repository
both pointing to the same partition? What do you have the "nifi.provenance.repository.max.storage.size"
property set to? How large is the actual disk?

Thanks
-Mark


> On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.work@gmail.com> wrote:
> 
> I've got this one... let me look for that
> 
> 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
> o.a.n.p.PersistentProvenanceRepository
> java.io.IOException: No space left on device
>        at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
>        at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[na:1.8.0_51]
>        at
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> ~[na:1.8.0_51]
>        at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> ~[na:1.8.0_51]
>        at
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
> ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
>        at
> org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>        at
> org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
> ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
>        at
> org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
> [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
> 
> On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <markap14@hotmail.com> wrote:
> 
>> Ryan, Elli,
>> 
>> Do you by chance have any error messages in your logs from the
>> FileSystemRepository?
>> 
>> I.e., if you perform:
>> 
>> grep FileSystemRepository logs/*
>> 
>> Do you get anything interesting in there?
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz
>> <eliezer_schwarz@yahoo.com.INVALID> wrote:
>>> 
>>> I've been working with Ryan. There appear to be a few issues here:
>>> 
>>>  - We upgraded from 0.2.0 to 0.3.0 and it appears that
>> content_repository archive is now true by default. In 0.2.0 it was false,
>> and the documentation still states it is false by default.
>>>  - When we ran out of disk space overnight, the problem was solved by
>> me simply restarting nifi, and that cleared out the archive by itself.
>>> 
>>>  - In order to clear up the archive, I had to set archive to true, and
>> set max usage to 1%, and restart nifi. That cleared it up, and then I set
>> archive to false and restarted again so we don't run out of space.
>>>  - Based on the above, it appears that something happened yesterday
>> that prevented Nifi from clearing out the archive even though disk usage
>> reached 100%. However, restarting nifi apparently enabled it to perform the
>> clearing of the archive. So apparently the max usage setting doesn't work
>> under some conditions, but we don't know what conditions occurred overnight
>> to cause this problem.
>>> 
>>> Thanks!-Elli
>>> 
>>> 
>>> 
>>>    On Friday, October 23, 2015 2:29 PM, Ryan H <
>> rhendrickson.work@gmail.com> wrote:
>>> 
>>> 
>>> 
>>> Agree, they concern the archive... although it sounds like there are 2
>>> archives?
>>> 
>>> Within the content_repository folder, there are subfolders with the name
>>> 'archive' and files inside them.
>>> 
>>> Example:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>> 
>>> Settings:
>>> nifi.content.repository.archive.max.retention.period=12 hours
>>> nifi.content.repository.archive.max.usage.percentage=50%
>>> nifi.content.repository.archive.enabled=true
>>> 
>>> Last night, our server ran out of disk space because the
>> content_repository
>>> grew too large.  Nifi didn't crash, but the log file contained errors
>>> saying the disk was full.
>>> 
>>> We're not sure how, but the content_repository did not respect the above
>>> settings.
>>> 
>>> We restarted Nifi, and it only then started to remove files, such as:
>>> ./nfii/content_repository/837/archive/1445611320767-837
>>> 
>>> We've turned off archiving for now.
>>> 
>>> Ryan
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinpiri@gmail.com>
>> wrote:
>>> 
>>>> Ryan,
>>>> 
>>>> Those items only concern the archive.  Did you have data enqueued in
>>>> connections in your flow?  If so, these items are not eligible and could
>>>> explain why your disk was filled.  Otherwise, can you please provide
>> some
>>>> additional information so we can dig into why this may have arisen.
>>>> 
>>>> Thanks!
>>>> 
>>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.work@gmail.com>
>>>> wrote:
>>>> 
>>>>> I've got the following set:
>>>>> 
>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>> nifi.content.repository.archive.enabled=true
>>>>> 
>>>>> Yet, the content repo filled my disk last night...
>>>>> 
>>>>> 
>>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinpiri@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> Ryan,
>>>>>> 
>>>>>> Those archive folders map to the
>>>> nifi.content.repository.archive.enabled
>>>>>> property.
>>>>>> 
>>>>>> What this property provides is a retention of files no longer in
the
>>>>> system
>>>>>> for historical context of your flow's processing and the ability
for
>>>>>> viewing this in conjunction with provenance events as well as allowing
>>>>>> replay.  The amount of the archive when enabled is bounded by the
>>>>>> properties nifi.content.repository.archive.max.retention.period and
>>>>>> nifi.content.repository.archive.max.usage.percentage.
>>>>>> 
>>>>>> Additional detail is available in the system properties of our
>>>>>> Administration Guide [1]
>>>>>> 
>>>>>> Let us know if you have additional questions.
>>>>>> 
>>>>>> --aldrin
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
>>>>>> 
>>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.work@gmail.com
>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Interesting.. So what would
>>>>>>> 
>>>>>>> ./nfii/content_repository/837/archive/1445611320767-837
>>>>>>> 
>>>>>>> typically be?
>>>>>>> 
>>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
>>>>> agrande@hortonworks.com
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Attachments don't go through, view at imagebin:
>>>>>>>> http://ibin.co/2K3SwR0z8yWX
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agrande@hortonworks.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Ryan,
>>>>>>>>> 
>>>>>>>>> ./conf/archive is to create a snapshot of your entire
flow, not
>>>> the
>>>>>>>> content repository data. See the attached screenshot (Settings
menu
>>>>> on
>>>>>>> the
>>>>>>>> right).
>>>>>>>>> 
>>>>>>>>> Andrew
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrickson@gmail.com
on
>>>> behalf
>>>>>> of
>>>>>>>> Ryan H" <ryan.andrew.hendrickson@gmail.com on behalf of
>>>>>>>> rhendrickson.work@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>>  I'm noticing my Content Repo growing large.  There's
a number
>>>> of
>>>>>>>> files...
>>>>>>>>>> 
>>>>>>>>>> content_repo/837/archive/144...-837
>>>>>>>>>> 
>>>>>>>>>>  Is this new in 3.0?  My conf file says any archiving
should be
>>>>>> going
>>>>>>>>>> into ./conf/archive, but i don't see anything in
there.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Ryan
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
>> 


 
  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message