nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Sequeiros <helloj...@gmail.com>
Subject Re: Content Repository Growing Too Large
Date Fri, 01 Sep 2017 22:58:05 GMT
My guess is that those smalls FlowFile are not terminating for x reason and
are causing the content-claim containing them not to clean out of the disk.

The content claim that they belong to could contain bigger files that have
terminated their flow life cycle but still linger on disk until all flow
files belonging to the content claim terminate / drop

On Fri, Sep 1, 2017 at 5:24 PM Alan Jackoway <alanj@cloudera.com> wrote:

> Hello,
>
> We have had issues with a few nifi instances recently where the content
> repository grew too large and we couldn't get nifi to clean it up.
>
> In our current instance of this, we added a second disk to keep nifi from
> consuming 100% of the available space, and now it is close to consuming all
> space on both disks.
>
> Our nifi.properties looks like this:
> # Content Repository
>
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=10 MB
> nifi.content.claim.max.flow.files=100
>
> nifi.content.repository.directory.default=/data/3/nifi_storage/edh-production-ingestion/content_repository
>
> nifi.content.repository.directory.content2=/data/5/nifi_storage/edh-production-ingestion/content_repository
> nifi.content.repository.archive.max.retention.period=1 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=false
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=/nifi-content-viewer/
>
> Each of those content_repository directories are now consuming about 750GB
> on their disks, which are 1TB. The nifi's queue/size is 53,084 / 3.42 MB
> but it does process a fairly large amount of data.
>
> How do we keep nifi from consuming our entire disks with content
> repository? We turned off archive enabled in the hope that it would bring
> the content repository size down to close to 0, but that has not worked for
> us.
>
> Thanks,
> Alan
>
> PS. Sorry for the Friday afternoon email. That's when disks always want to
> get full.
>

Mime
View raw message