nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Jackoway <al...@cloudera.com>
Subject Content Repository Growing Too Large
Date Fri, 01 Sep 2017 21:24:41 GMT
Hello,

We have had issues with a few nifi instances recently where the content
repository grew too large and we couldn't get nifi to clean it up.

In our current instance of this, we added a second disk to keep nifi from
consuming 100% of the available space, and now it is close to consuming all
space on both disks.

Our nifi.properties looks like this:
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/data/3/nifi_storage/edh-production-ingestion/content_repository
nifi.content.repository.directory.content2=/data/5/nifi_storage/edh-production-ingestion/content_repository
nifi.content.repository.archive.max.retention.period=1 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

Each of those content_repository directories are now consuming about 750GB
on their disks, which are 1TB. The nifi's queue/size is 53,084 / 3.42 MB
but it does process a fairly large amount of data.

How do we keep nifi from consuming our entire disks with content
repository? We turned off archive enabled in the hope that it would bring
the content repository size down to close to 0, but that has not worked for
us.

Thanks,
Alan

PS. Sorry for the Friday afternoon email. That's when disks always want to
get full.

Mime
View raw message