nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Skora <>
Subject Re: [jira] [Commented] (NIFI-994) Processor to tail files
Date Wed, 30 Sep 2015 14:10:03 GMT
Maybe this processor could copy the behavior of the GNU Coreutils "tail"
command, aka Linux *tail*?  (source is here

If it can, it appears to use the Linux iNotify service to track file
changes.  If the files are not local or it encounters iNotify problems, it
falls back to a manual tracking process.  I have never had problems with
that utility, and it can track files across logging restarts, rotation, etc.

On Thu, Sep 24, 2015 at 7:50 PM, Aldrin Piri (JIRA) <> wrote:

>     [
> ]
> Aldrin Piri commented on NIFI-994:
> ----------------------------------
> I think we can make a best effort at this, but I don't think there are any
> guarantees that we have all the data.  A lot of this comes down to logging
> provider configuration.  As an example, perhaps there are constraints on
> size or time that cause records to be rotated off.  It could take long
> outages for these environments to develop, but when the data has been
> rolled off, it is gone.  Depending on how markers and such work, this
> brings up some interesting cases to consider when implementing.
> As another point of consideration, it would be nice to have a
> property/properties that provide handling for rolling log formats.
> Consider logback and log4j with their date formatted log names.
> > Processor to tail files
> > -----------------------
> >
> >                 Key: NIFI-994
> >                 URL:
> >             Project: Apache NiFi
> >          Issue Type: New Feature
> >    Affects Versions: 0.4.0
> >            Reporter: Joseph Percivall
> >
> > It's a very common data ingest situation to want to input text into the
> system by "tailing" a file, most commonly log files. Currently we don't
> have an easy way to do this.
> > A simple processor to tail a file would benefit many users. There would
> need to be an option to not just tail a file but pick up where the
> processor left off if it is interrupted.
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message