nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: NiFi - create folders and files based on hostname and date
Date Thu, 23 May 2019 00:53:09 GMT
Hello,

In order to create the folders automatically, I would use PutFile like below:
- Assuming the incoming FlowFile has attribute named 'host' so that
PutFile's 'Directory' can refer it using NiFi Expression Language (EL)
- The date part can also be generated using 'now()' EL function
- Directory: /var/log/network/${host}/${now():format('yyyy')}/${now():format('MM')}/${now():format('dd')}/syslog.log
- Create Missing Directories: true

Thanks,
Koji

On Thu, May 23, 2019 at 8:36 AM N. J. <nadhemj@yahoo.com> wrote:
>
> Hi,
>
>
> Trying to "practically" replace multiple ingestion tools with NiFi. I have Syslog-NG
as one these tools storing syslog messages to files based on the hostname (syslog.hostname)
extracted from the syslog message and the date when the message was received. The following
shows the Syslog-NG destination configuration:
>
> "
>
> destination d_network {file("/var/log/network/$HOST/$YEAR/$MONTH/$DAY/syslog.log" owner(root)
group(root) perm(0666) dir_perm(0777) create_dirs(yes) ); };
>
> "
>
> Syslog-NG would create folders automatically based on the hostname, year, month and day,
and then store the data to a file called "syslog.log". The file would be appended if it already
exists.
>
>
> Couple of challenges I faced with NiFi when trying to do the same:
>
> 1. The regular file append issue (which a patch exists for)
>
> 2. The bigger problem is how to automatically create the folders without the need to
manually do this for each source, noting that I have 100s of sources.
>
>
> Suggestions on how to do this on NiFi would be appreciated.
>
>
> Thanks,

Mime
View raw message