nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "N. J." <nadh...@yahoo.com>
Subject NiFi - create folders and files based on hostname and date
Date Wed, 22 May 2019 23:36:40 GMT

Hi,




Trying to "practically" replace multiple ingestion tools with NiFi. I have Syslog-NG as one
these tools storing syslog messages to files based on the hostname (syslog.hostname) extracted
from the syslog message and the date when the message was received. The following shows the
Syslog-NG destination configuration:

"

destination d_network {file("/var/log/network/$HOST/$YEAR/$MONTH/$DAY/syslog.log" owner(root)
group(root) perm(0666) dir_perm(0777) create_dirs(yes) ); };

"

Syslog-NG would create folders automatically based on the hostname, year, month and day, and
then store the data to a file called "syslog.log". The file would be appended if it already
exists.




Couple of challenges I faced with NiFi when trying to do the same:

1. The regular file append issue (which a patch exists for)

2. The bigger problem is how to automatically create the folders without the need to manually
do this for each source, noting that I have 100s of sources.




Suggestions on how to do this on NiFi would be appreciated.

Thanks,
Mime
View raw message