nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "N. J." <nadh...@yahoo.com>
Subject Re: NiFi - create folders and files based on hostname and date
Date Thu, 23 May 2019 01:00:44 GMT
Thanks Koji. That was super quick. Let me try that. Might get back to you in case I face issues.

Sent from my iPhone

> On May 23, 2019, at 9:53 AM, Koji Kawamura <ijokarumawak@gmail.com> wrote:
> 
> Hello,
> 
> In order to create the folders automatically, I would use PutFile like below:
> - Assuming the incoming FlowFile has attribute named 'host' so that
> PutFile's 'Directory' can refer it using NiFi Expression Language (EL)
> - The date part can also be generated using 'now()' EL function
> - Directory: /var/log/network/${host}/${now():format('yyyy')}/${now():format('MM')}/${now():format('dd')}/syslog.log
> - Create Missing Directories: true
> 
> Thanks,
> Koji
> 
>> On Thu, May 23, 2019 at 8:36 AM N. J. <nadhemj@yahoo.com> wrote:
>> 
>> Hi,
>> 
>> 
>> Trying to "practically" replace multiple ingestion tools with NiFi. I have Syslog-NG
as one these tools storing syslog messages to files based on the hostname (syslog.hostname)
extracted from the syslog message and the date when the message was received. The following
shows the Syslog-NG destination configuration:
>> 
>> "
>> 
>> destination d_network {file("/var/log/network/$HOST/$YEAR/$MONTH/$DAY/syslog.log"
owner(root) group(root) perm(0666) dir_perm(0777) create_dirs(yes) ); };
>> 
>> "
>> 
>> Syslog-NG would create folders automatically based on the hostname, year, month and
day, and then store the data to a file called "syslog.log". The file would be appended if
it already exists.
>> 
>> 
>> Couple of challenges I faced with NiFi when trying to do the same:
>> 
>> 1. The regular file append issue (which a patch exists for)
>> 
>> 2. The bigger problem is how to automatically create the folders without the need
to manually do this for each source, noting that I have 100s of sources.
>> 
>> 
>> Suggestions on how to do this on NiFi would be appreciated.
>> 
>> 
>> Thanks,


Mime
View raw message