Hi

 

Currently I am using flume for data ingestion and my use case as follows

 

Log repository--------log copy Script-----à Staging directory for  copied logs

 

Staging directory for  copied logs folder structure----Machine1log----a.log

                                                                                                                              -----b.log

                                                                                                    Machine2log----a.log

                                                                                                                              -----b.log

 

Flume will copy these logs and replicate same structure in HDFS cluster. Beginning with which is :

                                                                                                /user/hdfs/Machine1log----a.log

                                                                                                                                          -----b.log

                                                                                                                  Machine2log----a.log

                                                                                                                                            -----b.log

 

 

And creates 0 byte dummy file with same name so that Script wont copy the same log again as it find 0 byte file already existing in source directory.

 

 

Can we do same things with apache nifi?

 

Keeping in mind two goals- same folder structure in HDFS and after moving file to HDFS it should crete 0 byte dummy file in source directory.

 

 

Please help

 

Thanks,

Deepak

 

 

 

 

With Best Regards,

Deepak Tripathi

Philips Innovation campus

Bangalore-560045

images

 



The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.