nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: archive files
Date Mon, 23 Nov 2015 16:08:05 GMT
Charlie,

Can give some pointers on how to get in the ballpark with this but
want to make sure we have a good alignment of purpose here.  NiFi has
from time to time come up as an intuitive way to build an archive
management tool and it is always "not quite right" because of the
subtle differences between continuous streams of information and
ad-hoc sort of one-time tasks.

Would this be a continuous task (always running) even if it is slow
(every few minutes, hours, days) or would it be a one-time thing to
move a bunch of data from one place to another?

The difference sounds very minor but it will help me to understand how
best to respond.

Thanks
Joe

On Mon, Nov 23, 2015 at 10:54 AM, Charlie Frasure
<charliefrasure@gmail.com> wrote:
> Use case: Archive and compress files by category and month, store like files
> in a common directory.
>
> I'm already processing the files, and have extracted the interesting
> attributes from each.  I ran them through MergeContent, but have not been
> able to produce a logical directory structure to store the results.  I would
> prefer something like archive/categoryA/201511/somefilename.tar.gz where
> somefilename is made up of all the categoryA files received in November
> 2015.
>
> I switched gears, and used PutFile to store the files in the preferred
> directory structure, but at a loss of how to archive them within their
> folders given hundreds of dynamic categories, and date additions every
> month.
>
> I'm playing with MergeContent's Correlation Attribute Name, but am also
> considering trying the "Degfragment" merge strategy by correlating the files
> earlier in the process.
>
> Any suggestions would be appreciated.

Mime
View raw message