nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Chaffelson <chaffel...@gmail.com>
Subject Re: How to use List File/Fetch File processor in a cluster mode.
Date Fri, 23 Feb 2018 08:54:28 GMT
Hi Mohit,

ListFile expects the directory to be on both nodes because the Primary node
could swap at some point, and so the configuration has to be valid for the
whole cluster.
If you want to have a single directory which ListFile/FetchFile pulls from,
you could try a cluster filesystem or an SFTP service as the source
perhaps?  Using a directory on the local file system is probably not going
to be a robust solution.

On Fri, Feb 23, 2018 at 8:42 AM <mohit.jain@open-insights.co.in> wrote:

>
>
> I tried it the same way, but ListFile expects the directory location to be
> present in both the nodes.
> I found  way to create a List/Fetch processor on standalone nifi and
> connect it to the cluster using Remote Process Group. But I’m looking for
> any other approach which doesnot involve Remote Process Group/ Standalone
> Nifi.
>
>
>
>
>
> *From:* Marco Gaido [mailto:marcogaido91@gmail.com]
> *Sent:* 23 February 2018 14:08
> *To:* users@nifi.apache.org
> *Subject:* Re: How to use List File/Fetch File processor in a cluster
> mode.
>
>
>
> Hi Mohit,
>
>
>
> I'd suggest you to schedule it only on the primary node instead of on all
> nodes and after that repartition the outcoming flow files among all the
> available nodes.
>
>
>
> Best regards,
>
> Marco
>
>
>
> 2018-02-23 6:34 GMT+01:00 <mohit.jain@open-insights.co.in>:
>
> Hi all,
>
>
>
> I created a template to ingest data from any NiFi local directory using
> ListFile/FetchFile processor. When running in a cluster mode it asks for
> the directory in both the nodes. What is the best way to use the
> combination of both?
>
>
>
>
>
> Regards,
>
> Mohit Jain
>
>
>

Mime
View raw message