nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ed B <bdes...@gmail.com>
Subject Re: Getting started
Date Sun, 05 Aug 2018 04:42:23 GMT
Hi Eric,
For monitoring, you have following tools:
1. Monitoring in web UI (
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#monitoring ,
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#routing-and-mediation
)
2. Using reporting tasks (
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks
 )

As for processing the data: NIFI is not ETL tool. it's a data FLOW
management tool. Yes, you have a lot of processors that can extract data
and process, make decisions, fork flows based on different attributes or
content, etc. But usually your NIFI clusters are tiny, and ETL tools a
sitting on much bigger and powerful clusters.
But, you obviously can do ETL too, Writing processors explained number of
time (e.g.
http://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/,
https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html
).

Regards,
Ed.

On Thu, Jul 19, 2018 at 8:59 AM Eric Desjardins <desjare@gmail.com> wrote:

> Hi,
>
> I am evaluating nifi to replace a local ETL system. I went through the
> video and the doc and it looks great.
>
> We have a large number of data feeds that ends up in different folder on a
> sftp server. I could use a SFTP processor to fetch data from the server.
> Than, I could write my own processor to process the data. For this, I would
> need to know information about the feed to fetch a configuration. How would
> I do that?
>
> Than, I am curious about monitoring all those feeds. I need to know what`s
> going on but also if data arrive on schedule.
>
> How would I setup my flow?
>
> Kind Regards,
> Eric
>
>

Mime
View raw message