nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Marin <Roger.Ma...@colesfinancialservices.com.au>
Subject Re: NiFi operational interface
Date Mon, 03 Oct 2016 08:19:18 GMT
Hi Pierre,

Yes, the search toolbar in the canvas works but it seems to be searching across processors
whereas what we really want to see is an interface centered around process groups.

We encapsulate all of our etl/elt jobs in individual process groups based around a specific
data source, hence why we really need a "process group" focused UI that can allow us to search
for and visualize process groups as well as to quickly start, stop and schedule a process
group, we can get away with going to the canvas view to see the details of a specific processor.

I'm open to any ideas or comments from anyone doing similar things in production, from what
we have seen so far (and please correct me if Im wrong) when we are talking about having hundreds
or even thousands of process groups running, the canvas view is perhaps not the best tool
for someone who needs to monitor and support a production ingestion batch on a per source
basis.

Regards,
Roger







-------- Original message --------
From: Pierre Villard <pierre.villard.fr@gmail.com>
Date: 3/10/2016 19:06 (GMT+10:00)
To: users@nifi.apache.org
Subject: Re: NiFi operational interface

Hi Roger,

I don't know if it answers your needs but you have a search tool bar to find any element on
the canvas and quickly access it. In terms of monitoring, you have the "Summary" panel available
from the menu. It gives you a way to quickly display information about everything on the canvas.

Pierre


2016-10-03 5:26 GMT+02:00 Roger Marin <Roger.Marin@colesfinancialservices.com.au<mailto:Roger.Marin@colesfinancialservices.com.au>>:
Hi all,

We are looking at using NiFi to replace some of our existing (batch, microbatch) Data Ingestion
& ETL/ELT processes but we have a few concerns around the "Canvas" interface.

>From a developer perspective the Canvas view is OK but we feel that it's not really suitable
from an operational point of view.  For instance, we are looking at eventually having hundreds,
perhaps even thousands of Process Groups running in production, we really need a way to easily
search and monitor any running process groups at any given time.

We were initially thinking of leveraging things like the ELK stack to build some kind of dashboard
but that doesn't give us the flexibility we need, e.g. if we need to stop a given process
group or restart it we would still need to use the canvas view for that, given this we are
looking at leveraging the NiFi API to build a new "operational" interface that we could use
to monitor and perform basic actions on running process groups (start, stop, schedule), as
well as quickly search and visualize running processors and process groups.

I am not sure if this is something that's already being worked on (I couldn't find anything
in JIRA) but I would like to know how other NiFi users in production are getting around this
(if it's even an issue for anyone else at all...).

Regards,
Roger




Mime
View raw message