nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Villard <pierre.villard...@gmail.com>
Subject Re: NiFi operational interface
Date Mon, 03 Oct 2016 09:52:39 GMT
In the Summary view, you have a tab "Process Group" but given the number of
process groups you have it may not be ideal.
In your situation, I guess, leveraging the REST API is probably the best
way to go.

Pierre

2016-10-03 10:19 GMT+02:00 Roger Marin <
Roger.Marin@colesfinancialservices.com.au>:

> Hi Pierre,
>
> Yes, the search toolbar in the canvas works but it seems to be searching
> across processors whereas what we really want to see is an interface
> centered around process groups.
>
> We encapsulate all of our etl/elt jobs in individual process groups based
> around a specific data source, hence why we really need a "process group"
> focused UI that can allow us to search for and visualize process groups as
> well as to quickly start, stop and schedule a process group, we can get
> away with going to the canvas view to see the details of a specific
> processor.
>
> I'm open to any ideas or comments from anyone doing similar things in
> production, from what we have seen so far (and please correct me if Im
> wrong) when we are talking about having hundreds or even thousands of
> process groups running, the canvas view is perhaps not the best tool for
> someone who needs to monitor and support a production ingestion batch on a
> per source basis.
>
> Regards,
> Roger
>
>
>
>
>
>
>
> -------- Original message --------
> From: Pierre Villard <pierre.villard.fr@gmail.com>
> Date: 3/10/2016 19:06 (GMT+10:00)
> To: users@nifi.apache.org
> Subject: Re: NiFi operational interface
>
> Hi Roger,
>
> I don't know if it answers your needs but you have a search tool bar to
> find any element on the canvas and quickly access it. In terms of
> monitoring, you have the "Summary" panel available from the menu. It gives
> you a way to quickly display information about everything on the canvas.
>
> Pierre
>
>
> 2016-10-03 5:26 GMT+02:00 Roger Marin <Roger.Marin@
> colesfinancialservices.com.au>:
>
>> Hi all,
>>
>>
>>
>> We are looking at using NiFi to replace some of our existing (batch,
>> microbatch) Data Ingestion & ETL/ELT processes but we have a few concerns
>> around the “Canvas” interface.
>>
>>
>>
>> From a developer perspective the Canvas view is OK but we feel that it’s
>> not really suitable from an operational point of view.  For instance, we
>> are looking at eventually having hundreds, perhaps even thousands of
>> Process Groups running in production, we really need a way to easily search
>> and monitor any running process groups at any given time.
>>
>>
>>
>> We were initially thinking of leveraging things like the ELK stack to
>> build some kind of dashboard but that doesn’t give us the flexibility we
>> need, e.g. if we need to stop a given process group or restart it we would
>> still need to use the canvas view for that, given this we are looking at
>> leveraging the NiFi API to build a new “operational” interface that we
>> could use to monitor and perform basic actions on running process groups
>> (start, stop, schedule), as well as quickly search and visualize running
>> processors and process groups.
>>
>>
>>
>> I am not sure if this is something that’s already being worked on (I
>> couldn’t find anything in JIRA) but I would like to know how other NiFi
>> users in production are getting around this (if it’s even an issue for
>> anyone else at all…).
>>
>>
>>
>> Regards,
>>
>> Roger
>>
>>
>>
>>
>>
>
>

Mime
View raw message