nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Chaffelson <chaffel...@gmail.com>
Subject Re: [EXT] Re: Get all Processors
Date Wed, 15 Aug 2018 06:40:04 GMT
Hi Karthik,
I have already implemented this in NiPyApi, assuming a Python automation
client is useful to you.

In the nipyapi.canvas.recurse_flow command (
https://github.com/Chaffelson/nipyapi/blob/28d7f74478e5e71253ce2de53fd22f56f455c338/nipyapi/canvas.py#L36
)
is the base functionality to step through the tree of ProcessGroups and
retrieve the attributes from each. This is the one-call-per-PG method
described above.

This is then leveraged in the list_all_process_groups and
list_all_processors commands, which respectively produce a flat list of all
process groups or processors anywhere on the canvas (
https://github.com/Chaffelson/nipyapi/blob/28d7f74478e5e71253ce2de53fd22f56f455c338/nipyapi/canvas.py#L144
 )
These are particularly useful for running Python list comprehensions
against NiFi entities, such as single line script commands to find all
instances of a certain processor, or purge the entire canvas in a test
environment.

If anyone can think of a different way to implement this that would be
useful, particularly in these larger deployments, I'm happy to take a run
at it.

Thanks,
Dan

On Tue, Aug 14, 2018 at 4:38 PM Karthik Kothareddy (karthikk) [CONT - Type
2] <karthikk@micron.com> wrote:

> Bryan,
>
> Thanks, I was thinking the same, may be get all the root level processors
> and filter them instead of going through each processor group(one call to
> each PG will be expensive).
>
> -Karthik
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Tuesday, August 14, 2018 9:17 AM
> To: users@nifi.apache.org
> Subject: Re: [EXT] Re: Get all Processors
>
> You could probably achieve the same thing but traversing the process
> groups and asking each one for its processors without the
> includeDescendantGroups=true.
>
> It would be more complex on the client side, but would avoid making one
> huge request.
>
> On Tue, Aug 14, 2018 at 11:07 AM, Karthik Kothareddy (karthikk) [CONT
> - Type 2] <karthikk@micron.com> wrote:
> > Hi Pierre,
> >
> >
> >
> > I tried this both on Standalone instance (1.7.1) and clustered
> > instance
> > (1.6.0) where the nifi.cluster.node.read.timeout is set to 60 secs.
> >
> >
> >
> > -Karthik
> >
> >
> >
> >
> >
> > From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
> > Sent: Tuesday, August 14, 2018 2:20 AM
> > To: users@nifi.apache.org
> > Subject: Re: [EXT] Re: Get all Processors
> >
> >
> >
> > Hi Karthik,
> >
> >
> >
> > Are you running a cluster or standalone NiFi? What's your default read
> > timeout value in nifi.properties? I believe the default value is a bit
> > low when you start reaching thousands of processors.
> >
> >
> >
> > nifi.cluster.node.read.timeout=5 sec
> >
> >
> >
> > Pierre
> >
> >
> >
> >
> >
> > 2018-08-14 0:12 GMT+02:00 Karthik Kothareddy (karthikk) [CONT - Type
> > 2]
> > <karthikk@micron.com>:
> >
> > Joe,
> >
> > I tried this call on both 1.7.1 and 1.6.0 and still getting the
> > timeout exception. I know that this is a very expensive call and
> > requires lot of caching from serverside. I was looking for a way to
> > get all processors and the controller Services they refer (if any?).
> > Not sure how to get the information I need in one call.
> >
> > -Karthik
> >
> >
> > -----Original Message-----
> > From: Joe Witt [mailto:joe.witt@gmail.com]
> > Sent: Monday, August 13, 2018 2:07 PM
> > To: users@nifi.apache.org
> > Subject: [EXT] Re: Get all Processors
> >
> > Karthik
> >
> > I believe that call is/was very expensive on the server side.  You
> > might want to experiment with the latest release of NiFi against the
> > same flow configuration.  From conversations I have had I feel like
> > this is an addressed issue though admittedly i'm not sure which JIRA
> > would address it if that is the case.
> >
> > Others might have better data offhand.
> >
> > Thanks
> > On Mon, Aug 13, 2018 at 3:08 PM Karthik Kothareddy (karthikk) [CONT -
> > Type 2] <karthikk@micron.com> wrote:
> >>
> >> All,
> >>
> >>
> >>
> >> I was trying to get all processors from “root” Process group with the
> >> following rest call -
> >> /nifi-api/process-groups/root/processors?includeDescendantGroups=true
> >> and this call keeps timing out with the below exception
> >>
> >>
> >>
> >>  javax.ws.rs.ProcessingException: java.net.SocketTimeoutException:
> >> Read timed out
> >>
> >>
> >>
> >> We have around 2000 processors on that instance and if I change the
> >> process group from root to a lower level group with less number of
> >> processors, the call will return the ProcessorsEntity json. Any idea
> >> on why this is timing out where as more bulkier rest calls such as
> >> /flow/process-groups/root/status?recursive=true Will return results
> >> immediately ?
> >>
> >>
> >>
> >>
> >>
> >> -Karthik
> >
> >
>

Mime
View raw message