nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Issue with nifi startup performance
Date Sat, 17 Mar 2018 13:18:16 GMT
Hello,

I would recommend grabbing a thread dump while nifi is starting up, in the part that takes
the longest. Typically, the startup time is related more to restoring the repositories more
than it is to restoring the flow.

Also, are you running on physical hardware or in a VM? How many cpus are available and how
many/what kind of disks (spinning vs ssd vs network attached storage, etc) are you using?

Sent from my iPhone

> On Mar 17, 2018, at 6:57 AM, 李磊 <l225402l@163.com> wrote:
> 
> Hi all,
> 
> I use KYLO with NIFI-1.3.0 now. In KYLO, I create 100 feed with data-ingest template
to transform data form RDBMS to HDFS, which provided by kylo.Finally, more than 1000 processors
were created and there are 29 processors in the reuseable_templates. NIFI uses 6g memory.
> 
> The problem is that when I restart the NIFI, more than 7 mins cost. By modify the source
code and print log, I find that nearly 6 min costs in org.apache.nifi.controller.StandardFlowSynchronizer
Class<sync>, especially method <addProcessGroup>. By read the source code, I think
the function of <addProcessGroup> is analyze flow.xml.gz and restore the NIFI status
before turned off.
> 
> Is there any ways to improve the performance of <addProcessGroup>? We will create
more than 1000 feeds in KYLO in production, means more than 10000 processors will be existed
in NIFI.
> 
> Thanks!
Mime
View raw message