nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Finding Performance bottleneck issue
Date Fri, 16 Mar 2018 14:46:01 GMT
Milan,

We dont offer any load testing specific tooling.  You can though write
integration tests which setup a more complete system.  The most common
model i see people do though for true performance/bottleneck analysis
though is setup a running nifi instance with a flow through your
custom processor and attach a profiler at runtime.

This is easy to do by editing conf/bootstrap.conf

#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

change that line to

#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

Then with your running nifi you can attach something like JVisualVM
from the JDK which is a really great tool.  You could look at more
powerful options like Yourkit as well.

These let you really learn where the bottlenecks are.

You might also want to look at the existing library of 200+ processors
code and see if any have similar characteristics to yours.

Design things:
1) Does how your processor works lend to idempotent (safe to do over
and over) operation?  If yes you can enable session batching.  This
means instead of one flowfile at a time we can do hundreds or
thousands and merge them into a single flowfile repo transaction
2) Are you using a lot of heap/memory?  This can be if you're reading
content fully into memory or worse doing array copies, etc..

If you need more help and pointers please share more details on what
the processor is doing/the design of it as there are many folks on
this list which can give general pointers.

Thanks

On Fri, Mar 16, 2018 at 2:39 PM, Milan Das <mdas@interset.com> wrote:
> I have a custom processor, it works as expected. But I feel there is some performance
measure need to be done. I see that my processor is actually queuing up  records at source.
>
> Is there a run a load  test and do performance measure for Custom Processor?
>
>
>
> Regards,
>
> Milan Das
>

Mime
View raw message