nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Lifecycle of a Processor instance
Date Tue, 19 Sep 2017 12:53:01 GMT
Peter,

What you outlined here is all correct, with a small exception. If you have multiple versions
of the same processor loaded into your flow (applicable only to 1.3, or perhaps 1.2?), you
will also see the Processor class re-instantiated when you right-click on a Processor and
choose to change the version.

Cheers
-Mark


> On Sep 19, 2017, at 1:27 AM, Peter Wicks (pwicks) <pwicks@micron.com> wrote:
> 
> There had been a lot of internal discussions at work about the actual life cycle of the
class instance instantiated by NiFi; and the variables scoped at the class level. When do
Processors "reset", are new instances created for each run, or are instances recycled? What
about concurrent threads and thread safety?
> 
> I'm sure several developers here on the list could have easily answered these questions
:), but I decided to do some research on my own. I built a test processor that either increments
a local private non-thread safe or thread safe integer, based upon the property choice you
make in the processor.  Just to share and discuss, below are my tests and results. The value
is stored only in the private variable, no state management is used.
> 
> 
> -          Test 1: 1 Concurrent Thread, Non-Thread Safe
> 
> o   The purpose of this test is to find out what happens to a Processors state between
execution of Flow Files
> 
> o   After 10,000 files the value was 10,000 on the last file. This means that state is
maintained in a processor between runs (this was what I assumed, but a good place to start).
Each execution of the processor used the same instance of the class.
> 
> -          Test 2: Stop the processor, then start it again and run a single file
> 
> o   The purpose of this test is to figure out when a processor "resets" it's state.
> 
> o   After 1 file the value was 10,001.
> 
> o   This means that stopping and starting a processor does not reset the processor. The
same class instance is still used.
> 
> -          Test 3: Stop, Disable, Enable, and then start again.
> 
> o   The purpose of this test is to see if disabling a processor causes the class instance
to be disposed of.
> 
> o   After 1 file the value was 10,002.
> 
> o   This means that disabling and re-enabling a processor does not reset its state. The
same class instance persists.
> 
> -          Test 4: Starting with a new copy of the test processor, run 10 concurrent
threads, non-thread safe
> 
> o   The purpose of this test is to see if each thread uses its own instance of the class,
or if a shared instance of the class is used.
> 
> o   After 10,000 files, the value was 9,975 on the last file (I didn't run this test
more than once, but the value should fluctuate from run to run due to thread contention).
> 
> o   I saw at least 8 concurrent threads running at one point. Combined with this, and
the resulting value, I'm fairly confident that the same class instance is used for all concurrent
threads.
> 
> -          Test 5: Starting with a new copy of the test processor, run 1 Concurrent Thread,
Thread-Safe
> 
> o   The purpose of this test is to find out what happens to a Processors state between
execution of Flow Files
> 
> o   After 10,000 files the value was 10,000 on the last file
> 
> o   This means that state is maintained in a processor between runs (this matches the
non-thread safe results, which makes sense for 1 concurrent thread).
> 
> -          Test 6: Starting with a new copy of the test processor, run 10 concurrent
Thread, Thread-Safe
> 
> o   The purpose of this test is to contrast with the non-thread safe approach, and verify
that a thread-safe object will work across concurrent threads.
> 
> o   After 10,000 files the value was 10,000 on the last file
> 
> o   This means that thread synchronization works with multiple concurrent threads for
a single class instance.
> 
> These tests ran on a single NiFi instance, with no clustering and are not designed to
say anything about clustering.
> 
> Based upon my limited test results, a Processor class is never re-instantiated unless
the Processor is deleted from the flow (... yea, kind of like cheating) or NiFi restarts.
There are of course other tests that could be run, welcome any feedback!
> 
> Thanks,
>  Peter
> 


Mime
View raw message