nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: NiFi data HA in cluster mode
Date Tue, 09 Jan 2018 06:37:58 GMT
I had someone from Hortonworks suggest to me that I should also set any PutSQL processors to
only execute on primary. The reasoning was due to flooding of the JDBC pool.

> On 9 Jan 2018, at 17:25, Joe Witt <joe.witt@gmail.com> wrote:
> 
> I'd avoid setting any processor to primary node only unless it is a
> source processor (something that brings data into the system).
> 
> But, yes, I believe your description is accurate as of now.
> 
> Thanks
> 
>> On Mon, Jan 8, 2018 at 11:21 PM, 尹文才 <batman713@gmail.com> wrote:
>> Thanks Joe, so you mean for example, if I set one processor to run only on
>> primary node in the cluster and there're 100 FlowFiles in the incoming
>> queue of the processor
>> waiting to be processed by this processor, and the processor suddenly goes
>> down and then another node is elected as the primary node, those 100
>> FlowFiles will be kept locally
>> in the node that went down and will continue to be processed by the node
>> when it goes back online, these FlowFiles will not be available to the new
>> primary node and other nodes,
>> am I correct?
>> 
>> Regards,
>> Ben
>> 
>> 
>> 2018-01-09 14:08 GMT+08:00 Joe Witt <joe.witt@gmail.com>:
>> 
>>> Ben,
>>> 
>>> Data already mid-flow within a node will be kept on the node and
>>> processed when the node is back on-line.  All other data coming into
>>> the cluster can fail-over to other nodes provided you're sourcing data
>>> with queuing semantics or automated load balancing or fail-over as-is
>>> present in the Apache NiFi Site to Site protocol.
>>> 
>>> Thanks
>>> Joe
>>> 
>>>> On Mon, Jan 8, 2018 at 11:05 PM, 尹文才 <batman713@gmail.com> wrote:
>>>> Hi guys, I have a question about data HA when NiFi is run in clustered
>>>> mode, if one node goes down, will the flowfiles owned by this node taken
>>>> over and processed by another node?
>>>> Or will the flowfiles be kept locally to that node and will only be
>>>> processed when that node is back online? Thanks.
>>>> 
>>>> Regards,
>>>> Ben
>>> 

Mime
View raw message