nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject RE: PutJMS performance reading from NIFI Queue
Date Wed, 20 May 2015 21:28:01 GMT
Chris,

If you're going straight from GetFile or GenerateFlowFile, neither of those is going to penalize
any FlowFiles, so that's not the issue.

I have two follow-up questions:

1. How many tasks is the Processor reporting in a 5-minute window (the bottom stat that is
shown on the Processor itself)?

2. Does there constantly seem to be a thread running or typically there are no threads running
(in the top-right corner of the Processor it shows how many threads are currently running.
If there is no number in the very top-right corner of the Processor, then no threads are running)?
If you keep right-clicking on the graph and then clicking "Refresh" , I'd like to know if
there is a number that stays there, or if it generally has no threads.

Thanks
-Mark

________________________________
> Date: Wed, 20 May 2015 16:44:29 -0400 
> Subject: Re: PutJMS performance reading from NIFI Queue 
> From: csm9105@gmail.com 
> To: users@nifi.incubator.apache.org 
> 
> Just going straight from the GetFile or GenerateFlowFile processor (via 
> a NIFI queue) to PutJMS. 
> 
> I guess I have to read up on FlowFile penalization, I am not familiar 
> with the concept. 
> 
> Chris 
> 
> 
> 
> On Wed, May 20, 2015 at 3:52 PM, Mark Payne 
> <markap14@hotmail.com<mailto:markap14@hotmail.com>> wrote: 
> Ohh, I totally misunderstood the problem. I thought you were saying 
> that you were pushing messages at 30 per second. 
> 
> When you say you have a GetFile or a GenerateFlowFile feeding the 
> PutJMS, are you feeding directly from those processors, or is there 
> something in between? 
> 
> I'm wondering if the FlowFiles are possibly penalized? 
> 
> Thanks 
> -Mark 
> 
> ________________________________ 
>> Date: Wed, 20 May 2015 15:44:37 -0400 
>> Subject: Re: PutJMS performance reading from NIFI Queue 
>> From: csm9105@gmail.com<mailto:csm9105@gmail.com> 
>> To: users@nifi.incubator.apache.org<mailto:users@nifi.incubator.apache.org>

>> 
>> Mark: 
>> 
>> I have been varying the "Message Batch Size" between 10 and 1000 
>> with no appreciable improvement in performance. I notice a larger 
>> batch of messages going into ActiveMQ but the Processor only is getting 
>> triggered every ~ 30 seconds. 
>> 
>> Chris 
>> 
>> On Wed, May 20, 2015 at 2:50 PM, Mark Payne 
>> 
> <markap14@hotmail.com<mailto:markap14@hotmail.com><mailto:markap14@hotmail.com<mailto:markap14@hotmail.com>>>

> wrote: 
>> Chris, 
>> 
>> What value do you have the "Batch Size" property set to? That can make 
>> a really big difference in throughput. 
>> 
>> ________________________________ 
>>> Date: Wed, 20 May 2015 13:56:46 -0400 
>>> Subject: PutJMS performance reading from NIFI Queue 
>>> From: 
> csm9105@gmail.com<mailto:csm9105@gmail.com><mailto:csm9105@gmail.com<mailto:csm9105@gmail.com>>

>>> To: 
> users@nifi.incubator.apache.org<mailto:users@nifi.incubator.apache.org><mailto:users@nifi.incubator.apache.org<mailto:users@nifi.incubator.apache.org>>

>>> 
>>> I am trying to push flowfile messages to ActiveMQ using a PutJMS 
>>> processor. I am generating flow files using either a GetFile or 
>>> GenerateFlowFile processor and feeding them to the PutJMS processor via 
>>> a NIFI queue. 
>>> 
>>> Issue is that the PutJMS processor never keeps up with the queue and 
>>> does not push messages out any quicker then at a rate of 30 seconds. I 
>>> have the PutJMS setup as timer driven with Run Schedule of 0 secs. 
>>> 
>>> Any suggestions. 
>>> 
>>> Thanks, 
>>> 
>>> Chris 
>>> 
>>> 
>>> 
>>> -- 
>>> Chris Mangold 
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> Chris Mangold 
>> 301-471-5758 (c) 
>> 301-898-7979 (h) 
> 
> 
> 
> 
> -- 
> Chris Mangold 
> 301-471-5758 (c) 
> 301-898-7979 (h) 
 		 	   		  
Mime
View raw message