nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mangold <csm9...@gmail.com>
Subject Re: PutJMS performance reading from NIFI Queue
Date Thu, 21 May 2015 02:19:32 GMT
Thanks Mark and Jennifer.  I was able to improve my throughput by adjusting
the Penalty and Yield durations.

Chris

On Wed, May 20, 2015 at 5:07 PM, Jennifer Barnabee <
jennifer.barnabee@gmail.com> wrote:

> Chris,
> FlowFile penalization is discussed in the User Guide section on
> configuring processors [1]. Here is the text, which discusses setting the
> Penalty Duration on the Settings tab of a given processor:
>
> During the normal course of processing a piece of data (a FlowFile), an
> event may occur that indicates that the data cannot be processed at this
> time but the data may be processable at a later time. When this occurs, the
> Processor may choose to Penalize the FlowFile. This will prevent the
> FlowFile from being Processed for some period of time. For example, if the
> Processor is to push the data to a remote service, but the remote service
> already has a file with the same name, the Processor may penalize the
> FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
> FlowFile should be penalized. The default value is 30 seconds.
>
> [1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab
>
> -Jenn
>
> On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <csm9105@gmail.com> wrote:
>
>> Just going straight from the GetFile or GenerateFlowFile processor (via a
>> NIFI queue) to PutJMS.
>>
>> I guess I have to read up on FlowFile penalization, I am not familiar
>> with the concept.
>>
>> Chris
>>
>>
>>
>> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <markap14@hotmail.com> wrote:
>>
>>> Ohh, I totally misunderstood the problem. I thought you were saying that
>>> you were pushing messages at 30 per second.
>>>
>>> When you say you have a GetFile or a GenerateFlowFile feeding the
>>> PutJMS, are you feeding directly from those processors, or is there
>>> something in between?
>>>
>>> I'm wondering if the FlowFiles are possibly penalized?
>>>
>>> Thanks
>>> -Mark
>>>
>>> ________________________________
>>> > Date: Wed, 20 May 2015 15:44:37 -0400
>>> > Subject: Re: PutJMS performance reading from NIFI Queue
>>> > From: csm9105@gmail.com
>>> > To: users@nifi.incubator.apache.org
>>> >
>>> > Mark:
>>> >
>>> > I have been varying the "Message Batch Size" between 10 and 1000
>>> > with no appreciable improvement in performance. I notice a larger
>>> > batch of messages going into ActiveMQ but the Processor only is getting
>>> > triggered every ~ 30 seconds.
>>> >
>>> > Chris
>>> >
>>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>>> > <markap14@hotmail.com<mailto:markap14@hotmail.com>> wrote:
>>> > Chris,
>>> >
>>> > What value do you have the "Batch Size" property set to? That can make
>>> > a really big difference in throughput.
>>> >
>>> > ________________________________
>>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>>> >> Subject: PutJMS performance reading from NIFI Queue
>>> >> From: csm9105@gmail.com<mailto:csm9105@gmail.com>
>>> >> To: users@nifi.incubator.apache.org<mailto:
>>> users@nifi.incubator.apache.org>
>>> >>
>>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>>> >> processor. I am generating flow files using either a GetFile or
>>> >> GenerateFlowFile processor and feeding them to the PutJMS processor
>>> via
>>> >> a NIFI queue.
>>> >>
>>> >> Issue is that the PutJMS processor never keeps up with the queue and
>>> >> does not push messages out any quicker then at a rate of 30 seconds.
I
>>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>> >>
>>> >> Any suggestions.
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Chris
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Chris Mangold
>>> >>
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Chris Mangold
>>> > 301-471-5758 (c)
>>> > 301-898-7979 (h)
>>>
>>>
>>
>>
>>
>> --
>> Chris Mangold
>> 301-471-5758 (c)
>> 301-898-7979 (h)
>>
>
>


-- 
Chris Mangold
301-471-5758 (c)
301-898-7979 (h)

Mime
View raw message