nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: PutJMS performance reading from NIFI Queue
Date Thu, 21 May 2015 02:26:17 GMT
Mark,

Happy to hear you are seeing performance more like you would
anticipate.  I do wonder why the processors are having to yield and
penalize data so much.  That suggests errors are occurring.  Do you
see bulletins about that?

Thanks
Joe

On Wed, May 20, 2015 at 10:19 PM, Chris Mangold <csm9105@gmail.com> wrote:
> Thanks Mark and Jennifer.  I was able to improve my throughput by adjusting
> the Penalty and Yield durations.
>
> Chris
>
> On Wed, May 20, 2015 at 5:07 PM, Jennifer Barnabee
> <jennifer.barnabee@gmail.com> wrote:
>>
>> Chris,
>> FlowFile penalization is discussed in the User Guide section on
>> configuring processors [1]. Here is the text, which discusses setting the
>> Penalty Duration on the Settings tab of a given processor:
>>
>> During the normal course of processing a piece of data (a FlowFile), an
>> event may occur that indicates that the data cannot be processed at this
>> time but the data may be processable at a later time. When this occurs, the
>> Processor may choose to Penalize the FlowFile. This will prevent the
>> FlowFile from being Processed for some period of time. For example, if the
>> Processor is to push the data to a remote service, but the remote service
>> already has a file with the same name, the Processor may penalize the
>> FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
>> FlowFile should be penalized. The default value is 30 seconds.
>>
>> [1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab
>>
>> -Jenn
>>
>> On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <csm9105@gmail.com> wrote:
>>>
>>> Just going straight from the GetFile or GenerateFlowFile processor (via a
>>> NIFI queue) to PutJMS.
>>>
>>> I guess I have to read up on FlowFile penalization, I am not familiar
>>> with the concept.
>>>
>>> Chris
>>>
>>>
>>>
>>> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <markap14@hotmail.com> wrote:
>>>>
>>>> Ohh, I totally misunderstood the problem. I thought you were saying that
>>>> you were pushing messages at 30 per second.
>>>>
>>>> When you say you have a GetFile or a GenerateFlowFile feeding the
>>>> PutJMS, are you feeding directly from those processors, or is there
>>>> something in between?
>>>>
>>>> I'm wondering if the FlowFiles are possibly penalized?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>> ________________________________
>>>> > Date: Wed, 20 May 2015 15:44:37 -0400
>>>> > Subject: Re: PutJMS performance reading from NIFI Queue
>>>> > From: csm9105@gmail.com
>>>> > To: users@nifi.incubator.apache.org
>>>> >
>>>> > Mark:
>>>> >
>>>> > I have been varying the "Message Batch Size" between 10 and 1000
>>>> > with no appreciable improvement in performance. I notice a larger
>>>> > batch of messages going into ActiveMQ but the Processor only is
>>>> > getting
>>>> > triggered every ~ 30 seconds.
>>>> >
>>>> > Chris
>>>> >
>>>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>>>> > <markap14@hotmail.com<mailto:markap14@hotmail.com>> wrote:
>>>> > Chris,
>>>> >
>>>> > What value do you have the "Batch Size" property set to? That can make
>>>> > a really big difference in throughput.
>>>> >
>>>> > ________________________________
>>>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>>>> >> Subject: PutJMS performance reading from NIFI Queue
>>>> >> From: csm9105@gmail.com<mailto:csm9105@gmail.com>
>>>> >> To:
>>>> >> users@nifi.incubator.apache.org<mailto:users@nifi.incubator.apache.org>
>>>> >>
>>>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>>>> >> processor. I am generating flow files using either a GetFile or
>>>> >> GenerateFlowFile processor and feeding them to the PutJMS processor
>>>> >> via
>>>> >> a NIFI queue.
>>>> >>
>>>> >> Issue is that the PutJMS processor never keeps up with the queue
and
>>>> >> does not push messages out any quicker then at a rate of 30 seconds.
>>>> >> I
>>>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>>>> >>
>>>> >> Any suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> Chris
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Chris Mangold
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Chris Mangold
>>>> > 301-471-5758 (c)
>>>> > 301-898-7979 (h)
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Chris Mangold
>>> 301-471-5758 (c)
>>> 301-898-7979 (h)
>>
>>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)

Mime
View raw message