nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jennifer Barnabee <jennifer.barna...@gmail.com>
Subject Re: PutJMS performance reading from NIFI Queue
Date Wed, 20 May 2015 21:07:28 GMT
Chris,
FlowFile penalization is discussed in the User Guide section on configuring
processors [1]. Here is the text, which discusses setting the Penalty
Duration on the Settings tab of a given processor:

During the normal course of processing a piece of data (a FlowFile), an
event may occur that indicates that the data cannot be processed at this
time but the data may be processable at a later time. When this occurs, the
Processor may choose to Penalize the FlowFile. This will prevent the
FlowFile from being Processed for some period of time. For example, if the
Processor is to push the data to a remote service, but the remote service
already has a file with the same name, the Processor may penalize the
FlowFile. The ‘Penalty duration’ allows the DFM to specify how long the
FlowFile should be penalized. The default value is 30 seconds.

[1] https://nifi.incubator.apache.org/docs/user-guide.html#settings-tab

-Jenn

On Wed, May 20, 2015 at 4:44 PM, Chris Mangold <csm9105@gmail.com> wrote:

> Just going straight from the GetFile or GenerateFlowFile processor (via a
> NIFI queue) to PutJMS.
>
> I guess I have to read up on FlowFile penalization, I am not familiar with
> the concept.
>
> Chris
>
>
>
> On Wed, May 20, 2015 at 3:52 PM, Mark Payne <markap14@hotmail.com> wrote:
>
>> Ohh, I totally misunderstood the problem. I thought you were saying that
>> you were pushing messages at 30 per second.
>>
>> When you say you have a GetFile or a GenerateFlowFile feeding the PutJMS,
>> are you feeding directly from those processors, or is there something in
>> between?
>>
>> I'm wondering if the FlowFiles are possibly penalized?
>>
>> Thanks
>> -Mark
>>
>> ________________________________
>> > Date: Wed, 20 May 2015 15:44:37 -0400
>> > Subject: Re: PutJMS performance reading from NIFI Queue
>> > From: csm9105@gmail.com
>> > To: users@nifi.incubator.apache.org
>> >
>> > Mark:
>> >
>> > I have been varying the "Message Batch Size" between 10 and 1000
>> > with no appreciable improvement in performance. I notice a larger
>> > batch of messages going into ActiveMQ but the Processor only is getting
>> > triggered every ~ 30 seconds.
>> >
>> > Chris
>> >
>> > On Wed, May 20, 2015 at 2:50 PM, Mark Payne
>> > <markap14@hotmail.com<mailto:markap14@hotmail.com>> wrote:
>> > Chris,
>> >
>> > What value do you have the "Batch Size" property set to? That can make
>> > a really big difference in throughput.
>> >
>> > ________________________________
>> >> Date: Wed, 20 May 2015 13:56:46 -0400
>> >> Subject: PutJMS performance reading from NIFI Queue
>> >> From: csm9105@gmail.com<mailto:csm9105@gmail.com>
>> >> To: users@nifi.incubator.apache.org<mailto:
>> users@nifi.incubator.apache.org>
>> >>
>> >> I am trying to push flowfile messages to ActiveMQ using a PutJMS
>> >> processor. I am generating flow files using either a GetFile or
>> >> GenerateFlowFile processor and feeding them to the PutJMS processor via
>> >> a NIFI queue.
>> >>
>> >> Issue is that the PutJMS processor never keeps up with the queue and
>> >> does not push messages out any quicker then at a rate of 30 seconds. I
>> >> have the PutJMS setup as timer driven with Run Schedule of 0 secs.
>> >>
>> >> Any suggestions.
>> >>
>> >> Thanks,
>> >>
>> >> Chris
>> >>
>> >>
>> >>
>> >> --
>> >> Chris Mangold
>> >>
>> >
>> >
>> >
>> >
>> > --
>> > Chris Mangold
>> > 301-471-5758 (c)
>> > 301-898-7979 (h)
>>
>>
>
>
>
> --
> Chris Mangold
> 301-471-5758 (c)
> 301-898-7979 (h)
>

Mime
View raw message