nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Williams <colin.williams.seat...@gmail.com>
Subject Re: Tasks / Time extremely high measure of tasks for LogAttribute
Date Sat, 10 Nov 2018 07:40:27 GMT
That sounds reasonable. Then it sounds like I should look out for those
high task counts when logging or otherwise.

On Fri, Nov 9, 2018, 9:10 AM Matt Burgess <mattyb149@gmail.com wrote:

> Building off Pierre’s idea, you could set the Penalty Duration to 0 so the
> the flow files  go immediately to LogMessage, then send along to
> ExecuteScript to “re-penalize” the file
>
> Regards,
> Matt
>
> On Nov 9, 2018, at 11:39 AM, Colin Williams <
> colin.williams.seattle@gmail.com> wrote:
>
> Hi Pierre,
>
> Thanks for explaining. Could this cause a performance penalty? I assume
> there's no way the next processor can be aware of the penalized flows
> state, or this would not be an issue.
>
> On Fri, Nov 9, 2018, 1:36 AM Pierre Villard <pierre.villard.fr@gmail.com
> wrote:
>
>> Hi Colin,
>>
>> That's because when the flow file is routed to failure by the PutS3, the
>> flow file is penalized and cannot be processed before a given penalty
>> duration by the LogMessage processor. However the LogMessage processor sees
>> there is a flow file to process, and then detect that it cannot be
>> processed because of the penalty duration. There is a JIRA for that [1]. An
>> option could be to set the penalty duration to 0s in the PutS3 processor
>> but I don't think that's a great idea as the processor would keep trying
>> sending the flow file and it could lead to unwanted pressure on S3.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3229
>>
>> Pierre
>>
>>
>> Le ven. 9 nov. 2018 à 00:25, Colin Williams <
>> colin.williams.seattle@gmail.com> a écrit :
>>
>>>
>>>
>>> On Thu, Nov 8, 2018 at 3:24 PM Colin Williams <
>>> colin.williams.seattle@gmail.com> wrote:
>>>
>>>> [image: airflow_tasks_time.png]
>>>>
>>>>
>>>> On Thu, Nov 8, 2018 at 3:23 PM Colin Williams <
>>>> colin.williams.seattle@gmail.com> wrote:
>>>>
>>>>> I have a LogAttribute processor connected to the failure of an
>>>>> S3PutObject processor. Then I noticed when the Put fails the
>>>>> LogAttribute Tasks/Time shot to 42,045,444 / 00:01:31.664 for In 14
>>>>> (109KB) and Out 14 (109KB).
>>>>>
>>>>> I'm not sure about HTML email or attachments then will attach
>>>>> separately. Just curious what this high task value means.
>>>>>
>>>>

Mime
View raw message