nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: GetSQS causes high CPU usage
Date Wed, 04 Nov 2015 21:33:57 GMT
Adam,

I wonder if this ticket that I just created [1] is actually the same issue that you're seeing
here.

When GetSQS determines there is nothing to do, it will "yield", essentially pausing itself
for some amount of time
(by default 1 second). But the framework wasn't properly pausing the processor. Instead, it
continually ran a task that
simply asks "are you yielded?" Since the answer was yes, it finished that task and ran it
again. This can cause the
CPU usage to be significantly higher when there's nothing to do than when there is actually
work to do.

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-1111 <https://issues.apache.org/jira/browse/NIFI-1111>


> On Nov 3, 2015, at 1:05 PM, Adam Lamar <adamonduty@gmail.com> wrote:
> 
> Hey Joe,
> 
> I think there are two possible JIRAs.
> 
> 1) Add long polling support using setWaitTimeSeconds() - should be really easy. I can
take a crack at a pull request. Here's a JIRA: https://issues.apache.org/jira/browse/NIFI-1103
> 
> 2) Investigate the high CPU usage. I saw this initially for several days, but it went
away after I adjusted the run schedule (from 1 second to 10 seconds back to 1 second). I have
CPU charts showing the high usage and corresponding drop, but I need to reproduce the issue.
> 
> I'll circle back in a few days when I get some time to work on it.
> 
> Cheers,
> Adam
> 
> On 11/3/15 2:41 AM, Joe Witt wrote:
>> Adam,
>> 
>> Just wanted to follow up on this.  Have you had any better results and
>> should we put a JIRA in behind what you're seeing?
>> 
>> Thanks
>> Joe
>> 
>> On Tue, Oct 20, 2015 at 7:58 PM, Adam Lamar <adamonduty@gmail.com> wrote:
>>> Adam,
>>> 
>>> Thanks for the reply!
>>> 
>>> Amazon supports (and recommends) long polling on SQS queues[1]. The GetSQS
>>> code doesn't attempt long polling at all, but I wasn't sure if this was
>>> intentional or if the option had just never been added. With a 20 second
>>> long poll, the processor would make 3 requests per minute instead of 60,
>>> assuming the queue was empty during that time.
>>> 
>>> Another data point - even during high CPU usage, the GetSQS processor was
>>> only making one request per second to SQS (verified via tcpdump). While not
>>> ideal from a billing perspective, doesn't it seem wrong that 1 request a
>>> second is causing such high CPU?
>>> 
>>> Perhaps to muddy the waters a bit, I played with the run schedule yesterday,
>>> and even now that I've turned it back to 1 second, CPU usage is remaining
>>> low. Before I could start/stop GetSQS repeatedly and observe the high CPU
>>> usage, but now I can't reproduce it. If I'm able to consistently reproduce
>>> the issue in the future, I'll be sure to post again.
>>> 
>>> Cheers,
>>> Adam
>>> 
>>> 
>>> [1]
>>> http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
>>> 
>>> 
>>> On 10/20/15 4:37 AM, Adam Estrada wrote:
>>>> Adam,
>>>> 
>>>> I suspect that getSQS is polling Amazon to check for data. It's not
>>>> exactly like your standard message broker in that you have to force the
>>>> poll. Anyway, throw a wait time in there and see if that fixes it. This will
>>>> also help lower your monthly Amazon bill...
>>>> 
>>>> Adam
>>>> 
>>>> 
>>>>> On Oct 19, 2015, at 11:41 PM, Adam Lamar <adamonduty@gmail.com>
wrote:
>>>>> 
>>>>> Hi everybody!
>>>>> 
>>>>> I've been testing NiFi 0.3.0 with the GetSQS processor to fetch objects
>>>>> from an AWS bucket as they're created. My flow looks like this:
>>>>> 
>>>>> GetSQS
>>>>> SplitJson
>>>>> ExtractText
>>>>> FetchS3Object
>>>>> PutFile
>>>>> 
>>>>> I noticed that GetSQS causes a high amount of CPU usage - about 90% of
>>>>> one core. If I turn off GetSQS, CPU usage immediately drops to 2%. If
I turn
>>>>> GetSQS back on with the run schedule at 10, it stays at 2%.
>>>>> 
>>>>> Would it be worth using setWaitTimeSeconds [1] to make the SQS receive
a
>>>>> blocking call? Alternatively, should GetSQS default to a longer run
>>>>> schedule?
>>>>> 
>>>>> Cheers,
>>>>> Adam
>>>>> 
>>>>> 
>>>>> [1]
>>>>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/ReceiveMessageRequest.html#setWaitTimeSeconds(java.lang.Integer)
>>> 
> 


Mime
View raw message