nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Percivall <jperciv...@apache.org>
Subject Re: Memory Leak in NiFi
Date Mon, 28 Jan 2019 03:50:13 GMT
Hello Asanka,

Apologies for the late reply, but this is due to a known issue[1] with the
InvokeHttp processor where it did not close the response in all cases. This
affected versions 1.4 through 1.6 and was fixed in 1.7.

[1] https://issues.apache.org/jira/browse/NIFI-5085

Joe

On Thu, Dec 20, 2018 at 3:08 AM Asanka Sanjaya <angal.sh@gmail.com> wrote:

> Hi All,
>
> NiFi version: 1.4.0, The same problem in v1.6.0 as well.
>
> I have a nifi flow with multiple process groups and I have deployed nifi
>  in kubernetes using a docker container.  I could see a very strange
> behavior of memory consumption in nifi docker container.  The only process
> which runs in the docker container is nifi. See the attached screenshot of
> the memory usage graph.
>
> [image: image.png]
>
>
> I tried stopping and starting the process groups to replicate the issue.
> And I found out that the issue could be replicated by enabling a process
> group which as a lot of ReplaceText processors.  When I enable the process
> group and parse several flow files through it, I can see a sudden spike in
> the graph (more than 1 GB sometimes). See the screenshot below.
>
> [image: image.png]
>
>
> I reduced the Maximum buffer size of each processor to 100KB from 1MB
> while the evaluation mode is set to Entire text mode. Then I ran the flow
> several times and I couldn't reproduce the problem. So I assumed that the
> problem was resolved.
>
> Today I noticed the strange behavior again. See this graph.
> [image: image.png]
>
>  But this hike was not due to this process group for sure, because no flow
> file was going through this group at the time (This is a daily running job).
>
> When I looked at the nifi log I could see this error message several
> times. This error is the only error I can find in the log and this error
> has logged exactly when the memory hike has happened.
>
> 018-12-20 06:08:20,939 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.standard.InvokeHTTP
> InvokeHTTP[id=11791003-1f8c-1365-6043-eb6322e072f1] Routing to Failure due
> to exception: java.net.SocketTimeoutException: connect timed out:
> java.net.SocketTimeoutException: connect timed out
> java.net.SocketTimeoutException: connect timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
> at
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:220)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:146)
> at
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
> at
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
> at
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179)
> at okhttp3.RealCall.execute(RealCall.java:63)
> at
> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:692)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Above error message is due to a different processor which tries to connect
> to a different server. At the time of the incident, the external server was
> down so that the InvokeHTTP processor has done several retries and because
> of that a sudden spike in memory. The memory is never getting released.
>
>
> Can somebody assist with me regarding this problem, please?
>
>
>
> --
>
> *Thanks,*
>
> Asanka Sanjaya Herath
>
> Senior Software Engineer | Zone24x7
>


-- 
*Joe Percivall*
linkedin.com/in/Percivall
e: jpercivall@apache.com

Mime
View raw message