nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Viewing flow file contents results in error
Date Mon, 23 Nov 2015 14:48:00 GMT
Chris,

We describe the upgrade considerations here [1].  We don't have an
upgrade guide yet but will soon [2].

Yes SplitText exposes a defect which will use too much memory (for no
good reason).  There is a pretty straightforward workaround for now
and we'll fix it in an upcoming release [3].

The workaround is to do the splitting in two stages with back-pressure
applied.  So, chain together two SplitText processors where the first
one does 5K lines at a time and the second one does single lines.
Then between the two set backpressure to say 10 or 100 or whatever
makes sense for you.  This way regardless of the size of an input file
in terms of lines NiFi will only ever have N/5000 flow files in the
first processor and 5000 flow files in the second.

[1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
[2] https://issues.apache.org/jira/browse/NIFI-590
[3] https://issues.apache.org/jira/browse/NIFI-1008

Thanks
Joe

On Mon, Nov 23, 2015 at 9:28 AM, Chris Teoh <chris.teoh@gmail.com> wrote:
> Yea this is NiFi 0.21. Is there an upgrade path? I have had it run out of
> memory before because I was using SplitText on an unrelated log file
> containing millions of lines. I guess that might be a limitation I'll have
> to find a workaround for. I've managed to recover it but I only noticed this
> trying to troubleshoot why the system stopped running the jobs properly.
>
> On Tue, 24 Nov 2015 at 1:06 AM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>> Chris,
>>
>> Questions:
>> Is this is NiFi 0.2.1?
>> Does it always fail when trying to view content?
>>
>> I just built 021 and verified it worked as expected.  In addition to
>> the classloading issue i notice the JSP null found issues at the
>> bottom of what you pasted.  I wonder if the work directory was
>> incomplete on load or if space issues on that partition have cropped
>> up while running (large guesses admittedly)
>>
>> Thanks
>> Joe
>>
>> On Mon, Nov 23, 2015 at 8:43 AM, Chris Teoh <chris.teoh@gmail.com> wrote:
>> > Hi folks,
>> >
>> > I tried to view the contents of flowfile from the provenance window and
>> > I
>> > get a 500 error.
>> >
>> > From Web page:-
>> > HTTP ERROR 500
>> >
>> > Problem accessing /nifi-content-viewer/. Reason:
>> >
>> >     Server Error
>> >
>> > Caused by:
>> >
>> > java.lang.NoClassDefFoundError:
>> > org/apache/nifi/web/ContentViewerController$2
>> >  at
>> >
>> > org.apache.nifi.web.ContentViewerController.getContentRequest(ContentViewerController.java:277)
>> >  at
>> >
>> > org.apache.nifi.web.ContentViewerController.doGet(ContentViewerController.java:74)
>> >  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>> >  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> >  at
>> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>> >  at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
>> >  at
>> >
>> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
>> >  at
>> >
>> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
>> >  at
>> >
>> > org.apache.nifi.web.security.x509.X509AuthenticationFilter.doFilter(X509AuthenticationFilter.java:111)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.apache.nifi.web.security.authorization.NodeAuthorizedUserFilter.doFilter(NodeAuthorizedUserFilter.java:121)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
>> >  at
>> >
>> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>> >  at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>> >  at
>> >
>> > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
>> >  at
>> >
>> > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
>> >  at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> >  at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> >  at
>> >
>> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> >  at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> >  at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> >  at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> >  at
>> >
>> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> >  at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> >  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> >  at
>> >
>> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> >  at
>> >
>> > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> >  at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> >  at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> >  at java.lang.Thread.run(Thread.java:745)
>> >
>> > Powered by Jetty://
>> >
>> >
>> > From nifi-app.log:
>> > 2015-11-23 00:40:18,756 WARN [NiFi Web Server-311490]
>> > org.eclipse.jetty.servlet.ServletHandler Error for /nifi-content-viewer/
>> > java.lang.NoClassDefFoundError:
>> > org/apache/nifi/web/ContentViewerController$2
>> >         at
>> >
>> > org.apache.nifi.web.ContentViewerController.getContentRequest(ContentViewerController.java:277)
>> > ~[classes/:0.2.1]
>> >         at
>> >
>> > org.apache.nifi.web.ContentViewerController.doGet(ContentViewerController.java:74)
>> > ~[classes/:0.2.1]
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>> > ~[javax.servlet-api-3.1.0.jar:3.1.0]
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> > ~[javax.servlet-api-3.1.0.jar:3.1.0]
>> >         at
>> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.apache.nifi.web.security.x509.X509AuthenticationFilter.doFilter(X509AuthenticationFilter.java:111)
>> > ~[nifi-web-security-0.2.1.jar:0.2.1]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.apache.nifi.web.security.authorization.NodeAuthorizedUserFilter.doFilter(NodeAuthorizedUserFilter.java:121)
>> > ~[nifi-web-security-0.2.1.jar:0.2.1]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >        at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE]
>> >         at
>> >
>> > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
>> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
>> >         at
>> >
>> > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
>> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE]
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> > [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> > [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> > [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> > [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> > [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> > [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.nifi.web.ContentViewerController$2
>> >         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> > ~[na:1.7.0_67]
>> >         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> > ~[na:1.7.0_67]
>> >         at java.security.AccessController.doPrivileged(Native Method)
>> > ~[na:1.7.0_67]
>> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> > ~[na:1.7.0_67]
>> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> > ~[na:1.7.0_67]
>> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> > ~[na:1.7.0_67]
>> >         at
>> >
>> > org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:450)
>> > ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         at
>> >
>> > org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:403)
>> > ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
>> >         ... 49 common frames omitted
>> > 2015-11-23 00:40:18,945 ERROR [NiFi Web Server-311493]
>> > org.apache.jasper.servlet.JspServlet PWC6117: File "null" not found
>> > 2015-11-23 00:40:18,946 ERROR [NiFi Web Server-311493]
>> > org.apache.jasper.servlet.JspServlet PWC6117: File "null" not found

Mime
View raw message