falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seetharam Venkatesh <venkat...@innerzeal.com>
Subject Re: Process NPE
Date Thu, 14 Aug 2014 00:36:42 GMT
James, feel free to create a jira and provide a patch. Thanks!


On Wed, Aug 13, 2014 at 10:52 AM, James Grant <james@queeg.org> wrote:

> I'm having the same issue and was just about to send a mail when you
> posted.
>
> I've had a look at the code and the problem seems to be when the process
> has no inputs. The check in getStorageType used to short circuit if there
> were no inputs but that changed as part of the changes in FALCON-443 to
> only short circuit if the process has no inputs and no outputs.
>
> Below is a patch that will prevent the exception (currently untested).
>
> diff --git
> a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> index 59361e8..33f0aad 100644
> --- a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> +++ b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> @@ -59,11 +59,13 @@ public final class ProcessHelper {
>              return storageType;
>          }
>
> -        for (Input input : process.getInputs().getInputs()) {
> -            Feed feed = EntityUtil.getEntity(EntityType.FEED,
> input.getFeed());
> -            storageType = FeedHelper.getStorageType(feed, cluster);
> -            if (Storage.TYPE.TABLE == storageType) {
> -                break;
> +        if (process.getInputs() != null) {
> +            for (Input input : process.getInputs().getInputs()) {
> +                Feed feed = EntityUtil.getEntity(EntityType.FEED,
> input.getFeed());
> +                storageType = FeedHelper.getStorageType(feed, cluster);
> +                if (Storage.TYPE.TABLE == storageType) {
> +                    break;
> +                }
>              }
>          }
>
> James
>
>
> On 13 August 2014 17:54, Josh Clum <joshclum@gmail.com> wrote:
>
> > Hi, I'm getting the following NPE when i launch with "falcon entity -type
> > process -schedule -name raw-cc-bp-ratio-lcms-comp-process". At the bottom
> > is my process which has no inputs.
> >
> > If I look at
> >
> >
> https://github.com/apache/incubator-falcon/blob/branch-0.5/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java
> > it seems that this should never happen because process.getInputs() ==
> > null should
> > be true and should avoid the NPE. Any thoughts?
> >
> > 2014-08-13 16:49:05,152 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Unable to schedule workflow
> > (AbstractSchedulableEntityManager:60)
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
> >  at
> >
> >
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> > at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
> >  at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> > at
> >
> >
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
> >  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >  at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
> >  at
> > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> > at
> >
> >
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >  at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> >  at
> >
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> >  at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >  at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> > at
> >
> >
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >  at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >  at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >  at org.mortbay.jetty.Server.handle(Server.java:326)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >  at
> >
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >  at
> >
> >
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> > at
> >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > 2014-08-13 16:49:05,154 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Failure reason
> > (FalconWebException:40)
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206)
> >  at
> >
> >
> org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142)
> > at
> >
> >
> org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111)
> >  at
> >
> >
> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119)
> > at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68)
> >  at
> >
> >
> org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57)
> > at
> >
> >
> org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85)
> >  at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >  at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326)
> > at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1)
> >  at
> > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
> > at
> >
> >
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
> >  at
> >
> >
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310)
> > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >  at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> >  at
> >
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> >  at
> >
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >  at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >  at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > at
> >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >  at
> >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >  at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183)
> > at
> >
> >
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
> >  at
> >
> >
> org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221)
> > at
> >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >  at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >  at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> > at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >  at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> > at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >  at org.mortbay.jetty.Server.handle(Server.java:326)
> > at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >  at
> >
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >  at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >  at
> >
> >
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> > at
> >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > 2014-08-13 16:49:05,154 ERROR -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Action failed: Bad Request
> > Error:null
> >  (FalconWebException:66)
> > 2014-08-13 16:49:05,155 INFO  -
> > [6221571@qtp-1696932027-3
> >
> :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process
> > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ {Action:schedule,
> > Dimensions:{entityType=process, colo=*,
> > entityName=raw-cc-bp-ratio-lcms-comp-process}, Status: FAILED,
> > Time-taken:161608565 ns} (METRIC:32)
> >
> > My process looks like this:
> >
> > <process name="raw-cc-bp-ratio-lcms-comp-process"
> > xmlns="uri:falcon:process:0.1">
> >
> >   <tags>type=raw</tags>
> >
> >   <clusters>
> >     <cluster name="minicluster">
> >       <validity start="2014-08-13T00:00Z" end="2016-01-01T00:00Z"/>
> >     </cluster>
> >   </clusters>
> >
> >   <parallel>1</parallel>
> >   <order>FIFO</order>
> >   <timeout>hours(2)</timeout>
> >   <frequency>days(1)</frequency>
> >
> >   <outputs>
> >     <output name="output" feed="raw-cc-bp-ratio-lcms-comp-feed"
> > instance="now(0,0)" />
> >   </outputs>
> >
> >   <properties>
> > ...
> >   </properties>
> >
> >   <workflow name="sqoop_wf" version="2.0.0" engine="oozie"
> > path="/apps/huron/ingest/standard_ingest/sqoop_wf.xml" />
> >
> >   <retry policy="periodic" delay="minutes(10)" attempts="3" />
> >
> > </process>
> >
>



-- 
Regards,
Venkatesh

“Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.”
- Antoine de Saint-Exupéry

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message