falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jainendra Rai <jainendr...@gmail.com>
Subject Falcon Incremental Import support not available
Date Wed, 01 Jun 2016 08:11:52 GMT
Hi Team,

We were trying to run an Import job with Apache Falcon. The document says
that

   - [incremental, append] - data is extracted incrementally using the key
   specified in the deltacolumn is supported


<import>
                <source name="mysql-db" tableName="simple">
                    <extract type="incremental">
                        <deltacolumn>modified_time</deltacolumn>
                        <mergepolicy>append</mergepolicy>
                    </extract>
                    <fields>
                        <includes>
                            <field>id</field>
                            <field>name</field>
                        </includes>
                    </fields>
                </source>
                <arguments>
                    <argument name="--split-by" value="id"/>
                    <argument name="--num-mappers" value="2"/>
                </arguments>

</import>


Whille [incremental, snapshot] -  this is not supported.

We were trying to work with the former one and suprisingly when we submit
the feed we get an Error

ERROR: Bad Request;default/Feed feed-employee2 is using unsupported
extraction mechanism incremental


Log:


 2016-06-01 08:04:24,761 INFO  - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
All locks released on feed-employee2 (AbstractEntityManager:371)

2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Unable to persist entity object (AbstractEntityManager:194)

org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is
using unsupported extraction mechanism incremental

        at
org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)

        at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)

        at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)

        at
org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)

        at
org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)

        at
org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)

        at
org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)

        at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)

        at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)

        at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)

        at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)

        at
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)

        at
org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)

        at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)

        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

        at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)

        at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

        at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

        at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

        at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

        at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)

        at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)

        at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)

        at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)

        at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

        at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)

        at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

        at
org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)

        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

        at
org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)

        at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)

        at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)

        at
org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)

        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

        at
org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)

        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

        at
org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)

        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

        at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

        at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

        at org.mortbay.jetty.Server.handle(Server.java:326)

        at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

        at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

        at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)

        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2016-06-01 08:04:24,763 INFO  - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
{Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED,
Time-taken:11824449 ns} (METRIC:38)

2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Exception in sending Notification from
EmailNotificationPlugin:java.lang.NullPointerException
(EmailNotificationPlugin:71)

2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1 performed
request https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO  - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop@127.0.0.1 performed
https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
2016-06-01T08:04Z (AUDIT:48




Should this error come up.


PLease have a look at the submitted feed too.


<?xml version="1.0" encoding="UTF-8"?>

      <!--

       A feed representing Hourly customer email data retained for 90 days

       -->

      <feed description="Raw customer email feed" name="feed1"
xmlns="uri:falcon:feed:0.1">


<tags>externalSystem=USWestEmailServers,classification=secure</tags>



          <groups>DataImportPipeline</groups>

          <frequency>minutes(6)</frequency>

          <late-arrival cut-off="hours(4)"/>

          <clusters>

              <cluster name="primaryCluster" type="source">

                  <validity start="2016-05-23T00:00Z"
end="2017-05-31T00:00Z"/>

                  <retention limit="days(90)" action="delete"/>

                  <import>

                      <source name="mysql-db" tableName="employee">



 <extract type="incremental">

                        <deltacolumn>modified_item</deltacolumn>

                        <mergepolicy>append</mergepolicy>

                    </extract>







                      </source>

 <arguments>

                          <argument name="--split-by" value="id"/>

                          <argument name="--num-mappers" value="2"/>

                      </arguments>



                  </import>





              </cluster>

          </clusters>





          <locations>

              <location type="data"
path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>

              <location type="stats" path="/none"/>

              <location type="meta" path="/none"/>

          </locations>

<notification type="email" to="xyz.com"/>

          <ACL owner="hadoop" group="hadoop" permission="0755"/>

          <schema location="/none" provider="none"/>


      </feed>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message