sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Error in Sqoop workflow using Oozie
Date Mon, 13 Apr 2015 18:02:30 GMT
Could you provide your oozie command? It should run as the user who
submitted the job.

On Mon, Apr 13, 2015 at 5:40 AM, Vishwakarma, Chhaya <
Chhaya.Vishwakarma@teradata.com> wrote:

>  I tried –doas <username> no luck with that.
>
>
>
> Below is the error log
>
>
>
> 31075 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - Loading data
> to table ecsdashboard.test_oozie
>
> 31175 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - Failed with
> exception Unable to move source hdfs://user/qjdht93/test/_SUCCESS to
> destination hdfs://apps/hive/warehouse/ecsdashboard.db/test_oozie/_SUCCESS
>
> 31176 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - FAILED:
> Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
>
> 31728 [main] ERROR org.apache.sqoop.tool.ImportTool  - Encountered
> IOException running import job: java.io.IOException: Hive exited with
> status 1
>
> at
> org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
>
> at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
>
> at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
>
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:425)
>
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
>
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>
> at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
>
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
>
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
>
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
>
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
>
>
>
>
> *From:* Abraham Elmahrek [mailto:abe@cloudera.com]
> *Sent:* Friday, April 10, 2015 1:46 AM
>
> *To:* user@sqoop.apache.org
> *Subject:* Re: Error in Sqoop workflow using Oozie
>
>
>
> I don't fully understand. Do you have the full error?
>
>
>
> On Thu, Apr 9, 2015 at 7:40 AM, Vishwakarma, Chhaya <
> Chhaya.Vishwakarma@teradata.com> wrote:
>
> Thanks Abe
> Permissions for jars is r-x for "others" user then too its giving
> permission denied error
>
> Can you suggest something
> Sent from my android device.
>
>
>
> -----Original Message-----
> From: Abraham Elmahrek <abe@cloudera.com>
> To: "user@sqoop.apache.org" <user@sqoop.apache.org>
> Sent: Wed, 08 Apr 2015 8:57 pm
> Subject: Re: Error in Sqoop workflow using Oozie
>
> Not sure. Could be a yarn configuration thing. Sqoop shouldn't be
> explicitly creating a directories there unless you're the yarn user.
>
>
>
> On Wed, Apr 8, 2015 at 4:15 AM, Vishwakarma, Chhaya <
> Chhaya.Vishwakarma@teradata.com> wrote:
>
>
>
> Created /user/yarn directory but why it is referring to this directory ?
>
> I have received a user ID from Hadoop admin that I’m using for running
> Sqoop jobs
>
>
>
> Thanks ,
>
>
>
>
>
> *From:* Abraham Elmahrek [mailto:abe@cloudera.com]
> *Sent:* Tuesday, April 07, 2015 11:23 PM
>
>
> *To:* user@sqoop.apache.org
> *Subject:* Re: Error in Sqoop workflow using Oozie
>
>
>
> Seems like you need to create the "/user/yarn" directory. Also, what user
> are you running sqoop as?
>
>
>
> On Tue, Apr 7, 2015 at 3:21 AM, Vishwakarma, Chhaya <
> Chhaya.Vishwakarma@teradata.com> wrote:
>
> Thanks Abe,
>
>
>
> Below is the Error I'm getting in Map Reduce logs
>
>
>
> error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission
denied)
>
> error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient
    .jar (Permission denied)
>
> error: error reading /usr/lib/hadoop/lib/aster-  networking.jar; /usr/lib/hadoop/lib/aster-networking.jar
(Permission  denied)
>
> error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar
(Permission denied)
>
> error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission
denied)
>
> error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission
denied)
>
> Note: /tmp/sqoop yarn/compile/f3aca99f37fa19e505b47bab1499bf5d/EXT_EVNT_ERR_ARC.java
uses or  overrides a deprecated API.
>
> Note: Recompile with -Xlint:deprecation for details.
>
> Intercepting System.exit(1)
>
> Failing Oozie Launcher, Main class   [org.apache.oozie.action.hadoop.SqoopMain], exit
code [1]
>
> che.sqoop.hive.HiveImport  - **Caused by: java.io.FileNotFoundException: File does not
exist: hdfs:/user/yarn**`
>
> Please help
>
>  ------------------------------
>
> *From:* Abraham Elmahrek [abe@cloudera.com]
> *Sent:* Tuesday, April 07, 2015 2:12 AM
> *To:* user@sqoop.apache.org
> *Subject:* Re: Error in Sqoop workflow using Oozie
>
> A couple of thoughts:
>
>    1. Oozie will run the sqoop job as the user oozie possibly. So
>    "hive-site.xml" won't be picked up. Try putting it in a shared location and
>    adding a symlink: <file>/tmp/hive-site.xml#hive-site.xml</file>. The same
>    is true for terajdbc4.jar.
>    2. The oozie launcher task logs should have more information. It will
>    be easier to debug if you drill down into the mapreduce job launching Sqoop.
>
>  -Abe
>
>
>
> On Mon, Apr 6, 2015 at 7:40 AM, Vishwakarma, Chhaya <
> Chhaya.Vishwakarma@teradata.com> wrote:
>
> I have written a Sqoop import script to import data from teradata to hive.
> It's working fine when I run it from command line ,but when I put it in
> Oozie workflow and try to execute through oozie I get the below error
>
>
>
>
>
> 2015-04-02 08:50:55,440  INFO ActionEndXCommand:539 - USER[qjdht93]
> GROUP[-TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] end executor for
> wf action 0000069-150114201015959-oozie-oozi-W with wf job
> 0000069-150114201015959-oozie-oozi-W
>
> 2015-04-02 08:50:55,459  INFO ActionEndXCommand:539 - USER[qjdht93]
> GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] ERROR is
> considered as FAILED for SLA
>
> 2015-04-02 08:50:55,505  INFO ActionStartXCommand:539 - USER[qjdht93]
> GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@fail] Start action
> [0000069-150114201015959-oozie-oozi-W@fail] with user-retry state :
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>
> 2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93]
> GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@fail]
> [***0000069-150114201015959-oozie-oozi-W@fail***]Action status=DONE
>
> 2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93]
> GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@fail]
> [***0000069-150114201015959-oozie-oozi-W@fail***]Action updated in DB!
>
> 2015-04-02 08:50:55,522  INFO ActionEndXCommand:539 - USER[qjdht93]
> GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W]
> ACTION[0000069-150114201015959-oozie-oozi-W@fail] end executor for wf
> action 0000069-150114201015959-oozie-oozi-W with wf job
> 0000069-150114201015959-oozie-oozi-W
>
> 2015-04-02 08:50:55,556  WARN CoordActionUpdateXCommand:542 -
> USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
> JOB[0000069-150114201015959-oozie-oozi-W] ACTION[-] **E1100: Command
> precondition does not hold before execution, [, coord action is null],
> Error Code: E1100**`
>
>
>
> Below is my workflow.xml
>
> <workflow-app name="sqoop-to-hive" xmlns="uri:oozie:workflow:0.4">
>
>     <start to="sqoop2hive"/>
>
>     <action name="sqoop2hive">
>
>         <sqoop xmlns="uri:oozie:sqoop-action:0.2">
>
>             <job-tracker>${jobTracker}</job-tracker>
>
>             <name-node>${nameNode}</name-node>
>
>             <command>import --connect "jdbc:teradata://server.co/database=TS"-–driver
Com.teradata.jdbc.TeraDriver--username sqoop --password sqoop --table test --hive-import --hive-
 table test</command>
>
>              <archive>tdgssconfig.jar</archive>
>
>             <archive>terajdbc4.jar</archive>
>
>             <file>hive-site.xml</file>
>
>         </sqoop>
>
>         <ok to="end"/>
>
>         <error to="kill"/>
>
>     </action>
>
>     <kill name="kill">
>
>         <message>Action failed</message>
>
>     </kill>
>
>     <end name="end"/>
>
> </workflow-app>
>
>
>
> Please suggest
>
>
>
> Regards,
>
> Chhaya
>
>
>
>
>
>
>
>
>

Mime
View raw message