sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishwakarma, Chhaya" <Chhaya.Vishwaka...@Teradata.com>
Subject RE: Error in Sqoop workflow using Oozie
Date Mon, 13 Apr 2015 12:40:32 GMT
I tried –doas <username> no luck with that.

Below is the error log

31075 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - Loading data to table ecsdashboard.test_oozie
31175 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception Unable to
move source hdfs://user/qjdht93/test/_SUCCESS to destination hdfs://apps/hive/warehouse/ecsdashboard.db/test_oozie/_SUCCESS
31176 [Thread-35] INFO  org.apache.sqoop.hive.HiveImport  - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
31728 [main] ERROR org.apache.sqoop.tool.ImportTool  - Encountered IOException running import
job: java.io.IOException: Hive exited with status 1
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:425)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)


From: Abraham Elmahrek [mailto:abe@cloudera.com]
Sent: Friday, April 10, 2015 1:46 AM
To: user@sqoop.apache.org
Subject: Re: Error in Sqoop workflow using Oozie

I don't fully understand. Do you have the full error?

On Thu, Apr 9, 2015 at 7:40 AM, Vishwakarma, Chhaya <Chhaya.Vishwakarma@teradata.com<mailto:Chhaya.Vishwakarma@teradata.com>>
wrote:
Thanks Abe
Permissions for jars is r-x for "others" user then too its giving permission denied error

Can you suggest something
Sent from my android device.


-----Original Message-----
From: Abraham Elmahrek <abe@cloudera.com<mailto:abe@cloudera.com>>
To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Sent: Wed, 08 Apr 2015 8:57 pm
Subject: Re: Error in Sqoop workflow using Oozie
Not sure. Could be a yarn configuration thing. Sqoop shouldn't be explicitly creating a directories
there unless you're the yarn user.

On Wed, Apr 8, 2015 at 4:15 AM, Vishwakarma, Chhaya <Chhaya.Vishwakarma@teradata.com<mailto:Chhaya.Vishwakarma@teradata.com>>
wrote:

Created /user/yarn directory but why it is referring to this directory ?
I have received a user ID from Hadoop admin that I’m using for running Sqoop jobs

Thanks ,


From: Abraham Elmahrek [mailto:abe@cloudera.com<mailto:abe@cloudera.com>]
Sent: Tuesday, April 07, 2015 11:23 PM

To: user@sqoop.apache.org<mailto:user@sqoop.apache.org>
Subject: Re: Error in Sqoop workflow using Oozie

Seems like you need to create the "/user/yarn" directory. Also, what user are you running
sqoop as?

On Tue, Apr 7, 2015 at 3:21 AM, Vishwakarma, Chhaya <Chhaya.Vishwakarma@teradata.com<mailto:Chhaya.Vishwakarma@teradata.com>>
wrote:
Thanks Abe,

Below is the Error I'm getting in Map Reduce logs


error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission
denied)

error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient
    .jar (Permission denied)

error: error reading /usr/lib/hadoop/lib/aster-  networking.jar; /usr/lib/hadoop/lib/aster-networking.jar
(Permission  denied)

error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar
(Permission denied)

error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission
denied)

error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission
denied)

Note: /tmp/sqoop yarn/compile/f3aca99f37fa19e505b47bab1499bf5d/EXT_EVNT_ERR_ARC.java uses
or  overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

Intercepting System.exit(1)

Failing Oozie Launcher, Main class   [org.apache.oozie.action.hadoop.SqoopMain], exit code
[1]

che.sqoop.hive.HiveImport  - **Caused by: java.io.FileNotFoundException: File does not exist:
hdfs:/user/yarn**`

Please help

________________________________
From: Abraham Elmahrek [abe@cloudera.com<mailto:abe@cloudera.com>]
Sent: Tuesday, April 07, 2015 2:12 AM
To: user@sqoop.apache.org<mailto:user@sqoop.apache.org>
Subject: Re: Error in Sqoop workflow using Oozie
A couple of thoughts:

  1.  Oozie will run the sqoop job as the user oozie possibly. So "hive-site.xml" won't be
picked up. Try putting it in a shared location and adding a symlink: <file>/tmp/hive-site.xml#hive-site.xml</file>.
The same is true for terajdbc4.jar.
  2.  The oozie launcher task logs should have more information. It will be easier to debug
if you drill down into the mapreduce job launching Sqoop.
-Abe

On Mon, Apr 6, 2015 at 7:40 AM, Vishwakarma, Chhaya <Chhaya.Vishwakarma@teradata.com<mailto:Chhaya.Vishwakarma@teradata.com>>
wrote:
I have written a Sqoop import script to import data from teradata to hive. It's working fine
when I run it from command line ,but when I put it in Oozie workflow and try to execute through
oozie I get the below error


2015-04-02 08:50:55,440  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell]
end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
2015-04-02 08:50:55,459  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell]
ERROR is considered as FAILED for SLA
2015-04-02 08:50:55,505  INFO ActionStartXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail]
Start action [0000069-150114201015959-oozie-oozi-W@fail] with user-retry state : userRetryCount
[0], userRetryMax [0], userRetryInterval [10]
2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail]
[***0000069-150114201015959-oozie-oozi-W@fail***]Action status=DONE
2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail]
[***0000069-150114201015959-oozie-oozi-W@fail***]Action updated in DB!
2015-04-02 08:50:55,522  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf]
JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail]
end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
2015-04-02 08:50:55,556  WARN CoordActionUpdateXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[]
APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[-] **E1100: Command precondition
does not hold before execution, [, coord action is null], Error Code: E1100**`


Below is my workflow.xml

<workflow-app name="sqoop-to-hive" xmlns="uri:oozie:workflow:0.4">

    <start to="sqoop2hive"/>

    <action name="sqoop2hive">

        <sqoop xmlns="uri:oozie:sqoop-action:0.2">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <command>import --connect "jdbc:teradata://server.co/database=TS<http://server.co/database=TS>"-–driver
Com.teradata.jdbc.TeraDriver--username sqoop --password sqoop --table test --hive-import --hive-
 table test</command>

             <archive>tdgssconfig.jar</archive>

            <archive>terajdbc4.jar</archive>

            <file>hive-site.xml</file>

        </sqoop>

        <ok to="end"/>

        <error to="kill"/>

    </action>

    <kill name="kill">

        <message>Action failed</message>

    </kill>

    <end name="end"/>

</workflow-app>

Please suggest

Regards,
Chhaya




Mime
View raw message