sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gäde, Sebastian <s116...@hft-leipzig.de>
Subject AW: using Sqoop 1.4.4 with Hadoop 2.4.0
Date Mon, 16 Jun 2014 07:00:46 GMT
Thanks Abe,

This is the output:
<snip>
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-flume-1.4.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/16 08:54:19 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/06/16 08:54:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/06/16 08:54:19 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql:
14/06/16 08:54:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@1537e1d
14/06/16 08:54:19 INFO tool.CodeGenTool: Beginning code generation
14/06/16 08:54:19 DEBUG manager.SqlManager: Execute getColumnTypesRawQuery : SELECT t.* FROM
`test1` AS t LIMIT 1
14/06/16 08:54:19 DEBUG manager.SqlManager: No connection paramenters specified. Using regular
API for making connection.
14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1`
AS t LIMIT 1
14/06/16 08:54:24 DEBUG orm.ClassWriter: selected columns:
14/06/16 08:54:24 DEBUG orm.ClassWriter:   index
14/06/16 08:54:24 DEBUG orm.ClassWriter:   bel
14/06/16 08:54:24 DEBUG orm.ClassWriter:   gesp
14/06/16 08:54:24 DEBUG orm.ClassWriter:   bd
14/06/16 08:54:24 DEBUG orm.ClassWriter:   gd1
14/06/16 08:54:24 DEBUG orm.ClassWriter:   gd2
14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1`
AS t LIMIT 1
14/06/16 08:54:24 DEBUG orm.ClassWriter: Writing source file: /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/test1.java
14/06/16 08:54:24 DEBUG orm.ClassWriter: Table name: test1
14/06/16 08:54:24 DEBUG orm.ClassWriter: Columns: index:12, bel:4, gesp:4, bd:4, gd1:4, gd2:4,
14/06/16 08:54:24 DEBUG orm.ClassWriter: sourceFilename is test1.java
14/06/16 08:54:24 DEBUG orm.CompilationManager: Found existing /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/
14/06/16 08:54:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.4.0/share/hadoop/mapreduce
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
        at org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager.java:109)
        at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
        at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
        at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 11 more
</snip>

In the past such issues turned out to be caused by a wrong classpath. However, I assume Sqoop
is building the classpath on its own depending on the input for HADOOP_*_HOME?!

There are no MR logs.

Cheers
Seb.
________________________________
Von: Abraham Elmahrek [abe@cloudera.com]
Gesendet: Sonntag, 15. Juni 2014 23:54
An: user@sqoop.apache.org
Betreff: Re: using Sqoop 1.4.4 with Hadoop 2.4.0

You can add --verbose to the beginning of your command (right after the word "export") and
you'll get some more info. Could you copy/paste the output of your sqoop command with this
flag? There should also be logs in the MapReduce job/task logs. In this case, I think the
application and container logs may have more information. They are usually located under '/var/log/hadoop-yarn/'.
Have you considered installing these components using a vendor package or BigTop?


On Sun, Jun 15, 2014 at 12:42 AM, Sebastian Gäde <s116102@hft-leipzig.de<mailto:s116102@hft-leipzig.de>>
wrote:
Hi,

Thanks Abe.

Yes, Sqoop and Hadoop were downloaded from a tarball.

I set HADOOP_COMMON_HOME to /opt/hadoop-2.4.0 and HADOOP_MAPRED_HOME to /opt/hadoop-2.4.0/share/hadoop/yarn
using the conf/sqoop-env.sh. I also tried /opt/hadoop-2.4.0/share/hadoop/mapreduce for HADOOP_MAPRED_HOME
as I did not know which JAR I need. However, the error was the same.

Is there a way to enable DEBUG messages? This sometimes helped me in the passed with other
Hadoop issues.

Cheers
Seb.

Am 14.06.2014 06:37, schrieb Abraham Elmahrek:
Hey there,

The exception is indicative of a missing jar from the class path. It
seems like the hadoop-mapreduce-client-core.jar file is missing. Have
you set HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME?

How did you install Sqoop and Hadoop? If you've downloaded a tarball,
you may need to set the environment variables specified above.

For more info see
http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_controlling_the_hadoop_installation.

-Abe


On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <s116102@hft-leipzig.de<mailto:s116102@hft-leipzig.de>
<mailto:s116102@hft-leipzig.de<mailto:s116102@hft-leipzig.de>>> wrote:

    Hi,

    I’d like to export MR result (text) files from HDFS to MySQL. My
    cluster is
    running Hadoop 2.4.0.

    When I’m running
    ./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user
    –password pass –table test1 –export-dir /output
    I get the below exception. I assume, sqoop is using the old MR API?
    Can I
    change this somehow to use the new API, then I'd be able to point
    $HADOOP_MAPRED_HOME to the right position?!

    14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the
    command-line is insecure. Consider using -P instead.
    14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
    streaming resultset.
    14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
    14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
    SELECT
    t.* FROM `test1` AS t LIMIT 1
    14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
    SELECT
    t.* FROM `test1` AS t LIMIT 1
    14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
    /opt/hadoop-2.4.0/share/hadoop/yarn
    Exception in thread "main" java.lang.NoClassDefFoundError:
    org/apache/hadoop/mapred/JobConf
             at
    org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager
    .java:109)
             at
    org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
             at
    org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
             at
    org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
             at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
             at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
             at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
             at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
             at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
             at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
             at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

    Thanks very much!
    Sebastian





Mime
View raw message