sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 呂佳鴻 <chiahung1...@gmail.com>
Subject Re: sqoop 1.4.5 for hadoop 2.4 question (can't find method)
Date Wed, 19 Nov 2014 05:27:40 GMT
Hi, i try not to  change build.xml now  , and still have the same problem.
(just use "ant package" to build)

I modified the sqoop.env.sh to add HADOOP_COMMON_HOME & HADOOP_MAPRED_HOME

Then I add  mysql-connector-java-5.1.34-bin.jar to sqoop lib
try the same command

sqoop import --connect jdbc:mysql://localhost:3306/test --username xxx
--password xxxx --table user1 -m 1

How can I modified the error?
so I can run sqoop on hadoop 2.4.1?
if there is other solution or file for hadoop 2.4.1 that i can check .


Thank you!


follow are appear message
Warning: /opt/sqoop-1.4.5/../hcatalog does not exist! HCatalog jobs will
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop-1.4.5/../accumulo does not exist! Accumulo imports will
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.5/../zookeeper does not exist! Accumulo imports
will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
14/11/19 13:21:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
14/11/19 13:21:47 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/11/19 13:21:48 INFO manager.MySQLManager: Preparing to use a MySQL
streaming resultset.
14/11/19 13:21:48 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/11/19 13:21:48 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `user1` AS t LIMIT 1
14/11/19 13:21:48 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `user1` AS t LIMIT 1
14/11/19 13:21:48 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
Note: /tmp/sqoop-hadoop/compile/4e64e5e221ace9afdb32576604ed074b/user1.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/11/19 13:21:49 INFO orm.CompilationManager: Writing jar file:
14/11/19 13:21:50 WARN manager.MySQLManager: It looks like you are
importing from mysql.
14/11/19 13:21:50 WARN manager.MySQLManager: This transfer can be faster!
Use the --direct
14/11/19 13:21:50 WARN manager.MySQLManager: option to exercise a
MySQL-specific fast path.
14/11/19 13:21:50 INFO manager.MySQLManager: Setting zero DATETIME behavior
to convertToNull (mysql)
14/11/19 13:21:50 INFO mapreduce.ImportJobBase: Beginning import of user1
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library
/opt/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled
stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/11/19 13:21:50 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/11/19 13:21:50 INFO Configuration.deprecation: mapred.jar is deprecated.
Instead, use mapreduce.job.jar
14/11/19 13:21:50 INFO Configuration.deprecation: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
14/11/19 13:21:50 INFO client.RMProxy: Connecting to ResourceManager at /
14/11/19 13:21:52 INFO db.DBInputFormat: Using read commited transaction
14/11/19 13:21:52 INFO mapreduce.JobSubmitter: number of splits:1
14/11/19 13:21:52 INFO mapreduce.JobSubmitter: Submitting tokens for job:
14/11/19 13:21:52 INFO impl.YarnClientImpl: Submitted application
14/11/19 13:21:52 INFO mapreduce.JobSubmitter: Cleaning up the staging area
Exception in thread "main" java.lang.NoSuchMethodError:
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
On Tue Nov 18 2014 at 上午1:38:33 Abraham Elmahrek <abe@cloudera.com> wrote:

> You shouldn't need to change the versions in build.xml. If should just
> work out of the box with any Hadoop 2 distribution.
> -Abe
> On Mon, Nov 17, 2014 at 2:33 AM, lu <chiahung1227@gmail.com> wrote:
>> hi ,
>> i download sqoop-1.4.5.tar.gz ,and modified build.xml
>> (just change hadoop version to 2.4.1 )
>> <equals arg1="${hadoopversion}" arg2="200" />
>>       <then>
>>         <property name="hadoop.version" value="2.4.1" />
>>         <property name="hbase94.version" value="0.94.2" />
>>         <property name="zookeeper.version" value="3.4.2" />
>>         <property name="hadoop.version.full" value="2.4.1" />
>>         <property name="hcatalog.version" value="0.13.0" />
>>         <property name="hbasecompatprofile" value="2" />
>>         <property name="avrohadoopprofile" value="2" />
>>       </then>
>> after build and install
>> i try to convert data from mysql to hdfs
>> "sqoop import --connect jdbc:mysql://localhost:3306/test --username xxx
>> --password xxxx --table user1 -m 1 --verbose"
>> it's appear
>> *Exception in thread "main" java.lang.NoSuchMethodError:
>> org.apache.hadoop.http.HttpConfig.getSchemePrefix()Ljava/lang/String;*
>> this method seem to change in hadoop 2.4
>> how can i fix this bug and if sqoop 1.4.5 can support hadoop 2.4.1?
>> i can't find solution on web
>> Thank you!
>> Luke

View raw message