sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason Yang <lin.yang.ja...@gmail.com>
Subject Should the source database instance be in the same network with Hadoop cluster?
Date Thu, 17 May 2012 09:15:45 GMT
Hello,

Recently, I'm trying to import MySQL data into Hive by using sqoop. but I
have encountered the "Communications link failure" problem (the output is
attached at the end of this mail)

To solve this problem, I have checked the user guide of sqoop and do the
solution mentioned in the troubleshooting section. however, It doesn't
work. After that, I read some posts about the basic process of sqoop
import, and I found that during the process of import, the sqoop
would submit a map-only Hadoop job to the cluster, which actually do the
data transfer works. so I was wondering that should the source database be
in the same network with the hadoop cluster?

Any suggestion would be appreciated.


*--output--*
*[root@node10 ~]# sqoop import --connect
jdbc:mysql://172.18.11.54:3306/hive--username root -P --table test
--hive-import -m1
*
*Enter password: *
*12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
for output. You can override*
*12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with
--fields-terminated-by, etc.*
*12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation*
*12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement:
SELECT t.* FROM `test` AS t LIMIT 1*
*12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement:
SELECT t.* FROM `test` AS t LIMIT 1*
*12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is
/srv/hadoop-0.20.2/bin/..*
*12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at:
/srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar*
*12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename
/tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java to
/root/./test.java*
*12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar*
*12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are
importing from mysql.*
*12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be faster!
Use the --direct*
*12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a
MySQL-specific fast path.*
*12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME
behavior to convertToNull (mysql)*
*12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test*
*12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement:
SELECT t.* FROM `test` AS t LIMIT 1*
*12/05/17 15:33:13 INFO mapred.JobClient: Running job: job_201205151001_0019
*
*12/05/17 15:33:14 INFO mapred.JobClient:  map 0% reduce 0%*
*12/05/17 15:33:23 INFO mapred.JobClient: Task Id :
attempt_201205151001_0019_m_000000_0, Status : FAILED*
*java.lang.RuntimeException: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
*
*        at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)*
*        at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
*
*        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)*
*        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)*
*        at org.apache.hadoop.mapred.Child.main(Child.java:170)*
*Caused by: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
*
*        ... 5 more*
*Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)*
*        at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)*
*        at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)*
*        at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)*
*        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)*
*        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)*
*        at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:582)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:185)*
*        at
com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
*
*        ... 6 more*
*Caused by: java.net.SocketException: Network is unreachable*
*        at java.net.PlainSocketImpl.socketConnect(Native Method)*
*        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)*
*        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)*
*        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)*
*        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)*
*        at java.net.Socket.connect(Socket.java:529)*
*        at java.net.Socket.connect(Socket.java:478)*
*        at java.net.Socket.<init>(Socket.java:375)*
*        at java.net.Socket.<init>(Socket.java:218)*
*        at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)*
*        ... 22 more*
*
*
* 12/05/17 15:33:29 INFO mapred.JobClient: Task Id :
attempt_201205151001_0019_m_000000_1, Status : FAILED*
*java.lang.RuntimeException: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
*
*        at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)*
*        at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
*
*        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)*
*        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)*
*        at org.apache.hadoop.mapred.Child.main(Child.java:170)*
*Caused by: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
*
*        ... 5 more*
*Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)*
*        at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)*
*        at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)*
*        at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)*
*        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)*
*        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)*
*        at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:582)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:185)*
*        at
com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
*
*        ... 6 more*
*Caused by: java.net.SocketException: Network is unreachable*
*        at java.net.PlainSocketImpl.socketConnect(Native Method)*
*        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)*
*        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)*
*        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)*
*        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)*
*        at java.net.Socket.connect(Socket.java:529)*
*        at java.net.Socket.connect(Socket.java:478)*
*        at java.net.Socket.<init>(Socket.java:375)*
*        at java.net.Socket.<init>(Socket.java:218)*
*        at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)*
*        ... 22 more*
*
*
* 12/05/17 15:33:35 INFO mapred.JobClient: Task Id :
attempt_201205151001_0019_m_000000_2, Status : FAILED*
*java.lang.RuntimeException: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
*
*        at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)*
*        at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
*
*        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)*
*        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)*
*        at org.apache.hadoop.mapred.Child.main(Child.java:170)*
*Caused by: java.lang.RuntimeException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
*
*        ... 5 more*
*Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure*
*
*
*The last packet sent successfully to the server was 0 milliseconds ago.
The driver has not received any packets from the server.*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)*
*        at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)*
*        at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)*
*        at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)*
*        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)*
*        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)*
*        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)*
*        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
*
*        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)*
*        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)*
*        at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)*
*        at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:582)*
*        at java.sql.DriverManager.getConnection(DriverManager.java:185)*
*        at
com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
*
*        at
com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
*
*        ... 6 more*
*Caused by: java.net.SocketException: Network is unreachable*
*        at java.net.PlainSocketImpl.socketConnect(Native Method)*
*        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)*
*        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)*
*        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)*
*        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)*
*        at java.net.Socket.connect(Socket.java:529)*
*        at java.net.Socket.connect(Socket.java:478)*
*        at java.net.Socket.<init>(Socket.java:375)*
*        at java.net.Socket.<init>(Socket.java:218)*
*        at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
*
*        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)*
*        ... 22 more*
*
*
* 12/05/17 15:33:44 INFO mapred.JobClient: Job complete:
job_201205151001_0019*
*12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2*
*12/05/17 15:33:44 INFO mapred.JobClient:   Job Counters *
*12/05/17 15:33:44 INFO mapred.JobClient:     Launched map tasks=4*
*12/05/17 15:33:44 INFO mapred.JobClient:     Failed map tasks=1*
*12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in
32.2704 seconds (0 bytes/sec)*
*12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records.*
*12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job
failed!*
*[root@node10 ~]#*
*------*

-- 
YANG, Lin

Mime
View raw message