sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Should the source database instance be in the same network with Hadoop cluster?
Date Thu, 17 May 2012 09:55:28 GMT
Hi Jason,
database and hadoop don't have to be on the same network. However sqoop is generating map
reduce job, so each your node have to be able to create connection to database.  Based on
the exception "java.net.SocketException: Network is unreachable" I would guess that some nodes
where are you running TaskTrackers are not allowed to reach your database server.

You can verify whether particular node is allowed to access the database using following command
(execute from that particular node):

mysql --host=<IP Address> --database=test --user=<username> --password=<password>

Jarcec

On May 17, 2012, at 11:15 AM, jason Yang wrote:

> Hello, 
> 
> Recently, I'm trying to import MySQL data into Hive by using sqoop. but I have encountered
the "Communications link failure" problem (the output is attached at the end of this mail)
> 
> To solve this problem, I have checked the user guide of sqoop and do the solution mentioned
in the troubleshooting section. however, It doesn't work. After that, I read some posts about
the basic process of sqoop import, and I found that during the process of import, the sqoop
would submit a map-only Hadoop job to the cluster, which actually do the data transfer works.
so I was wondering that should the source database be in the same network with the hadoop
cluster?
> 
> Any suggestion would be appreciated.
> 
> 
> --output--
> [root@node10 ~]# sqoop import --connect jdbc:mysql://172.18.11.54:3306/hive --username
root -P --table test --hive-import -m1
> Enter password: 
> 12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output.
You can override
> 12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
> 12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation
> 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM
`test` AS t LIMIT 1
> 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM
`test` AS t LIMIT 1
> 12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is /srv/hadoop-0.20.2/bin/..
> 12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at: /srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar
> 12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java
to /root/./test.java
> 12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar
> 12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are importing from mysql.
> 12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
> 12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a MySQL-specific fast
path.
> 12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull
(mysql)
> 12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test
> 12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM
`test` AS t LIMIT 1
> 12/05/17 15:33:13 INFO mapred.JobClient: Running job: job_201205151001_0019
> 12/05/17 15:33:14 INFO mapred.JobClient:  map 0% reduce 0%
> 12/05/17 15:33:23 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_0,
Status : FAILED
> java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
>        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
>        ... 5 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
>        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
>        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
>        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
>        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
>        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
>        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
>        at java.sql.DriverManager.getConnection(DriverManager.java:582)
>        at java.sql.DriverManager.getConnection(DriverManager.java:185)
>        at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
>        ... 6 more
> Caused by: java.net.SocketException: Network is unreachable
>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>        at java.net.Socket.connect(Socket.java:529)
>        at java.net.Socket.connect(Socket.java:478)
>        at java.net.Socket.<init>(Socket.java:375)
>        at java.net.Socket.<init>(Socket.java:218)
>        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
>        ... 22 more
> 
> 12/05/17 15:33:29 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_1,
Status : FAILED
> java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
>        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
>        ... 5 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
>        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
>        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
>        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
>        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
>        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
>        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
>        at java.sql.DriverManager.getConnection(DriverManager.java:582)
>        at java.sql.DriverManager.getConnection(DriverManager.java:185)
>        at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
>        ... 6 more
> Caused by: java.net.SocketException: Network is unreachable
>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>        at java.net.Socket.connect(Socket.java:529)
>        at java.net.Socket.connect(Socket.java:478)
>        at java.net.Socket.<init>(Socket.java:375)
>        at java.net.Socket.<init>(Socket.java:218)
>        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
>        ... 22 more
> 
> 12/05/17 15:33:35 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_2,
Status : FAILED
> java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
>        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
>        ... 5 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
>        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
>        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
>        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
>        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
>        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
>        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
>        at java.sql.DriverManager.getConnection(DriverManager.java:582)
>        at java.sql.DriverManager.getConnection(DriverManager.java:185)
>        at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
>        at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
>        ... 6 more
> Caused by: java.net.SocketException: Network is unreachable
>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>        at java.net.Socket.connect(Socket.java:529)
>        at java.net.Socket.connect(Socket.java:478)
>        at java.net.Socket.<init>(Socket.java:375)
>        at java.net.Socket.<init>(Socket.java:218)
>        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
>        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
>        ... 22 more
> 
> 12/05/17 15:33:44 INFO mapred.JobClient: Job complete: job_201205151001_0019
> 12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2
> 12/05/17 15:33:44 INFO mapred.JobClient:   Job Counters 
> 12/05/17 15:33:44 INFO mapred.JobClient:     Launched map tasks=4
> 12/05/17 15:33:44 INFO mapred.JobClient:     Failed map tasks=1
> 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 32.2704 seconds
(0 bytes/sec)
> 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records.
> 12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job failed!
> [root@node10 ~]#
> ------
> 
> -- 
> YANG, Lin
> 



Mime
View raw message