sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason Yang <lin.yang.ja...@gmail.com>
Subject Re: Should the source database instance be in the same network with Hadoop cluster?
Date Thu, 17 May 2012 10:18:14 GMT
Hi, Cheolsoo and Jarcec,

Thanks for your reply, it helps a lot.

I checked my network topology again and I found some nodes in the cluster
can not connect to the database, I think that's why I got the error.

2012/5/17 Jarek Jarcec Cecho <jarcec@apache.org>

> Hi Jason,
> database and hadoop don't have to be on the same network. However sqoop is
> generating map reduce job, so each your node have to be able to create
> connection to database.  Based on the exception "java.net.SocketException:
> Network is unreachable" I would guess that some nodes where are you running
> TaskTrackers are not allowed to reach your database server.
>
> You can verify whether particular node is allowed to access the database
> using following command (execute from that particular node):
>
> mysql --host=<IP Address> --database=test --user=<username>
> --password=<password>
>
> Jarcec
>
> On May 17, 2012, at 11:15 AM, jason Yang wrote:
>
> > Hello,
> >
> > Recently, I'm trying to import MySQL data into Hive by using sqoop. but
> I have encountered the "Communications link failure" problem (the output is
> attached at the end of this mail)
> >
> > To solve this problem, I have checked the user guide of sqoop and do the
> solution mentioned in the troubleshooting section. however, It doesn't
> work. After that, I read some posts about the basic process of sqoop
> import, and I found that during the process of import, the sqoop would
> submit a map-only Hadoop job to the cluster, which actually do the data
> transfer works. so I was wondering that should the source database be in
> the same network with the hadoop cluster?
> >
> > Any suggestion would be appreciated.
> >
> >
> > --output--
> > [root@node10 ~]# sqoop import --connect jdbc:mysql://
> 172.18.11.54:3306/hive --username root -P --table test --hive-import -m1
> > Enter password:
> > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific
> delimiters for output. You can override
> > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with
> --fields-terminated-by, etc.
> > 12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation
> > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement:
> SELECT t.* FROM `test` AS t LIMIT 1
> > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement:
> SELECT t.* FROM `test` AS t LIMIT 1
> > 12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is
> /srv/hadoop-0.20.2/bin/..
> > 12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at:
> /srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar
> > 12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename
> /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java to
> /root/./test.java
> > 12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar
> > 12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are
> importing from mysql.
> > 12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be
> faster! Use the --direct
> > 12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a
> MySQL-specific fast path.
> > 12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME
> behavior to convertToNull (mysql)
> > 12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test
> > 12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement:
> SELECT t.* FROM `test` AS t LIMIT 1
> > 12/05/17 15:33:13 INFO mapred.JobClient: Running job:
> job_201205151001_0019
> > 12/05/17 15:33:14 INFO mapred.JobClient:  map 0% reduce 0%
> > 12/05/17 15:33:23 INFO mapred.JobClient: Task Id :
> attempt_201205151001_0019_m_000000_0, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
> >        at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> >        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
> >        ... 5 more
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
> >        at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
> >        at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
> >        at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
> >        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
> >        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
> >        at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:582)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:185)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
> >        ... 6 more
> > Caused by: java.net.SocketException: Network is unreachable
> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
> >        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> >        at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> >        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> >        at java.net.Socket.connect(Socket.java:529)
> >        at java.net.Socket.connect(Socket.java:478)
> >        at java.net.Socket.<init>(Socket.java:375)
> >        at java.net.Socket.<init>(Socket.java:218)
> >        at
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
> >        ... 22 more
> >
> > 12/05/17 15:33:29 INFO mapred.JobClient: Task Id :
> attempt_201205151001_0019_m_000000_1, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
> >        at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> >        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
> >        ... 5 more
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
> >        at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
> >        at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
> >        at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
> >        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
> >        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
> >        at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:582)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:185)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
> >        ... 6 more
> > Caused by: java.net.SocketException: Network is unreachable
> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
> >        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> >        at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> >        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> >        at java.net.Socket.connect(Socket.java:529)
> >        at java.net.Socket.connect(Socket.java:478)
> >        at java.net.Socket.<init>(Socket.java:375)
> >        at java.net.Socket.<init>(Socket.java:218)
> >        at
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
> >        ... 22 more
> >
> > 12/05/17 15:33:35 INFO mapred.JobClient: Task Id :
> attempt_201205151001_0019_m_000000_2, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
> >        at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> >        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.RuntimeException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
> >        ... 5 more
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
> >        at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
> >        at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
> >        at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
> >        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
> >        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >        at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
> >        at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:582)
> >        at java.sql.DriverManager.getConnection(DriverManager.java:185)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
> >        at
> com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
> >        ... 6 more
> > Caused by: java.net.SocketException: Network is unreachable
> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
> >        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> >        at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> >        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> >        at java.net.Socket.connect(Socket.java:529)
> >        at java.net.Socket.connect(Socket.java:478)
> >        at java.net.Socket.<init>(Socket.java:375)
> >        at java.net.Socket.<init>(Socket.java:218)
> >        at
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
> >        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
> >        ... 22 more
> >
> > 12/05/17 15:33:44 INFO mapred.JobClient: Job complete:
> job_201205151001_0019
> > 12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2
> > 12/05/17 15:33:44 INFO mapred.JobClient:   Job Counters
> > 12/05/17 15:33:44 INFO mapred.JobClient:     Launched map tasks=4
> > 12/05/17 15:33:44 INFO mapred.JobClient:     Failed map tasks=1
> > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in
> 32.2704 seconds (0 bytes/sec)
> > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records.
> > 12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job
> failed!
> > [root@node10 ~]#
> > ------
> >
> > --
> > YANG, Lin
> >
>
>
>


-- 
YANG, Lin

Mime
View raw message