sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sunil kumar (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-3144) Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema -
Date Tue, 17 Oct 2017 19:59:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204301#comment-16204301
] 

sunil kumar edited comment on SQOOP-3144 at 10/17/17 7:58 PM:
--------------------------------------------------------------

Is there any way to migrate data from Mysql to HDFS using 1.99.7
a) schema/table combination doesn't work
b) sql statement doesn't work either.
example SQL statement: select record_id from test.test1 where ${CONDITIONS};

please let us know if you have any way to pull data from mysql.

update 10/17/2017
it could be semicolon is my query , however  by using sql and providing partition column,
the partitioner doesn't take a column that is INT,LONG, String, throws error related to converting
partition key from String to some other type. I have spend more than a day on this version
(1.99.7) and no luck. I have used previous version 1.99.5 it took only few hours to get it
working. Cloudera is excluding scoop2 from it's CDH suite. I think this could be the reason.


was (Author: sunelsnn):
Is there any way to migrate data from Mysql to HDFS using 1.99.7
a) schema/table combination doesn't work
b) sql statement doesn't work either.
example SQL statement: select record_id from test.test1 where ${CONDITIONS};

please let us know if you have any way to pull data from mysql.

> Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0016:Can't
fetch schema -
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-3144
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3144
>             Project: Sqoop
>          Issue Type: Bug
>          Components: sqoop2-jdbc-connector
>    Affects Versions: 1.99.7
>         Environment: hduser@cai-hadoop01:/usr/local/sqoop/bin$ cat /etc/*release*
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
> NAME="Ubuntu"
> VERSION="16.04.1 LTS (Xenial Xerus)"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 16.04.1 LTS"
> VERSION_ID="16.04"
> HOME_URL="http://www.ubuntu.com/"
> SUPPORT_URL="http://help.ubuntu.com/"
> BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
> UBUNTU_CODENAME=xenial
> hduser@cai-hadoop01:/usr/local/sqoop/bin$
> -----------------------------------------------------------
> hduser@cai-hadoop01:/usr/local/sqoop/bin$ uname -a
> Linux cai-hadoop01 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64
x86_64 x86_64 GNU/Linux
>            Reporter: Mohamed El-Sayyad
>              Labels: beginner, easyfix, easytest, newbie
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Requirement: Importing MySQL Database to HDFS
> Minor Issue: No Unique Primary Key included in DB
> Workaround: Added Year as partition Column 
> Current Issue:
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema
> Cause:
> This is because a double quote is appended, which is not parsed by MySQL DB.
>  Using min/max query: SELECT MIN("year"), MAX("year") FROM "sports"."players"
> ======================================================================================================
> # SQOOP LOG #
> 2017-03-02 10:24:10,576 INFO  [org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.configurePartitionProperties(GenericJdbcFromInitializer.java:152)]
Using partition column: year
> 2017-03-02 10:24:10,576 INFO  [org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.configurePartitionProperties(GenericJdbcFromInitializer.java:238)]
Using min/max query: SELECT MIN("
> year"), MAX("year") FROM "sports"."players"
> 2017-03-02 10:24:10,576 ERROR [org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:86)]
Exception in PUT http://localhost:12000/sqoop/v1/job/MysqlToHdfs/start
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0016:Can't fetch schema
>         at org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.initialize(GenericJdbcFromInitializer.java:64)
>         at org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer.initialize(GenericJdbcFromInitializer.java:47)
>         at org.apache.sqoop.driver.JobManager$1.call(JobManager.java:520)
>         at org.apache.sqoop.driver.JobManager$1.call(JobManager.java:517)
>         at org.apache.sqoop.utils.ClassUtils.executeWithClassLoader(ClassUtils.java:281)
>         at org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:516)
>         at org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:423)
>         at org.apache.sqoop.driver.JobManager.start(JobManager.java:317)
>         at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:353)
>         at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:114)
>         at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:84)
>         at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:81)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>         at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:291)
>         at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>         at org.eclipse.jetty.server.Server.handle(Server.java:499)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
in your SQL syntax; check the manual that corresponds to your MySQL server version for the
right syntax to use near '"sports"."players"' at line 1
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> ======================================================================================================
> # Running the Query on MYSQL #
> With Double Quotes:
> mysql> SELECT MIN("year"), MAX("year") FROM "sports"."players";
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds
to your MySQL server version for the right syntax to use near '"sports"."players"' at line
1
> Without Double Quotes:
> mysql> SELECT MIN(year), MAX(year) FROM sports.players;
> +-----------+-----------+
> | MIN(year) | MAX(year) |
> +-----------+-----------+
> |      1950 |      2009 |
> +-----------+-----------+
> 1 row in set (0.01 sec)
> mysql>
> # MySQL Version #
> mysql> SHOW VARIABLES LIKE "%version%";
> +-------------------------+-------------------------+
> | Variable_name           | Value                   |
> +-------------------------+-------------------------+
> | innodb_version          | 5.7.17                  |
> | protocol_version        | 10                      |
> | slave_type_conversions  |                         |
> | tls_version             | TLSv1,TLSv1.1           |
> | version                 | 5.7.17-0ubuntu0.16.04.1 |
> | version_comment         | (Ubuntu)                |
> | version_compile_machine | x86_64                  |
> | version_compile_os      | Linux                   |
> +-------------------------+-------------------------+
> 8 rows in set (0.00 sec)
> # MySQL Driver Version #
> hduser@cai-hadoop01:/usr/local/sqoop/bin$ ls -l ../lib
> total 968
> -rwxrwxrwx 1 root root 990927 Feb 12 15:26 mysql-connector-java-5.1.40-bin.jar
> hduser@cai-hadoop01:/usr/local/sqoop/bin$
> # SQOOP ENV Vars  #
> export SQOOP_HOME=/usr/local/sqoop
> export SQOOP_CONF_DIR=$SQOOP_HOME/conf
> export SQOOP_CLASSPATH=$SQOOP_CONF_DIR
> export SQOOP_SERVER_EXTRA_LIB=/usr/local/sqoop/lib
> export PATH=$SQOOP_HOME/bin:/bin:/sbin:$PATH



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message