sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: SQOOP Export to Netezza fails with "--direct" option: Table Owner not recognized
Date Mon, 28 Jul 2014 17:03:16 GMT
Netezza driver does not support multiple schema support introduced in
Netezza 7.x.

Thanks

Venkat

On Sun, Jul 27, 2014 at 7:02 PM, Munjuluri, Shyam <MunjuluriS@aetna.com> wrote:
> I was able to successfully import data from Netezza to HDFS with & without
> the “--direct” option. I’m also able to export data to Netezza from HDFS
> successfully without the “—direct’ option.
>
> But SQOOP fails when I try to export data to Netezza from HDFS with the
> “—direct” option. It surprisingly does not recognize the “owner” of the
> table properly. Can you please help check the issue?
>
>
>
>  NETEZZA DBMS Version: Release 7.0.4.3-P2 [Build 35925]
>
> NETEZZA driver: nzjdbc3.jar
>
>
>
>  Here is the SQOOP statement & the error;
>
>
>
>  /sqoop_hdp2/bin/sqoop export
>
> --direct
>
> --connect "jdbc:netezza://netezza-server:5480/NZ_DB;schema=ADMIN;"
>
> --username nzuser --password nzpass
>
> --export-dir /user/nzuser/NZ_DB/NZ_TBL_GEOG --table "NZ_TBL_GEOG"
>
> --num-mappers 8
>
> --input-fields-terminated-by ","
>
> --input-escaped-by '\\'
>
> --verbose
>
>
>
>  Error:
>
>  14/07/25 00:56:45 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.1
>
> 14/07/25 00:56:45 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>
> 14/07/25 00:56:45 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
>
> 14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Loaded manager factory:
>
>  com.cloudera.sqoop.manager.DefaultManagerFactory
>
> 14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>
>  com.cloudera.sqoop.manager.DefaultManagerFactory
>
> 14/07/25 00:56:45 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:netezza:
>
> 14/07/25 00:56:45 INFO manager.SqlManager: Using default fetchSize of 1000
>
> 14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>
>  org.apache.sqoop.manager.DirectNetezzaManager@643c0007
>
> 14/07/25 00:56:45 INFO tool.CodeGenTool: The connection manager declares
> that it self manages mapping between records & fields and rows & columns.
> No class will will be generated.
>
> 14/07/25 00:56:46 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
>
> 14/07/25 00:56:47 ERROR manager.DirectNetezzaManager: NZ_TBL_GEOG is not a
> valid Netezza table.  Please make sure that you have connected to the
> Netezza DB and the table name is right.  The current values are
>
>
>
>            connection string : jdbc:netezza://
> netezza-server:5480/NZ_DB;schema=admin;
>
>            table owner : shyam (windows network login ID)
>
>            table name : NZ_TBL_GEOG
>
>
>
>
>
>  Now, I removed the “schema” from the JDBC connection string and added
> “owner” name to the table name in the “—table” switch. The owner name of the
> table repeats itself 2 times in the error log below.
> “NZ_DB.ADMIN.ADMIN.NZ_TBL_GEOG”.
>
>
>
>  Here is the SQOOP statement;
>
>
>
>  /sqoop_hdp2/bin/sqoop export
>
> --direct
>
> --connect "jdbc:netezza://netezza-server:5480/NZ_DB"
>
> --username nzuser --password nzpass
>
> --export-dir /user/nzuser/NZ_DB/NZ_TBL_GEOG
>
> --table  "ADMIN.NZ_TBL_GEOG"
>
> --num-mappers 8
>
> --input-fields-terminated-by ","
>
> --input-escaped-by '\\'
>
> --verbose
>
>
>
>  Error:
>
>
>
>  14/07/25 01:09:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.1
>
> 14/07/25 01:09:07 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>
> 14/07/25 01:09:07 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
>
> 14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Loaded manager factory:
>
>  com.cloudera.sqoop.manager.DefaultManagerFactory
>
> 14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>
>  com.cloudera.sqoop.manager.DefaultManagerFactory
>
> 14/07/25 01:09:07 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:netezza:
>
> 14/07/25 01:09:07 INFO manager.SqlManager: Using default fetchSize of
>
>  1000
>
> 14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>
>  org.apache.sqoop.manager.DirectNetezzaManager@4cc39a20
>
> 14/07/25 01:09:07 INFO tool.CodeGenTool: The connection manager declares
> that it self manages mapping between records & fields and rows & columns.
> No class will will be generated.
>
> 14/07/25 01:09:07 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
>
> 14/07/25 01:09:09 INFO mapreduce.ExportJobBase: Beginning export of
>
>  ADMIN.NZ_TBL_GEOG
>
> 14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Attempting to load jar
> through URL:
>
>  jar:file:/usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar!/
>
> 14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Previous classloader is
>
>  sun.misc.Launcher$AppClassLoader@546b97fd
>
> 14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Loaded jar into current
>
>  JVM: jar:file:/usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar!/
>
> 14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Added classloader for
>
>  jar /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar:
>
>  java.net.FactoryURLClassLoader@6655bb93
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
>  [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/
>
> StaticLoggerBinder.class]
>
> SLF4J: Found binding in
>
>  [jar:file:/usr/lib/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/St
>
> aticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/07/25 01:09:09 INFO Configuration.deprecation: mapred.jar is
>
>  deprecated. Instead, use mapreduce.job.jar
>
> 14/07/25 01:09:09 DEBUG mapreduce.JobBase: Using InputFormat: class
>
>  org.apache.sqoop.mapreduce.ExportInputFormat
>
> 14/07/25 01:09:10 DEBUG db.DBConfiguration: Securing password into job
>
>  credentials store
>
> 14/07/25 01:09:10 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
>
> 14/07/25 01:09:11 ERROR manager.SqlManager: Error executing statement:
>
>   org.netezza.error.NzSQLException: ERROR:  relation does not exist:
> NZ_DB.ADMIN.ADMIN.NZ_TBL_GEOG
>
>          at
> org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:280)
>
>          at
> org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76)
>
>          at org.netezza.sql.NzConnection.execute(NzConnection.java:2819)
>
>          at
> org.netezza.sql.NzPreparedStatament._execute(NzPreparedStatament.java:1126)
>
>          at
> org.netezza.sql.NzPreparedStatament.prepare(NzPreparedStatament.java:1143)
>
>          at
> org.netezza.sql.NzPreparedStatament.<init(NzPreparedStatament.java:89)
>
>          at
> org.netezza.sql.NzConnection.prepareStatement(NzConnection.java:1588)
>
>          at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:689)
>
>          at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:712)
>
>          at
> org.apache.sqoop.manager.SqlManager.getColumnNamesForRawQuery(SqlManager.java:132)
>
>          at
> org.apache.sqoop.manager.SqlManager.getColumnNames(SqlManager.java:116)
>
>          at
> org.apache.sqoop.mapreduce.netezza.NetezzaExternalTableExportJob.configureOutputFormat(NetezzaExternalTableExportJob.java:128)
>
>          at
> org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:398)
>
>          at
> org.apache.sqoop.manager.DirectNetezzaManager.exportTable(DirectNetezzaManager.java:179)
>
>          at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
>
>          at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
>
>          at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
>          at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>
>          at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>
>          at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>
>          at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>
> 14/07/25 01:09:11 DEBUG util.ClassLoaderStack: Restoring classloader:
>
>  sun.misc.Launcher$AppClassLoader@546b97fd
>
> 14/07/25 01:09:11 ERROR sqoop.Sqoop: Got exception running Sqoop:
>
>  java.lang.NullPointerException java.lang.NullPointerException
>
>          at
> org.apache.sqoop.mapreduce.db.DBOutputFormat.setOutput(DBOutputFormat.java:139)
>
>          at
> org.apache.sqoop.mapreduce.netezza.NetezzaExternalTableExportJob.configureOutputFormat(NetezzaExternalTableExportJob.java:134)
>
>          at
> org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:398)
>
>          at
> org.apache.sqoop.manager.DirectNetezzaManager.exportTable(DirectNetezzaManager.java:179)
>
>          at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
>
>          at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
>
>          at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
>          at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>
>          at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>
>          at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>
>          at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>
>
>
>
>
>  Thanks,
>
>
>
>
>
> Shyam Munjuluri
> Integrated Systems Engineering
> Office: (M, W, F) 860-273-0595
> WAH:  (Tu, Th)  860-404-5219
>
>
>
> This e-mail may contain confidential or privileged information. If you think
> you have received this e-mail in error, please advise the sender by reply
> e-mail and then delete this e-mail immediately. Thank you. Aetna

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message