sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ying Cao <angela.caoy...@gmail.com>
Subject Re: Review Request 58477: support DB2 xml data type when sqoop import with parquet
Date Mon, 15 May 2017 02:03:05 GMT


> On 五月 3, 2017, 12:15 p.m., Boglarka Egyed wrote:
> > Hi Ying,
> > 
> > Thanks for the updated diff however the manual test case added by you still fails
with the following:
> > 
> > 2017-05-03 13:53:39,234 (main) [ERROR - org.apache.sqoop.tool.ImportTool.run(ImportTool.java:630)]
Import failed: Cannot convert SQL type 2009
> > 2017-05-03 13:53:39,234 (main) [ERROR - org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:185)]
Got exception running Sqoop: java.lang.IllegalArgumentException: Cannot convert SQL type 2009
> > java.lang.IllegalArgumentException: Cannot convert SQL type 2009
> > 	at org.apache.sqoop.manager.ConnManager.toAvroType(ConnManager.java:223)
> > 	at org.apache.sqoop.manager.ConnManager.toAvroType(ConnManager.java:277)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.toAvroType(AvroSchemaGenerator.java:166)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.toAvroSchema(AvroSchemaGenerator.java:136)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchemaGenerator.java:96)
> > 	at org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroSchema(DataDrivenImportJob.java:156)
> > 	at org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:110)
> > 	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:264)
> > 	at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
> > 	at org.apache.sqoop.manager.Db2Manager.importTable(Db2Manager.java:72)
> > 	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
> > 	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
> > 	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> > 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > 	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> > 	at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:42)
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:228)
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:243)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.runDb2Test(DB2XmlTypeParquetImportManualTest.java:241)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.testParquetDb2Import(DB2XmlTypeParquetImportManualTest.java:212)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:498)
> > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> > 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> > 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
> > 2017-05-03 13:53:39,234 (main) [ERROR - com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:230)]
Got exception running Sqoop: java.lang.IllegalArgumentException: Cannot convert SQL type 2009
> > java.lang.IllegalArgumentException: Cannot convert SQL type 2009
> > 	at org.apache.sqoop.manager.ConnManager.toAvroType(ConnManager.java:223)
> > 	at org.apache.sqoop.manager.ConnManager.toAvroType(ConnManager.java:277)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.toAvroType(AvroSchemaGenerator.java:166)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.toAvroSchema(AvroSchemaGenerator.java:136)
> > 	at org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchemaGenerator.java:96)
> > 	at org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroSchema(DataDrivenImportJob.java:156)
> > 	at org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:110)
> > 	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:264)
> > 	at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
> > 	at org.apache.sqoop.manager.Db2Manager.importTable(Db2Manager.java:72)
> > 	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
> > 	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
> > 	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> > 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > 	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> > 	at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:42)
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:228)
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:243)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.runDb2Test(DB2XmlTypeParquetImportManualTest.java:241)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.testParquetDb2Import(DB2XmlTypeParquetImportManualTest.java:212)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:498)
> > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> > 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> > 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
> > 2017-05-03 13:53:39,235 (main) [ERROR - org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.runDb2Test(DB2XmlTypeParquetImportManualTest.java:244)]
Got IOException during import: java.io.IOException: Failure during job; return status 1
> > java.io.IOException: Failure during job; return status 1
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:237)
> > 	at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:243)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.runDb2Test(DB2XmlTypeParquetImportManualTest.java:241)
> > 	at org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.testParquetDb2Import(DB2XmlTypeParquetImportManualTest.java:212)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:498)
> > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> > 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> > 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
> > 
> > Is there any extra setting which should be made before running maybe? I ran the
command below:
> > 
> > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty -Dsqoop.test.db2.connectstring.host_url=connectstring
-Dsqoop.test.db2.connectstring.database=databasename -Dsqoop.test.db2.connectstring.username=username
-Dsqoop.test.db2.connectstring.password=password -Dtestcase=DB2XmlTypeParquetImportManualTest
> > 
> > Thanks,
> > Bogi

Hi Bogi,

The error seems happened UT run using invalidate import argvs, but I can not reproduce in
my environment.

I have update an new patch to LOG the import argvs, it will LOG import args under string of
"runImport with argv is"


- Ying


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58477/#review173726
-----------------------------------------------------------


On 五月 8, 2017, 7:35 a.m., Ying Cao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58477/
> -----------------------------------------------------------
> 
> (Updated 五月 8, 2017, 7:35 a.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3173
>     https://issues.apache.org/jira/browse/SQOOP-3173
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> SQOOP-3173 : support DB2 xml data type when sqoop import with parquet
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/manager/ConnManager.java 1811ce0 
>   src/test/org/apache/sqoop/manager/db2/DB2XmlTypeParquetImportManualTest.java PRE-CREATION

> 
> 
> Diff: https://reviews.apache.org/r/58477/diff/3/
> 
> 
> Testing
> -------
> 
> UT passed by manually
> 
> This is third-part test, I ran it by :
> 
> ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty -Dsqoop.test.db2.connectstring.host_url=connectstring
-Dsqoop.test.db2.connectstring.database=databasename -Dsqoop.test.db2.connectstring.username=username
-Dsqoop.test.db2.connectstring.password=password -Dtestcase=DB2XmlTypeParquetImportManualTest
> 
> 
> Thanks,
> 
> Ying Cao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message