sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Teoh <chris.t...@gmail.com>
Subject Re: Review Request 62653: SQOOP-3225: Mainframe module FTP listing parser should cater for larger datasets on disk
Date Thu, 19 Oct 2017 10:24:45 GMT


> On Sept. 29, 2017, 10:27 a.m., Boglarka Egyed wrote:
> > Hi Chris,
> > 
> > Thanks for your patch!
> > 
> > I ran unit tests with your patch and TestMainframeDatasetInputFormat test case failed
for me with the following error message:
> > 
> > [ERROR - org.apache.sqoop.util.MainframeFTPClientUtils.listSequentialDatasets(MainframeFTPClientUtils.java:64)]
MainframeDatasetPath helper class incorrectly initialised
> > java.lang.NullPointerException
> > 	at org.apache.sqoop.mapreduce.mainframe.MainframeDatasetPath.setMainframeDatasetType(MainframeDatasetPath.java:110)
> > 	at org.apache.sqoop.mapreduce.mainframe.MainframeDatasetPath.<init>(MainframeDatasetPath.java:54)
> > 	at org.apache.sqoop.util.MainframeFTPClientUtils.listSequentialDatasets(MainframeFTPClientUtils.java:61)
> > 	at org.apache.sqoop.mapreduce.mainframe.MainframeDatasetInputFormat.retrieveDatasets(MainframeDatasetInputFormat.java:93)
> > 	at org.apache.sqoop.mapreduce.mainframe.MainframeDatasetInputFormat.getSplits(MainframeDatasetInputFormat.java:67)
> > 	at org.apache.sqoop.mapreduce.mainframe.TestMainframeDatasetInputFormat.testRetrieveDatasets(TestMainframeDatasetInputFormat.java:106)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:498)
> > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> > 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> > 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
> > 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
> >     
> > Please review your changes there. Also, please avoid any reformatting of any file
as it makes difficult to review your change.
> > 
> > Thanks,
> > Bogi

Thanks for picking this up. I have updated the code so the unit test passes.


- Chris


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62653/#review186658
-----------------------------------------------------------


On Oct. 2, 2017, 11:12 p.m., Chris Teoh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62653/
> -----------------------------------------------------------
> 
> (Updated Oct. 2, 2017, 11:12 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3225
>     https://issues.apache.org/jira/browse/SQOOP-3225
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> There were some cases where the FTP listing parser was not used for mainframe datasets,
only the default was used. This caused some datasets to not be seen by Sqoop as the default
parser couldn't find them. This patch addresses this behaviour where it is used to parse FTP
listings for sequential and gdg datasets on disk and tape, only partitioned datasets are excluded
as their FTP listing looks very different and is handled by the default FTP listing parser.
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java f61b9838 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java d87c75df 
> 
> 
> Diff: https://reviews.apache.org/r/62653/diff/5/
> 
> 
> Testing
> -------
> 
> Unit testing.
> Functional testing on a real mainframe with a previous combined patch.
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message