sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anna Szonyi via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open
Date Fri, 13 Apr 2018 09:19:14 GMT


> On March 23, 2018, 4:33 p.m., Szabolcs Vasas wrote:
> > Hi Anna,
> > 
> > Awesome, thanks for the quick update.
> > 
> > I have tried running the third party tests today, it was successful, however there
is a small problem with the cleanup.
> > If I run ./gradlew test first and then the third party tests then org.apache.sqoop.manager.sqlserver.SQLServerHiveImportTest.testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
fails. This test case succeeds if I run the third party tests only after a ./gradlew clean.
I think the problem is that SQLServerHiveImportTest extends HiveImportTest which does not
do a proper Hive metastore database cleanup, so a table which has to be created by the test
already exists.
> > Anyway, changing the Hive metastore DB to an in-memory DB solves the problem so
I think can incorporate it into your patch (https://github.com/szvasas/sqoop/blob/gradletest2/testdata/hcatalog/conf/hive-site.xml#L37).
I am not sure why but I did not experience this issue when running the tests with ant.
> > 
> > An interesting thing I have noticed is that when I try to execute one test case
Gradle seems to start up lots of JVMs even after the test has successfully finished.
> > 
> > ./gradlew test --tests org.apache.sqoop.hive.TestHiveImport
> > :test
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support
was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support
was removed in 8.0
> > objc[10180]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support
was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support
was removed in 8.0
> > objc[10183]: Class JavaLaunchHelper is implemented in both 
> > ...
> > ...
> > ...
> > ...
> > 
> > org.apache.sqoop.hive.TestHiveImport > testDecimalMapColumnHive STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testDecimalMapColumnHive PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelims STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelims PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testGenerateOnly STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testGenerateOnly PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveExitFails STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveExitFails PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testDate STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testDate PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelimsReplacement STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelimsReplacement PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCustomDelimiters STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCustomDelimiters PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveDropAndReplaceOptionValidation
STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveDropAndReplaceOptionValidation
PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImport STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImport PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateHiveImportAsParquet STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOnlyHiveImport STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOnlyHiveImport PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testAppendHiveImportAsParquet STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testAppendHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImportAsParquet
STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImportAsParquet
PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNumeric STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNumeric PASSED
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support
was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support
was removed in 8.0
> > objc[11118]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support
was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support
was removed in 8.0
> > objc[11119]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
> > ...
> > ...
> > 
> > BUILD SUCCESSFUL
> > 
> > Total time: 2 mins 54.654 secs
> > 
> > 
> > I am not sure what happens here, my guess is that a new JVM is started for every
test case even if they are not executed. Have you experienced this problem too?
> 
> Anna Szonyi wrote:
>     I've had similar issues running with the --tests option, I usually use -Dtest.single,
unfortunately if you have a buildSrc folder in your project that seems to mess everything
up for test.single, so I can rename that directory to solve this for now. Oddly enough it
doesn't affect thirdParty.single.
>     Maybe in the long run we can also try to make test runs independent, so we don't
have to use forkevery 1...

Hey Szabi,

Please try with test.single with the new patch - you shouldn't face the same issues!

Thanks,
Anna


- Anna


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199896
-----------------------------------------------------------


On April 13, 2018, 9:06 a.m., Anna Szonyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> -----------------------------------------------------------
> 
> (Updated April 13, 2018, 9:06 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
>     https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer friendly
/ open
> 
> 
> Diffs
> -----
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/7/
> 
> 
> Testing
> -------
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and compare them
to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier to combine
all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as previously
- though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message