sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Szabolcs Vasas <vasas.szabo...@gmail.com>
Subject Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open
Date Fri, 23 Mar 2018 16:33:07 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199896
-----------------------------------------------------------



Hi Anna,

Awesome, thanks for the quick update.

I have tried running the third party tests today, it was successful, however there is a small
problem with the cleanup.
If I run ./gradlew test first and then the third party tests then org.apache.sqoop.manager.sqlserver.SQLServerHiveImportTest.testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
fails. This test case succeeds if I run the third party tests only after a ./gradlew clean.
I think the problem is that SQLServerHiveImportTest extends HiveImportTest which does not
do a proper Hive metastore database cleanup, so a table which has to be created by the test
already exists.
Anyway, changing the Hive metastore DB to an in-memory DB solves the problem so I think can
incorporate it into your patch (https://github.com/szvasas/sqoop/blob/gradletest2/testdata/hcatalog/conf/hive-site.xml#L37).
I am not sure why but I did not experience this issue when running the tests with ant.

An interesting thing I have noticed is that when I try to execute one test case Gradle seems
to start up lots of JVMs even after the test has successfully finished.

./gradlew test --tests org.apache.sqoop.hive.TestHiveImport
:test
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed
in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed
in 8.0
objc[10180]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed
in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed
in 8.0
objc[10183]: Class JavaLaunchHelper is implemented in both 
...
...
...
...

org.apache.sqoop.hive.TestHiveImport > testDecimalMapColumnHive STARTED

org.apache.sqoop.hive.TestHiveImport > testDecimalMapColumnHive PASSED

org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelims STARTED

org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelims PASSED

org.apache.sqoop.hive.TestHiveImport > testGenerateOnly STARTED

org.apache.sqoop.hive.TestHiveImport > testGenerateOnly PASSED

org.apache.sqoop.hive.TestHiveImport > testHiveExitFails STARTED

org.apache.sqoop.hive.TestHiveImport > testHiveExitFails PASSED

org.apache.sqoop.hive.TestHiveImport > testDate STARTED

org.apache.sqoop.hive.TestHiveImport > testDate PASSED

org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelimsReplacement STARTED

org.apache.sqoop.hive.TestHiveImport > testFieldWithHiveDelimsReplacement PASSED

org.apache.sqoop.hive.TestHiveImport > testCustomDelimiters STARTED

org.apache.sqoop.hive.TestHiveImport > testCustomDelimiters PASSED

org.apache.sqoop.hive.TestHiveImport > testHiveDropAndReplaceOptionValidation STARTED

org.apache.sqoop.hive.TestHiveImport > testHiveDropAndReplaceOptionValidation PASSED

org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImport STARTED

org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImport PASSED

org.apache.sqoop.hive.TestHiveImport > testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
STARTED

org.apache.sqoop.hive.TestHiveImport > testHiveImportAsParquetWhenTableExistsWithIncompatibleSchema
PASSED

org.apache.sqoop.hive.TestHiveImport > testCreateHiveImportAsParquet STARTED

org.apache.sqoop.hive.TestHiveImport > testCreateHiveImportAsParquet PASSED

org.apache.sqoop.hive.TestHiveImport > testCreateOnlyHiveImport STARTED

org.apache.sqoop.hive.TestHiveImport > testCreateOnlyHiveImport PASSED

org.apache.sqoop.hive.TestHiveImport > testAppendHiveImportAsParquet STARTED

org.apache.sqoop.hive.TestHiveImport > testAppendHiveImportAsParquet PASSED

org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport STARTED

org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport PASSED

org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet STARTED

org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet PASSED

org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey STARTED

org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey PASSED

org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImportAsParquet STARTED

org.apache.sqoop.hive.TestHiveImport > testCreateOverwriteHiveImportAsParquet PASSED

org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions STARTED

org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions PASSED

org.apache.sqoop.hive.TestHiveImport > testNumeric STARTED

org.apache.sqoop.hive.TestHiveImport > testNumeric PASSED
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed
in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed
in 8.0
objc[11118]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed
in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed
in 8.0
objc[11119]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
...
...

BUILD SUCCESSFUL

Total time: 2 mins 54.654 secs


I am not sure what happens here, my guess is that a new JVM is started for every test case
even if they are not executed. Have you experienced this problem too?

- Szabolcs Vasas


On March 23, 2018, 10:28 a.m., Anna Szonyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> -----------------------------------------------------------
> 
> (Updated March 23, 2018, 10:28 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
>     https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer friendly
/ open
> 
> 
> Diffs
> -----
> 
>   .gitignore 68cbe28 
>   COMPILING.txt 3b82250 
>   build.gradle PRE-CREATION 
>   buildSrc/customUnixStartScript.txt PRE-CREATION 
>   buildSrc/customWindowsStartScript.txt PRE-CREATION 
>   buildSrc/sqoop-package.gradle PRE-CREATION 
>   buildSrc/sqoop-version-gen.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/6/
> 
> 
> Testing
> -------
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and compare them
to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier to combine
all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as previously
- though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message