sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daniel voros <daniel.vo...@gmail.com>
Subject Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open
Date Thu, 26 Apr 2018 09:23:31 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review201990
-----------------------------------------------------------



Hey Anna,

Thank you for addressing my previous concerns! As a final touch I've tried to compare the
contents of the tar.gz we're creating. I've used
```
ant tar
./gradlew tar
```

and ended up with the following differences (right side is gradle):

```
11a12
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-codegen.bat
12a14
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-create-hive-table.bat
13a16
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-eval.bat
14a18
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-export.bat
15a20
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-help.bat
17a23
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-import-all-tables.bat
18a25,26
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-import-mainframe.bat
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-import.bat
19a28
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-job.bat
20a30
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-list-databases.bat
21a32
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-list-tables.bat
22a34
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-merge.bat
23a36
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-metastore.bat
24a38
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/bin/sqoop-version.bat
72,88d85
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/BigDecimalSerializer.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/BlobRef.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/BooleanParser.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/ClobRef.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/DelimiterSet.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/FieldFormatter.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/FieldMapProcessor.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/FieldMappable.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/JdbcWritableBridge.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/LargeObjectLoader.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/LobRef.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/LobSerializer.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/ProcessingException.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/RecordParser.ParseError.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/RecordParser.html
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/class-use/SqoopRecord.html
92d88
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/docs/api/org/apache/sqoop/lib/package-use.html
163a160
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/avro-ipc-1.8.1.jar
165c162
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/commons-codec-1.4.jar
---
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/commons-codec-1.9.jar
189a187
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/parquet-hive-bundle-1.6.0.jar
191c189
< sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/slf4j-api-1.6.1.jar
---
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/lib/slf4j-api-1.7.7.jar
342a341
> sqoop-1.5.0-SNAPSHOT.bin__hadoop-2.6.0/src/java/org/apache/sqoop/SqoopVersion.java
```

These fall into the following four categories (in order):

1) Windows executables. There were some conditional parts in build.xml regarding windows.
I don't see these in our [latest release](http://www-us.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz).
Shouldn't we just get rid of all these?
2) Javadoc used to include class usage info (see [here](https://github.com/apache/sqoop/blob/72c5cd717e3fad6d5f5a3a2b3d185ffbacd876cf/build.xml#L1096).
I think the 'use' option does the same in gradle (see [here](https://docs.gradle.org/current/javadoc/org/gradle/external/javadoc/StandardJavadocDocletOptions.html#isUse--)).
3) Jar differences under `lib/`. This is what worries me the most. I've had trouble with having
(wrong version of) `parquet-hive-bundle` on the classpath. I think we should try to make contents
of lib/ the same. One difference I've spotted is that we're not excluding parquet-hive-bundle
from kite as we [used to](https://github.com/apache/sqoop/blob/72c5cd717e3fad6d5f5a3a2b3d185ffbacd876cf/ivy.xml#L118).
4) SqoopVersion.java is now included. I think it makes sense to keep it. Any objections?

Regards,
Daniel

- daniel voros


On April 24, 2018, 2:23 p.m., Anna Szonyi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> -----------------------------------------------------------
> 
> (Updated April 24, 2018, 2:23 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
>     https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer friendly
/ open
> 
> 
> Diffs
> -----
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
>   testdata/hcatalog/conf/hive-site.xml edac7aa9087a84b7a0c660907794adae684ae313 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/10/
> 
> 
> Testing
> -------
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and compare them
to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier to combine
all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as previously
- though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message