sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Cecho" <jar...@apache.org>
Subject Re: Review Request 14085: Review request for SQOOP-1192 Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib folder when launched by Oozie and use Oozie share lib
Date Mon, 14 Oct 2013 22:15:55 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14085/#review26986
-----------------------------------------------------------

Ship it!


Hi sir, please just fix the following two nits and publish the updated patch on the JIRA,
I'll be more then happy to commit it!


src/docs/user/import.txt
<https://reviews.apache.org/r/14085/#comment52554>

    Nit: please use the linux notation $SQOOP_HOME instead of the windows notation %SQOOP_HOME%
(entire guide is written in a way to be compatible with linux, so it would be great to stay
consistent).



src/docs/user/import.txt
<https://reviews.apache.org/r/14085/#comment52553>

    Nit: s/massave/massive/


Jarcec

- Jarek Cecho


On Oct. 14, 2013, 9:31 p.m., Shuaishuai Nie wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14085/
> -----------------------------------------------------------
> 
> (Updated Oct. 14, 2013, 9:31 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1192
>     https://issues.apache.org/jira/browse/SQOOP-1192
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> Now Sqoop will copy jar files in %SQOOP_HOME%\lib folder to the job cache every time
a Sqoop job is launched. When Oozie launch a Sqoop job, this behavior can be optimized by
add these jars in Oozie Sqoop sharelib. In this case, the jar files in share lib only needed
be localized to each worker node once and reuse by all Sqoop job launched by Oozie. This can
reduce massive disk I/O on worker node when using Sqoop by Oozie. To enable this, Sqoop need
to have an option which enable the job to skip adding lib jars to the job cache. For now,
this option should only be used by Oozie started Sqoop job. The patch attached introduce "--skip-dist-cache"
option to enable this feature.
> 
> 
> Diffs
> -----
> 
>   src/docs/user/import.txt 71b50d8 
>   src/java/org/apache/sqoop/SqoopOptions.java 01805f9 
>   src/java/org/apache/sqoop/mapreduce/JobBase.java 322df1c 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java b05f587 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java ebb1857 
>   src/test/com/cloudera/sqoop/TestSqoopOptions.java 03e2504 
> 
> Diff: https://reviews.apache.org/r/14085/diff/
> 
> 
> Testing
> -------
> 
> Tested the new option with Oozie-Sqoop workflow to ensure it doesn't break Sqoop library
dependencies when launched by Oozie
> 
> 
> Thanks,
> 
> Shuaishuai Nie
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message