sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1192) Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib folder when launched by Oozie and use Oozie share lib
Date Mon, 14 Oct 2013 23:02:42 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794605#comment-13794605
] 

Hudson commented on SQOOP-1192:
-------------------------------

SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #821 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/821/])
SQOOP-1192: Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib
folder when launched by Oozie and use Oozie share lib (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=84071181265f98959ffdfc41425022f8251d2429)
* src/docs/user/import.txt
* src/java/org/apache/sqoop/mapreduce/TextExportMapper.java
* src/test/com/cloudera/sqoop/TestSqoopOptions.java
* src/java/org/apache/sqoop/tool/BaseSqoopTool.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/java/org/apache/sqoop/mapreduce/JobBase.java
* src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java


> Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib folder
when launched by Oozie and use Oozie share lib
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-1192
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1192
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Shuaishuai Nie
>            Assignee: Shuaishuai Nie
>             Fix For: 1.4.5
>
>         Attachments: SQOOP-1192.1.patch, SQOOP-1192.2.patch, SQOOP-1192.3.patch
>
>
> Now Sqoop will copy jar files in %SQOOP_HOME%\lib folder to the job cache every time
a Sqoop job is launched. When Oozie launch a Sqoop job, this behavior can be optimized by
add these jars in Oozie Sqoop sharelib. In this case, the jar files in share lib only needed
be localized to each worker node once and reuse by all Sqoop job launched by Oozie. This can
reduce massive disk I/O on worker node when using Sqoop by Oozie. To enable this, Sqoop need
to have an option which enable the job to skip adding lib jars to the job cache. For now,
this option should only be used by Oozie started Sqoop job. The patch attached introduce "--skip-dist-cache"
option to enable this feature.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message