sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh Sharma <raksha...@expedia.com>
Subject Custom Options for User defined plugins for sqoop
Date Thu, 30 Jul 2015 05:58:52 GMT
Hi,

Sqoop is based on Hadoop Tools infrastructure, which allows users to develop their own plugins
for sqoop and extend sqoop as per their requirements without changing the base sqoop. Users
can develop their plugins as separate jars, deploy them in $SQOOP_LIB and register with sqoop
as :

<property>
    <name>sqoop.tool.plugins</name>
    <value>com.expedia.sqoop.tool.HiveMergePlugin</value>
    <description>A comma-delimited list of ToolPlugin implementations
      which are consulted, in order, to register SqoopTool instances which
      allow third-party tools to be used.
    </description>
</property>

This functionality is already there and we are able to use this develop our own plugins. Infact
there are a couple of teams in Expedia which are using their own plugins with sqoop. I am
sure there might be others who will be using this or willing to use this functionality.. Our
Team have also a tool called hive-merge for sqoop which merges two hive tables,  but this
tool needs to use some option names which are not available in SqoopOptions. The limitation
with user defined plugin is that it doesn't support custom defined options as all the sqoop
options for sqoop are tied to names and types in SqoopOptions. We have made a small enhancement
which adds a map member to SqoopOptions. This is a very small change but it opens up Sqoop
for user defined tools with custom options. It will open up sqoop for user defined plugins.
We have also added support for its persistence into metastore. The patch is already available
for the issue :
https://issues.apache.org/jira/browse/SQOOP-2333

The review request for the fix is available at :
https://reviews.apache.org/r/36042/

Although the change is small, it will be a great addition to Sqoop and will enable developers
to develop their own plugins with user defined options and contribute back to the community
without knowing much details of sqoop. We have also added documentation about how to develop
user defined plugins and custom options for user defined plugins.

Please review and suggest.

Thanks and Regards,
Rakesh.



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message