sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Install a new connector jar into sqoop2
Date Wed, 12 Nov 2014 22:04:48 GMT
Hey there,

It seems like you've downloaded the Sqoop2 tarball. The common.loader
should be fine.

Here's an example setting:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar

With the above setting, you should be able to just drop your jar where the
rest of the Sqoop jars are.

I think the best way to do this is use bigtop or one of the Hadoop
distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2.

Sqoop uses tomcat currently and searches for connectors via classloaders.

-Abe

On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <vinoth@uber.com> wrote:

> Thanks for the help!!..
>
> >> If you cannot use **any** connector, change “common.loader” to load
> correct jars will help.
> Can you please elaborate what you mean by 'any' connector? As I mentioned,
> I placed by connector jar in common.loader and I only got a
> NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector,
> which is packaged into the WAR..
>
>
> In other words, I think if my connector jar is part of the WAR, things
> will work (effectively what I achieved by adding my connector also to the
> pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats
> the right way to do this..
>
> @Ravi , I tried the lib approach also (which is based off common.loader
> again). did not help. Details below.
>
> I have the sqoop2 branch checked out.
> 1) I first did the following to build out a binary package
>     $ mvn clean package -DskipTests -Dbinary
>
> 2) Hopping into the install directory from there on
> $ pwd
> ~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200
> $ ls server/lib/*connector*jar
> server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib
> $ jar tf ./server/target/sqoop.war | grep -i my # no match
> $ jar tf ./server/webapps/sqoop.war | grep -i my # no match
>
> When I start up the server, it cannot find the SqoopConnector class which
> my jar references since I am extending SqoopConnector
>
> Thanks
> Vinoth
>
>
> On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <qian.a.xu@intel.com> wrote:
>
>>  Hi Vinoth,
>>
>>
>>
>> If you cannot use **any** connector, change “common.loader” to load
>> correct jars will help.
>>
>> If only your connector meets NoClassDefFound issue, it means your jars
>> are not packaged with sqoop.war.
>>
>>
>>
>> --Stanley
>>
>>
>>
>> *From:* Vinoth Chandar [mailto:vinoth@uber.com]
>> *Sent:* Wednesday, November 12, 2014 9:33 AM
>> *To:* user@sqoop.apache.org
>> *Subject:* Install a new connector jar into sqoop2
>>
>>
>>
>> Hey guys,
>>
>>
>>
>> I am working on a connector which connects to a sharded mysql cluster as
>> the source. I built out the connector jar with its
>> sqoopconnector.properties.
>>
>>
>>
>> And I tried to make sqoop pick up the jar a couple of ways
>>
>>
>>
>> 1) Add the path to the jar in "common.loader" property in
>> catalina.properties: This gives a NoClassDefFound on the
>> SqoopConnector.class since it is probably in tomcat's system loader, which
>> may be referenceable by external libs.
>>
>>
>>
>> 2) Randomly placing the connector jar in shared/server loader, which also
>> did not work.
>>
>>
>>
>> Wound up adding the connector as a dependency in sqoop-server's pom and
>> it worked. (sort of validates the theory in 1)
>>
>>
>>
>> I am probably a tomcat noob, so please let me know if there is a cleaner
>> way to do this or supported documentation .. Or if this is an open issue,
>> happy to work on a patch..
>>
>> Thanks
>>
>> Vinoth
>>
>
>

Mime
View raw message