sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinoth Chandar <vin...@uber.com>
Subject Re: Install a new connector jar into sqoop2
Date Thu, 13 Nov 2014 16:31:19 GMT
Thanks Abe.

Right now, I am playing around with my connector. Will try out your
solution when I hit production.

On Wed, Nov 12, 2014 at 2:04 PM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Hey there,
>
> It seems like you've downloaded the Sqoop2 tarball. The common.loader
> should be fine.
>
> Here's an example setting:
>
> common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar
>
> With the above setting, you should be able to just drop your jar where the
> rest of the Sqoop jars are.
>
> I think the best way to do this is use bigtop or one of the Hadoop
> distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2.
>
> Sqoop uses tomcat currently and searches for connectors via classloaders.
>
> -Abe
>
> On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <vinoth@uber.com> wrote:
>
>> Thanks for the help!!..
>>
>> >> If you cannot use **any** connector, change “common.loader” to load
>> correct jars will help.
>> Can you please elaborate what you mean by 'any' connector? As I
>> mentioned, I placed by connector jar in common.loader and I only got a
>> NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector,
>> which is packaged into the WAR..
>>
>>
>> In other words, I think if my connector jar is part of the WAR, things
>> will work (effectively what I achieved by adding my connector also to the
>> pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats
>> the right way to do this..
>>
>> @Ravi , I tried the lib approach also (which is based off common.loader
>> again). did not help. Details below.
>>
>> I have the sqoop2 branch checked out.
>> 1) I first did the following to build out a binary package
>>     $ mvn clean package -DskipTests -Dbinary
>>
>> 2) Hopping into the install directory from there on
>> $ pwd
>> ~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200
>> $ ls server/lib/*connector*jar
>> server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib
>> $ jar tf ./server/target/sqoop.war | grep -i my # no match
>> $ jar tf ./server/webapps/sqoop.war | grep -i my # no match
>>
>> When I start up the server, it cannot find the SqoopConnector class which
>> my jar references since I am extending SqoopConnector
>>
>> Thanks
>> Vinoth
>>
>>
>> On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <qian.a.xu@intel.com> wrote:
>>
>>>  Hi Vinoth,
>>>
>>>
>>>
>>> If you cannot use **any** connector, change “common.loader” to load
>>> correct jars will help.
>>>
>>> If only your connector meets NoClassDefFound issue, it means your jars
>>> are not packaged with sqoop.war.
>>>
>>>
>>>
>>> --Stanley
>>>
>>>
>>>
>>> *From:* Vinoth Chandar [mailto:vinoth@uber.com]
>>> *Sent:* Wednesday, November 12, 2014 9:33 AM
>>> *To:* user@sqoop.apache.org
>>> *Subject:* Install a new connector jar into sqoop2
>>>
>>>
>>>
>>> Hey guys,
>>>
>>>
>>>
>>> I am working on a connector which connects to a sharded mysql cluster as
>>> the source. I built out the connector jar with its
>>> sqoopconnector.properties.
>>>
>>>
>>>
>>> And I tried to make sqoop pick up the jar a couple of ways
>>>
>>>
>>>
>>> 1) Add the path to the jar in "common.loader" property in
>>> catalina.properties: This gives a NoClassDefFound on the
>>> SqoopConnector.class since it is probably in tomcat's system loader, which
>>> may be referenceable by external libs.
>>>
>>>
>>>
>>> 2) Randomly placing the connector jar in shared/server loader, which
>>> also did not work.
>>>
>>>
>>>
>>> Wound up adding the connector as a dependency in sqoop-server's pom and
>>> it worked. (sort of validates the theory in 1)
>>>
>>>
>>>
>>> I am probably a tomcat noob, so please let me know if there is a cleaner
>>> way to do this or supported documentation .. Or if this is an open issue,
>>> happy to work on a patch..
>>>
>>> Thanks
>>>
>>> Vinoth
>>>
>>
>>
>

Mime
View raw message