sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Veena Basavaraj <vbasava...@cloudera.com>
Subject Re: Install a new connector jar into sqoop2
Date Wed, 21 Jan 2015 17:27:13 GMT
Vinoth,

We finally have this fixed in the SQOOP-1821. Kindly let us know if you run
into other issues.

Thanks again for all the inputs.

--Veena




Best,
*./Vee*

On Thu, Nov 13, 2014 at 8:31 AM, Vinoth Chandar <vinoth@uber.com> wrote:

> Thanks Abe.
>
> Right now, I am playing around with my connector. Will try out your
> solution when I hit production.
>
> On Wed, Nov 12, 2014 at 2:04 PM, Abraham Elmahrek <abe@cloudera.com>
> wrote:
>
>> Hey there,
>>
>> It seems like you've downloaded the Sqoop2 tarball. The common.loader
>> should be fine.
>>
>> Here's an example setting:
>>
>> common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar
>>
>> With the above setting, you should be able to just drop your jar where
>> the rest of the Sqoop jars are.
>>
>> I think the best way to do this is use bigtop or one of the Hadoop
>> distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2.
>>
>> Sqoop uses tomcat currently and searches for connectors via classloaders.
>>
>> -Abe
>>
>> On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <vinoth@uber.com> wrote:
>>
>>> Thanks for the help!!..
>>>
>>> >> If you cannot use **any** connector, change “common.loader” to load
>>> correct jars will help.
>>> Can you please elaborate what you mean by 'any' connector? As I
>>> mentioned, I placed by connector jar in common.loader and I only got a
>>> NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector,
>>> which is packaged into the WAR..
>>>
>>>
>>> In other words, I think if my connector jar is part of the WAR, things
>>> will work (effectively what I achieved by adding my connector also to the
>>> pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats
>>> the right way to do this..
>>>
>>> @Ravi , I tried the lib approach also (which is based off common.loader
>>> again). did not help. Details below.
>>>
>>> I have the sqoop2 branch checked out.
>>> 1) I first did the following to build out a binary package
>>>     $ mvn clean package -DskipTests -Dbinary
>>>
>>> 2) Hopping into the install directory from there on
>>> $ pwd
>>> ~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200
>>> $ ls server/lib/*connector*jar
>>> server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib
>>> $ jar tf ./server/target/sqoop.war | grep -i my # no match
>>> $ jar tf ./server/webapps/sqoop.war | grep -i my # no match
>>>
>>> When I start up the server, it cannot find the SqoopConnector class
>>> which my jar references since I am extending SqoopConnector
>>>
>>> Thanks
>>> Vinoth
>>>
>>>
>>> On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <qian.a.xu@intel.com> wrote:
>>>
>>>>  Hi Vinoth,
>>>>
>>>>
>>>>
>>>> If you cannot use **any** connector, change “common.loader” to load
>>>> correct jars will help.
>>>>
>>>> If only your connector meets NoClassDefFound issue, it means your jars
>>>> are not packaged with sqoop.war.
>>>>
>>>>
>>>>
>>>> --Stanley
>>>>
>>>>
>>>>
>>>> *From:* Vinoth Chandar [mailto:vinoth@uber.com]
>>>> *Sent:* Wednesday, November 12, 2014 9:33 AM
>>>> *To:* user@sqoop.apache.org
>>>> *Subject:* Install a new connector jar into sqoop2
>>>>
>>>>
>>>>
>>>> Hey guys,
>>>>
>>>>
>>>>
>>>> I am working on a connector which connects to a sharded mysql cluster
>>>> as the source. I built out the connector jar with its
>>>> sqoopconnector.properties.
>>>>
>>>>
>>>>
>>>> And I tried to make sqoop pick up the jar a couple of ways
>>>>
>>>>
>>>>
>>>> 1) Add the path to the jar in "common.loader" property in
>>>> catalina.properties: This gives a NoClassDefFound on the
>>>> SqoopConnector.class since it is probably in tomcat's system loader, which
>>>> may be referenceable by external libs.
>>>>
>>>>
>>>>
>>>> 2) Randomly placing the connector jar in shared/server loader, which
>>>> also did not work.
>>>>
>>>>
>>>>
>>>> Wound up adding the connector as a dependency in sqoop-server's pom and
>>>> it worked. (sort of validates the theory in 1)
>>>>
>>>>
>>>>
>>>> I am probably a tomcat noob, so please let me know if there is a
>>>> cleaner way to do this or supported documentation .. Or if this is an open
>>>> issue, happy to work on a patch..
>>>>
>>>> Thanks
>>>>
>>>> Vinoth
>>>>
>>>
>>>
>>
>

Mime
View raw message