Vinoth,

We finally have this fixed in the SQOOP-1821. Kindly let us know if you run into other issues.

Thanks again for all the inputs.

--Veena




Best,
./Vee

On Thu, Nov 13, 2014 at 8:31 AM, Vinoth Chandar <vinoth@uber.com> wrote:
Thanks Abe. 

Right now, I am playing around with my connector. Will try out your solution when I hit production. 

On Wed, Nov 12, 2014 at 2:04 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
Hey there,

It seems like you've downloaded the Sqoop2 tarball. The common.loader should be fine.

Here's an example setting:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar

With the above setting, you should be able to just drop your jar where the rest of the Sqoop jars are.

I think the best way to do this is use bigtop or one of the Hadoop distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2.

Sqoop uses tomcat currently and searches for connectors via classloaders.

-Abe

On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <vinoth@uber.com> wrote:
Thanks for the help!!..

>> If you cannot use *any* connector, change “common.loader” to load correct jars will help.
Can you please elaborate what you mean by 'any' connector? As I mentioned, I placed by connector jar in common.loader and I only got a NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector, which is packaged into the WAR..


In other words, I think if my connector jar is part of the WAR, things will work (effectively what I achieved by adding my connector also to the pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats the right way to do this.. 

@Ravi , I tried the lib approach also (which is based off common.loader again). did not help. Details below. 

I have the sqoop2 branch checked out. 
1) I first did the following to build out a binary package
    $ mvn clean package -DskipTests -Dbinary

2) Hopping into the install directory from there on
$ pwd
~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200
$ ls server/lib/*connector*jar
server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib
$ jar tf ./server/target/sqoop.war | grep -i my # no match
$ jar tf ./server/webapps/sqoop.war | grep -i my # no match
 
When I start up the server, it cannot find the SqoopConnector class which my jar references since I am extending SqoopConnector 

Thanks
Vinoth


On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <qian.a.xu@intel.com> wrote:

Hi Vinoth,

 

If you cannot use *any* connector, change “common.loader” to load correct jars will help.

If only your connector meets NoClassDefFound issue, it means your jars are not packaged with sqoop.war.

 

--Stanley

 

From: Vinoth Chandar [mailto:vinoth@uber.com]
Sent: Wednesday, November 12, 2014 9:33 AM
To: user@sqoop.apache.org
Subject: Install a new connector jar into sqoop2

 

Hey guys, 

 

I am working on a connector which connects to a sharded mysql cluster as the source. I built out the connector jar with its sqoopconnector.properties.  

 

And I tried to make sqoop pick up the jar a couple of ways 

 

1) Add the path to the jar in "common.loader" property in catalina.properties: This gives a NoClassDefFound on the SqoopConnector.class since it is probably in tomcat's system loader, which may be referenceable by external libs.

 

2) Randomly placing the connector jar in shared/server loader, which also did not work. 

 

Wound up adding the connector as a dependency in sqoop-server's pom and it worked. (sort of validates the theory in 1)

 

I am probably a tomcat noob, so please let me know if there is a cleaner way to do this or supported documentation .. Or if this is an open issue, happy to work on a patch.. 

Thanks

Vinoth