sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Vohra <dvohr...@yahoo.com>
Subject Re: can we use -libjars with the Sqoop tool?
Date Thu, 06 Jun 2013 13:28:55 GMT
can we use -libjars with the Sqoop tool?


Yes.
-libjars <comma separated list of jars>    specify comma separated jar files to include
in the classpath.

________________________________
 From: Jane Wayne <jane.wayne2978@gmail.com>
To: user@sqoop.apache.org 
Sent: Thursday, June 6, 2013 5:56 AM
Subject: Re: can we use -libjars with the Sqoop tool?
 


i am running the Sqoop tool as follows.

export LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar

hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop
import -libjars ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
mypass --table data --verbose --append --warehouse-dir /sqoop

i am getting this stack trace.

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
        at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

when i looked at the sqoop-1.4.3.jar, clearly, org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class
exists. 

any ideas what's going on?





On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <vranganathan@hortonworks.com> wrote:

sqoop tools  implement the ToolRunner interface.   So, you can pass -libjars to sqoop (but
make sure it is the first option among the list of the options
>
>
>For example, sqoop import -libjars $LIBJARS other options
>
>Venkat
>
>
>
>On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <jane.wayne2978@gmail.com> wrote:
>
>hi,
>>
>>
>>i am wondering if i can run the org.apache.sqoop.Sqoop tool like other tools from
the command line using the -libjars flag/option/parameter?
>>
>>
>>i have tried the following.
>>
>>
>>export LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>>
>>hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop
-libjars ${LIBJARS} import --connect jdbc:mysql://mysql/mydb --username myuser --password
mypass --table data --verbose --append --warehouse-dir /sqoop  
>>
>>
>>what i get is the message below.
>>
>>
>>No such sqoop tool: -libjars. See 'sqoop help'.
>>
>>
>>
>>obviously, this problem has something to do with sqoop's parsing of the command line
options. shouldn't sqoop allow hadoop's command line options and its own at the same time?
>>
>>
>>in the meanwhile, i will try to tinker a bit with that Sqoop class to see if i can
get it to work with -libjars.
>>
>>
>>references that i found regarding -libjars.
>>1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>2. http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>
>>
>>thanks,
>
Mime
View raw message