sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jane Wayne <jane.wayne2...@gmail.com>
Subject Re: can we use -libjars with the Sqoop tool?
Date Thu, 06 Jun 2013 20:11:09 GMT
venkat,

could you clarify? do you mean to not use the hadoop jar command (option
1)? or to use the hadoop jar command and omit -config (option 2)?

option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop

option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
mypass --table data --verbose --append --warehouse-dir /sqoop

i need -libjars because i am trying to run sqoop from an edge node that is
not part of the hadoop cluster. is this even possible?

i posted this question on the dev mailing list, since i thought i might
have to modify sqoop. but what i am trying to explore is this scenario.

W: web server (JEE container, tomcat)
H: hadoop cluster
D: database server (MySQL)

i need to invoke sqoop on W to import data from D to H. i read that sqoop2
has a client API, which should have met my requirements. however, i am not
allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
cluster).

i am opened to any approach subject to that one constraint (cannot install
sqoop onto the hadoop cluster or put third-party libs onto the hadoop
cluster).

thanks.



On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:

> Don't use hadoop -config ...
>
> Use
>  sqoop import -libjars ....   And I am not sure why you need libjars
> option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
> and just run sqoop import
>
>
> Venkat
>
>
> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <jane.wayne2978@gmail.com>wrote:
>
>> i am running the Sqoop tool as follows.
>>
>> export
>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>> --verbose --append --warehouse-dir /sqoop
>>
>> i am getting this stack trace.
>>
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>         at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>         at
>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>
>> when i looked at the sqoop-1.4.3.jar, clearly,
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>>
>> any ideas what's going on?
>>
>>
>>
>>
>> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
>> vranganathan@hortonworks.com> wrote:
>>
>>> sqoop tools  implement the ToolRunner interface.   So, you can pass
>>> -libjars to sqoop (but make sure it is the first option among the list of
>>> the options
>>>
>>> For example, sqoop import -libjars $LIBJARS other options
>>>
>>> Venkat
>>>
>>>
>>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <jane.wayne2978@gmail.com>wrote:
>>>
>>>> hi,
>>>>
>>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>>> tools from the command line using the -libjars flag/option/parameter?
>>>>
>>>> i have tried the following.
>>>>
>>>> export
>>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>>
>>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>>> --verbose --append --warehouse-dir /sqoop
>>>>
>>>> what i get is the message below.
>>>>
>>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>>
>>>> obviously, this problem has something to do with sqoop's parsing of the
>>>> command line options. shouldn't sqoop allow hadoop's command line options
>>>> and its own at the same time?
>>>>
>>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>>> see if i can get it to work with -libjars.
>>>>
>>>> references that i found regarding -libjars.
>>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>>> 2.
>>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>>>
>>>> thanks,
>>>>
>>>
>>>
>>
>

Mime
View raw message