sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abhijeet gaikwad <abygaikwa...@gmail.com>
Subject Re: can we use -libjars with the Sqoop tool?
Date Sat, 08 Jun 2013 05:05:18 GMT
With the approach you are taking, see if this helps:
export HADOOP_CLASSPATH=$LIBJARS

-libjars makes sure each MR task gets these jars in classpath but for the
local machine that kick starts hadoop job does not get these jars, so the
CNFE. You need to explicitly do that as above.

A better solution:
If your edge node can submit a hadoop job to the cluster, I think this
should be easy: Install Sqoop on Edge node and run the command that Venkat
suggested.

Let me know how it goes!

Thanks,
Abhijeet


On Fri, Jun 7, 2013 at 1:41 AM, Jane Wayne <jane.wayne2978@gmail.com> wrote:

> venkat,
>
> could you clarify? do you mean to not use the hadoop jar command (option
> 1)? or to use the hadoop jar command and omit -config (option 2)?
>
> option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
> sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
> ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
> mypass --table data --verbose --append --warehouse-dir /sqoop
>
> i need -libjars because i am trying to run sqoop from an edge node that is
> not part of the hadoop cluster. is this even possible?
>
> i posted this question on the dev mailing list, since i thought i might
> have to modify sqoop. but what i am trying to explore is this scenario.
>
> W: web server (JEE container, tomcat)
> H: hadoop cluster
> D: database server (MySQL)
>
> i need to invoke sqoop on W to import data from D to H. i read that sqoop2
> has a client API, which should have met my requirements. however, i am not
> allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
> cluster).
>
> i am opened to any approach subject to that one constraint (cannot install
> sqoop onto the hadoop cluster or put third-party libs onto the hadoop
> cluster).
>
> thanks.
>
>
>
> On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> Don't use hadoop -config ...
>>
>> Use
>>  sqoop import -libjars ....   And I am not sure why you need libjars
>> option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
>> and just run sqoop import
>>
>>
>> Venkat
>>
>>
>> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <jane.wayne2978@gmail.com>wrote:
>>
>>> i am running the Sqoop tool as follows.
>>>
>>> export
>>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> i am getting this stack trace.
>>>
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>>         at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>>         at
>>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>>>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>
>>> when i looked at the sqoop-1.4.3.jar, clearly,
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>>>
>>> any ideas what's going on?
>>>
>>>
>>>
>>>
>>> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
>>> vranganathan@hortonworks.com> wrote:
>>>
>>>> sqoop tools  implement the ToolRunner interface.   So, you can pass
>>>> -libjars to sqoop (but make sure it is the first option among the list of
>>>> the options
>>>>
>>>> For example, sqoop import -libjars $LIBJARS other options
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <jane.wayne2978@gmail.com>wrote:
>>>>
>>>>> hi,
>>>>>
>>>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>>>> tools from the command line using the -libjars flag/option/parameter?
>>>>>
>>>>> i have tried the following.
>>>>>
>>>>> export
>>>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>>>
>>>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>>>> --verbose --append --warehouse-dir /sqoop
>>>>>
>>>>> what i get is the message below.
>>>>>
>>>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>>>
>>>>> obviously, this problem has something to do with sqoop's parsing of
>>>>> the command line options. shouldn't sqoop allow hadoop's command line
>>>>> options and its own at the same time?
>>>>>
>>>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>>>> see if i can get it to work with -libjars.
>>>>>
>>>>> references that i found regarding -libjars.
>>>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>>>> 2.
>>>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>>>>
>>>>> thanks,
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message