sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhanmohan Savadamuthu <ermad...@gmail.com>
Subject Re: Sqoop 1.99.2 + Cloud Era Hadoop 200 error
Date Fri, 05 Jul 2013 18:42:40 GMT
Mengwei,

Issue is solved. MapReduce configuration file had invalid configuration and
was creating problem.

*File Name:* /etc/hadoop/conf/mapred-site.xml
*Parameter Name:* mapred.job.tracker
*Original Value: *neededForHive:999999
*Modified Value: *<machinename>:9001

After this change, this are working fine now. Note that I have followed
suggestions provided by Mengwei in this thread.

Regards,
Madhan

On Thu, Jul 4, 2013 at 11:08 PM, Mengwei Ding <mengwei.ding@cloudera.com>wrote:

> Ok, Madhan, why not.
>
> Could you kindly provide you availabilities of time and communication
> tools. I will be more than happy to help you out with this.
>
> Best,
> Mengwei
>
>
> On Thu, Jul 4, 2013 at 1:15 AM, Madhanmohan Savadamuthu <
> ermadhan@gmail.com> wrote:
>
>> After doing changes in catalina.properties also, same issue is coming.
>>
>> is there any possibility for interactive discussion on this issue?
>>
>> Regards,
>> Madhan
>>
>>  On Wed, Jul 3, 2013 at 11:05 PM, Mengwei Ding <mengwei.ding@cloudera.com
>> > wrote:
>>
>>> Thank you for your prompt response, sir. Please don't worry, I can help
>>> you out with this until your problem is done.
>>>
>>> Well, let's try out our new method of adding dependency jar files, and
>>> forget about the addtowar.sh script.
>>>
>>> Please following these instructions:
>>>
>>>  "
>>> Installing Dependencies
>>>
>>> Hadoop libraries must be available on node where you are planning to run
>>> Sqoop server with proper configuration for major services - NameNode and
>>> either JobTracker or ResourceManager depending whether you are running
>>> Hadoop 1 or 2. There is no need to run any Hadoop service on the same node
>>> as Sqoop server, just the libraries and configuration must be available.
>>>
>>> Path to Hadoop libraries is stored in file catalina.properties inside
>>> directory server/conf. You need to change property called common.loader to
>>> contain all directories with your Hadoop libraries. The default expected
>>> locations are /usr/lib/hadoop and /usr/lib/hadoop/lib/. Please check out
>>> the comments in the file for further description how to configure different
>>> locations.
>>>
>>> Lastly you might need to install JDBC drivers that are not bundled with
>>> Sqoop because of incompatible licenses. You can add any arbitrary Java jar
>>> file to Sqoop server by copying it into lib/ directory. You can create this
>>> directory if it do not exists already.
>>> "
>>>
>>> I can give you my configuration as an example. So in my
>>> catalina.properties file, I have the following line:
>>>
>>> *
>>> common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/client-0.20/*.jar,/home/mengweid/Downloads/mysql-connector-java-5.1.25-bin.jar
>>> *
>>>
>>> The */usr/lib/hadoop/client-0.20/*.jar *is used to include all
>>> hadoop-related jars, and *mysql-connector-java-5.1.25-bin.jar *is used
>>> for JDBC driver.
>>>
>>> Please try this, and let me know whether it works. Thank you.
>>>
>>> Best,
>>> Mengwei
>>>
>>>
>>> On Wed, Jul 3, 2013 at 9:18 AM, Madhanmohan Savadamuthu <
>>> ermadhan@gmail.com> wrote:
>>>
>>>> I did deployment as sugggested in below thread. I am not able to
>>>> successfully use sqoop2. I am attaching the services log for your
>>>> references.
>>>>
>>>> I made sure that exact same set of JAR files in appropriate location
>>>> and also deleted sqoop folder before starting the sqoop server.
>>>>
>>>> *Error Message:*
>>>>  Exception has occurred during processing command
>>>> Exception: com.sun.jersey.api.client.UniformInterfaceException Message:
>>>> GET http://<ipaddress>:12013/sqoop/version returned a response status
>>>> of 404 Not Found
>>>>
>>>> Regards,
>>>> Madhan
>>>>
>>>>  On Wed, Jul 3, 2013 at 7:30 PM, Mengwei Ding <
>>>> mengwei.ding@cloudera.com> wrote:
>>>>
>>>>> Hi Madhanmohan,
>>>>>
>>>>> Thank you for providing all these detailed information. Help a lot to
>>>>> diagnose the problem.
>>>>>
>>>>> First, the addtowar.sh is not good enough for every situation,
>>>>> we apologize for that. We have already figured out a new way to add
>>>>> dependency library, which will coming out along with next version of
Sqoop2.
>>>>>
>>>>> Currently, it seems like the hadoop-core.jar has not been added. I
>>>>> could show you all the libraries existing in the
>>>>> webapps/sqoop/WEB-INF/lib folder, please check below:
>>>>>  avro-1.7.4.jar
>>>>>  commons-cli-1.2.jar
>>>>> commons-configuration-1.6.jar
>>>>> commons-dbcp-1.4.jar
>>>>> commons-lang-2.5.jar
>>>>> commons-logging-1.1.1.jar
>>>>> commons-pool-1.5.4.jar
>>>>> derby-10.8.2.2.jar
>>>>> guava-11.0.2.jar
>>>>> hadoop-auth-2.0.0-cdh4.3.0.jar
>>>>> hadoop-common-2.0.0-cdh4.3.0.jar
>>>>> *hadoop-core-2.0.0-mr1-cdh4.3.0.jar*
>>>>> hadoop-hdfs-2.0.0-cdh4.3.0.jar
>>>>> hadoop-mapreduce-client-app-2.0.0-cdh4.3.0.jar
>>>>> hadoop-mapreduce-client-common-2.0.0-cdh4.3.0.jar
>>>>> hadoop-mapreduce-client-core-2.0.0-cdh4.3.0.jar
>>>>> hadoop-mapreduce-client-jobclient-2.0.0-cdh4.3.0.jar
>>>>> hadoop-yarn-api-2.0.0-cdh4.3.0.jar
>>>>> hadoop-yarn-common-2.0.0-cdh4.3.0.jar
>>>>>  jackson-core-asl-1.8.8.jar
>>>>> jackson-mapper-asl-1.8.8.jar
>>>>> json-simple-1.1.jar
>>>>> log4j-1.2.16.jar
>>>>> mysql-connector-java-5.1.25-bin.jar
>>>>> protobuf-java-2.4.0a.jar
>>>>> slf4j-api-1.6.1.jar
>>>>> slf4j-log4j12-1.6.1.jar
>>>>> sqoop-common-1.99.2.jar
>>>>> sqoop-connector-generic-jdbc-1.99.2.jar
>>>>> sqoop-core-1.99.2.jar
>>>>> sqoop-execution-mapreduce-1.99.2-hadoop200.jar
>>>>> sqoop-repository-derby-1.99.2.jar
>>>>> sqoop-spi-1.99.2.jar
>>>>> sqoop-submission-mapreduce-1.99.2-hadoop200.jar
>>>>>
>>>>> I have the same Hadoop and Sqoop2 installation directories with you.
>>>>> And I am running a pseudo cluster in a single Ubuntu virtual machine.
>>>>>
>>>>> So, now, you could try to add the hadoop-core.jar manually, and then
>>>>> go ahead to see whether the sqoop2 server could run. Please follow the
>>>>> following steps:
>>>>>
>>>>> *./bin/addtowar.sh
>>>>> -jars /usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.3.0.jar
>>>>> *
>>>>>
>>>>> Please find the hadoop-core.jar in your own machine. It should be in
a
>>>>> similar place. But still, if you have problem, please let me know.
>>>>>
>>>>>
>>>>> The reason why it's better to remove the "sqoop" folder is to clear
>>>>> the cached old servlet. Because Tomcat cannot alway extract the sqoop.war
>>>>> file immediately after you add dependency library to sqoop.war file.
By
>>>>> removing the sqoop folder, the Tomcat is forced to extract the sqoop.war
to
>>>>> keep the sqoop folder up-to-date. So in this way, you could know whether
>>>>> you have correctly setup the dependency library. Does this explanation
>>>>> help?
>>>>>
>>>>> Best,
>>>>> Mengwei
>>>>>
>>>>>
>>>>> On Tue, Jul 2, 2013 at 9:19 PM, Madhanmohan Savadamuthu <
>>>>> ermadhan@gmail.com> wrote:
>>>>>
>>>>>> Hi  Mengwei,
>>>>>>
>>>>>> Following are details
>>>>>>
>>>>>> Hadoop Version: Hadoop 2.0.0-cdh4.2.1
>>>>>> Linux Version: Linux version 2.6.32-358.2.1.el6.x86_64 (
>>>>>> mockbuild@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7
>>>>>> 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Feb 20 12:17:37 EST
2013
>>>>>> Hadoop Installation Location: /usr/lib/hadoop
>>>>>> Sqoop2 Installation Location: /usr/lib/sqoop2
>>>>>> Sqoop2 Dependency Configuration  Command Used: ./bin/addtowar.sh
>>>>>> -hadoop-auto
>>>>>> Files in :
>>>>>>
>>>>>> avro-1.7.3.jar
>>>>>> commons-cli-1.2.jar
>>>>>> commons-configuration-1.6.jar
>>>>>> commons-dbcp-1.4.jar
>>>>>> commons-lang-2.5.jar
>>>>>> commons-logging-1.1.1.jar
>>>>>> commons-pool-1.5.4.jar
>>>>>> derby-10.8.2.2.jar
>>>>>> guava-11.0.2.jar
>>>>>> hadoop-auth-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-common-2.0.0-cdh4.2.1-tests.jar
>>>>>> hadoop-hdfs-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-mapreduce-client-app-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-mapreduce-client-common-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-mapreduce-client-core-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-yarn-api-2.0.0-cdh4.2.1.jar
>>>>>> hadoop-yarn-common-2.0.0-cdh4.2.1.jar
>>>>>> jackson-core-asl-1.8.8.jar
>>>>>> jackson-mapper-asl-1.8.8.jar
>>>>>> json-simple-1.1.jar
>>>>>> log4j-1.2.16.jar
>>>>>> mysql-connector-java-5.1.25-bin.jar
>>>>>> protobuf-java-2.4.0a.jar
>>>>>> slf4j-api-1.6.1.jar
>>>>>> slf4j-log4j12-1.6.1.jar
>>>>>> sqoop-common-1.99.2.jar
>>>>>> sqoop-connector-generic-jdbc-1.99.2.jar
>>>>>> sqoop-core-1.99.2.jar
>>>>>> sqoop-execution-mapreduce-1.99.2-hadoop200.jar
>>>>>> sqoop-repository-derby-1.99.2.jar
>>>>>> sqoop-spi-1.99.2.jar
>>>>>> sqoop-submission-mapreduce-1.99.2-hadoop200.jar
>>>>>>
>>>>>> Can you elaborate more about deletion of 'sqoop' folder?
>>>>>>
>>>>>> Regards,
>>>>>> Madhanmohan S
>>>>>>
>>>>>>  On Tue, Jul 2, 2013 at 10:50 PM, Mengwei Ding <
>>>>>> mengwei.ding@cloudera.com> wrote:
>>>>>>
>>>>>>> Hi Madhanmohan,
>>>>>>>
>>>>>>> Thank you for your interest in Sqoop2. It's really great to hear
>>>>>>> this. And thank you for providing details for your question.
Let me help
>>>>>>> you out with this.
>>>>>>>
>>>>>>> This main reason for your situation is that the Sqoop servlet
has
>>>>>>> not been started successfully, so the client get connection refused.
I have
>>>>>>> gone through you attachments. The reason of servlet failure is
that your
>>>>>>> Hadoop dependency library has not be configured correctly. Could
you kindly
>>>>>>> answer my following questions, so that I could help with you
further.
>>>>>>>
>>>>>>> 1. Your Hadoop version and installation location? You operating
>>>>>>> system?
>>>>>>> 2. The details of how you configure the dependency library for
sqoop?
>>>>>>> 3. Could you kindly go to
>>>>>>> [sqoop_install_dir]/server/server/webapps/sqoop/WEB-INF/lib and
list all
>>>>>>> the jar files?
>>>>>>>
>>>>>>> PS: remember to delete the sqoop folder under server/server/webapps
>>>>>>> every time after you configure the dependency library.
>>>>>>>
>>>>>>> Best,
>>>>>>> Mengwei
>>>>>>>
>>>>>>>
>>>>>>>  On Tue, Jul 2, 2013 at 10:05 AM, Madhanmohan Savadamuthu <
>>>>>>> ermadhan@gmail.com> wrote:
>>>>>>>
>>>>>>>>  I have setup Sqoop 1.99.2 as mentioned in sqoop.apache.orginstruction.
When I try to show version --all command, following error is
>>>>>>>> coming.
>>>>>>>>
>>>>>>>> Sqoop 1.99.2 revision 3e31b7d3eefb3696d4970704364dea05a9ea2a59
>>>>>>>>   Compiled by homeuser on Mon Apr 15 20:50:13 PDT 2013
>>>>>>>> Exception has occurred during processing command
>>>>>>>> Exception: com.sun.jersey.api.client.ClientHandlerException
>>>>>>>> Message: java.net.ConnectException: Connection refused
>>>>>>>>
>>>>>>>> all log files are attached for reference.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>> Madhanmohan S
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Madhanmohan S
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Madhanmohan S
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Madhanmohan S
>>
>
>


-- 
Thanks and Regards,
Madhanmohan S

Mime
View raw message