sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arvind Prabhakar <arv...@apache.org>
Subject Re: need help on merge tool in sqoop
Date Wed, 22 Feb 2012 05:35:33 GMT
Hi Narasimharao,

When you invoke the merge command, try specifying the complete path of the
jar file to use. From the errors you have described, it appears that Sqoop
is unable to load the jar file correctly.

Thanks,
Arvind

On Tue, Feb 21, 2012 at 8:28 PM, chelikani narasimharao <
narasimharaoc@gmail.com> wrote:

> hi guys,
>               Can any one please help me for the below issue.
>
> Advanced Thanks,
> Narasimharao
>
>
> On Thu, Feb 16, 2012 at 8:24 PM, chelikani narasimharao <
> narasimharaoc@gmail.com> wrote:
>
>> Hi All,
>>
>>     I am trying to merge the two datasets using the merge tool
>> functionality in sqoop.But i got struck up with the following error.
>>
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class*
>>
>> *---->below is my command executed to get the first data set specifying
>> the location to store the first dataset to dataset1 under narasimharao
>> directory*
>>
>> sqoop import --connect jdbc:hpt4jdbc://
>> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
>> com.hp.t4jdbc.HPT4Driver --username narasimharoa@hp.com -P --verbose
>> --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and CNTCT_ID
>> in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
>> --target-dir /home/narasimharao/dataset1
>>
>> successfully able to import the dataset1 and also the .class and .jar
>> files were written to the following location
>>
>> */tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2*
>>
>>
>> *--->below is my command executed to get the second data set  after
>> updating the few records which exists in the dataset1 with the same
>> merge-key*
>>
>>
>> sqoop import --connect jdbc:hpt4jdbc://
>> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
>> com.hp.t4jdbc.HPT4Driver --username narasimharoa.chelikani@hp.com -P
>> --verbose --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and
>> CNTCT_ID in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
>> --target-dir /home/narasimharao/dataset2
>>
>> successfully able to import the dataset1 and also the .class and .jar
>> files were written to the following location
>>
>> *
>> /tmp/sqoop-narasimharao/compile/e26282f14a672e7722352fb424781e5a/CNTCT_D.jar
>> *
>>
>>
>> *--->Now the actually Sqoop merge command was given like below:*
>>
>> *sqoop merge --new-data /home/narasimharao/dataset1 --onto
>> /home/narasimharao/dataset2 --target-dir /home/narasimharao/merged
>> --jar-file CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID*
>>
>>
>> Can you please help me which class name i need to mention over here under
>> the class-name attribute and what's wrong in this statement.
>>
>> I tried the below options
>>
>> 1.copied the dataset1 corresponding class and jar files into my home
>> direcotry and tried like below...but getting the same error as below
>>
>> sqoop merge --new-data /home/narasimharao/dataset2 --onto
>> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
>> --jar-file ./CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID
>>
>> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
>> Please set $HBASE_HOME to the root of your HBase installation.
>>
>> 12/02/16 13:56:08 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 12/02/16 13:56:08 WARN snappy.LoadSnappy: Snappy native library is
>> available
>> 12/02/16 13:56:08 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 12/02/16 13:56:08 INFO snappy.LoadSnappy: Snappy native library loaded
>> 12/02/16 13:56:10 INFO mapred.JobClient: Running job:
>> job_201202161202_0005
>> 12/02/16 13:56:11 INFO mapred.JobClient:  map 0% reduce 0%
>> 12/02/16 13:56:55 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_0, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:56:59 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_0, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:22 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_1, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>>  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:23 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_1, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:50 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_2, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:51 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>>  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>  at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLo
>> attempt_201202161202_0005_m_000001_2: log4j:WARN No appenders could be
>> found for logger (org.apache.hadoop.mapred.Task).
>> attempt_201202161202_0005_m_000001_2: log4j:WARN Please initialize the
>> log4j system properly.
>> 12/02/16 13:58:27 INFO mapred.JobClient: Job complete:
>> job_201202161202_0005
>> 12/02/16 13:58:28 INFO mapred.JobClient: Counters: 7
>> 12/02/16 13:58:28 INFO mapred.JobClient:   Job Counters
>> 12/02/16 13:58:28 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=249623
>> 12/02/16 13:58:28 INFO mapred.JobClient:     Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 12/02/16 13:58:28 INFO mapred.JobClient:     Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 12/02/16 13:58:28 INFO mapred.JobClient:     Launched map tasks=8
>> 12/02/16 13:58:28 INFO mapred.JobClient:     Data-local map tasks=8
>> 12/02/16 13:58:28 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
>> 12/02/16 13:58:28 INFO mapred.JobClient:     Failed map tasks=1
>> 12/02/16 13:58:28 ERROR tool.MergeTool: MapReduce job failed!
>>
>> 2. given second import dataset2 corresponding classname..but the same
>> error it is also throwing
>>
>> 3.mentioned with full path for both .class and .jar file
>>
>> sqoop merge --new-data /home/narasimharao/dataset2 --onto
>> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
>> --jar-file
>> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.jar
>> --class-name
>> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.class
>> --merge-key CNTCT_ID
>> 4.tried the above stmt3 with second dataset .class and .jar files
>> also..getting the same errors
>>
>>
>>
>> Any Help Appreciated!!!!!!
>>
>> Thanks,
>> Narasimharao
>>
>>
>>
>
>

Mime
View raw message