sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: SQOOP export from Hadoop Hive to MySQL
Date Fri, 24 May 2013 08:56:00 GMT
Hi Andy,
I'm glad to hear that Sqoop has started working for you!

There is great article on Cloudera blog [1] that describes what sort of logs is Hadoop generating
and how to get them. The task logs are describe there as well.

Jarcec

Links:
1: http://blog.cloudera.com/blog/2009/09/apache-hadoop-log-files-where-to-find-them-in-cdh-and-what-info-they-contain/

On Thu, May 23, 2013 at 06:08:47AM -0700, Han Sen Tey wrote:
> Good morning Jarcec,
> 
>    Thank you very much for your insight. I would have never attempted to remove all
single quote from my sqoop command. Upon removing all quotes from my sqoop command, my Oozie
workflow that comprised of a pig-, hive- and sqoop- actions flow like a river.
> 
>    export --verbose --connect jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test
--table CHRYSLER_Anametrix_Aggregated --update-key VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession
--update-mode allowinsert --export-dir /user/hive/warehouse/chrysler_anametrix_aggregated
--username root -m 1 --input-fields-terminated-by | --input-null-string \\N --input-null-non-string
\\N
> 
> 
>    One question to help with my future debugging. Where or how can I find the map task
log in CDH4 ?
> 
> Thanks,
> Andy
> 
>    
> 
> 
> 
> ________________________________
>  From: Jarek Jarcec Cecho <jarcec@apache.org>
> To: user@sqoop.apache.org; Han Sen Tey <axemen_99@yahoo.com> 
> Sent: Thursday, May 23, 2013 4:45:55 AM
> Subject: Re: SQOOP export from Hadoop Hive to MySQL
>  
> 
> Hi Han,
> would you mind sharing entire Sqoop log generated with parameter --verbose as an attachment?
It might be also beneficial to share the map task log.
> 
> Oozie is not using bash nor any other shell to execute Sqoop, so you should remove any
escaping introduced especially for shell. Looking at your example, I think that it should
be something like:
> 
>   ... --innput-fields-terminated-by | --input-null-string \\N ...
> 
> Jarcec
> 
> On Tue, May 21, 2013 at 05:56:16AM -0700, Han Sen Tey wrote:
> > Greeting experts,
> > 
> >    I have a table "CHRYSLER_Anametrix_Aggregated" in Hive and in MySQL. 
> > Similarly I can find this table in hdfs -- 
> > /user/hive/warehouse/chrysler_anametrix_aggregated.
> > 
> > 
> >    I can successfully execute the following SQOOP command from namenode CLI -
> > 
> >    sqoop export --verbose --connect 
> > jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test --table 
> > CHRYSLER_Anametrix_Aggregated --update-key 
> > VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession 
> > --update-mode allowinsert --export-dir 
> > /user/hive/warehouse/chrysler_anametrix_aggregated --username root -m 1 
> > --input-fields-terminated-by '|' --input-null-string '\\N' 
> > --input-null-non-string '\\N'
> > 
> >    However, when I want to run the above as part of Oozie workflow action ...

> > 
> > 
> >     <action name="sqoop-node">
> >         <sqoop xmlns="uri:oozie:sqoop-action:0.2">
> >             <job-tracker>${jobTracker}</job-tracker>
> >             <name-node>${nameNode}</name-node>
> >            
> >  <configuration>
> >                 <property>
> >                     <name>sqoop.connection.factories</name>
> >                     <value>com.cloudera.sqoop.manager.DefaultManagerFactory</value>
> >                 </property>
> >                 <property>
> >                    
> >  <name>mapred.job.queue.name</name>
> >                     <value>${queueName}</value>
> >                 </property>
> >                 <property>
> >                     <name>mapred.compress.map.output</name>
> >                     <value>true</value>
> >                 </property>
> >                
> >  <property>
> >                     <name>oozie.service.WorkflowAppService.system.libpath</name>
> >                     <value>${nameNode}/user/oozie/share/lib/sqoop</value>
> >                 </property>
> >             </configuration>
> >            
> >  <command>export --verbose --connect 
> > jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test --table 
> > CHRYSLER_Anametrix_Aggregated --update-key 
> > VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession 
> > --update-mode allowinsert --export-dir
> >  /user/hive/warehouse/chrysler_anametrix_aggregated --username root -m 1
> >  --input-fields-terminated-by '|' --input-null-string '\\N' 
> > --input-null-non-string '\\N'</command>
> >         </sqoop>
> >         <ok to="end"/>
> >         <error to="fail"/>
> >     </action>
> > 
> >    I get the following error message. I have "mysql-connector-java.5.1.24-bin.jar"
shared library - /user/oozie/share/lib/sqoop.
> > 
> >    Hadoop version - 2.00 - CDH4.2.0
> >    Sqoop version - 1.4.2. - CDH4.2.0
> > 
> > 
> >    Can you experts please guide me  ... a rookie to Hadoop and its projects.
> > 
> > 2013-05-21 00:48:35,135 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter
is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2013-05-21 00:48:37,394
WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
2013-05-21 00:48:37,408 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
with processName=MAP, sessionId= 2013-05-21 00:48:38,042 INFO org.apache.hadoop.util.ProcessTree:
setsid exited with exit code 0 2013-05-21 00:48:38,058 INFO org.apache.hadoop.mapred.Task: 
Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5dce1bea
2013-05-21 00:48:38,423 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://ip-10-147-191-48.ec2.internal:8020/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/input/dummy.txt:0+5
2013-05-21 00:48:38,453 WARN mapreduce.Counters: Counter name
>  MAP_INPUT_BYTES
> >  is deprecated. Use FileInputFormatCounters as group name and  BYTES_READ as counter
name instead 2013-05-21 00:48:38,459 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks:
0 2013-05-21 00:48:39,743 WARN org.apache.sqoop.tool.SqoopTool: $SQOOP_CONF_DIR has not been
set in the environment. Cannot check for additional configuration. 2013-05-21 00:48:40,032
DEBUG org.apache.sqoop.tool.BaseSqoopTool: Enabled debug logging. 2013-05-21 00:48:40,036
WARN org.apache.sqoop.SqoopOptions: Character argument '|' has multiple characters; only the
first will be used. 2013-05-21 00:48:40,084 DEBUG org.apache.sqoop.ConnFactory: sqoop.connection.factories
is set; ignoring managers.d 2013-05-21 00:48:40,139 DEBUG org.apache.sqoop.ConnFactory: Loaded
manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory 2013-05-21 00:48:40,139
DEBUG org.apache.sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
2013-05-21
>  00:48:40,140
> >  DEBUG org.apache.sqoop.manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql:
2013-05-21 00:48:40,183 INFO org.apache.sqoop.manager.MySQLManager: Preparing to use a MySQL
streaming resultset. 2013-05-21 00:48:40,183 DEBUG org.apache.sqoop.ConnFactory: Instantiated
ConnManager org.apache.sqoop.manager.MySQLManager@795e0c2b 2013-05-21 00:48:40,184 INFO org.apache.sqoop.tool.CodeGenTool:
Beginning code generation 2013-05-21 00:48:40,233 DEBUG org.apache.sqoop.manager.SqlManager:
No connection paramenters specified. Using regular API for making connection. 2013-05-21 00:48:40,911
DEBUG org.apache.sqoop.manager.SqlManager: Using fetchSize for next query: -2147483648 2013-05-21
00:48:40,911 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.*
FROM `CHRYSLER_Anametrix_Aggregated` AS t LIMIT 1 2013-05-21 00:48:40,930 DEBUG org.apache.sqoop.orm.ClassWriter:
selected columns: 2013-05-21 00:48:40,931 DEBUG
> >  org.apache.sqoop.orm.ClassWriter:   VisitorID 2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: 
 VisitorDay 2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   VisitorMonth
2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   VisitorYear 2013-05-21
00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   VisitorSession 2013-05-21 00:48:40,931
DEBUG org.apache.sqoop.orm.ClassWriter:   Brand 2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: 
 DeviceType 2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   DeviceBrowser
2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   TrafficSource 2013-05-21
00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter:   SiteID 2013-05-21 00:48:40,931 DEBUG
org.apache.sqoop.orm.ClassWriter:   BuyID 2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: 
 AdvertisementID 2013-05-21 00:48:40,931 DEBUG
>  org.apache.sqoop.orm.ClassWriter:  
> >  PlacementID 2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter: 
 CreativeID 2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter:   PaidKeyword
2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter:   Model200 2013-05-21 00:48:40,934
DEBUG org.apache.sqoop.orm.ClassWriter:   Model200Conv 2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter: 
 Model300 2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter:   Model300TownCountry
2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter:   BMOLineup 2013-05-21 00:48:40,935
DEBUG org.apache.sqoop.orm.ClassWriter:   BMOModels 2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: 
 BMOColors 2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   BMOOptions 2013-05-21
00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   BMOSummary 2013-05-21 00:48:40,935
DEBUG org.apache.sqoop.orm.ClassWriter:   BrochureLineup
>  2013-05-21
> >  00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   BrochureForm 2013-05-21
00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   BrochureMailLink 2013-05-21 00:48:40,935
DEBUG org.apache.sqoop.orm.ClassWriter:   BrochureDownloadLink 2013-05-21 00:48:40,935 DEBUG
org.apache.sqoop.orm.ClassWriter:   PreDL 2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: 
 DealerLocate 2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   DLHours 2013-05-21
00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter:   DLServices 2013-05-21 00:48:40,936
DEBUG org.apache.sqoop.orm.ClassWriter:   DLDealerSite 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: 
 PreSNI 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter:   SearchNewInventory
2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter:   SNIVIN 2013-05-21 00:48:40,936
DEBUG org.apache.sqoop.orm.ClassWriter:   SNIDetails 2013-05-21 00:48:40,936
>  DEBUG
> >  org.apache.sqoop.orm.ClassWriter:   SNIDealer 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: 
 TestDriveForm 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter:   TestDriveComplete
2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter:   Inventives 2013-05-21 00:48:40,937
DEBUG org.apache.sqoop.orm.ClassWriter:   PreCal 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: 
 PaymentCalculator 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter:   PreGAQ
2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter:   GetAQuote 2013-05-21 00:48:40,937
DEBUG org.apache.sqoop.orm.ClassWriter:   LandingPage 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: 
 Models 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter:   PhotosGallery 2013-05-21
00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter:   Interior 2013-05-21 00:48:40,937 DEBUG
>  org.apache.sqoop.orm.ClassWriter:  
> >  Exterior 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter:   Performance
2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter:   Safety 2013-05-21 00:48:40,938
DEBUG org.apache.sqoop.orm.ClassWriter:   Specifications 2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter: 
 ModelHighlights 2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter:   Dimensions
2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter:   Warranty 2013-05-21 00:48:40,965
DEBUG org.apache.sqoop.manager.SqlManager: Using fetchSize for next query: -2147483648 2013-05-21
00:48:40,965 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.*
FROM `CHRYSLER_Anametrix_Aggregated` AS t LIMIT 1 2013-05-21 00:48:40,990 DEBUG org.apache.sqoop.orm.ClassWriter:
Writing source file: /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.java
2013-05-21 00:48:40,990 DEBUG
> >  org.apache.sqoop.orm.ClassWriter: Table name: CHRYSLER_Anametrix_Aggregated 2013-05-21
00:48:40,990 DEBUG org.apache.sqoop.orm.ClassWriter: Columns: VisitorID:12, VisitorDay:12,
VisitorMonth:12, VisitorYear:12, VisitorSession:4, Brand:12, DeviceType:12, DeviceBrowser:12,
TrafficSource:12, SiteID:12, BuyID:12, AdvertisementID:12, PlacementID:12, CreativeID:12,
PaidKeyword:12, Model200:4, Model200Conv:4, Model300:4, Model300TownCountry:4, BMOLineup:4,
BMOModels:4, BMOColors:4, BMOOptions:4, BMOSummary:4, BrochureLineup:4, BrochureForm:4, BrochureMailLink:4,
BrochureDownloadLink:4, PreDL:4, DealerLocate:4, DLHours:4, DLServices:4, DLDealerSite:4,
PreSNI:4, SearchNewInventory:4, SNIVIN:4, SNIDetails:4, SNIDealer:4, TestDriveForm:4, TestDriveComplete:4,
Inventives:4, PreCal:4, PaymentCalculator:4, PreGAQ:4, GetAQuote:4, LandingPage:4, Models:4,
PhotosGallery:4, Interior:4, Exterior:4, Performance:4, Safety:4, Specifications:4, ModelHighlights:4,
> >  Dimensions:4, Warranty:4, 2013-05-21 00:48:40,990 DEBUG org.apache.sqoop.orm.ClassWriter:
sourceFilename is CHRYSLER_Anametrix_Aggregated.java 2013-05-21 00:48:40,991 DEBUG org.apache.sqoop.orm.CompilationManager:
Found existing /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/ 2013-05-21 00:48:40,992
INFO org.apache.sqoop.orm.CompilationManager: $HADOOP_MAPRED_HOME is not set 2013-05-21 00:48:41,098
DEBUG org.apache.sqoop.orm.CompilationManager: Adding source file: /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.java
2013-05-21 00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager: Invoking javac with
args: 2013-05-21 00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager:   -sourcepath
2013-05-21 00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager:   /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/
2013-05-21 00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager: 
>   -d
> >  2013-05-21 00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager:   /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/
2013-05-21 00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager:   -classpath 2013-05-21
00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager:  
> > 
>  /run/cloudera-scm-agent/process/76-mapreduce-TASKTRACKER:/usr/lib/jvm/j2sdk1.6-oracle/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-
> > 0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/j
> > asper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-map
> > reduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/navigator-plugin-4.5.1-shaded.jar:/usr/share/cmf/lib/plugins/event-publish-4.5.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.5.1.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:
> > /usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/
> > usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-tests.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/avro-1.7.3.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-c
> > onfiguration-1.6.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hado
> > op/lib/hue-plugins-2.2.0-cdh4.2.0.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.2.0-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-common.jar:/mnt/mapred/local/taskTracker/distcache/-3359360379488511723_1277082233_1150217957/ip-10-147-191-48.ec2.internal/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/sqoop-launcher.jar:/mnt/mapred/local/taskTracker/distcache/-3926870669093555941_811939984_1868217431/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoo
> > p/avro-ipc-1.7.3-tests.jar:/mnt/mapred/local/taskTracker/distcache/4298483340499219542_2138859044_1868217453/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/avro-ipc-1.7.3.jar:/mnt/mapred/local/taskTracker/distcache/-1975491928978283532_-1185784129_1868217475/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/avro-mapred-1.7.3.jar:/mnt/mapred/local/taskTracker/distcache/8448357694523829347_-1491317642_1868217500/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-beanutils-1.7.0.jar:/mnt/mapred/local/taskTracker/distcache/-334079317866666788_-232552041_1868217523/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-beanutils-core-1.8.0.jar:/mnt/mapred/local/taskTracker/distcache/-644448586073609072_-372575484_1868217547/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-configuration-1.6.jar:/mnt/mapred/local/taskTracker/distcache/246054495199880993_357051411_1868217589/ip-10-147-191-48.ec2.internal/
> > user/oozie/share/lib/sqoop/commons-digester-1.8.jar:/mnt/mapred/local/taskTracker/distcache/-4144140590487253933_766503300_1868217614/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-io-2.1.jar:/mnt/mapred/local/taskTracker/distcache/5753546228468760323_-553874526_1868217665/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/guava-11.0.2.jar:/mnt/mapred/local/taskTracker/distcache/8662866019753074519_-1003108238_1868217924/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/hbase-0.94.2-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-4840377369495417802_845918870_1868217954/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/high-scale-lib-1.1.1.jar:/mnt/mapred/local/taskTracker/distcache/-142623586016294918_445946554_1868217996/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/hsqldb-1.8.0.7.jar:/mnt/mapred/local/taskTracker/distcache/5142583573507929188_-105788043_1868218025/ip-10-147-191-48.ec2.internal/us
> > er/oozie/share/lib/sqoop/httpclient-4.1.3.jar:/mnt/mapred/local/taskTracker/distcache/-6413369533723682352_-180904991_1868218051/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/httpcore-4.1.3.jar:/mnt/mapred/local/taskTracker/distcache/3446550291828223401_633399508_1868218088/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/jsr305-1.3.9.jar:/mnt/mapred/local/taskTracker/distcache/-6492962697058022370_-616322509_1868218133/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/libthrift-0.9.0.jar:/mnt/mapred/local/taskTracker/distcache/7049151581973813525_2017354882_1868218156/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/metrics-core-2.1.2.jar:/mnt/mapred/local/taskTracker/distcache/623583253106481693_846556431_1142287025/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/mysql-connector-java-5.1.24-bin.jar:/mnt/mapred/local/taskTracker/distcache/-7982675208196719520_-1207761111_1868218197/ip-10-147-191-48.ec2.interna
> > l/user/oozie/share/lib/sqoop/netty-3.4.0.Final.jar:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-412267918101123238_-517736257_1868215515/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/oozie/json-simple-1.1.jar:/mnt/mapred/local/taskTracker/ubuntu/jobcache/job_201305131705_0181/attempt_201305131705_0181_m_000000_0/work:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar
2013-05-21 00:48:48,206 INFO org.apache.sqoop.orm.CompilationManager: Writing jar file: /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.jar
2013-05-21 00:48:48,210 DEBUG
> >  org.apache.sqoop.orm.CompilationManager: Scanning for .class files in directory:
/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca 2013-05-21 00:48:48,221 DEBUG org.apache.sqoop.orm.CompilationManager:
Got classfile: /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.class
-> CHRYSLER_Anametrix_Aggregated.class 2013-05-21 00:48:48,236 DEBUG org.apache.sqoop.orm.CompilationManager:
Finished writing jar file /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.jar
2013-05-21 00:48:48,243 WARN org.apache.sqoop.manager.MySQLManager: MySQL Connector upsert
functionality is using INSERT ON 2013-05-21 00:48:48,243 WARN org.apache.sqoop.manager.MySQLManager:
DUPLICATE KEY UPDATE clause that relies on table's unique key. 2013-05-21 00:48:48,243 WARN
org.apache.sqoop.manager.MySQLManager: Insert/update distinction is therefore independent
on column 2013-05-21 00:48:48,243 WARN
> >  org.apache.sqoop.manager.MySQLManager: names specified in --update-key parameter.
Please see MySQL 2013-05-21 00:48:48,245 WARN org.apache.sqoop.manager.MySQLManager: documentation
for additional limitations. 2013-05-21 00:48:48,290 INFO org.apache.sqoop.mapreduce.ExportJobBase:
Beginning export of CHRYSLER_Anametrix_Aggregated 2013-05-21 00:48:48,319 DEBUG org.apache.sqoop.mapreduce.JobBase:
Using InputFormat: class org.apache.sqoop.mapreduce.ExportInputFormat 2013-05-21 00:48:48,575
DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to job classpath: file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar
2013-05-21 00:48:48,581 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to job classpath:
> >  file:/mnt/mapred/local/taskTracker/distcache/623583253106481693_846556431_1142287025/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/mysql-connector-java-5.1.24-bin.jar
2013-05-21 00:48:48,585 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to job classpath:
file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar
2013-05-21 00:48:48,588 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to job classpath:
file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar
2013-05-21 00:48:48,588 WARN org.apache.sqoop.mapreduce.JobBase: SQOOP_HOME is unset. May
not be able to find all job dependencies. 2013-05-21 00:48:48,739 WARN org.apache.hadoop.mapred.JobClient:
Use GenericOptionsParser for parsing the arguments. Applications
> >  should implement Tool for the same. 2013-05-21 00:48:49,058 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat:
Total input paths to process : 1 2013-05-21 00:48:49,059 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat:
Target numMapTasks=1 2013-05-21 00:48:49,059 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat:
Total input bytes=8187428 2013-05-21 00:48:49,059 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat:
maxSplitSize=8187428 2013-05-21 00:48:49,063 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat:
Total input paths to process : 1 2013-05-21 00:48:49,073 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat:
Generated splits: 2013-05-21 00:48:49,073 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat: 
 Paths:/user/hive/warehouse/chrysler_anametrix_aggregated/part-AG-00000-1.txt:0+8187428 Locations:ip-10-147-191-48.ec2.internal:;
2013-05-21 00:48:49,403 INFO org.apache.hadoop.mapred.JobClient: Running job:
>  job_201305131705_0182
> >  2013-05-21 00:48:50,417 INFO org.apache.hadoop.mapred.JobClient:  map 0% reduce
0% 2013-05-21 00:49:06,478 INFO org.apache.hadoop.mapred.JobClient: Task Id : attempt_201305131705_0182_m_000000_0,
Status : FAILED 2013-05-21 00:49:16,703 INFO org.apache.hadoop.mapred.JobClient: Task Id :
attempt_201305131705_0182_m_000000_1, Status : FAILED 2013-05-21 00:49:25,861 INFO org.apache.hadoop.mapred.JobClient:
Task Id : attempt_201305131705_0182_m_000000_2, Status : FAILED 2013-05-21 00:49:42,012 INFO
org.apache.hadoop.mapred.JobClient: Job complete: job_201305131705_0182 2013-05-21 00:49:42,015
INFO org.apache.hadoop.mapred.JobClient: Counters: 7 2013-05-21 00:49:42,020 INFO org.apache.hadoop.mapred.JobClient: 
 Job Counters 2013-05-21 00:49:42,023 INFO org.apache.hadoop.mapred.JobClient:     Failed
map tasks=1 2013-05-21 00:49:42,023 INFO org.apache.hadoop.mapred.JobClient:     Launched
map tasks=4 2013-05-21 00:49:42,023 INFO
> >  org.apache.hadoop.mapred.JobClient:     Data-local map tasks=4 2013-05-21 00:49:42,024
INFO org.apache.hadoop.mapred.JobClient:     Total time spent by all maps in occupied slots
(ms)=50187 2013-05-21 00:49:42,024 INFO org.apache.hadoop.mapred.JobClient:     Total time
spent by all reduces in occupied slots (ms)=0 2013-05-21 00:49:42,024 INFO org.apache.hadoop.mapred.JobClient: 
   Total time spent by all maps waiting after reserving slots (ms)=0 2013-05-21 00:49:42,024
INFO org.apache.hadoop.mapred.JobClient:     Total time spent by all reduces waiting after
reserving slots (ms)=0 2013-05-21 00:49:42,032 WARN mapreduce.Counters: Group FileSystemCounters
is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2013-05-21 00:49:42,033
INFO org.apache.sqoop.mapreduce.ExportJobBase: Transferred 0 bytes in 53.4336 seconds (0 bytes/sec)
2013-05-21 00:49:42,035 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter
>  is deprecated.
> >  Use org.apache.hadoop.mapreduce.TaskCounter instead 2013-05-21 00:49:42,035 INFO
org.apache.sqoop.mapreduce.ExportJobBase: Exported 0 records. 2013-05-21 00:49:42,035 ERROR
org.apache.sqoop.tool.ExportTool: Error during export: Export job failed! 2013-05-21 00:49:42,063
INFO org.apache.hadoop.mapred.Task: Task:attempt_201305131705_0181_m_000000_0 is done. And
is in the process of commiting 2013-05-21 00:49:43,124 INFO org.apache.hadoop.mapred.Task:
Task attempt_201305131705_0181_m_000000_0 is allowed to commit now 2013-05-21 00:49:43,292
INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task 'attempt_201305131705_0181_m_000000_0'
to hdfs://ip-10-147-191-48.ec2.internal:8020/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/output
2013-05-21 00:49:43,300 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201305131705_0181_m_000000_0'
done. 2013-05-21 00:49:43,309 INFO
>  org.apache.hadoop.mapred.TaskLogsTruncater:
> >  Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> > 
> > Regards,
> > Andy Tey
> > Student !!

Mime
View raw message