sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: import to hive hangs
Date Wed, 07 Mar 2012 14:28:40 GMT
I'm not sure whether it's connected to your problem, but sqoop is designed to firstly import
data into HDFS to a temporary directory and than using hive command "LOAD DATA" move them
to target location in HIVE warehouse directory. You seems to be importing data directly into
default HIVE warehouse directory (--targer-dir /user/hive/warehouse/sample_rm2). I would firstly
try to change --target-dir parameter to simple "--target-dir sample_rm2" (directory with this
name will be created in your HDFS home directory) or any other HDFS path outside /user/hive/warehouse.

Jarcec

On Wed, Mar 07, 2012 at 03:10:57PM +0100, Marcin Cylke wrote:
> Hi
> 
> I'm trying to import some tables into Hive. I'm using this kind of
> CLI command:
> 
> sqoop import \
> --driver oracle.jdbc.driver.OracleDriver \
> --connect jdbc:oracle:thin:@localhost:1521:XE \
> --username had_test --password abc1234 \
> --query "select * from all_tables where \$CONDITIONS" \
> --num-mappers 1 \
> --hive-import \
> --hive-table "sample_rm2" \
> --target-dir /user/hive/warehouse/sample_rm2 \
> --mysql-delimiters \
> --verbose
> 
> The output table is created, everything looks good, but sqoop does
> not exit properly. It just hangs indefinitelly on the last line
> (look in the attached log file).
> 
> My environment is:
> - Sqoop 1.4.1-incubating
>   git commit id
>   Compiled by jarcec on Tue Feb  7 20:54:25 CET 2012
> - Hadoop 1.0.0
> - hive 0.9.0-SNAPSHOT - built from git a few days ago
> 
> Is this a know issue?
> 
> Thanks for any help
> Regards, Marcin

> 
> 12/02/28 13:54:13 WARN tool.BaseSqoopTool: Setting your password on the command-line
is insecure. Consider using -P instead.
> 12/02/28 13:54:13 WARN tool.BaseSqoopTool: Hive does not support escape characters in
fields;
> 12/02/28 13:54:13 WARN tool.BaseSqoopTool: parse errors in Hive may result from using
--escaped-by.
> 12/02/28 13:54:13 WARN tool.BaseSqoopTool: Hive does not support quoted strings; parse
errors
> 12/02/28 13:54:13 WARN tool.BaseSqoopTool: in Hive may result from using --enclosed-by.
> 12/02/28 13:54:13 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/02/28 13:54:13 INFO tool.CodeGenTool: Beginning code generation
> 12/02/28 13:54:14 INFO manager.SqlManager: Executing SQL statement: select * from all_tables
where (1 = 0)
> 12/02/28 13:54:14 INFO manager.SqlManager: Executing SQL statement: select * from all_tables
where (1 = 0)
> 12/02/28 13:54:14 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop-1.0.0/libexec/..
> Note: /tmp/sqoop-m/compile/e30db5359f37c6d21004e22c78085f74/QueryResult.java uses or
overrides a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 12/02/28 13:54:16 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-michler/compile/e30db5359f37c6d21004e22c78085f74/QueryResult.java
to /home /michler/./QueryResult.java
> java.io.IOException: Destination '/home/michler/./QueryResult.java' already exists
>         at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:1811)
>         at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:227)
>         at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:83)
>         at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:367)
>         at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:453)
>         at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>         at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> 12/02/28 13:54:16 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-m/compile/e30db5359f37c6d21004e22c78085f74/QueryResult.jar
> 12/02/28 13:54:16 INFO mapreduce.ImportJobBase: Beginning query import.
> 12/02/28 13:54:18 INFO mapred.JobClient: Running job: job_201202281147_0003
> 12/02/28 13:54:19 INFO mapred.JobClient: map 0% reduce 0%
> 12/02/28 13:54:39 INFO mapred.JobClient: map 100% reduce 0%
> 12/02/28 13:54:44 INFO mapred.JobClient: Job complete: job_201202281147_0003
> 12/02/28 13:54:44 INFO mapred.JobClient: Counters: 18
> 12/02/28 13:54:44 INFO mapred.JobClient: Job Counters
> 12/02/28 13:54:44 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=18733
> 12/02/28 13:54:44 INFO mapred.JobClient: Total time spent by all reduces waiting after
reserving slots (ms)=0
> 12/02/28 13:54:44 INFO mapred.JobClient: Total time spent by all maps waiting after reserving
slots (ms)=0
> 12/02/28 13:54:44 INFO mapred.JobClient: Launched map tasks=1
> 12/02/28 13:54:44 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 12/02/28 13:54:44 INFO mapred.JobClient: File Output Format Counters
> 12/02/28 13:54:44 INFO mapred.JobClient: Bytes Written=28941
> 12/02/28 13:54:44 INFO mapred.JobClient: FileSystemCounters
> 12/02/28 13:54:44 INFO mapred.JobClient: HDFS_BYTES_READ=87
> 12/02/28 13:54:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=29027
> 12/02/28 13:54:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=28941
> 12/02/28 13:54:44 INFO mapred.JobClient: File Input Format Counters
> 12/02/28 13:54:44 INFO mapred.JobClient: Bytes Read=0
> 12/02/28 13:54:44 INFO mapred.JobClient: Map-Reduce Framework
> 12/02/28 13:54:44 INFO mapred.JobClient: Map input records=95
> 12/02/28 13:54:44 INFO mapred.JobClient: Physical memory (bytes) snapshot=112734208
> 12/02/28 13:54:44 INFO mapred.JobClient: Spilled Records=0
> 12/02/28 13:54:44 INFO mapred.JobClient: CPU time spent (ms)=3640
> 12/02/28 13:54:44 INFO mapred.JobClient: Total committed heap usage (bytes)=99221504
> 12/02/28 13:54:44 INFO mapred.JobClient: Virtual memory (bytes) snapshot=623640576
> 12/02/28 13:54:44 INFO mapred.JobClient: Map output records=95
> 12/02/28 13:54:44 INFO mapred.JobClient: SPLIT_RAW_BYTES=87
> 12/02/28 13:54:44 INFO mapreduce.ImportJobBase: Transferred 28.2627 KB in 28.1098 seconds
(1.0054 KB/sec)
> 12/02/28 13:54:44 INFO mapreduce.ImportJobBase: Retrieved 95 records.
> 12/02/28 13:54:44 INFO hive.HiveImport: Removing temporary files from import process:
/user/hive/warehouse/sample_rm2/_logs
> 12/02/28 13:54:44 INFO hive.HiveImport: Loading uploaded data into Hive
> 12/02/28 13:54:44 INFO manager.SqlManager: Executing SQL statement: select * from all_tables
where (1 = 0)
> 12/02/28 13:54:44 INFO manager.SqlManager: Executing SQL statement: select * from all_tables
where (1 = 0)
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column PCT_FREE had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column PCT_USED had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column INI_TRANS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column MAX_TRANS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column INITIAL_EXTENT had to be cast to a
less precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column NEXT_EXTENT had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column MIN_EXTENTS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column MAX_EXTENTS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column PCT_INCREASE had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column FREELISTS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column FREELIST_GROUPS had to be cast to
a less precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column NUM_ROWS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column BLOCKS had to be cast to a less precise
type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column EMPTY_BLOCKS had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column AVG_SPACE had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column CHAIN_CNT had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column AVG_ROW_LEN had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column AVG_SPACE_FREELIST_BLOCKS had to be
cast to a less precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column NUM_FREELIST_BLOCKS had to be cast
to a less precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column SAMPLE_SIZE had to be cast to a less
precise type in Hive
> 12/02/28 13:54:44 WARN hive.TableDefWriter: Column LAST_ANALYZED had to be cast to a
less precise type in Hive
> 12/02/28 13:54:45 INFO hive.HiveImport: WARNING: org.apache.hadoop.metrics.jvm.EventCounter
is deprecated. Please use org.apache.hadoop.log.metrics.Ev entCounter in all the log4j.properties
files.
> 12/02/28 13:54:46 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/home/m/hive/lib/hive-common-0.9.0-SNAPSHOT.jar!/hiv
e-log4j.properties
> 12/02/28 13:54:46 INFO hive.HiveImport: Hive history file=/tmp/m/hive_job_log_michler_201202281354_287717854.txt
> 12/02/28 13:54:51 INFO hive.HiveImport: OK
> 12/02/28 13:54:51 INFO hive.HiveImport: Time taken: 4.591 seconds
> 12/02/28 13:54:51 INFO hive.HiveImport: Loading data to table default.sample_rm2


Mime
View raw message