sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "arvind@cloudera.com" <arv...@cloudera.com>
Subject Re: [sqoop-user] Re: Sqoop import job failure?
Date Mon, 08 Aug 2011 16:51:43 GMT
[bcc: sqoop-user@cloudera.org (deprecated), to: sqoop-user@incubator.apache.org]

Kevin,

The project page for Sqoop at Apache Incubator is at [1]. Also,
information regarding relevant mailing lists that you should direct
your comments to is at [2].

>From the information you have provided below, it appears that the data
transfer was completed successfully but invocation of Hive client to
populate that into a table failed for some reason. What version of
Hive are you using? It would be good if you could try manually
creating a table in Hive to make sure that it is working as expected.

[1] http://incubator.apache.org/projects/sqoop.html
[2] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists

Thanks,
Arvind

On Fri, Aug 5, 2011 at 6:21 PM, Kevin <kevinfifteen@gmail.com> wrote:
> http://incubator.apache.org/sqoop/ seems to be down, so I guess I'll
> have to continue the post here. Sorry Arvind!
>
> But I ended up doing my project on a new machine, so I had to start
> again from scratch. The problem I was having from before doesn't
> occur
> anymore.
>
> Instead, I encountered some other problems.
>
> I ran:
> ./sqoop import --verbose --fields-terminated-by , --hive-drop-import-
> delims --direct --connect jdbc:postgresql://query-4.redfintest.com:
> 5432/stingray_6_5_d --username redfin_oltp -P --table brokerages --
> hive-import
>
> And got:
> 11/08/05 17:47:05 DEBUG util.DirectImportUtils: Writing to filesystem:
> file:///
> 11/08/05 17:47:05 DEBUG util.DirectImportUtils: Creating destination
> directory brokerages
> 11/08/05 17:47:05 DEBUG io.SplittingOutputStream: Opening next output
> file: brokerages/data-00000
> 11/08/05 17:47:05 WARN util.NativeCodeLoader: Unable to load native-
> hadoop library for your platform... using builtin-java classes where
> applicable
> 11/08/05 17:47:05 DEBUG manager.DirectPostgresqlManager: Starting
> stream sink
> 11/08/05 17:47:05 DEBUG manager.DirectPostgresqlManager: Waiting for
> process completion
> 11/08/05 17:47:05 INFO manager.DirectPostgresqlManager: Transfer loop
> complete.
> 11/08/05 17:47:05 INFO manager.DirectPostgresqlManager: Transferred
> 79.166 KB in 0.0401 seconds (1.9274 MB/sec)
> 11/08/05 17:47:05 INFO hive.HiveImport: Loading uploaded data into
> Hive
> 11/08/05 17:47:05 DEBUG hive.HiveImport: Hive.inputTable: brokerages
> 11/08/05 17:47:05 DEBUG hive.HiveImport: Hive.outputTable: brokerages
> 11/08/05 17:47:05 DEBUG manager.SqlManager: Using fetchSize for next
> query: 1000
> 11/08/05 17:47:05 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM "brokerages" AS t LIMIT 1
> 11/08/05 17:47:05 DEBUG manager.SqlManager: Using fetchSize for next
> query: 1000
> 11/08/05 17:47:05 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM "brokerages" AS t LIMIT 1
> 11/08/05 17:47:05 WARN hive.TableDefWriter: Column created_date had to
> be cast to a less precise type in Hive
> 11/08/05 17:47:05 DEBUG hive.TableDefWriter: Create statement: CREATE
> TABLE IF NOT EXISTS `brokerages` ( `brokerage_id` BIGINT,
> `brokerage_name` STRING, `managing_broker` STRING, `phone` STRING,
> `created_date` STRING) COMMENT 'Imported by sqoop on 2011/08/05
> 17:47:05' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054' LINES
> TERMINATED BY '\012' STORED AS TEXTFILE
> 11/08/05 17:47:05 DEBUG hive.TableDefWriter: Load statement: LOAD DATA
> INPATH 'file:/usr/lib/sqoop-1.3.0/bin/brokerages' INTO TABLE
> `brokerages`
> 11/08/05 17:47:05 DEBUG hive.HiveImport: Using external Hive process.
> 11/08/05 17:47:06 INFO hive.HiveImport: hive-log4j.properties not
> found
> 11/08/05 17:47:06 INFO hive.HiveImport: Hive history file=/tmp/root/
> hive_job_log_root_201108051747_1516665945.txt
> 11/08/05 17:47:06 INFO hive.HiveImport: 11/08/05 17:47:06 INFO
> exec.HiveHistory: Hive history file=/tmp/root/
> hive_job_log_root_201108051747_1516665945.txt
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS
> `brokerages` ( `brokerage_id` BIGINT, `brokerage_name` STRING,
> `managing_broker` STRING, `phone` STRING, `created_date` STRING)
> COMMENT 'Imported by sqoop on 2011/08/05 17:47:05' ROW FORMAT
> DELIMITED FIELDS TERMINATED BY '\054' LINES TERMINATED BY '\012'
> STORED AS TEXTFILE
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> parse.ParseDriver: Parse Completed
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> parse.SemanticAnalyzer: Starting Semantic Analysis
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> parse.SemanticAnalyzer: Creating table brokerages position=27
> 11/08/05 17:47:07 INFO hive.HiveImport:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 919)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 904)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:
> 7074)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:
> 6573)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:
> 238)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:262)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:269)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:430)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
> 25)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> java.lang.reflect.Method.invoke(Method.java:597)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> 11/08/05 17:47:07 INFO hive.HiveImport: Caused by:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:
> 127)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:
> 1855)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1865)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
> 917)
> 11/08/05 17:47:07 INFO hive.HiveImport:         ... 16 more
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> ql.Driver: Semantic Analysis Completed
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> ql.Driver: Returning Hive schema: Schema(fieldSchemas:null,
> properties:null)
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> ql.Driver: Starting command: CREATE TABLE IF NOT EXISTS `brokerages`
> ( `brokerage_id` BIGINT, `brokerage_name` STRING, `managing_broker`
> STRING, `phone` STRING, `created_date` STRING) COMMENT 'Imported by
> sqoop on 2011/08/05 17:47:05' ROW FORMAT DELIMITED FIELDS TERMINATED
> BY '\054' LINES TERMINATED BY '\012' STORED AS TEXTFILE
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> exec.DDLTask: Default to LazySimpleSerDe for table brokerages
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO
> hive.log: DDL: struct brokerages { i64 brokerage_id, string
> brokerage_name, string managing_broker, string phone, string
> created_date}
> 11/08/05 17:47:07 INFO hive.HiveImport: FAILED: Error in metadata:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 ERROR
> exec.DDLTask: FAILED: Error in metadata:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:476)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3146)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:213)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
> 57)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:262)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:269)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:430)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
> 25)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> java.lang.reflect.Method.invoke(Method.java:597)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> 11/08/05 17:47:07 INFO hive.HiveImport: Caused by:
> java.lang.IllegalArgumentException: URI:  does not have a scheme
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:
> 127)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:
> 1855)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1865)
> 11/08/05 17:47:07 INFO hive.HiveImport:         at
> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:470)
> 11/08/05 17:47:07 INFO hive.HiveImport:         ... 17 more
> 11/08/05 17:47:07 INFO hive.HiveImport:
> 11/08/05 17:47:07 INFO hive.HiveImport: FAILED: Execution Error,
> return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
> 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 ERROR
> ql.Driver: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> 11/08/05 17:47:07 ERROR tool.ImportTool: Encountered IOException
> running import job: java.io.IOException: Hive exited with status 9
>        at
> com.cloudera.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:
> 326)
>        at
> com.cloudera.sqoop.hive.HiveImport.executeScript(HiveImport.java:276)
>        at
> com.cloudera.sqoop.hive.HiveImport.importTable(HiveImport.java:218)
>        at
> com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:362)
>        at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423)
>        at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>        at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>        at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>        at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>
> Data seems to have transferred correctly, but it transferred to file:///
> which is my local machine. I thought by default to should transfer to
> hdfs:///? Is there a way for me to force it to transfer hdfs? On the
> other hand, I'm having troubles invoking Hive once again. It seems as
> though Sqoop doesn't recognize Hive even though I have set $HIVE_HOME
> to the correct directory. I'm running Hive 0.7.0.
> Thanks Arvind, I greatly appreciate your responses!
>
> On Aug 4, 6:10 pm, "arv...@cloudera.com" <arv...@cloudera.com> wrote:
>> [bcc:sqoop-u...@cloudera.org, to:sqoop-u...@incubator.apache.org]
>>
>> Kevin,
>>
>> It looks like the transfer of data was successful but there was a
>> problem in invoking Hive. What version of Hive are you using?
>>
>> Hive by default generates a session log file in /tmp. Search for
>> hive.log in /tmp and let us know what you find in there. From the
>> looks of it, it appears to be a classpath issue.
>>
>> Thanks,
>> Arvind
>>
>> Note: Please subscribe to sqoop-u...@incubator.apache.org and direct
>> further repsonses there.
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Aug 4, 2011 at 5:38 PM, Kevin <kevinfift...@gmail.com> wrote:
>> > I appreciate your response Arvind. I tried the --direct route you
>> > mentioned and it seems to have fixed the problems I mentioned earlier.
>> > Unfortunately, I haven't been successful with Sqoop yet. I'm running
>> > into this problem:
>>
>> > After executing:
>> > sqoop import --direct --connect jdbc:postgresql://
>> > query-4.redfintest.com:5432/stingray_6_5_d --username redfin_oltp -P --
>> > table brokerages --hive-import --hive-home /usr/share/brisk/hive/ --
>> > target-dir /data/qa-metrics/
>>
>> > I get:
>> > 11/08/04 17:26:46 INFO tool.BaseSqoopTool: Using Hive-specific
>> > delimiters for output. You can override
>> > 11/08/04 17:26:46 INFO tool.BaseSqoopTool: delimiters with --fields-
>> > terminated-by, etc.
>> > 11/08/04 17:26:46 INFO manager.SqlManager: Using default fetchSize of
>> > 1000
>> > 11/08/04 17:26:46 INFO tool.CodeGenTool: Beginning code generation
>> > 11/08/04 17:26:47 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT t.* FROM "brokerages" AS t LIMIT 1
>> > 11/08/04 17:26:47 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT t.* FROM "brokerages" AS t LIMIT 1
>> > 11/08/04 17:26:47 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/
>> > hadoop
>> > 11/08/04 17:26:47 INFO orm.CompilationManager: Found hadoop core jar
>> > at: /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar
>> > 11/08/04 17:26:48 INFO orm.CompilationManager: Writing jar file: /tmp/
>> > sqoop-root/compile/8d97f121b1707576d1574cb5ba4653b0/brokerages.jar
>> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Beginning psql
>> > fast path import
>> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT t.* FROM "brokerages" AS t LIMIT 1
>> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Performing
>> > import of table brokerages from database stingray_6_5_d
>> > 11/08/04 17:26:48 WARN util.NativeCodeLoader: Unable to load native-
>> > hadoop library for your platform... using builtin-java classes where
>> > applicable
>> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Transfer loop
>> > complete.
>> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Transferred
>> > 78.8574 KB in 0.0396 seconds (1.9445 MB/sec)
>> > 11/08/04 17:26:48 INFO hive.HiveImport: Loading uploaded data into
>> > Hive
>> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT t.* FROM "brokerages" AS t LIMIT 1
>> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT t.* FROM "brokerages" AS t LIMIT 1
>> > 11/08/04 17:26:48 WARN hive.TableDefWriter: Column created_date had to
>> > be cast to a less precise type in Hive
>> > 11/08/04 17:26:48 INFO hive.HiveImport: Exception in thread "main"
>> > java.lang.NoClassDefFoundError: jline/ArgumentCompletor
>> > $ArgumentDelimiter
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.lang.Class.forName0(Native Method)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.lang.Class.forName(Class.java:247)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > org.apache.hadoop.util.RunJar.main(RunJar.java:179)
>> > 11/08/04 17:26:48 INFO hive.HiveImport: Caused by:
>> > java.lang.ClassNotFoundException: jline.ArgumentCompletor
>> > $ArgumentDelimiter
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.security.AccessController.doPrivileged(Native Method)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at sun.misc.Launcher
>> > $AppClassLoader.loadClass(Launcher.java:301)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         at
>> > java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > 11/08/04 17:26:48 INFO hive.HiveImport:         ... 3 more
>> > 11/08/04 17:26:48 ERROR tool.ImportTool: Encountered IOException
>> > running import job: java.io.IOException: Hive exited with status 1
>> >        at
>> > com.cloudera.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.jav
a:
>> > 326)
>> >        at
>> > com.cloudera.sqoop.hive.HiveImport.executeScript(HiveImport.java:276)
>> >        at
>> > com.cloudera.sqoop.hive.HiveImport.importTable(HiveImport.java:218)
>> >        at
>> > com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:362)
>> >        at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423)
>> >        at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >        at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>> >        at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>> >        at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>> >        at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>>
>> > It seems to be a Hive issue. I haven't had luck trying to figure out a
>> > solution. Is it possible that my Hive is corrupt?
>>
>> > On Aug 2, 6:18 pm, "arv...@cloudera.com" <arv...@cloudera.com> wrote:
>> >> [bcc:sqoop-u...@cloudera.org, to:sqoop-u...@incubator.apache.org.
>> >> Please move the conversation over to Apache mailing list.]
>>
>> >> Kevin,
>>
>> >> The OOM error you pointed out is raised when the proportion of the VM
>> >> time spent is GC crosses a high threshold that should normally not be
>> >> reached. This could happen if your existing heap space for the map
>> >> task is small enough that it compares with the record size you are
>> >> dealing with. You could try increasing your heap space by specifying
>> >> the property mapred.child.java.opts to something like -Xmx4096m,
>> >> assuming your nodes have that much memory to spare. You can also add
>> >> another switch to this property -XX:-UseGCOverheadLimit which will
>> >> disable the policy of the VM that results in OOM errors like you are
>> >> seeing, however doing that may not be of any help.
>>
>> >> Alternatively, you could try using the direct mode of import from
>> >> PostgreSQL server by specifying --direct option during the import.
>> >> This option will require that you have the PostgreSQL client (psql)
>> >> installed on the nodes where the map task will be executed.
>>
>> >> Thanks,
>> >> Arvin
>>
>> >> On Tue, Aug 2, 2011 at 5:09 PM, Kevin <kevinfift...@gmail.com> wrote:
>> >> > Hi all,
>>
>> >> > I am trying to use Sqoop alongside Brisk. For those who don't know,
>> >> > Brisk is DataStax's is a Hadoop/Hive distribution powered by
>> >> > Cassandra.http://www.datastax.com/products/brisk
>>
>> >> > I'm attempting to use Sqoop to transfer data from a PostgreSQL db to
>> >> > Hive.
>> >> > I used this command: sqoop import --connect jdbc:postgresql://
>> >> > idb.corp.redfin.com:5432/metrics --username redfin_readonly -P --table
>> >> > metrics --target-dir /data/qatest --hive-import
>>
>> >> > A end of the output is:
>>
>> >> > 11/08/02 16:50:15 INFO mapred.LocalJobRunner:
>> >> > 11/08/02 16:50:16 INFO mapred.JobClient:  map 100% reduce 0%
>> >> > 11/08/02 16:50:18 INFO mapreduce.AutoProgressMapper: Auto-progress
>> >> > thread is finished. keepGoing=false
>> >> > 11/08/02 16:50:18 WARN mapred.LocalJobRunner: job_local_0001
>> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/
>> >> > 6026311161398729600_27875872_1046092330/file/usr/lib/sqoop/lib/ant-
>> >> > contrib-1.0b3.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/
>> >> > archive/-2957435182115348485_1795484550_1046091330/file/usr/lib/sqoop/
>> >> > lib/ant-eclipse-1.0-jvm1.2.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/
>> >> > 8769703156041621920_1132758636_1046087330/file/usr/lib/sqoop/lib/
>> >> > avro-1.5.1.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/
>> >> > archive/-9111846482501201198_-633786885_1046093330/file/usr/lib/sqoop/
>> >> > lib/avro-ipc-1.5.1.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/
>> >> > archive/-7340432634222599452_756368084_1046091330/file/usr/lib/sqoop/
>> >> > lib/avro-mapred-1.5.1.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/
>> >> > archive/-5046079240639376542_-1808425119_1046090330/file/usr/lib/sqoop/
>> >> > lib/commons-io-1.4.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/
>> >> > 8537290295187062884_-810674145_1046086330/file/usr/lib/sqoop/lib/
>> >> > ivy-2.0.0-rc2.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/
>> >> > archive/-3739620623688167588_-1832479804_1046082330/file/usr/lib/sqoop/
>> >> > lib/jackson-core-asl-1.7.3.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/
>> >> > 3083352659231038596_-1724007002_1046089330/file/usr/lib/sqoop/lib/
>> >> > jackson-mapper-asl-1.7.3.jar
>> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager:
>> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/
>> >> > 2334745090627744860_-1029425194_1046082330/file/usr/lib/sqoop/lib/jopt-
>> >> > simple-3.2.jar
>>
>> ...
>>
>> read more »
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop
mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to
incubator-sqoop-user-subscribe@apache.org.
>

Mime
View raw message