sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sonya Ling (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-931) Integrate HCatalog with Sqoop
Date Thu, 15 Aug 2013 00:10:48 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740468#comment-13740468

Sonya Ling commented on SQOOP-931:

I tried sqoop-1.4.4 for this feature today.  I have Hive 0.11.0 that has hcatalog merged into
it. I have Hadoop 2.0.0-cdh4.3.0.    I run statement similar to the followings:
sqoop import --connect jdbc:mysql://<host>/<database> --username <user>
--password <password> --table transaction --hcatalog-table transaction --create-hcatalog-table
--where "date >= 08-01-2012" --hive-partition-key date

It went through and started hadoop job but halt due the following error: 
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext,
but class was expected
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94)
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:82)
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)

I know this is due to Hadoop version conflict issue. HCatBaseOutputFormat is expecting Hadoop
1.0.x but I have Hadoop 2.0.x.   I saw similar error when I ran Ozzie.  I could get around
by add mapred.mapper.new-api and mapred.reducer.new-api to true in workflow.xml of ozzie.
  I added the same properties to proto-hive-site.xml in hcatalog/etc.  It did not work.  
I checked hcatalog 0.50 source codes.  It does not use those properties.

How can I get around this issue?   Please advise.  Thanks.
> Integrate HCatalog with Sqoop
> -----------------------------
>                 Key: SQOOP-931
>                 URL: https://issues.apache.org/jira/browse/SQOOP-931
>             Project: Sqoop
>          Issue Type: New Feature
>    Affects Versions: 1.4.2, 1.4.3
>         Environment: All 1.x sqoop version
>            Reporter: Venkat Ranganathan
>            Assignee: Venkat Ranganathan
>             Fix For: 1.4.4
>         Attachments: SQOOP-931.patch, SQOOP-931.patch.14, SQOOP HCatalog Integration
- 2.pdf, SQOOP HCatalog Integration - 3.pdf, SQOOP HCatalog Integration.pdf
>  Apache HCatalog is a table and storage management service that provides a shared schema,
data types and table abstraction freeing users from being concerned about where or how their
data is stored.  It provides interoperability across  Pig, Map Reduce, and Hive.
> A sqoop hcatalog connector will help in supporting storage formats that are abstracted
by HCatalog.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message