sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sonya Ling (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-931) Integrate HCatalog with Sqoop
Date Thu, 15 Aug 2013 00:10:48 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740468#comment-13740468
] 

Sonya Ling commented on SQOOP-931:
----------------------------------

I tried sqoop-1.4.4 for this feature today.  I have Hive 0.11.0 that has hcatalog merged into
it. I have Hadoop 2.0.0-cdh4.3.0.    I run statement similar to the followings:
 
sqoop import --connect jdbc:mysql://<host>/<database> --username <user>
--password <password> --table transaction --hcatalog-table transaction --create-hcatalog-table
--where "date >= 08-01-2012" --hive-partition-key date

It went through and started hadoop job but halt due the following error: 
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext,
but class was expected
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94)
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:82)
	at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)

I know this is due to Hadoop version conflict issue. HCatBaseOutputFormat is expecting Hadoop
1.0.x but I have Hadoop 2.0.x.   I saw similar error when I ran Ozzie.  I could get around
by add mapred.mapper.new-api and mapred.reducer.new-api to true in workflow.xml of ozzie.
  I added the same properties to proto-hive-site.xml in hcatalog/etc.  It did not work.  
I checked hcatalog 0.50 source codes.  It does not use those properties.

How can I get around this issue?   Please advise.  Thanks.
                
> Integrate HCatalog with Sqoop
> -----------------------------
>
>                 Key: SQOOP-931
>                 URL: https://issues.apache.org/jira/browse/SQOOP-931
>             Project: Sqoop
>          Issue Type: New Feature
>    Affects Versions: 1.4.2, 1.4.3
>         Environment: All 1.x sqoop version
>            Reporter: Venkat Ranganathan
>            Assignee: Venkat Ranganathan
>             Fix For: 1.4.4
>
>         Attachments: SQOOP-931.patch, SQOOP-931.patch.14, SQOOP HCatalog Integration
- 2.pdf, SQOOP HCatalog Integration - 3.pdf, SQOOP HCatalog Integration.pdf
>
>
>  Apache HCatalog is a table and storage management service that provides a shared schema,
data types and table abstraction freeing users from being concerned about where or how their
data is stored.  It provides interoperability across  Pig, Map Reduce, and Hive.
> A sqoop hcatalog connector will help in supporting storage formats that are abstracted
by HCatalog.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message