sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1685) HCatalog integration is not working on JDK8
Date Thu, 06 Nov 2014 05:17:33 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199812#comment-14199812

ASF subversion and git services commented on SQOOP-1685:

Commit 933e71ef303f5330c541a6e8f3c5641f368d8105 in sqoop's branch refs/heads/trunk from [~abec]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=933e71e ]

SQOOP-1685: HCatalog integration is not working on JDK8

(Jarek Jarcec Cecho via Abraham Elmahrek)

> HCatalog integration is not working on JDK8
> -------------------------------------------
>                 Key: SQOOP-1685
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1685
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.5
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.4.6
>         Attachments: SQOOP-1685.patch
> I was looking into JDK8 compatibility in Sqoop 1 and I found out that HCatalog tests
are broken, failing on exceptions such as this one:
> {code}
> 14/11/05 16:29:06 ERROR tool.ImportTool: Encountered IOException running import job:
java.io.IOException: Projected column id not in list of columns from database
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.initDBColumnInfo(SqoopHCatUtilities.java:483)
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:314)
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:753)
> {code}
> Looking deeper into our HCatalog code, we do have class {{[LCKeyMap|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L169]}}
that is suppose to store all it's keys in lower case form and provide case independent lookup.
Indeed we're overriding methods {{put}} and {{get}} and lowercasing the key parameter to achieve
desired behavior.
> Now to the actual problem. We are in the code calling method {{[putAll|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L483]}}
to bulk insert values from another Map. This worked in JDK7 as the [underlaying implementation|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.putAll%28java.util.Map%29]
of {{putAll}} method is calling {{put}} method that we're overriding. It seems that this [has
changed in JDK8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java#HashMap.putAll%28java.util.Map%29]
where the {{putAll}} method is no longer calling {{put}} and hence when we call {{putAll}}
all the keys are stored with their original case and subsequent lookup fails

This message was sent by Atlassian JIRA

View raw message