sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1685) HCatalog integration is not working on JDK8
Date Thu, 06 Nov 2014 02:35:34 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jarek Jarcec Cecho updated SQOOP-1685:
    Attachment: SQOOP-1685.patch

> HCatalog integration is not working on JDK8
> -------------------------------------------
>                 Key: SQOOP-1685
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1685
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.5
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.4.6
>         Attachments: SQOOP-1685.patch
> I was looking into JDK8 compatibility in Sqoop 1 and I found out that HCatalog tests
are broken, failing on exceptions such as this one:
> {code}
> 14/11/05 16:29:06 ERROR tool.ImportTool: Encountered IOException running import job:
java.io.IOException: Projected column id not in list of columns from database
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.initDBColumnInfo(SqoopHCatUtilities.java:483)
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:314)
> 	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:753)
> {code}
> Looking deeper into our HCatalog code, we do have class {{[LCKeyMap|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L169]}}
that is suppose to store all it's keys in lower case form and provide case independent lookup.
Indeed we're overriding methods {{put}} and {{get}} and lowercasing the key parameter to achieve
desired behavior.
> Now to the actual problem. We are in the code calling method {{[putAll|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L483]}}
to bulk insert values from another Map. This worked in JDK7 as the [underlaying implementation|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.putAll%28java.util.Map%29]
of {{putAll}} method is calling {{put}} method that we're overriding. It seems that this [has
changed in JDK8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java#HashMap.putAll%28java.util.Map%29]
where the {{putAll}} method is no longer calling {{put}} and hence when we call {{putAll}}
all the keys are stored with their original case and subsequent lookup fails

This message was sent by Atlassian JIRA

View raw message