sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2632) Sqoop2: add partition class to FROM
Date Fri, 23 Oct 2015 20:22:27 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971771#comment-14971771
] 

ASF subversion and git services commented on SQOOP-2632:
--------------------------------------------------------

Commit 0773c10f83d2714cd05ce37393ec968f940ad302 in sqoop's branch refs/heads/sqoop2 from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=0773c10 ]

SQOOP-2632: Sqoop2: add partition class to FROM

(Dian Fu via Jarek Jarcec Cecho)


> Sqoop2: add partition class to FROM
> -----------------------------------
>
>                 Key: SQOOP-2632
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2632
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Dian Fu
>            Assignee: Dian Fu
>             Fix For: 1.99.7
>
>         Attachments: SQOOP-2632.001.patch
>
>
> Each {{SqoopMapper}} will process one {{SqoopSplit}}. {{SqoopSplit}} is instantiated
by the MR framework. Let's suppose that MR framework loads {{SqoopSplit}} with classloader
A. As {{SqoopSplit}} has one field {{partition}}, so when {{SqoopSplit}} is loaded by classloader
A, class {{Partition}} will also be loaded by classloader A and field {{partition}} will be
of type {{A.Partition}}. 
> In {{SqoopMapper}}, we will call {{extractor.extract(extractorContext, fromConfig, fromJob,
partition)}}. To provide classpath isolation, we will load class {{Extractor}} with a custom
classloader {{B}}, so class {{Partition}} in {{Extractor}} will be loaded by classloader {{B}}.
This will cause type miss-match between {{A.Partition}} and {{B.Partition}} when we call {{extractor.extract(extractorContext,
fromConfig, fromJob, partition}}.
> To solve this issue, we need to make sure class {{Partition}} not be loaded by the custom
classloader {{B}}. To achieve this, we need firstly provide a mechanism to let {{SqoopMapper}}
know the class name of {{Partition}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message