sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Voros (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3336) Splitting on integer column can create more splits than necessary
Date Fri, 22 Jun 2018 09:42:00 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520176#comment-16520176
] 

Daniel Voros commented on SQOOP-3336:
-------------------------------------

Attached review request.

This also affects splitting on date/timestamp columns, since DateSplitter uses the same logic.

> Splitting on integer column can create more splits than necessary
> -----------------------------------------------------------------
>
>                 Key: SQOOP-3336
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3336
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.7
>            Reporter: Daniel Voros
>            Assignee: Daniel Voros
>            Priority: Major
>             Fix For: 1.5.0, 3.0.0
>
>
> Running an import with {{-m 2}} will result in three splits if there are only three consecutive
integers in the table ({{\{1, 2, 3\}}}).
> Work is (probably) spread more evenly between mappers this way, but ending up with more
files than expected could be an issue.
> Split-limit can also result in more values than asked for in the last chunk (due to the
closed interval in the end).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message