sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-603) Support small intervals in IntegerSplitter implementation
Date Thu, 20 Sep 2012 20:53:07 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459943#comment-13459943
] 

Hudson commented on SQOOP-603:
------------------------------

Integrated in Sqoop-ant-jdk-1.6-hadoop200 #56 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/56/])
    SQOOP-603 Support small intervals in IntegerSplitter implementation (Revision 5616152ac4c96d6c0589768b982cf67f3277df74)

     Result = SUCCESS
cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5616152ac4c96d6c0589768b982cf67f3277df74
Files : 
* src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java
* src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
* src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java

                
> Support small intervals in IntegerSplitter implementation
> ---------------------------------------------------------
>
>                 Key: SQOOP-603
>                 URL: https://issues.apache.org/jira/browse/SQOOP-603
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.2
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 1.4.3
>
>         Attachments: SQOOP-603.patch
>
>
> IntegerSplitter is currently creating splits of following nature:
> {code}
> minimal value <= x < splitPoint1
> splitPoint1 <= x < splitPoint2
> ...
> splitPointN <= x <= maximal value
> {code}
> Please notice that upper bound is always with using condition "<" with exception of
the last split that is using condition "<=". This is perfectly fine when creating reasonable
amount of splits on very huge interval.
> This approach will however cause issues on very small intervals. For example following
splits will be created on interval [0, 5] with 5 splits:
> * 0 <= x < 1
> * 1 <= x < 2 
> * 2 <= x < 3 
> * 3 <= x < 4 
> * 4 <= x <= 5
> Notice that all splits have equal count of numbers except the last one having two numbers
- 4 and 5. This becomes very huge issue when for example user needs to create one split per
one partition as one mapper will end up with moving two partitions and thus entire job will
take twice as long as the other ones.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message