sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sai Karthik Ganguru (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1125) Out of memory errors when number of records to import < 0.5 * splitSize
Date Wed, 19 Nov 2014 22:32:34 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218623#comment-14218623
] 

Sai Karthik Ganguru commented on SQOOP-1125:
--------------------------------------------

Well. I have created separate patches, one for the development part of it and the other which
has the test cases in it. Please try to patch them up separately and let us see if that works.

Attached them now [~jarcec]

> Out of memory errors when number of records to import < 0.5 * splitSize
> -----------------------------------------------------------------------
>
>                 Key: SQOOP-1125
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1125
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.3
>            Reporter: Dave Kincaid
>            Assignee: Sai Karthik Ganguru
>            Priority: Critical
>              Labels: newbie
>         Attachments: sqoop-1125-1.patch, sqoop-1125-2.patch
>
>
> We are getting out of memory errors during import if the number of records to import
is less than 0.5*splitSize (and is nonterminating decimal).
> For example, if the numSplits = 3, minVal = 100, maxVal = 101 then in BigDecimalSplitter.split()
an extraordinary number of tiny values will be added to the splits List and run out of memory
eventually.
> I also noticed that there are no tests for BigDecimalSplitter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message