sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Kincaid (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-1125) Out of memory errors when number of records to import < 0.5 * splitSize
Date Mon, 08 Jul 2013 19:19:48 GMT
Dave Kincaid created SQOOP-1125:
-----------------------------------

             Summary: Out of memory errors when number of records to import < 0.5 * splitSize
                 Key: SQOOP-1125
                 URL: https://issues.apache.org/jira/browse/SQOOP-1125
             Project: Sqoop
          Issue Type: Bug
    Affects Versions: 1.4.3
            Reporter: Dave Kincaid
            Priority: Critical


We are getting out of memory errors during import if the number of records to import is less
than 0.5*splitSize (and is nonterminating decimal).

For example, if the numSplits = 3, minVal = 100, maxVal = 101 then in BigDecimalSplitter.split()
an extraordinary number of tiny values will be added to the splits List and run out of memory
eventually.

I also noticed that there are no tests for BigDecimalSplitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message