sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1617) MySQL fetch-size behavior changed with SQOOP-1400
Date Mon, 03 Nov 2014 01:52:33 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jarek Jarcec Cecho updated SQOOP-1617:
--------------------------------------
    Attachment: SQOOP-1617.patch

It seems that our change in SQOOP-1400 has changed the default behavior from "streaming row
by row" to "cache everything in memory" which is causing various out of memory problems to
our users as Sqoop has been design to transfer huge amount of data. Hence I'm recommending
to put the code back.

> MySQL fetch-size behavior changed with SQOOP-1400
> -------------------------------------------------
>
>                 Key: SQOOP-1617
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1617
>             Project: Sqoop
>          Issue Type: Bug
>          Components: connectors/mysql
>    Affects Versions: 1.4.6
>         Environment: CDH 5.2
> sqoop 1.4.5 (seems to include SQOOP-1400)
> mysql connector version 5.1.33
>            Reporter: J├╝rgen Thomann
>            Priority: Minor
>             Fix For: 1.4.6
>
>         Attachments: SQOOP-1617.patch
>
>
> SQOOP-1400 changed the default behavior for the connector to load everything in memory.
The only working way to get the old streaming back is to use --fetch-size -2147483648 (Integer.MIN_VALUE)
> It would be nice if that could be changed and/or documented that mysql does not support
a fetch size and does only support row-by-row or loading everything in memory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message