sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Szabo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3022) sqoop export for Oracle generates tremendous amounts of redo logs
Date Tue, 11 Oct 2016 22:49:22 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566863#comment-15566863

Attila Szabo commented on SQOOP-3022:

Hi [~Tagar],

I think for this related problem had the designers created the PreparedStatement interface
( https://docs.oracle.com/javase/7/docs/api/java/sql/PreparedStatement.html ) in the JDBC
API and generally batching ( see addBatch and executeBatch methods on both PreparedStatement
and Statement interfaces ).

I'm not 100% sure (because it's been a while I'd dig deep into Oracle's proprietary JDBC driver
through a decompiler) but I do think every batch execute is doing only 1 network roundtrip
+ AFAIR JDBC is also supporting compression over the network.

Especially in case of Sqoop we're depending on JDBC PreparedStatement batching functionality.

So I think in our current case it would not much matter, if we would concatenate all the statements
as String on Java side and send out one query, or rather with the help of PreparedStatement
we would be able to send it out as a batch (and AFAIR PreparedStatement could be cached on
Oracle side, so even query parsing is cheaper, but I'm not sure if it's true all the cases,
or just in case of named params).

What do you think? Do you still have concerns and suggesting the usage INSERT ALL statements?


> sqoop export for Oracle generates tremendous amounts of redo logs
> -----------------------------------------------------------------
>                 Key: SQOOP-3022
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3022
>             Project: Sqoop
>          Issue Type: Bug
>          Components: codegen, connectors, connectors/oracle
>    Affects Versions: 1.4.3, 1.4.4, 1.4.5, 1.4.6
>            Reporter: Ruslan Dautkhanov
>              Labels: export, oracle
> Sqoop export for Oracle generates tremendous amounts of redo logs (comparable to export
size or more).
> We have put target tables in nologgin mode, but Oracle will still generate redo logs
unless +APPEND Oracle insert hint is used.
> See https://oracle-base.com/articles/misc/append-hint for examples.
> Please add an option for sqoop to generate insert statements in Oracle with APPEND statement.
Our databases are swamped with redo/archived logs whenever we sqoop data to them. This is
easily avoidable. And from business prospective sqooping to staging tables in nologgin mode
is totally fine.
> Thank you.

This message was sent by Atlassian JIRA

View raw message