sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Robson <david.rob...@quest.com>
Subject Re: Review Request 50155: SQOOP-2983: OraOop export has degraded performance with wide tables
Date Thu, 11 Aug 2016 05:05:51 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50155/#review145463
-----------------------------------------------------------



Import the TST_PRODUCT table which is created by the automated tests:

import --direct --connect jdbc:oracle:thin:@//host:1521/service --username oraoop --password
oraoop --table TST_PRODUCT --target-dir tst_product

Export it back to a new table:

export -Doraoop.template.table=tst_product --direct --connect jdbc:oracle:thin:@//host:1521/service
--username oraoop --password oraoop --table TST_PRODUCT_TEST --export-dir tst_product

Then do an update export onto the newly created table:

export --direct --connect jdbc:oracle:thin:@//host:1521/service --username oraoop --password
oraoop --table TST_PRODUCT_TEST --export-dir tst_product --update-key PRODUCT_ID

It doesn't actually update any data due to the following warning:

WARN oracle.OraOopOutputFormatUpdate: 346 records were skipped due to a NULL value within
one of the update-key column(s).

It looks like this is something to do with the quoting of columns but it was working before
this patch. I think the whole table / column quoting is actually quite broken at the moment
so we probably need to do further testing here and possibly raise another issue.

The new automated tests fail for me as well - the second one looks like it is caused by this
quoting issue. The first one is saying the value is too large - perhaps I am running a different
character set to you or something?

Testsuite: org.apache.sqoop.manager.oracle.OraOopTypesTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.532 sec

Testcase: ensureTypesAfterExportMappedAsExpected took 2.519 sec
        Caused an ERROR
ORA-12899: value too large for column "SQOOPTEST"."ORACLE_DATATYPES_TEMPLATE"."C11_CHAR" (actual:
47, maximum: 29)

java.sql.SQLException: ORA-12899: value too large for column "SQOOPTEST"."ORACLE_DATATYPES_TEMPLATE"."C11_CHAR"
(actual: 47, maximum: 29)

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:30)
        at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:931)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1150)
        at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1792)
        at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1745)
        at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:334)
        at org.apache.sqoop.manager.oracle.OraOopTypesTest.ensureTypesAfterExportMappedAsExpected(OraOopTypesTest.java:39)

Testsuite: org.apache.sqoop.manager.oracle.OraOopUpdateKeyTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.195 sec

java.lang.Exception: java.io.IOException: java.sql.SQLSyntaxErrorException: ORA-00904: "B"."product_id":
invalid identifier

        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:399)
Caused by: java.io.IOException: java.sql.SQLSyntaxErrorException: ORA-00904: "B"."product_id":
invalid identifier

        at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:211)
        at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:555)
        at org.apache.sqoop.manager.oracle.OraOopOutputFormatUpdate$OraOopDBRecordWriterUpdate.close(OraOopOutputFormatUpdate.java:113)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:641)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "B"."product_id": invalid identifier

- David Robson


On Aug. 9, 2016, 10:50 p.m., Attila Szabo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50155/
> -----------------------------------------------------------
> 
> (Updated Aug. 9, 2016, 10:50 p.m.)
> 
> 
> Review request for Sqoop, David Robson, Jarek Cecho, and Kathleen Ting.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> Proposed changes for SQOOP-2983
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/manager/oracle/OraOopConnManager.java 216c771 
>   src/java/org/apache/sqoop/manager/oracle/OraOopOracleQueries.java 6b27bd8 
>   src/java/org/apache/sqoop/manager/oracle/OraOopOutputFormatBase.java 8f94cf8 
>   src/java/org/apache/sqoop/manager/oracle/OraOopOutputFormatInsert.java d5eebf4 
>   src/java/org/apache/sqoop/manager/oracle/OraOopOutputFormatUpdate.java a33768f 
>   src/java/org/apache/sqoop/manager/oracle/OraOopUtilities.java e81588c 
>   src/java/org/apache/sqoop/orm/ClassWriter.java 9d91887 
>   src/test/org/apache/sqoop/manager/oracle/ExportTest.java 991b221 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 9fe4821 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java PRE-CREATION 
>   src/test/org/apache/sqoop/manager/oracle/OraOopUpdateKeyTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50155/diff/
> 
> 
> Testing
> -------
> 
> 800 columns with table
> 100.000 lines (156mb data)
> 1.000.000 lines (1.56 GB data)
> 3.000.000 lines (4.5 GB data)
> 
> 
> Thanks,
> 
> Attila Szabo
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message