sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arvind Prabhakar" <arv...@apache.org>
Subject Re: Review Request: SQOOP-314 Basic export hangs when target database does not support INSERT syntax with multiple rows of values
Date Thu, 18 Aug 2011 22:35:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1531
-----------------------------------------------------------


Changes look good Bilung. One high-level suggestion is to modify OracleManager and SQLServerManager
to use the new ExportBatchOutputFormat. The classes for OracleExportOutputFormat and SQLSErverExportOutputFormat
can then be deprecated.


src/java/com/cloudera/sqoop/SqoopOptions.java
<https://reviews.apache.org/r/1585/#comment3483>

    This will trigger a checkstyle warning. Should change the argument name to mode or something
else.



src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java
<https://reviews.apache.org/r/1585/#comment3486>

    Thread.stop() is unsafe operation that can lead to corruption. Would it help if the join
call was part of the try block instead of in finally? In that case, throwing the IOException
would automatically get rid of the thread.



src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java
<https://reviews.apache.org/r/1585/#comment3484>

    Longer than 80.


- Arvind


On 2011-08-18 17:46:40, Bilung Lee wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/1585/
> -----------------------------------------------------------
> 
> (Updated 2011-08-18 17:46:40)
> 
> 
> Review request for Sqoop, Arvind Prabhakar and jmhsieh.
> 
> 
> Summary
> -------
> 
> Basic export job will hang when the target database does not support insert syntax with
multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23,
24)
> 
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread
to finish even when an SQLException is thrown underneath.
> 
> The fix is to introduce a new "--batch" option to execute underlying statements in batch
mode instead of a single multirow insert statement.
> 
> 
> This addresses bug SQOOP-314.
>     https://issues.apache.org/jira/browse/SQOOP-314
> 
> 
> Diffs
> -----
> 
>   src/docs/man/sqoop-export.txt 6090ea1 
>   src/docs/user/export.txt 4f87886 
>   src/java/com/cloudera/sqoop/SqoopOptions.java d760d39 
>   src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41 
>   src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION 
>   src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37 
>   src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6 
>   src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1 
>   src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091 
>   src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c 
> 
> Diff: https://reviews.apache.org/r/1585/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Bilung
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message