sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bilung Lee" <bleeapa...@gmail.com>
Subject Re: Review Request: SQOOP-314 Basic export hangs when target database does not support INSERT syntax with multiple rows of values
Date Fri, 19 Aug 2011 00:49:17 GMT


> On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
> > Changes look good Bilung. One high-level suggestion is to modify OracleManager and
SQLServerManager to use the new ExportBatchOutputFormat. The classes for OracleExportOutputFormat
and SQLSErverExportOutputFormat can then be deprecated.

Done in the new patch.  Thanks.


> On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
> > src/java/com/cloudera/sqoop/SqoopOptions.java, line 1026
> > <https://reviews.apache.org/r/1585/diff/1/?file=33455#file33455line1026>
> >
> >     This will trigger a checkstyle warning. Should change the argument name to mode
or something else.

Done in the new patch.  Thanks.


> On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
> > src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java, line 41
> > <https://reviews.apache.org/r/1585/diff/1/?file=33457#file33457line41>
> >
> >     Longer than 80.

Done in the new patch.  Thanks.


> On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
> > src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java, line 179
> > <https://reviews.apache.org/r/1585/diff/1/?file=33456#file33456line179>
> >
> >     Thread.stop() is unsafe operation that can lead to corruption. Would it help
if the join call was part of the try block instead of in finally? In that case, throwing the
IOException would automatically get rid of the thread.

Done in the new patch.  Thanks.


- Bilung


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1531
-----------------------------------------------------------


On 2011-08-19 00:49:12, Bilung Lee wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/1585/
> -----------------------------------------------------------
> 
> (Updated 2011-08-19 00:49:12)
> 
> 
> Review request for Sqoop, Arvind Prabhakar and jmhsieh.
> 
> 
> Summary
> -------
> 
> Basic export job will hang when the target database does not support insert syntax with
multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23,
24)
> 
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread
to finish even when an SQLException is thrown underneath.
> 
> The fix is to introduce a new "--batch" option to execute underlying statements in batch
mode instead of a single multirow insert statement.
> 
> 
> This addresses bug SQOOP-314.
>     https://issues.apache.org/jira/browse/SQOOP-314
> 
> 
> Diffs
> -----
> 
>   src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6 
>   src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1 
>   src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091 
>   src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c 
>   src/docs/man/sqoop-export.txt 6090ea1 
>   src/docs/user/export.txt 4f87886 
>   src/java/com/cloudera/sqoop/SqoopOptions.java d760d39 
>   src/java/com/cloudera/sqoop/manager/OracleManager.java 6a55312 
>   src/java/com/cloudera/sqoop/manager/SQLServerManager.java e1ce2af 
>   src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41 
>   src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION 
>   src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37 
> 
> Diff: https://reviews.apache.org/r/1585/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Bilung
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message