sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki" <iwasak...@nttdata.co.jp>
Subject Re: Review Request: SQOOP-390: PostgreSQL connector for direct export with pg_bulkload
Date Mon, 30 Jul 2012 00:57:09 GMT


> On July 26, 2012, 9:13 p.m., Jarek Cecho wrote:
> > /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportReducer.java, line 83
> > <https://reviews.apache.org/r/2724/diff/3/?file=129307#file129307line83>
> >
> >     I'm thinking whether it would be beneficial here not to die immediately. For
example if user is exporting 30 partitions and there will be some issues with the first. He
would be forced to manually do all 30 of them. But if we would try all them first and log
only the failed ones, user would have to fix only that one. What do you think sir?

The motivation for the current design is keeping destination table clean by staging in a single
atomic transaction. I think making a number of reduce tasks configurable is a good solution.
Users can choose single atomic staging or parallel staging.


- Masatake


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2724/#review9514
-----------------------------------------------------------


On July 26, 2012, 10:41 a.m., Masatake Iwasaki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2724/
> -----------------------------------------------------------
> 
> (Updated July 26, 2012, 10:41 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Description
> -------
> 
> Patch for SQOOP-390
> https://issues.apache.org/jira/browse/SQOOP-390
> 
> 
> This addresses bug SQOOP-390.
>     https://issues.apache.org/jira/browse/SQOOP-390
> 
> 
> Diffs
> -----
> 
>   /src/java/org/apache/sqoop/manager/PGBulkloadManager.java PRE-CREATION 
>   /src/java/org/apache/sqoop/mapreduce/AutoProgressReducer.java PRE-CREATION 
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportJob.java PRE-CREATION 
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportMapper.java PRE-CREATION 
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportReducer.java PRE-CREATION 
>   /src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/2724/diff/
> 
> 
> Testing
> -------
> 
> This patch include the test class PGBulkloadManagerTest.
> I've tested "ant test" and passed.
> 
> 
> Thanks,
> 
> Masatake Iwasaki
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message