sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Illya Yalovyy <yalov...@amazon.com>
Subject Re: Review Request 56770: SQOOP-3136 Sqoop should work well with not default file systems
Date Fri, 24 Feb 2017 16:21:06 GMT


> On Feb. 23, 2017, 11:01 p.m., Attila Szabo wrote:
> > Hey Illya,
> > 
> > Thank you for you contribution. This improvement looks great from both testing and
implementation POV.
> > I do hope we will see same great Sqoop patches from you in the future as well!
> > 
> > Cheers,
> > Attila

Thank you!


- Illya


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56770/#review166601
-----------------------------------------------------------


On Feb. 22, 2017, 10:01 p.m., Illya Yalovyy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56770/
> -----------------------------------------------------------
> 
> (Updated Feb. 22, 2017, 10:01 p.m.)
> 
> 
> Review request for Sqoop, Jarek Cecho, Attila Szabo, and Venkat Ranganathan.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> Currently Sqoop assumes default file system when it comes to IO operations. It makes
it hard to use other FileSystem implementations as source or destination.
> 
> https://issues.apache.org/jira/browse/SQOOP-3136
> 
> 
> Diffs
> -----
> 
>   src/java/com/cloudera/sqoop/io/LobReaderCache.java 3394296 
>   src/java/org/apache/sqoop/hive/HiveImport.java 4828375 
>   src/java/org/apache/sqoop/hive/TableDefWriter.java c9962e9 
>   src/java/org/apache/sqoop/io/LobReaderCache.java bd75374 
>   src/java/org/apache/sqoop/io/SplittingOutputStream.java 5f98192 
>   src/java/org/apache/sqoop/lib/LargeObjectLoader.java 70c0f4e 
>   src/java/org/apache/sqoop/manager/oracle/OraOopUtilities.java e81588c 
>   src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java e08f997 
>   src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 260bc29 
>   src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 27f84da 
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java b32cdd1 
>   src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 626119b 
>   src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java f911280 
>   src/java/org/apache/sqoop/mapreduce/MergeJob.java 5b6c4df 
>   src/java/org/apache/sqoop/tool/ImportTool.java 258ef79 
>   src/java/org/apache/sqoop/util/FileSystemUtil.java PRE-CREATION 
>   src/java/org/apache/sqoop/util/FileUploader.java 155cffc 
>   src/test/org/apache/sqoop/util/TestFileSystemUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/56770/diff/
> 
> 
> Testing
> -------
> 
> ** Build:
> ant clean package
> ...
> BUILD SUCCESSFUL
> Total time: 51 seconds
> 
> 
> ** Test:
> ant test
> ...
> BUILD SUCCESSFUL
> Total time: 7 minutes 21 seconds
> 
> * On Hadoop Cluster:
> 
> ** original version of sqoop:
> sqoop import --connect <JDBC URL> --table table1 --driver <JDBC DRIVER> --username
root --password **** --delete-target-dir --target-dir s3a://some-bucket/tmp/sqoop
> ...
> 17/02/15 19:16:59 ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://some-bucket/tmp/sqoop,
expected: hdfs://<DNS>:8020
> 
> ** updated version of sqoop:
> sqoop import --connect <JDBC URL> --table table1 --driver <JDBC DRIVER> --username
root --password **** --delete-target-dir --target-dir s3a://some-bucket/tmp/sqoop
> ...
> 17/02/15 22:24:42 INFO mapreduce.Job: Running job: job_1487183144282_0004
> 17/02/15 22:24:52 INFO mapreduce.Job: Job job_1487183144282_0004 running in uber mode
: false
> 17/02/15 22:24:52 INFO mapreduce.Job:  map 0% reduce 0%
> 17/02/15 22:25:04 INFO mapreduce.Job:  map 25% reduce 0%
> 17/02/15 22:25:06 INFO mapreduce.Job:  map 50% reduce 0%
> 17/02/15 22:25:07 INFO mapreduce.Job:  map 75% reduce 0%
> 17/02/15 22:25:08 INFO mapreduce.Job:  map 100% reduce 0%
> 17/02/15 22:25:08 INFO mapreduce.Job: Job job_1487183144282_0004 completed successfully
> 17/02/15 22:25:08 INFO mapreduce.Job: Counters: 36
> ...
> 17/02/15 22:25:08 INFO mapreduce.ImportJobBase: Retrieved 4993 records.
> 
> 
> Thanks,
> 
> Illya Yalovyy
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message