sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lin via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename
Date Fri, 13 Jul 2018 05:26:44 GMT


> On July 10, 2018, 4:05 p.m., Boglarka Egyed wrote:
> > .gitignore
> > Lines 39 (patched)
> > <https://reviews.apache.org/r/67424/diff/1/?file=2034783#file2034783line39>
> >
> >     Just as in https://reviews.apache.org/r/54528/ why do we need this here?
> 
> Eric Lin wrote:
>     It happened to me a few times that "out" directory was created after running "ant
jar-all" or "ant test" commands, as those are the ones I run most often. However, not every
time though, so I am not sure what exactly will create this directory, but definitely happened
to me several times. (I tested again, it was not created anymore)

Removed


- Eric


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/#review205902
-----------------------------------------------------------


On July 13, 2018, 5:25 a.m., Eric Lin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67424/
> -----------------------------------------------------------
> 
> (Updated July 13, 2018, 5:25 a.m.)
> 
> 
> Review request for Sqoop and Boglarka Egyed.
> 
> 
> Bugs: SQOOP-3330
>     https://issues.apache.org/jira/browse/SQOOP-3330
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> When adding --append to Sqoop directory import with -Dmapreduce.output.basename, all
files will be ignored, which end up nothing being imported. See below DEBUG output:
> 
> {code}
> sqoop import -Dmapreduce.output.basename="eric-test" --connect jdbc:mysql://mysql-host.com/test
--username root --password 'root' --table test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by
'\t' --verbose --append
> 
> 18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 14935e396acc4ea7b9a6236c66064c9b_test
> {code}
> 
> This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"
> 
> https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
>   src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 
> 
> 
> Diff: https://reviews.apache.org/r/67424/diff/4/
> 
> 
> Testing
> -------
> 
> Added test case + manual test in VM
> 
> 
> Thanks,
> 
> Eric Lin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message