sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lin via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename
Date Wed, 11 Jul 2018 07:24:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/
-----------------------------------------------------------

(Updated July 11, 2018, 7:24 a.m.)


Review request for Sqoop and Boglarka Egyed.


Changes
-------

updated based on latest review.


Bugs: SQOOP-3330
    https://issues.apache.org/jira/browse/SQOOP-3330


Repository: sqoop-trunk


Description
-------

When adding --append to Sqoop directory import with -Dmapreduce.output.basename, all files
will be ignored, which end up nothing being imported. See below DEBUG output:

{code}
sqoop import -Dmapreduce.output.basename="eric-test" --connect jdbc:mysql://mysql-host.com/test
--username root --password 'root' --table test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by
'\t' --verbose --append

18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 14935e396acc4ea7b9a6236c66064c9b_test
{code}

This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"

https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46


Diffs (updated)
-----

  .gitignore 68cbe287 
  src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
  src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 


Diff: https://reviews.apache.org/r/67424/diff/2/

Changes: https://reviews.apache.org/r/67424/diff/1-2/


Testing
-------

Added test case + manual test in VM


Thanks,

Eric Lin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message