sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Foerster <anthonymfoers...@gmail.com>
Subject Review Request 60453: Add --timestamp-query for sqoop-job to get timestamp from database
Date Wed, 12 Jul 2017 13:14:22 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60453/
-----------------------------------------------------------

Review request for Sqoop and Jarek Cecho.


Bugs: 2402, 2985 and 3045
    https://issues.apache.org/jira/browse/2402
    https://issues.apache.org/jira/browse/2985
    https://issues.apache.org/jira/browse/3045


Repository: sqoop-trunk


Description
-------

This patch adds a `--timestamp-query` option to be used with incremental pulls. Sqoop goes
out to the source database to get the timestamp but if there is no ConnectionManager for the
source database it uses 'SELECT CURRENT_TIMESTAMP()' by default which doesn't work for Hana,
Sybase, Teradata, and others.

The alternative would be to write ConnectionManagers for each type of database, but I thought
this generic solution would be better.

I did this for Hana and Sybase specifically, but there are three existing bugs for other databases
it would fix.

The patch should fix these open issues with the new --timestamp-query option:
https://issues.apache.org/jira/browse/SQOOP-2402
https://issues.apache.org/jira/browse/SQOOP-2421
https://issues.apache.org/jira/browse/SQOOP-2627


Diffs
-----

  src/java/org/apache/sqoop/SqoopOptions.java 30b47058 
  src/java/org/apache/sqoop/manager/SqlManager.java 768507b0 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 468bf346 
  src/java/org/apache/sqoop/tool/ImportTool.java d3f8b935 


Diff: https://reviews.apache.org/r/60453/diff/1/


Testing
-------

Has been run with sqoop and sqoop job. Property serializes into sqoop metastore. Has been
tested against Hana, Sybase ASE, and DB2. I didn't write any unit tests, the only test that
would make sense would be for SqlManager.java but no test exists and I'm only adding one conditional
there.


Thanks,

Tony Foerster


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message