sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szabolcs Vasas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2994) Incremental lastmodified lower bound value not getting updated
Date Thu, 18 Aug 2016 07:32:21 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426024#comment-15426024

Szabolcs Vasas commented on SQOOP-2994:

Hi [~ektadhusia],

Do you use local Sqoop metastore and do you run more than one jobs at the same time?
Local Sqoop metastore stores the values in one file (as a series of INSERT commands) and in
case of multiple simultaneous Sqoop jobs it might not get updated properly.
There is a JIRA created for documenting this deficiency of the local metastore: https://issues.apache.org/jira/browse/SQOOP-2988

I would recommend using a shared metastore.


> Incremental lastmodified lower bound value not getting updated
> --------------------------------------------------------------
>                 Key: SQOOP-2994
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2994
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Ekta Dhusia
> I am using incremental  lastmodified import and when I am running the job after the first
run the lower bound is not getting updated.
> Suppose I am running the sqoop import job on 3/08/2016, all the data from the oracle
table is imported properly. The incremental last value is changed to the current TS. Now if
I run the on 4/08/2016, the incremental last value is not updated to current TS. This is going
to affect the run for 5/08/2016.
> My sqoop command:
> sqoop job --create jobname -- import --connect connectionstring --username username --password-file
password_file --table tablename --columns "col1,col2,col3" --target-dir target_dir --incremental
lastmodified --check-column UPDATED_TS --last-value '2016-07-11 00:00:00.0' 
> I was getting some file already exist error for the target dir, so I am recreating the
target dir for each load.
> Sqoop version: Sqoop

This message was sent by Atlassian JIRA

View raw message