sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Byron (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-1086) Running multiple incremental sqoop jobs in parallel resets the first sqoop job's --last-value
Date Mon, 17 Jun 2013 15:32:20 GMT
Byron created SQOOP-1086:
----------------------------

             Summary: Running multiple incremental sqoop jobs in parallel resets the first
sqoop job's --last-value
                 Key: SQOOP-1086
                 URL: https://issues.apache.org/jira/browse/SQOOP-1086
             Project: Sqoop
          Issue Type: Bug
    Affects Versions: 1.4.0-incubating
         Environment: Ubuntu 12.04.2
            Reporter: Byron
            Priority: Critical


I've created 2 jobs (different names) that pull from the same database(MSSQL), but 2 different
tables.
They both use incremental append.

If I run the jobs in sequence, I got no issue and the meta store for both jobs remembers the
--last-value per job.

If I run the jobs in parallel, when the 1st job finished the meta is updated with the --last-value
correctly, but once the 2nd job finished the 1st job's meta --last-value is reset.

First Job
# create the import job into the incremental table
$ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 1" --create "import-events" -- import
--connect "$ENV_TRACKING_CONNECTION" --table "$TABLE1" --split-by "dtmDBDateTime" --target-dir
"$OUTPUT1" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000"
--fields-terminated-by \\t --null-string '' --null-non-string '';

Second Job
# create the import job into the table
$ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 2" --create "import-impressions" --
import --connect "$ENV_TRACKING_CONNECTION" --table "$TABLE2" --split-by "dtmDBDateTime" --target-dir
"$OUTPUT2" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000"
--fields-terminated-by \\t --null-string '' --null-non-string '';

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message