sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Boglarka Egyed (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1086) Running multiple incremental sqoop jobs in parallel resets the first sqoop job's --last-value
Date Mon, 25 Jul 2016 09:47:20 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391617#comment-15391617

Boglarka Egyed commented on SQOOP-1086:

I could reproduce the bug on three different tables executing parallel incremental import
jobs with the same --check-column parameter using built-in metastore but it didn't occur when
metastore was running as a service.
A possible root cause is some file locking during the writing of local /var/lib/hadoop-hdfs/.sqoop/metastore.db.script
file which stores incremental.last.value in case of built in metastore usage. Further investigation
is in progress regarding a proper resolution.

> Running multiple incremental sqoop jobs in parallel resets the first sqoop job's --last-value
> ---------------------------------------------------------------------------------------------
>                 Key: SQOOP-1086
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1086
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.0-incubating
>         Environment: Ubuntu 12.04.2
>            Reporter: Byron
>            Assignee: Boglarka Egyed
>            Priority: Critical
>              Labels: import, incremental, job, parallel
> I've created 2 jobs (different names) that pull from the same database(MSSQL), but 2
different tables.
> They both use incremental append.
> If I run the jobs in sequence, I got no issue and the meta store for both jobs remembers
the --last-value per job.
> If I run the jobs in parallel, when the 1st job finished the meta is updated with the
--last-value correctly, but once the 2nd job finished the 1st job's meta --last-value is reset.
> First Job
> # create the import job into the incremental table
> $ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 1" --create "import-events" --
import --connect "$ENV_TRACKING_CONNECTION" --table "$TABLE1" --split-by "dtmDBDateTime" --target-dir
"$OUTPUT1" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000"
--fields-terminated-by \\t --null-string '' --null-non-string '';
> Second Job
> # create the import job into the table
> $ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 2" --create "import-impressions"
-- import --connect "$ENV_TRACKING_CONNECTION" --table "$TABLE2" --split-by "dtmDBDateTime"
--target-dir "$OUTPUT2" --incremental append --check-column "dtmDBDateTime" --last-value "2012-01-01
00:00:00.000" --fields-terminated-by \\t --null-string '' --null-non-string '';

This message was sent by Atlassian JIRA

View raw message