sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1666) Sqoop2: JobManager code on failure should call SqoopDestroyerExecutor
Date Tue, 04 Nov 2014 19:23:33 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196640#comment-14196640
] 

Jarek Jarcec Cecho commented on SQOOP-1666:
-------------------------------------------

Comparing {{[SqoopDestroyerExecutor|https://github.com/apache/sqoop/blob/sqoop2/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java]}}
and {{[JobManager|https://github.com/apache/sqoop/blob/sqoop2/core/src/main/java/org/apache/sqoop/driver/JobManager.java#L505]}}
code that executes the destroyer, it seems to me that the main difference is in the way we
get all the parameters. The Executor runs inside mapreduce job so it needs to load all of
them from jobconf whereas the version in {{JobManager}} runs directly in server and hence
loading all parameters from jobconf is not necessary. One thing that stands out is that the
Executor performs schema matching that is not done in {{JobManager}} that we should probably
fix (even though it's probably not super critical).

> Sqoop2: JobManager code  on failure should call SqoopDestroyerExecutor
> ----------------------------------------------------------------------
>
>                 Key: SQOOP-1666
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1666
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> There is more to just calling the destroyer.destroy method in the SqoopDestroyerExecutor
> Should not the code in JobManger call this utility class?
> The matching code is missing in the JobManager
> {code}
>  public static void executeDestroyer(boolean success, Configuration configuration, Direction
direction) {
>     String destroyerPropertyName, prefixPropertyName;
>     switch (direction) {
>       default:
>       case FROM:
>         destroyerPropertyName = MRJobConstants.JOB_ETL_FROM_DESTROYER;
>         prefixPropertyName = MRJobConstants.PREFIX_CONNECTOR_FROM_CONTEXT;
>         break;
>       case TO:
>         destroyerPropertyName = MRJobConstants.JOB_ETL_TO_DESTROYER;
>         prefixPropertyName = MRJobConstants.PREFIX_CONNECTOR_TO_CONTEXT;
>         break;
>     }
>     Destroyer destroyer = (Destroyer) ClassUtils.instantiate(configuration.get(destroyerPropertyName));
>     if(destroyer == null) {
>       LOG.info("Skipping running destroyer as non was defined.");
>       return;
>     }
>     // Objects that should be pass to the Destroyer execution
>     PrefixContext subContext = new PrefixContext(configuration, prefixPropertyName);
>     Object configConnection = MRConfigurationUtils.getConnectorConnectionConfig(direction,
configuration);
>     Object configJob = MRConfigurationUtils.getConnectorJobConfig(direction, configuration);
>     // Propagate connector schema in every case for now
>     Matcher matcher = MatcherFactory.getMatcher(
>         MRConfigurationUtils.getConnectorSchema(Direction.FROM, configuration),
>         MRConfigurationUtils.getConnectorSchema(Direction.TO, configuration));
>     Schema schema = direction == Direction.FROM ?
>         matcher.getFromSchema() : matcher.getToSchema();
>     DestroyerContext destroyerContext = new DestroyerContext(subContext, success, schema);
>     LOG.info("Executing destroyer class " + destroyer.getClass());
>     destroyer.destroy(destroyerContext, configConnection, configJob);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message