sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1799) Ability for connector to indicate if its FROM and TO support incremental reading/ writing
Date Mon, 24 Nov 2014 22:21:13 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Veena Basavaraj updated SQOOP-1799:
-----------------------------------
    Description: 
One suggestion would be have a connector's FROM/ TO initializer to expose if it even supports
incremental. So this can be used to immediately validate the job creation.

{code}
 sqoop > create incremental-job -f 1 -t 2 
{code}

Both the from connector and to connector has to support this feature before we proceed. The
default will be false. The Initializer API will be updated to support this.


{code}

import java.util.LinkedList;
import java.util.List;

import org.apache.sqoop.schema.NullSchema;
import org.apache.sqoop.schema.Schema;

/**
 * This allows connector to define initialization work for execution,
 * for example, context configuration.
 */
public abstract class Initializer<LinkConfiguration, JobConfiguration> {

  /**
   * Initialize new submission based on given configuration properties. Any
   * needed temporary values might be saved to context object and they will be
   * promoted to all other part of the workflow automatically.
   *
   * @param context Initializer context object
   * @param linkConfiguration link configuration object
   * @param jobConfiguration job configuration object for the FROM and TO
   *        In case of the FROM initializer this will represent the FROM job configuration
   *        In case of the TO initializer this will represent the TO job configuration
   */
  public abstract void initialize(InitializerContext context, LinkConfiguration linkConfiguration,
      JobConfiguration jobConfiguration);

  /**
   * Return list of all jars that this particular connector needs to operate on
   * following job. This method will be called after running initialize method.
   * @param context Initializer context object
   * @param linkConfiguration link configuration object
   * @param jobConfiguration job configuration object for the FROM and TO
   *        In case of the FROM initializer this will represent the FROM job configuration
   *        In case of the TO initializer this will represent the TO job configuration
   * @return
   */
  public List<String> getJars(InitializerContext context, LinkConfiguration linkConfiguration,
      JobConfiguration jobConfiguration) {
    return new LinkedList<String>();
  }

  /**
   * Return schema associated with the connector for FROM and TO
   * By default we assume a null schema. Override the method if there a custom schema to provide
either for FROM or TO
   * @param context Initializer context object
   * @param linkConfiguration link configuration object
   * @param jobConfiguration job configuration object for the FROM and TO
   *        In case of the FROM initializer this will represent the FROM job configuration
   *        In case of the TO initializer this will represent the TO job configuration
   * @return
   */

  public Schema getSchema(InitializerContext context, LinkConfiguration linkConfiguration,
      JobConfiguration jobConfiguration) {
    return NullSchema.getInstance();
  }

{code}


  was:
One suggestion would be have a connector's FROM/ TO initializer to expose if it even supports
incremental. So this can be used to immediately validate the job creation.

{code}
 sqoop > create incremental-job -f 1 -t 2 
{code}

Both the from connector and to connector has to support this feature before we proceed. The
default will be false. The Initializer API will be updated to support this.



> Ability for connector to indicate if its FROM and TO support incremental reading/ writing
> -----------------------------------------------------------------------------------------
>
>                 Key: SQOOP-1799
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1799
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> One suggestion would be have a connector's FROM/ TO initializer to expose if it even
supports incremental. So this can be used to immediately validate the job creation.
> {code}
>  sqoop > create incremental-job -f 1 -t 2 
> {code}
> Both the from connector and to connector has to support this feature before we proceed.
The default will be false. The Initializer API will be updated to support this.
> {code}
> import java.util.LinkedList;
> import java.util.List;
> import org.apache.sqoop.schema.NullSchema;
> import org.apache.sqoop.schema.Schema;
> /**
>  * This allows connector to define initialization work for execution,
>  * for example, context configuration.
>  */
> public abstract class Initializer<LinkConfiguration, JobConfiguration> {
>   /**
>    * Initialize new submission based on given configuration properties. Any
>    * needed temporary values might be saved to context object and they will be
>    * promoted to all other part of the workflow automatically.
>    *
>    * @param context Initializer context object
>    * @param linkConfiguration link configuration object
>    * @param jobConfiguration job configuration object for the FROM and TO
>    *        In case of the FROM initializer this will represent the FROM job configuration
>    *        In case of the TO initializer this will represent the TO job configuration
>    */
>   public abstract void initialize(InitializerContext context, LinkConfiguration linkConfiguration,
>       JobConfiguration jobConfiguration);
>   /**
>    * Return list of all jars that this particular connector needs to operate on
>    * following job. This method will be called after running initialize method.
>    * @param context Initializer context object
>    * @param linkConfiguration link configuration object
>    * @param jobConfiguration job configuration object for the FROM and TO
>    *        In case of the FROM initializer this will represent the FROM job configuration
>    *        In case of the TO initializer this will represent the TO job configuration
>    * @return
>    */
>   public List<String> getJars(InitializerContext context, LinkConfiguration linkConfiguration,
>       JobConfiguration jobConfiguration) {
>     return new LinkedList<String>();
>   }
>   /**
>    * Return schema associated with the connector for FROM and TO
>    * By default we assume a null schema. Override the method if there a custom schema
to provide either for FROM or TO
>    * @param context Initializer context object
>    * @param linkConfiguration link configuration object
>    * @param jobConfiguration job configuration object for the FROM and TO
>    *        In case of the FROM initializer this will represent the FROM job configuration
>    *        In case of the TO initializer this will represent the TO job configuration
>    * @return
>    */
>   public Schema getSchema(InitializerContext context, LinkConfiguration linkConfiguration,
>       JobConfiguration jobConfiguration) {
>     return NullSchema.getInstance();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message