sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-1551) Repository Upgrader api - Extensibility
Date Mon, 06 Oct 2014 17:16:34 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160508#comment-14160508
] 

Veena Basavaraj edited comment on SQOOP-1551 at 10/6/14 5:15 PM:
-----------------------------------------------------------------

Proposal


1. We should be more descriptive and not called it repository upgrader API, since repository
upgrades are handled by the repository itself and not by the entities using it to store their
data. 

eg code in one of the repository implementations we support
{code}
  * {@inheritDoc}
   */
  @Override
  public void createOrUpdateInternals(Connection conn) {
    int version = detectVersion(conn);

    if(version <= 0) {
      runQuery(QUERY_CREATE_SCHEMA_SQOOP, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_CONNECTOR, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_CONFIG, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_INPUT, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_LINK, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_JOB, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_LINK_INPUT, conn);
   
{code}

Today we have connector and driver that can provide configs key/ values( hence the name configurables)

I suggest we call this api config upgrader, other apt names are welcome. 

2. The current api we have is not extensible. Today we support do config types, LINK/ JOB
( since we think these config values are required to create a link / create a job. Tomm we
can have other configs as well of different types, we can easily support this with a different
annotation, like {code}@FooConfig {code}
So the config upgrader api should be generic to add this new type for upgrade as well.

{code}
//todo
{code}

3. The version field in configurable.( should be config_version)
It is mostly for config upgrade. So we might be more careful on what we call this. I suggest
we call this config version, since that is all we need to track from these configurables.


4. Revisit when the repo upgrade and config upgrade should be run.

We should support ways for repo upgrade to happen independent of config upgrade or not?


5. The repository api should be agnostic of {code} configurables {code}

the following is not extensible either

{code}
 /**
   * Upgrade the connector with the same {@linkplain MConnector#uniqueName}
   * in the repository with values from <code>newConnector</code>.
   * <p/>
   * All links and jobs associated with this connector will be upgraded
   * automatically.
   *
   * @param oldConnector The old connector that should be upgraded.
   * @param newConnector New properties for the Connector that should be
   *                     upgraded.
   */
  public final void upgradeConnector(MConnector oldConnector, MConnector newConnector) {
    LOG.info("Upgrading connector: " + oldConnector.getUniqueName());
    long connectorID = oldConnector.getPersistenceId();
    newConnector.setPersistenceId(connectorID);

{code}



was (Author: vybs):
Proposal


1. We should be more descriptive and not called it repository upgrader API, since repository
upgrades are handled by the repository itself and not by the entities using it to store their
data. 

eg code in one of the repository implementations we support
{code}
  * {@inheritDoc}
   */
  @Override
  public void createOrUpdateInternals(Connection conn) {
    int version = detectVersion(conn);

    if(version <= 0) {
      runQuery(QUERY_CREATE_SCHEMA_SQOOP, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_CONNECTOR, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_CONFIG, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_INPUT, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_LINK, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_JOB, conn);
      runQuery(QUERY_CREATE_TABLE_SQ_LINK_INPUT, conn);
   
{code}

Today we have connector and driver that can provide configs key/ values( hence the name configurables)

I suggest we call this api config upgrader, other apt names are welcome. 

2. The current api we have is not extensible. Today we support do config types, LINK/ JOB
( since we think these config values are required to create a link / create a job. Tomm we
can have other configs as well of different types, we can easily support this with a different
annotation, like {code}@FooConfig {code}
So the config upgrader api should be generic to add this new type for upgrade as well.

{code}
//todo
{code}

3. The version field in configurable.( should be config_version)
It is mostly for config upgrade. So we might be more careful on what we call this. I suggest
we call this config version, since that is all we need to track from these configurables.


4. Revisit when the repo upgrade and config upgrade should be run.

We should support ways for repo upgrade to happen independent of config upgrade or not?

> Repository Upgrader api - Extensibility
> ---------------------------------------
>
>                 Key: SQOOP-1551
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1551
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>
> I am not sure if the current api is extensible enough. It only supports upgrading the
config info. Which actually can be now done via the rest api as well. So do we really need
this config upgrade api was my first thought?
> I am also not sure how this code supports upgrades across different versions, since there
seems to be no code in any of these that has knowledge of the repository version and what
type of repository it really belongs to
> Split the api into
> ConnectorConfigUpgrader
> upgradeLinkConfig
> upgradeFromJobConfig
> upgradeToJobConfig
> DriverConfig Upgrader
> upgradeDriverConfig



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message