sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj" <vbasava...@cloudera.com>
Subject Re: Review Request 26880: SQOOP-1551:Repository Upgrader api - Extensibility
Date Mon, 20 Oct 2014 20:45:22 GMT


> On Oct. 20, 2014, 1:11 p.m., Jarek Cecho wrote:
> > spi/src/main/java/org/apache/sqoop/connector/spi/ConnectorConfigurableUpgrader.java,
lines 42-66
> > <https://reviews.apache.org/r/26880/diff/7/?file=726213#file726213line42>
> >
> >     We should throw an exception in the default case because it means that Sqoop
is trying to upgrade something that the connector developper did not anticipated. I don't
think that no-oop is correct behaviour here - we should die early.
> 
> Veena Basavaraj wrote:
>     this is a empty method for reason. There will be cases in connector that may not
have the from part. see the HbaseToConnector.
>     
>     or Link config is missing in HDSF
>     
>     In that case, why would we throw an exception? Not making sense to me.
> 
> Jarek Cecho wrote:
>     If the connector don't support given entity than it's upgrade method should not be
even called in my mind. And we should ensure that it's the case rather then silently ignoring
such case. 
>     
>     Similarly if the connector developper "forgets" to implement upgrade for entity that
he is using, we should be very direct about that and fail as soon as possible, rather then
wait to see if for example validations will fail (because they don't).
> 
> Veena Basavaraj wrote:
>     this is not what the current design does.
>     
>     There is no way for a connector to tell that it supports only FromJobConfig. Even
if they told, we dont store such info.
>     
>     The upgrade code in Derby, blindly calls all the methods. Am I missing something
here? since you seem to allude at things I dont seen in the code.

this is the code in the the abstract class Repository, it will call these methods for every
connector. 


 public final void upgradeConnector(MConnector oldConnector, MConnector newConnector) {
    LOG.info("Upgrading connector: " + oldConnector.getUniqueName());
    long connectorID = oldConnector.getPersistenceId();
    newConnector.setPersistenceId(connectorID);

    RepositoryTransaction tx = null;
    try {
      SqoopConnector connector = ConnectorManager.getInstance().getSqoopConnector(
          newConnector.getUniqueName());

      Validator connectorConfigValidator = connector.getConfigValidator();
      boolean upgradeSuccessful = true;
      // 1. Get an upgrader for the connector
      ConnectorConfigurableUpgrader upgrader = connector.getConfigurableUpgrader();
      // 2. Get all links associated with the connector.
      List<MLink> existingLinksByConnector = findLinksForConnector(connectorID);
      // 3. Get all jobs associated with the connector.
      List<MJob> existingJobsByConnector = findJobsForConnector(connectorID);
      // -- BEGIN TXN --
      tx = getTransaction();
      tx.begin();
      // 4. Delete the inputs for all of the jobs and links (in that order) for
      // this connector
      deletelinksAndJobs(existingLinksByConnector, existingJobsByConnector, tx);
      // 5. Delete all inputs and configs associated with the connector, and
      // insert the new configs and inputs for this connector
      upgradeConnectorConfigs(newConnector, tx);
      // 6. Run upgrade logic for the configs related to the link objects
      for (MLink link : existingLinksByConnector) {
        // Make a new copy of the configs
        List<MConfig> linkConfig = newConnector.getLinkConfig().clone(false).getConfigs();
        MLinkConfig newLinkConfig = new MLinkConfig(linkConfig);
        MLinkConfig oldLinkConfig = link.getConnectorLinkConfig();
        upgrader.upgradeLinkConfig(oldLinkConfig, newLinkConfig);
        MLink newlink = new MLink(link, newLinkConfig);

        Object newConfigurationObject = ClassUtils.instantiate(connector
            .getLinkConfigurationClass());
        ConfigUtils.fromConfigs(newlink.getConnectorLinkConfig().getConfigs(),
            newConfigurationObject);
        // 7. Run link config validation
        ConfigValidator configValidator = connectorConfigValidator
            .validateConfigForLink(newConfigurationObject);
        if (configValidator.getStatus().canProceed()) {
          updateLink(newlink, tx);
        } else {


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26880/#review57419
-----------------------------------------------------------


On Oct. 20, 2014, 1:09 p.m., Veena Basavaraj wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26880/
> -----------------------------------------------------------
> 
> (Updated Oct. 20, 2014, 1:09 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> See JIRA : https://issues.apache.org/jira/browse/SQOOP-1551
> 
> for the details of the ticket and the changes in the RB
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/model/Configurable.java PRE-CREATION 
>   common/src/main/java/org/apache/sqoop/model/MConfigurableType.java PRE-CREATION 
>   common/src/main/java/org/apache/sqoop/model/MConnector.java 2f42191 
>   common/src/main/java/org/apache/sqoop/model/MDriver.java 685439e 
>   common/src/main/java/org/apache/sqoop/model/MJob.java b3dec27 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
87ac2af 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorUpgrader.java
a069b3e 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConfigUpgrader.java
b17aa21 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnector.java
606b9fa 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnectorUpgrader.java
PRE-CREATION 
>   core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 54bdd13 
>   core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 5226926 
>   core/src/main/java/org/apache/sqoop/driver/Driver.java f1b45bb 
>   core/src/main/java/org/apache/sqoop/driver/DriverConfigUpgrader.java 847b73d 
>   core/src/main/java/org/apache/sqoop/driver/DriverConfigValidator.java 9c3b660 
>   core/src/main/java/org/apache/sqoop/driver/DriverConfigurableUpgrader.java PRE-CREATION

>   core/src/main/java/org/apache/sqoop/driver/DriverUpgrader.java PRE-CREATION 
>   core/src/main/java/org/apache/sqoop/driver/JobManager.java df2a5ab 
>   core/src/main/java/org/apache/sqoop/driver/configuration/DriverConfiguration.java d4e2254

>   core/src/main/java/org/apache/sqoop/driver/configuration/JobConfiguration.java PRE-CREATION

>   core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java f06fd0c 
>   core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryHandler.java 5a8e026 
>   core/src/main/java/org/apache/sqoop/repository/Repository.java 74a9e12 
>   core/src/test/java/org/apache/sqoop/driver/TestDriverConfigUpgrader.java dc4e8c8 
>   core/src/test/java/org/apache/sqoop/driver/TestJobManager.java 3b475c6 
>   core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java 34bd8a5 
>   repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
c888910 
>   repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestDriverHandling.java
95fbe07 
>   server/src/main/java/org/apache/sqoop/handler/ConnectorRequestHandler.java 7109ae5

>   server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java 462579c 
>   server/src/main/java/org/apache/sqoop/handler/LinkRequestHandler.java 80e65b8 
>   spi/src/main/java/org/apache/sqoop/configurable/ConfigurableUpgradeUtil.java PRE-CREATION

>   spi/src/main/java/org/apache/sqoop/connector/spi/ConnectorConfigurableUpgrader.java
PRE-CREATION 
>   spi/src/main/java/org/apache/sqoop/connector/spi/RepositoryUpgrader.java 879e428 
>   spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 5315e1f 
>   tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java f89c546 
>   tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryLoadTool.java 76ebd3b 
> 
> Diff: https://reviews.apache.org/r/26880/diff/
> 
> 
> Testing
> -------
> 
> yes
> 
> 
> Thanks,
> 
> Veena Basavaraj
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message