sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-638) Add an optional, simple and extensible validation framework for sqoop
Date Mon, 22 Oct 2012 00:28:12 GMT
Venkatesh Seetharam created SQOOP-638:
-----------------------------------------

             Summary: Add an optional, simple and extensible validation framework for sqoop
                 Key: SQOOP-638
                 URL: https://issues.apache.org/jira/browse/SQOOP-638
             Project: Sqoop
          Issue Type: Improvement
    Affects Versions: 1.4.2
            Reporter: Venkatesh Seetharam
            Priority: Minor
             Fix For: 1.4.2
         Attachments: SQOOP-638.patch

Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
There are 3 basic interfaces:

ValidationThreshold - Determines if the error margin between the source and target are acceptable:
Absolute, Percentage Tolerant, etc.
Default implementation is AbsoluteValidationThreshold which ensures the row counts from source
and targets are the same.

ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort,
etc. Default implementation logs a warning message to the configured logger.

Validator - Drives the validation logic by delegating the decision to ValidationThreshold
and delegating failure handling to ValidationFailureHandler. The default implementation comes
with a RowCountValidator which validates the row counts from source and the target.

You could extend these interfaces for more specific implementations and override 'em in sqoop
configuration that is picked up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message