sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-638) Add an optional, simple and extensible validation framework for sqoop
Date Mon, 22 Oct 2012 00:28:12 GMT
Venkatesh Seetharam created SQOOP-638:

             Summary: Add an optional, simple and extensible validation framework for sqoop
                 Key: SQOOP-638
                 URL: https://issues.apache.org/jira/browse/SQOOP-638
             Project: Sqoop
          Issue Type: Improvement
    Affects Versions: 1.4.2
            Reporter: Venkatesh Seetharam
            Priority: Minor
             Fix For: 1.4.2
         Attachments: SQOOP-638.patch

Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
There are 3 basic interfaces:

ValidationThreshold - Determines if the error margin between the source and target are acceptable:
Absolute, Percentage Tolerant, etc.
Default implementation is AbsoluteValidationThreshold which ensures the row counts from source
and targets are the same.

ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort,
etc. Default implementation logs a warning message to the configured logger.

Validator - Drives the validation logic by delegating the decision to ValidationThreshold
and delegating failure handling to ValidationFailureHandler. The default implementation comes
with a RowCountValidator which validates the row counts from source and the target.

You could extend these interfaces for more specific implementations and override 'em in sqoop
configuration that is picked up.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message