sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1796) Support Incremental Semantics in command line tool for Job
Date Thu, 15 Jan 2015 17:14:34 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Veena Basavaraj updated SQOOP-1796:
-----------------------------------
    Description: 
The goal of this ticket is to make sure that additional delta configs from connectors are
displayed to the user. It is upto the connectors to define a new config class or add inputs
the fromJobConfig and toJobConfig based on the delta read support or delta write support


  was:
WIP ( so do not consider this as a final design)

This task will support the Incremental semantics to provide in the command line tool. In sqoop
1 here is the semantics, using the --incremental and the corresponding fields that specify
the column and value. The > is somehow not configurable. This can be easily made configurable
using predicates that specify the column/ value and the operator in one shot.

{code}
sqoop import \
--connect jdbc:mysql://mysql.example.com/sqoop \
--username sqoop \
--password sqoop \
--table visits \
--incremental append \
--check-column id \
--last-value 1
{code}

In Sqoop 2: When we create a job, we are asked for  ( wrt to the connectors)

FromJobConfigValues
ToJobConfigValues

Should the incremental be more generically a delta import, so this predicate can me more more
generic, such as values in between 'A' and 'B' or values < 20.  Its just going to provide
flexibility and avoid having 2 fields, incremental, check-column and value. It can all be
encapsulated in one field predicate.



> Support Incremental Semantics in command line tool for Job
> ----------------------------------------------------------
>
>                 Key: SQOOP-1796
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1796
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> The goal of this ticket is to make sure that additional delta configs from connectors
are displayed to the user. It is upto the connectors to define a new config class or add inputs
the fromJobConfig and toJobConfig based on the delta read support or delta write support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message