sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-1753) Sqoop2: CommandLineClient docs have formatting issues
Date Tue, 18 Nov 2014 16:38:33 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Veena Basavaraj updated SQOOP-1753:
-----------------------------------
    Description:     (was: From ConnectorDevelopment.java:
{noformat}

The diagram below describes the initialization phase of a job.
``SqoopInputFormat`` create splits using ``Partitioner``.
::

      ,----------------.          ,-----------.
      |SqoopInputFormat|          |Partitioner|
      `-------+--------'          `-----+-----'
   getSplits  |                         |
  ----------->|                         |
              |      getPartitions      |
              |------------------------>|
              |                         |         ,---------.
              |                         |-------> |Partition|
              |                         |         `----+----'
              |<- - - - - - - - - - - - |              |
              |                         |              |          ,----------.
              |-------------------------------------------------->|SqoopSplit|
              |                         |              |          `----+-----'

The diagram below describes the map phase of a job.
``SqoopMapper`` invokes ``From`` connector's extractor's ``extract`` method.
::

      ,-----------.
      |SqoopMapper|
      `-----+-----'
     run    |
  --------->|                                   ,------------------.
            |---------------------------------->|SqoopMapDataWriter|
            |                                   `------+-----------'
            |                ,---------.               |
            |--------------> |Extractor|               |
            |                `----+----'               |
            |      extract        |                    |
            |-------------------->|                    |
            |                     |                    |
           read from DB           |                    |
  <-------------------------------|      write*        |
            |                     |------------------->|
            |                     |                    |           ,----.
            |                     |                    |---------->|Data|
            |                     |                    |           `-+--'
            |                     |                    |
            |                     |                    |      context.write
            |                     |                    |-------------------------->

The diagram below decribes the reduce phase of a job.
``OutputFormat`` invokes ``To`` connector's loader's ``load`` method (via ``SqoopOutputFormatLoadExecutor``
).
::

    ,------------.  ,---------------------.
    |SqoopReducer|  |SqoopNullOutputFormat|
    `---+--------'  `----------+----------'
        |                 |   ,-----------------------------.
        |                 |-> |SqoopOutputFormatLoadExecutor|
        |                 |   `--------------+--------------'        ,----.
        |                 |                  |---------------------> |Data|
        |                 |                  |                       `-+--'
        |                 |                  |   ,-----------------.   |
        |                 |                  |-> |SqoopRecordWriter|   |
      getRecordWriter     |                  |   `--------+--------'   |
  ----------------------->| getRecordWriter  |            |            |
        |                 |----------------->|            |            |     ,--------------.
        |                 |                  |-----------------------------> |ConsumerThread|
        |                 |                  |            |            |     `------+-------'
        |                 |<- - - - - - - - -|            |            |            | 
  ,------.
  <- - - - - - - - - - - -|                  |            |            |            |--->|Loader|
        |                 |                  |            |            |            |    `--+---'
        |                 |                  |            |            |            |    
  |
        |                 |                  |            |            |            | load
 |
   run  |                 |                  |            |            |            |------>|
  ----->|                 |     write        |            |            |            | 
     |
        |------------------------------------------------>| setContent |            | read*
|
        |                 |                  |            |----------->| getContent |<------|
        |                 |                  |            |            |<-----------| 
     |
        |                 |                  |            |            |            | - -
->|
        |                 |                  |            |            |            |    
  | write into DB
        |                 |                  |            |            |            |    
  |-------------->



.. _`Intermediate representation`: https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Intermediate+representation
{noformat}

Notice the use of the "Data" entity.)

> Sqoop2: CommandLineClient docs have formatting issues
> -----------------------------------------------------
>
>                 Key: SQOOP-1753
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1753
>             Project: Sqoop
>          Issue Type: Bug
>          Components: docs
>    Affects Versions: 1.99.5
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.4
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message