sqoop-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a..@apache.org
Subject sqoop git commit: SQOOP-1717: Sqoop2: Remove Data class from docs
Date Tue, 18 Nov 2014 23:53:58 GMT
Repository: sqoop
Updated Branches:
  refs/heads/sqoop2 77eb52e24 -> 6bd8fe3e9


SQOOP-1717: Sqoop2: Remove Data class from docs

(Veena Basavaraj via Abraham Elmahrek)


Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/6bd8fe3e
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/6bd8fe3e
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/6bd8fe3e

Branch: refs/heads/sqoop2
Commit: 6bd8fe3e933df2c44833c7cdf7c2c33413b6c7a4
Parents: 77eb52e
Author: Abraham Elmahrek <abraham@elmahrek.com>
Authored: Tue Nov 18 15:48:23 2014 -0800
Committer: Abraham Elmahrek <abraham@elmahrek.com>
Committed: Tue Nov 18 15:48:23 2014 -0800

----------------------------------------------------------------------
 docs/src/site/sphinx/ConnectorDevelopment.rst | 72 +++++++++++-----------
 1 file changed, 36 insertions(+), 36 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/sqoop/blob/6bd8fe3e/docs/src/site/sphinx/ConnectorDevelopment.rst
----------------------------------------------------------------------
diff --git a/docs/src/site/sphinx/ConnectorDevelopment.rst b/docs/src/site/sphinx/ConnectorDevelopment.rst
index e740cd5..280b502 100644
--- a/docs/src/site/sphinx/ConnectorDevelopment.rst
+++ b/docs/src/site/sphinx/ConnectorDevelopment.rst
@@ -171,7 +171,8 @@ Extractor (E for ETL) extracts data from a given data source
                                JobConfiguration jobConfiguration,
                                SqoopPartition partition);
 
-The ``extract`` method extracts data from the data source using the link and job configuration
properties and writes it to the ``DataWriter`` (provided by the extractor context) as the
default `Intermediate representation`_ .
+The ``extract`` method extracts data from the data source using the link and job configuration
properties and writes it to the ``SqoopMapDataWriter`` (provided in the extractor context
given to the extract method).
+The ``SqoopMapDataWriter`` has the ``SqoopWritable`` thats holds the data read from the data
source in the `Intermediate Data Format representation`_
 
 Extractors use Writer's provided by the ExtractorContext to send a record through the sqoop
system.
 ::
@@ -225,7 +226,7 @@ A loader (L for ETL) receives data from the ``From`` instance of the sqoop
conne
                             ConnectionConfiguration connectionConfiguration,
                             JobConfiguration jobConfiguration) throws Exception;
 
-The ``load`` method reads data from ``DataReader`` (provided by context) in the default `Intermediate
representation`_ and loads it to data source.
+The ``load`` method reads data from ``SqoopOutputFormatDataReader`` (provided in the loader
context of the load methods). It reads the data in the `Intermediate Data Format representation`_
and loads it to the data source.
 
 Loader must iterate in the ``load`` method until the data from ``DataReader`` is exhausted.
 ::
@@ -414,15 +415,15 @@ The diagram below describes the map phase of a job.
             |      extract        |                    |
             |-------------------->|                    |
             |                     |                    |
-           read from DB           |                    |
+           read from Data Source  |                    |
   <-------------------------------|      write*        |
             |                     |------------------->|
-            |                     |                    |           ,----.
-            |                     |                    |---------->|Data|
-            |                     |                    |           `-+--'
-            |                     |                    |
-            |                     |                    |      context.write
-            |                     |                    |-------------------------->
+            |                     |                    |           ,-------------.
+            |                     |                    |---------->|SqoopWritable|
+            |                     |                    |           `----+--------'
+            |                     |                    |                |
+            |                     |                    |                |  context.write(writable,
..)
+            |                     |                    |                |---------------------------->
 
 The diagram below decribes the reduce phase of a job.
 ``OutputFormat`` invokes ``To`` connector's loader's ``load`` method (via ``SqoopOutputFormatLoadExecutor``
).
@@ -433,30 +434,29 @@ The diagram below decribes the reduce phase of a job.
     `---+--------'  `----------+----------'
         |                 |   ,-----------------------------.
         |                 |-> |SqoopOutputFormatLoadExecutor|
-        |                 |   `--------------+--------------'        ,----.
-        |                 |                  |---------------------> |Data|
-        |                 |                  |                       `-+--'
-        |                 |                  |   ,-----------------.   |
-        |                 |                  |-> |SqoopRecordWriter|   |
-      getRecordWriter     |                  |   `--------+--------'   |
-  ----------------------->| getRecordWriter  |            |            |
-        |                 |----------------->|            |            |     ,--------------.
-        |                 |                  |-----------------------------> |ConsumerThread|
-        |                 |                  |            |            |     `------+-------'
-        |                 |<- - - - - - - - -|            |            |            |
   ,------.
-  <- - - - - - - - - - - -|                  |            |            |            |--->|Loader|
-        |                 |                  |            |            |            |   
`--+---'
-        |                 |                  |            |            |            |   
   |
-        |                 |                  |            |            |            | load
 |
-   run  |                 |                  |            |            |            |------>|
-  ----->|                 |     write        |            |            |            |
      |
-        |------------------------------------------------>| setContent |            |
read* |
-        |                 |                  |            |----------->| getContent |<------|
-        |                 |                  |            |            |<-----------|
      |
-        |                 |                  |            |            |            | - -
->|
-        |                 |                  |            |            |            |   
   | write into DB
-        |                 |                  |            |            |            |   
   |-------------->
-
-
-
-.. _`Intermediate representation`: https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Intermediate+representation
+        |                 |   `--------------+--------------'              |
+        |                 |                  |                             |
+        |                 |                  |   ,-----------------.   ,-------------.
+        |                 |                  |-> |SqoopRecordWriter|-->|SqoopWritable|
+      getRecordWriter     |                  |   `--------+--------'   `---+---------'
+  ----------------------->| getRecordWriter  |            |                |
+        |                 |----------------->|            |                |     ,--------------.
+        |                 |                  |---------------------------------->|ConsumerThread|
+        |                 |                  |            |                |     `------+-------'
+        |                 |<- - - - - - - - -|            |                |         
  |    ,------.
+  <- - - - - - - - - - - -|                  |            |                |         
  |--->|Loader|
+        |                 |                  |            |                |            |
   `--+---'
+        |                 |                  |            |                |            |
      |
+        |                 |                  |            |                |            |
load  |
+   run  |                 |                  |            |                |            |------>|
+  ----->|                 |     write        |            |                |         
  |       |
+        |------------------------------------------------>| setContent     |         
  | read* |
+        |                 |                  |            |--------------->| getContent
|<------|
+        |                 |                  |            |                |<-----------|
      |
+        |                 |                  |            |                |            |
- - ->|
+        |                 |                  |            |                |            |
      | write into Data Source
+        |                 |                  |            |                |            |
      |----------------------->
+
+
+
+.. _`Intermediate Data Format representation`: https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Intermediate+representation


Mime
View raw message