spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segerlind, Nathan L" <nathan.l.segerl...@intel.com>
Subject RE: [Discuss] - Future plans for Spot-ingest
Date Thu, 13 Apr 2017 23:05:37 GMT
The diagram became garbled in the text format.
Could you resend it as a pdf?

Thanks,
Nate

-----Original Message-----
From: Nathanael Smith [mailto:nathanael@apache.org] 
Sent: Thursday, April 13, 2017 4:01 PM
To: private@spot.incubator.apache.org; dev@spot.incubator.apache.org; user@spot.incubator.apache.org
Subject: [Discuss] - Future plans for Spot-ingest

How would you like to see Spot-ingest change?

A. continue development on the Python Master/Worker with focus on performance / error handling
/ logging B. Develop Scala based ingest to be inline with code base from ingest, ml, to OA
(UI to continue being ipython/JS) C. Python ingest Worker with Scala based Spark code for
normalization and input into DB

Including the high level diagram:
+------------------------------------------------------------------------------------------+
| +--------------------------+                                  +-----------------+      
 |
| | Master                   |  A. B. C.                        | Worker          |      
 |
| |    A. Python             +---------------+      A.          |   A. Python     |      
 |
| |    B. Scala              |               |    +------------->                 +----+
  |
| |    C. Python             |               |    |             |                 |    | 
 |
| +---^------+---------------+               |    |             +-----------------+    | 
 |
|     |      |                               |    |                                    | 
 |
|     |      |                               |    |                                    | 
 |
|     |     +Note--------------+             |    |             +-----------------+    | 
 |
|     |     |Running on a      |             |    |             | Spark Streaming |    | 
 |
|     |     |worker node in    |             |    |      B. C.  | B. Scala        |    | 
 |
|     |     |the Hadoop cluster|             |    |    +--------> C. Scala        +-+ 
|   |
|     |     +------------------+             |    |    |        |                 | |  | 
 |
|   A.|                                      |    |    |        +-----------------+ |  | 
 |
|   B.|                                      |    |    |                            |  | 
 |
|   C.|                                      |    |    |                            |  | 
 |
| +----------------------+          +-v------+----+----+-+           +--------------v--v-+
|
| |                      |          |                    |           |                   |
|
| |   Local FS:          |          |    hdfs            |           |  Hive / Impala    |
|
| |  - Binary/Text       |          |                    |           |   - Parquet -     |
|
| |    Log files -       |          |                    |           |                   |
|
| |                      |          |                    |           |                   |
|
| +----------------------+          +--------------------+           +-------------------+
|
+------------------------------------------------------------------------------------------+

Please let me know your thoughts,

- Nathanael




Mime
View raw message