spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathanael Smith <nathan...@apache.org>
Subject [Discuss] - Future plans for Spot-ingest
Date Thu, 13 Apr 2017 23:01:13 GMT
How would you like to see Spot-ingest change?

A. continue development on the Python Master/Worker with focus on performance / error handling
/ logging
B. Develop Scala based ingest to be inline with code base from ingest, ml, to OA (UI to continue
being ipython/JS)
C. Python ingest Worker with Scala based Spark code for normalization and input into DB

Including the high level diagram:
+------------------------------------------------------------------------------------------+
| +--------------------------+                                  +-----------------+      
 |
| | Master                   |  A. B. C.                        | Worker          |      
 |
| |    A. Python             +---------------+      A.          |   A. Python     |      
 |
| |    B. Scala              |               |    +------------->                 +----+
  |
| |    C. Python             |               |    |             |                 |    | 
 |
| +---^------+---------------+               |    |             +-----------------+    | 
 |
|     |      |                               |    |                                    | 
 |
|     |      |                               |    |                                    | 
 |
|     |     +Note--------------+             |    |             +-----------------+    | 
 |
|     |     |Running on a      |             |    |             | Spark Streaming |    | 
 |
|     |     |worker node in    |             |    |      B. C.  | B. Scala        |    | 
 |
|     |     |the Hadoop cluster|             |    |    +--------> C. Scala        +-+ 
|   |
|     |     +------------------+             |    |    |        |                 | |  | 
 |
|   A.|                                      |    |    |        +-----------------+ |  | 
 |
|   B.|                                      |    |    |                            |  | 
 |
|   C.|                                      |    |    |                            |  | 
 |
| +----------------------+          +-v------+----+----+-+           +--------------v--v-+
|
| |                      |          |                    |           |                   |
|
| |   Local FS:          |          |    hdfs            |           |  Hive / Impala    |
|
| |  - Binary/Text       |          |                    |           |   - Parquet -     |
|
| |    Log files -       |          |                    |           |                   |
|
| |                      |          |                    |           |                   |
|
| +----------------------+          +--------------------+           +-------------------+
|
+------------------------------------------------------------------------------------------+

Please let me know your thoughts,

- Nathanael




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message