spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate Smith <natedogs...@gmail.com>
Subject Re: [Discuss] - Future plans for Spot-ingest
Date Thu, 13 Apr 2017 23:26:11 GMT
I was really hoping it came through ok,
Oh well :)
Here’s an image form:
http://imgur.com/a/DUDsD


> On Apr 13, 2017, at 4:05 PM, Segerlind, Nathan L <nathan.l.segerlind@intel.com>
wrote:
> 
> The diagram became garbled in the text format.
> Could you resend it as a pdf?
> 
> Thanks,
> Nate
> 
> -----Original Message-----
> From: Nathanael Smith [mailto:nathanael@apache.org] 
> Sent: Thursday, April 13, 2017 4:01 PM
> To: private@spot.incubator.apache.org; dev@spot.incubator.apache.org; user@spot.incubator.apache.org
> Subject: [Discuss] - Future plans for Spot-ingest
> 
> How would you like to see Spot-ingest change?
> 
> A. continue development on the Python Master/Worker with focus on performance / error
handling / logging B. Develop Scala based ingest to be inline with code base from ingest,
ml, to OA (UI to continue being ipython/JS) C. Python ingest Worker with Scala based Spark
code for normalization and input into DB
> 
> Including the high level diagram:
> +------------------------------------------------------------------------------------------+
> | +--------------------------+                                  +-----------------+ 
      |
> | | Master                   |  A. B. C.                        | Worker          | 
      |
> | |    A. Python             +---------------+      A.          |   A. Python     | 
      |
> | |    B. Scala              |               |    +------------->                
+----+   |
> | |    C. Python             |               |    |             |                 | 
  |   |
> | +---^------+---------------+               |    |             +-----------------+ 
  |   |
> |     |      |                               |    |                                 
  |   |
> |     |      |                               |    |                                 
  |   |
> |     |     +Note--------------+             |    |             +-----------------+ 
  |   |
> |     |     |Running on a      |             |    |             | Spark Streaming | 
  |   |
> |     |     |worker node in    |             |    |      B. C.  | B. Scala        | 
  |   |
> |     |     |the Hadoop cluster|             |    |    +--------> C. Scala       
+-+  |   |
> |     |     +------------------+             |    |    |        |                 | |
 |   |
> |   A.|                                      |    |    |        +-----------------+ |
 |   |
> |   B.|                                      |    |    |                            |
 |   |
> |   C.|                                      |    |    |                            |
 |   |
> | +----------------------+          +-v------+----+----+-+           +--------------v--v-+
|
> | |                      |          |                    |           |              
    | |
> | |   Local FS:          |          |    hdfs            |           |  Hive / Impala
   | |
> | |  - Binary/Text       |          |                    |           |   - Parquet -
    | |
> | |    Log files -       |          |                    |           |              
    | |
> | |                      |          |                    |           |              
    | |
> | +----------------------+          +--------------------+           +-------------------+
|
> +------------------------------------------------------------------------------------------+
> 
> Please let me know your thoughts,
> 
> - Nathanael
> 
> 
> 


Mime
View raw message