spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: [Discuss] - Future plans for Spot-ingest
Date Fri, 14 Apr 2017 15:26:17 GMT
Option C?

On Fri, Apr 14, 2017 at 8:23 AM, Chokha Palayamkottai <
chokha@integralops.com> wrote:

> I like option C.
>
> Thanks,
>
> Chokha.
>
>
> On 4/13/17 7:01 PM, Nathanael Smith wrote:
> > How would you like to see Spot-ingest change?
> >
> > A. continue development on the Python Master/Worker with focus on
> performance / error handling / logging
> > B. Develop Scala based ingest to be inline with code base from ingest,
> ml, to OA (UI to continue being ipython/JS)
> > C. Python ingest Worker with Scala based Spark code for normalization
> and input into DB
> >
> > Including the high level diagram:
> > +-----------------------------------------------------------
> -------------------------------+
> > | +--------------------------+
> +-----------------+        |
> > | | Master                   |  A. B. C.                        |
> Worker          |        |
> > | |    A. Python             +---------------+      A.          |   A.
> Python     |        |
> > | |    B. Scala              |               |    +------------->
>          +----+   |
> > | |    C. Python             |               |    |             |
>          |    |   |
> > | +---^------+---------------+               |    |
>  +-----------------+    |   |
> > |     |      |                               |    |
>               |   |
> > |     |      |                               |    |
>               |   |
> > |     |     +Note--------------+             |    |
>  +-----------------+    |   |
> > |     |     |Running on a      |             |    |             | Spark
> Streaming |    |   |
> > |     |     |worker node in    |             |    |      B. C.  | B.
> Scala        |    |   |
> > |     |     |the Hadoop cluster|             |    |    +--------> C.
> Scala        +-+  |   |
> > |     |     +------------------+             |    |    |        |
>          | |  |   |
> > |   A.|                                      |    |    |
> +-----------------+ |  |   |
> > |   B.|                                      |    |    |
>             |  |   |
> > |   C.|                                      |    |    |
>             |  |   |
> > | +----------------------+          +-v------+----+----+-+
>  +--------------v--v-+ |
> > | |                      |          |                    |           |
>                  | |
> > | |   Local FS:          |          |    hdfs            |           |
> Hive / Impala    | |
> > | |  - Binary/Text       |          |                    |           |
>  - Parquet -     | |
> > | |    Log files -       |          |                    |           |
>                  | |
> > | |                      |          |                    |           |
>                  | |
> > | +----------------------+          +--------------------+
>  +-------------------+ |
> > +-----------------------------------------------------------
> -------------------------------+
> >
> > Please let me know your thoughts,
> >
> > - Nathanael
> >
> >
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message