gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject [DISCUSS] Abstracting away Hadoop/MapReduce as Data Processing Layer
Date Mon, 07 Jul 2014 15:19:24 GMT
Hi Folks,
Many people know the way that things are going with regards to in-memory
computing being 'the' hot topic on the planet right now (outside of the
world cup).
We have made good strides in Gora to get it to where it is as a top level
project. It has also become aparent to me that something we embrace very
well is the notion of abstraction and flexability in the way we modules are
implemented via DataStore API.
One thing which is apparent to me though, is that we may be restricting the
project scope and capablities if we do not embrace new technologies within
our development model.
I am of course talking about embracing the Spark paradigm within Gora and
abstracting ourselves away from the traditional MapReduce Input/Output
Formats which we currently use.
A colleague of mine was at Spark Summit last week in San Francisco and
mentioned that there is ongoing work to move towards a connector-based
approach for IO so that different datastores can be used within Spark SQL.
The point I want to pose here is where can we take advantage of this in an
attempt to further grow the Gora community and improve the project?
Thanks in advance for any thoughts folks.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message