spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@cs.berkeley.edu>
Subject Re: Shark Queries on Streams?
Date Tue, 27 Aug 2013 19:09:39 GMT
It definitely makes sense. In the long run we definitely would like to make
Shark work for streaming queries.

There was a prototype Harvey did a while ago that makes Shark being able to
query streaming RDDs. I will let him comment on how he implemented that.

Note that this might help too: https://github.com/amplab/shark/pull/136



--
Reynold Xin, AMPLab, UC Berkeley
http://rxin.org



On Tue, Aug 27, 2013 at 11:03 AM, Paul Snively <psnively@icloud.com> wrote:

> Hi everyone!
>
> I'm continuing to investigate the Spark/Shark ecosystem and am fascinated
> by the potential. In noticing that I can cache a DStream, it occurred to me
> to wonder whether there's a way to run Shark queries against a cached
> DStream? I guess this would imply that a cached DStream has "similar
> enough" structure to a Shark table, or could somehow be treated as a
> (memory-based) "external table," for the sake of being representable in the
> metastore.
>
> Does this make any sense?
>
> Thanks!
> Paul

Mime
View raw message