spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Shark Queries on Streams?
Date Tue, 27 Aug 2013 19:09:39 GMT
It definitely makes sense. In the long run we definitely would like to make
Shark work for streaming queries.

There was a prototype Harvey did a while ago that makes Shark being able to
query streaming RDDs. I will let him comment on how he implemented that.

Note that this might help too:

Reynold Xin, AMPLab, UC Berkeley

On Tue, Aug 27, 2013 at 11:03 AM, Paul Snively <> wrote:

> Hi everyone!
> I'm continuing to investigate the Spark/Shark ecosystem and am fascinated
> by the potential. In noticing that I can cache a DStream, it occurred to me
> to wonder whether there's a way to run Shark queries against a cached
> DStream? I guess this would imply that a cached DStream has "similar
> enough" structure to a Shark table, or could somehow be treated as a
> (memory-based) "external table," for the sake of being representable in the
> metastore.
> Does this make any sense?
> Thanks!
> Paul

View raw message