spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron's Yahoo! <zlgonza...@yahoo.com.INVALID>
Subject Re: Spark streaming for synchronous API
Date Tue, 09 Sep 2014 03:59:27 GMT
Tobias,
 Let me explain a little more.
 I want to create a synchronous REST API that will process some data that is passed in as
some request.
 I would imagine that the Spark Streaming Job on YARN is a long running job that waits on
requests from something. What that something is is still not clear to me, but I would imagine
that it’s some queue. The goal is to be able to push a message onto a queue with some id,
and then  get the processed results back from Spark Streaming.
 The goal is for the REST API be able to respond to lots of calls with low latency.
 Hope that clarifies things...

Thanks,
Ron

On Sep 8, 2014, at 7:41 PM, Tobias Pfeiffer <tgp@preferred.jp> wrote:

> Ron,
> 
> On Tue, Sep 9, 2014 at 11:27 AM, Ron's Yahoo! <zlgonzalez@yahoo.com.invalid> wrote:
>   I’m trying to figure out how I can run Spark Streaming like an API.
>   The goal is to have a synchronous REST API that runs the spark data flow on YARN.
> 
> I guess I *may* develop something similar in the future.
> 
> By "a synchronous REST API", do you mean that submitting the job is synchronous and you
would fetch the processing results via a different call? Or do you want to submit a job and
get the processed data back as an HTTP stream?
> 
> To begin with, is it even possible to have Spark Streaming run as a yarn job?
> 
> I think it is very much possible to run Spark Streaming as a YARN job; at least it worked
well with Mesos.
> 
> Tobias
> 


Mime
View raw message