On Tue, Sep 9, 2014 at 11:27 AM, Ron's Yahoo! <firstname.lastname@example.org>
I’m trying to figure out how I can run Spark Streaming like an API.
The goal is to have a synchronous REST API that runs the spark data flow on YARN.
I guess I *may* develop something similar in the future.
By "a synchronous REST API", do you mean that submitting the job is synchronous and you would fetch the processing results via a different call? Or do you want to submit a job and get the processed data back as an HTTP stream?
To begin with, is it even possible to have Spark Streaming run as a yarn job?
I think it is very much possible to run Spark Streaming as a YARN job; at least it worked well with Mesos.