spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <hamishberri...@tutanota.com>
Subject Re: RESTful Operations
Date Mon, 20 Jan 2020 10:41:18 GMT
I see. Thanks for the advice! 


Jan 20, 2020, 09:47 by hemant2184@gmail.com:

> Livy has both statement based(scala) as well as batch processing(code jar packaged).
I think first statement based approach is what you might want to look at.
> Data has to residing in some source.
>
> Thanks,
> Hemant 
>
> On Mon, 20 Jan 2020 at 2:04 PM, <> hamishberridge@tutanota.com> > wrote:
>
>> Sorry didn't explain well. Livy seems to me similar to job server by interacting
through restful api interface for  job submission. The task I want to achieve is more like
general restful api interface where user would provide some parameters for business operations
(a bit like readText or a custom date source with embedded http). So when the spark receive
those requests in e.g custom date source, the spark job can then perform normal operations
such as  aggregation, window functions, and so on. 
>>
>> Thank you for the suggestions and help. 
>>
>>     
>>
>>
>> Jan 20, 2020, 00:26 by >> chris.teoh@gmail.com>> :
>>
>>>
>>>
>>> Maybe something like Livy, otherwise roll your own REST API and have it start
a Spark job.
>>>
>>> On Mon, 20 Jan 2020 at 06:55, <>>> hamishberridge@tutanota.com>>>
> wrote:
>>>
>>>> I am new to Spark. The task I want to accomplish is let client send http
requests, then spark process that request for further operations. However  searching Spark's
website docs 
>>>>
>>>>     https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
>>>>
>>>>     >>>> https://spark.apache.org/docs/latest/
>>>>
>>>> I do not find any places mentioning about this. Also most of the internet
result are more lated to spark job server. 
>>>>
>>>> Any places I should start if I want to use Spark for such purpose? 
>>>>
>>>> Thanks 
>>>>
>>>
>>>
>>> -- 
>>> Chris
>>>
>>
>>


Mime
View raw message