spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: spark 2.0 readStream from a REST API
Date Sun, 31 Jul 2016 20:24:27 GMT
Hi,

See https://github.com/jaceklaskowski/spark-workshop/tree/master/solutions/spark-mf-format.
There's a custom format that you can use to get started.

Basically, you need to develop the code behind "mysource" format and
register it using --packages or --jars or similar when you
spark-submit your Spark application.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sun, Jul 31, 2016 at 6:45 PM, Ayoub Benali
<benali.ayoub.info@gmail.com> wrote:
> Looks like the way to go in spark 2.0 is to implement StreamSourceProvider
> with DataSourceRegister. But now spark fails at loading the class when
> doing:
>
> spark.readStream.format("mysource").load()
>
> I get :
>
> java.lang.ClassNotFoundException: Failed to find data source: mysource.
> Please find packages at http://spark-packages.org
>
> Is there something I need to do in order to "load" the Stream source
> provider ?
>
> Thanks,
> Ayoub
>
> 2016-07-31 17:19 GMT+02:00 Jacek Laskowski <jacek@japila.pl>:
>>
>> On Sun, Jul 31, 2016 at 12:53 PM, Ayoub Benali
>> <benali.ayoub.info@gmail.com> wrote:
>>
>> > I started playing with the Structured Streaming API in spark 2.0 and I
>> > am
>> > looking for a way to create streaming Dataset/Dataframe from a rest HTTP
>> > endpoint but I am bit stuck.
>>
>> What a great idea! Why did I myself not think about this?!?!
>>
>> > What would be the easiest way to hack around it ? Do I need to implement
>> > the
>> > Datasource API ?
>>
>> Yes and perhaps Hadoop API too, but not sure which one exactly since I
>> haven't even thought about it (not even once).
>>
>> > Are there examples on how to create a DataSource from a REST endpoint ?
>>
>> Never heard of one.
>>
>> I'm hosting a Spark/Scala meetup this week so I'll definitely propose
>> it as a topic. Thanks a lot!
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message