spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: java TwitterUtils.createStream() how create "user stream" ???
Date Thu, 22 Oct 2015 12:16:25 GMT
I don't think the one that comes with spark would listen to specific user
feeds, but yes you can filter out the public tweets by passing the filters
argument. Here's an example for you to start
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/TwitterPopularTags.scala#L37

If you are really interested in specific user stream, then you would have
to probably create a custom receiver and add your logic to that (to fetch
only those tweets etc), you can read this doc to get started on that
http://spark.apache.org/docs/latest/streaming-custom-receivers.html

Thanks
Best Regards

On Tue, Oct 20, 2015 at 4:40 AM, Andy Davidson <
Andy@santacruzintegration.com> wrote:

> Hi
>
> I wrote a little prototype that created a “public stream” now I want to
> convert it to read tweets for a large number of explicit users.
>
> I to create a “user stream” or a “site stream". According to the twitter
> developer doc I should be able to set the “follows” parameter to a list of
> users I am interested in
>
> https://dev.twitter.com/streaming/overview/request-parameters#follow
> *follow*
>
> *A comma-separated list of user IDs, indicating the users whose Tweets
> should be delivered on the stream. *
>
>
> I am not sure how to do this? I found the doc for createStream. I am
> guessing I need to set filters? Can anyone provide a example?
>
> Kind regards
>
> Andy
>
> http://spark.apache.org/docs/latest/api/java/index.html
>
>
> createStream
>
> public static JavaReceiverInputDStream <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/api/java/JavaReceiverInputDStream.html><twitter4j.Status>
createStream(JavaStreamingContext <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/api/java/JavaStreamingContext.html>
jssc,
>                                                       java.lang.String[] filters)
>
> Create a input stream that returns tweets received from Twitter using
> Twitter4J's default OAuth authentication; this requires the system
> properties twitter4j.oauth.consumerKey, twitter4j.oauth.consumerSecret,
> twitter4j.oauth.accessToken and twitter4j.oauth.accessTokenSecret. Storage
> level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.
> Parameters:jssc - JavaStreamingContext objectfilters - Set of filter
> strings to get only those tweets that match themReturns:(undocumented)
>
>

Mime
View raw message