spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: basic twitter stream program not working.
Date Thu, 13 Nov 2014 09:54:39 GMT
Change this line

*val* sparkConf = *new* SparkConf().setAppName("TwitterPopularTags"
).setMaster("local").set("spark.eventLog.enabled","true")

to

*val* sparkConf = *new* SparkConf().setAppName("TwitterPopularTags"
).setMaster(*"local[4]"*).set("spark.eventLog.enabled","true")



Thanks
Best Regards

On Thu, Nov 13, 2014 at 2:58 PM, <jishnu.prathap@wipro.com> wrote:

>   *Hi *
>
> *       I am trying to run a basic twitter stream program but getting
> blank output. Please correct me if I am missing something.*
>
>
>
> *import* org.apache.spark.SparkConf
>
> *import* org.apache.spark.streaming.StreamingContext
>
> *import* org.apache.spark.streaming.twitter.TwitterUtils
>
> *import* org.apache.spark.streaming.Seconds
>
> *import* org.apache.log4j.LogManager
>
> *import* org.apache.log4j.Level
>
>
>
> *object* Sparktwiter1 {
>
>   *def* main(args: Array[String]) {
>
> LogManager.getRootLogger().setLevel(Level.ERROR);
>
>     System.setProperty("http.proxyHost", "proxy4.wipro.com");
>
>     System.setProperty("http.proxyPort", "8080");
>
>     System.setProperty("twitter4j.oauth.consumerKey", "")
>
>     System.setProperty("twitter4j.oauth.consumerSecret", "")
>
>     System.setProperty("twitter4j.oauth.accessToken", "")
>
>     System.setProperty("twitter4j.oauth.accessTokenSecret", "")
>
>     *val* sparkConf = *new* SparkConf().setAppName("TwitterPopularTags"
> ).setMaster("local").set("spark.eventLog.enabled", "true")
>
>     *val* ssc = *new* StreamingContext(sparkConf, Seconds(2))
>
>     *val* stream = TwitterUtils.createStream(ssc, None)//, filters)
>
>     stream.print
>
>     *val* s1 = stream.flatMap(status => *status.getText*)
>
>     s1.print
>
>     *val* hashTags = stream.flatMap(status => *status.getText.split(**" "*
> *).filter(_.startsWith(**"#"**))*)
>
>     hashTags.print
>
>      ssc.start()
>
>     ssc.awaitTermination()
>
>   }
>
> }
>
>
>
> Output
>
>
>
> -------------------------------------------
>
> Time: 1415869348000 ms
>
> -------------------------------------------
>
>
>
> -------------------------------------------
>
> Time: 1415869348000 ms
>
> -------------------------------------------
>
>
>
> -------------------------------------------
>
> Time: 1415869348000 ms
>
> -------------------------------------------
>
>
>
>
>
>
>
>
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email.
>
> www.wipro.com
>

Mime
View raw message