spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SK <>
Subject Spark Streaming: Sentiment Analysis of Twitter streams
Date Tue, 14 Oct 2014 20:13:11 GMT

I am trying to implement simple sentiment analysis of Twitter streams in
Spark/Scala.  I am getting an exception and it appears when I combine
SparkContext with StreamingContext in the same program. When I read the
positive and negative words using only SparkContext.textFile (without
creating a StreamingContext) and analyze static text files, the program
works. Likewise, when I just create the twitter stream using
StreamingContext (and dont create a SparkContext to create the vocabulary),
the program works. The exception seems to be appearing when I combine both
SparkContext and StreamingContext in the same program and I am not sure if
we are not allowed to  have both simultaneously. All the examples in the
streaming module contain only the StreamingContext. The error transcript and
my code appear below. I would appreciate your guidance  in fixing this error
and the right way to  read static files and streams in the same program or
any pointers to relevant examples.

----------------------Error transcript -----------------------------
Lost task 0.0 in stage 2.0 (TID 70, unexpected exception type
------------------------------ My code below ------------------------------
object TweetSentiment {
  def main(args: Array[String]) {

    val filters = args
    val sparkConf = new SparkConf().setAppName("TweetSentiment")
    val sc = new SparkContext(sparkConf)

    // get the list of positive words
    val pos_list =  sc.textFile("positive-words.txt")
                      .filter(line => !line.isEmpty())

    // get the list of negative words
    val neg_list =  sc.textFile("negative-words.txt")
                      .filter(line => !line.isEmpty())

   // create twitter stream
   val ssc = new StreamingContext(sparkConf, Seconds(60))
   val stream = TwitterUtils.createStream(ssc, None, filters)
   val tweets = => r.getText)
   tweets.print() // print tweet text

  sc.stop()   // I tried commenting this, but the exception still appeared.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message