spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filli Alem <>
Subject AW: Twitter live Streaming
Date Tue, 04 Aug 2015 12:29:59 GMT
Hi Sadaf,

Im currently struggling with Twitter Streaming as well. I cant get it working using the simple
setup bellow. I use spark 1.2 and I replaced twitter4j v3 with v4. Am I doing something wrong?
How are you doing this?

twitter4j.conf.Configuration conf = new twitter4j.conf.ConfigurationBuilder()

            TwitterFactory tf =new TwitterFactory(conf);
            Authorization a = new OAuthAuthorization(conf);
            Authorization a2 = tf.getInstance(a).getAuthorization();

            SparkConf sparkConf = new SparkConf().setAppName("TwitterStreamJob");

                JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,Durations.seconds(30));

                String[] filter = new String[]{"football"};

                JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc,
a2, filter);

        JavaDStream<String> tweets= Function<Status, String>()

                        public String call(Status tweet) throws Exception {
                                return tweet.getUser().getName() +"_" + tweet.getText() +"_"
+ tweet.getCreatedAt().getTime();

        tweets.foreachRDD(new Function<JavaRDD<String>, Void>() {

                        public Void call(JavaRDD<String> arg0) throws Exception {
                                arg0.saveAsTextFile("hdfs://myhost/results/twitter_" + UUID.randomUUID().toString());
                                return null;


-----Urspr√ľngliche Nachricht-----
Von: Sadaf []
Gesendet: Dienstag, 4. August 2015 10:29
Betreff: Twitter live Streaming

Is there any way to get all old tweets since when the account was created using spark streaming
and twitters api? Currently my connector is showing those tweets that get posted after the
program runs. I've done this task using spark streaming and a custom receiver using "twitter
user api".

Thanks in anticipation.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail: For additional commands, e-mail:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message