spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filli Alem <Alem.Fi...@ti8m.ch>
Subject AW: Twitter live Streaming
Date Tue, 04 Aug 2015 12:29:59 GMT
Hi Sadaf,

Im currently struggling with Twitter Streaming as well. I cant get it working using the simple
setup bellow. I use spark 1.2 and I replaced twitter4j v3 with v4. Am I doing something wrong?
How are you doing this?

twitter4j.conf.Configuration conf = new twitter4j.conf.ConfigurationBuilder()
                                        .setOAuthConsumerKey("********************")
                                        .setOAuthConsumerSecret("*********************")
                                        .setOAuthAccessToken("*****************")
                                        .setOAuthAccessTokenSecret("**************************").build();

            TwitterFactory tf =new TwitterFactory(conf);
            Authorization a = new OAuthAuthorization(conf);
            Authorization a2 = tf.getInstance(a).getAuthorization();

            SparkConf sparkConf = new SparkConf().setAppName("TwitterStreamJob");

                JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,Durations.seconds(30));

                String[] filter = new String[]{"football"};

                JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc,
a2, filter);

        JavaDStream<String> tweets= receiverStream.map(new Function<Status, String>()
{

                        @Override
                        public String call(Status tweet) throws Exception {
                                return tweet.getUser().getName() +"_" + tweet.getText() +"_"
+ tweet.getCreatedAt().getTime();
                        }

                });
        tweets.foreachRDD(new Function<JavaRDD<String>, Void>() {

                        @Override
                        public Void call(JavaRDD<String> arg0) throws Exception {
                                arg0.saveAsTextFile("hdfs://myhost/results/twitter_" + UUID.randomUUID().toString());
                                return null;
                        }
                });

         jssc.start();
         jssc.awaitTermination();

-----Urspr√ľngliche Nachricht-----
Von: Sadaf [mailto:sadaf@platalytics.com]
Gesendet: Dienstag, 4. August 2015 10:29
An: user@spark.apache.org
Betreff: Twitter live Streaming

Hi
Is there any way to get all old tweets since when the account was created using spark streaming
and twitters api? Currently my connector is showing those tweets that get posted after the
program runs. I've done this task using spark streaming and a custom receiver using "twitter
user api".

Thanks in anticipation.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-live-Streaming-tp24124.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail:
user-help@spark.apache.org

[http://www.ti8m.ch/fileadmin/daten/ti8m/Bilder/footer/Footer_Paymit_klein.jpg]<https://www.ti8m.ch/competences/garage.html>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message