spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "fat.wei" <>
Subject How to use window method with direct kafka streaming ?
Date Wed, 01 Aug 2018 09:17:02 GMT
Hi everyone,

I have the following scenario , and I tried to use window method with direct kafka streaming.
The program can run, but doesn't run right!

1. The data is stored in kafka.
2. Every single item of the data has its primary key.
3. Every single item of the data will be split into multipe parts,and these parts will arrive
at kafka in order.

Here's my sample code:

 JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(20));
 JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(jssc,
String.class, String.class,
 StringDecoder.class, StringDecoder.class, kafkaParams, topicsSet);
 messages.window(Durations.seconds(60), Durations.seconds(40)).print();

I couldn't get the data of  RDD@40 ,when I tried to print the data of windowed RDD@80.

Can I have some  suggestions!

View raw message