spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <shushantaror...@gmail.com>
Subject spark streaming get kafka individual message's offset and partition no
Date Tue, 28 Jul 2015 12:48:17 GMT
Hi

I am processing kafka messages using spark streaming 1.3.

I am using mapPartitions function to process kafka message.
 How can I access offset no of individual message getting being processed.


JavaPairInputDStream<byte[], byte[]> directKafkaStream
=KafkaUtils.createDirectStream(..);

directKafkaStream.mapPartitions(new
FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() {
public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t)
throws Exception {

while(t.hasNext()){
Tuple2<byte[], byte[]> tuple = t.next();
byte[] key = tuple._1();
byte[] msg = tuple._2();
 ///how to get kafka partition no and offset of this message
 }
}
});

Mime
View raw message