spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <>
Subject spark streaming get kafka individual message's offset and partition no
Date Tue, 28 Jul 2015 12:48:17 GMT

I am processing kafka messages using spark streaming 1.3.

I am using mapPartitions function to process kafka message.
 How can I access offset no of individual message getting being processed.

JavaPairInputDStream<byte[], byte[]> directKafkaStream

FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() {
public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t)
throws Exception {

Tuple2<byte[], byte[]> tuple =;
byte[] key = tuple._1();
byte[] msg = tuple._2();
 ///how to get kafka partition no and offset of this message

View raw message