spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for initial offsets?
Date Mon, 26 Aug 2019 11:03:24 GMT
Hi,

Just found out that KafkaSource [1] does not
use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
initial offsets.

Any reason for that? Should I report an issue? Just checking out as I'm
with 2.4.3 exclusively and have no idea what's coming for 3.0.

[1]
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102

[2]
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Mime
View raw message