spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [SS] ContinuousExecution.commit and excessive JSON serialization?
Date Mon, 03 Jun 2019 10:55:46 GMT
Hi,

Why does ContinuousExecution.commit serialize an offset to JSON format just
before deserializing it back (from JSON to an offset)? [1]

val offset =

sources(0).deserializeOffset(offsetLog.get(epoch).get.offsets(0).get.json)

[1]
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala#L341

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Mime
View raw message