spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Ott <alex...@gmail.com>
Subject Binary compatibility issues in 3.1.1?
Date Mon, 08 Feb 2021 18:01:21 GMT
Hi all

I've noticed following SO question about Spark 3.1.1 not working with Delta
0.7.0:
https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800
- I checked with Delta 0.8.0 and it has the same problem. It brokes here:
https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204

I'm thinking that it's caused by following change:
https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150,
but not 100% sure.  If it's, then maybe we could add the backward
compatible constructor for this case class?

P.S. It's not the first time when 3.1 breaks compatibility with existing
connectors, for example, Spark Cassandra Connector on 3.1 doesn't work
without changes:
https://github.com/datastax/spark-cassandra-connector/pull/1280


-- 
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)

Mime
View raw message