spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Ott <alex...@gmail.com>
Subject Re: Binary compatibility issues in 3.1.1?
Date Mon, 08 Feb 2021 18:25:16 GMT
although no, additional constructor won't work...

On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <alexott@gmail.com> wrote:

> Hi all
>
> I've noticed following SO question about Spark 3.1.1 not working with
> Delta 0.7.0:
> https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800
> - I checked with Delta 0.8.0 and it has the same problem. It brokes here:
> https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204
>
> I'm thinking that it's caused by following change:
> https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150,
> but not 100% sure.  If it's, then maybe we could add the backward
> compatible constructor for this case class?
>
> P.S. It's not the first time when 3.1 breaks compatibility with existing
> connectors, for example, Spark Cassandra Connector on 3.1 doesn't work
> without changes:
> https://github.com/datastax/spark-cassandra-connector/pull/1280
>
>
> --
> With best wishes,                    Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>


-- 
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)

Mime
View raw message