spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artemis User <arte...@dtechspace.com>
Subject Re: Apache Spark Connector for SQL Server and Azure SQL
Date Tue, 27 Oct 2020 02:26:13 GMT
The best option certainly would be to recompile the Spark Connector for 
MS SQL server using the Spark 3.0.1/Scala 2.12 dependencies, and just 
fix the compiler errors as you go. The code is open source on github 
(https://github.com/microsoft/sql-spark-connector).  Looks like this 
connector is using Data Frame instead of RDD, so I would expect there 
shouldn't be many API changes that would cause compiler errors.

You may also want to go through the forks and it looks like some people 
have already tried to convert the connector to Spark 3.0.1, like this 
one https://github.com/datarootsio/sql-spark-connector-spark-3

-- ND

On 10/26/20 6:18 AM, alejandra.lemmo@vattenfall.com wrote:
>
> Hi,
>
> In a project where I work with Databricks, we use this connector to 
> read / write data to Azure SQL Database. Currently with Spark 2.4.5 
> and Scala 2.11.
>
> But those setups are getting old. What happens if we update Spark to 
> 3.0.1 or higher and Scala 2.12.
>
> This connector does not work according to the versions it supports. 
> What should we do? Don't use the connector or is there another way to 
> work?
>
> I appreciate any type of information that helps me.
>
> Med vänlig hälsning / Best regards
>
> *Alejandra Lemmo*
> /Data Engineer
> /
> Customer Analytic
>
> Address: Evenemangsgatan 13, 169 56 Solna, 16956 Solna
>
> D +46735249832
>
> 	
>
> M +46735249832
>
> 	
>
>
> alejandra.lemmo@vattenfall.com
> www.vattenfall.se <http://www.vattenfall.se>
>
> Please consider the environment before printing this e-mail
>
>
> Confidentiality: C2 - Internal
>

Mime
View raw message