spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aviad Klein <aviad.kl...@fundbox.com.INVALID>
Subject Re: Referencing a scala/java PipelineStage from pyspark - constructor issues with HasInputCol
Date Tue, 25 Aug 2020 08:30:04 GMT
Hey Chris and Sean, thanks for taking the time to answer.

Perhaps my installation of pyspark is off, although I did use version 2.4.4
When developing in scala and pyspark how do you setup your environment?

I used sbt for scala spark

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.4.4",
  "org.apache.spark" %% "spark-sql" % "2.4.4",
  "org.scalactic" %% "scalactic" % "3.1.2",
  "org.scalatest" %% "scalatest" % "3.1.2" % "test",
  "org.apache.spark" %% "spark-mllib" % "2.4.4",
  "org.plotly-scala" %% "plotly-render" % "0.7.2",
  "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly()
)


and pip for pyspark (python 3.6.5)

pip3 install pyspark==2.4.4

Mime
View raw message