spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aviad Klein <aviad.kl...@fundbox.com.INVALID>
Subject Referencing a scala/java PipelineStage from pyspark - constructor issues with HasInputCol
Date Mon, 17 Aug 2020 07:17:30 GMT
Hi, I've referenced the same problem on stack overflow and can't seem to
find answers.

I have custom spark pipelinestages written in scala that are specific to my
organization. They work well on scala-spark.

However, when I try to wrap them as shown here, so I can use them in
pyspark, I get weird stuff that's happening. mostly around constructors of
the java objects

please refer to the stack overflow question
<https://stackoverflow.com/questions/63439162/referencing-a-scala-java-pipelinestage-from-pyspark-constructor-issues-with-ha>,
it's the most documented.

Thanks, any help is appreciated

-- 
*Aviad Klein*
Director of Data Science

Mime
View raw message