spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aviad Klein <>
Subject Referencing a scala/java PipelineStage from pyspark - constructor issues with HasInputCol
Date Mon, 17 Aug 2020 07:17:30 GMT
Hi, I've referenced the same problem on stack overflow and can't seem to
find answers.

I have custom spark pipelinestages written in scala that are specific to my
organization. They work well on scala-spark.

However, when I try to wrap them as shown here, so I can use them in
pyspark, I get weird stuff that's happening. mostly around constructors of
the java objects

please refer to the stack overflow question
it's the most documented.

Thanks, any help is appreciated

*Aviad Klein*
Director of Data Science

View raw message