spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pasha Finkelshteyn <>
Subject kotlin-spark-api
Date Sat, 04 Apr 2020 09:06:12 GMT
Hi everybody!

Me again. I'm developing Spark API for Kotlin and found interesting

As you know, Kotlin has concept of nullable types which I'm using in my
implementation of serializers/deserializers (slightly changed
ScalaReflection to be more concrete). But in one case it doesn't work
for me: when list contains nullable type

It outputs error like this:

Caused by: java.lang.NullPointerException: Null value appeared in
non-nullable field:
top level Product or row object
If the schema is inferred from a Scala tuple/case class, or a Java bean,
please try to use scala.Option[_] or other nullable types (e.g.
java.lang.Integer instead of int/scala.Int).

What is even more interesting that if I disable
"spark.sql.codegen.wholeStage" than program works as intended.

Any ideas on this?


To unsubscribe e-mail:

View raw message