spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pasha Finkelshteyn <pavel.finkelsht...@gmail.com>
Subject kotlin-spark-api
Date Sat, 04 Apr 2020 09:06:12 GMT
Hi everybody!

Me again. I'm developing Spark API for Kotlin and found interesting
behaviour:

As you know, Kotlin has concept of nullable types which I'm using in my
implementation of serializers/deserializers (slightly changed
ScalaReflection to be more concrete). But in one case it doesn't work
for me: when list contains nullable type

It outputs error like this:

Caused by: java.lang.NullPointerException: Null value appeared in
non-nullable field:
top level Product or row object
If the schema is inferred from a Scala tuple/case class, or a Java bean,
please try to use scala.Option[_] or other nullable types (e.g.
java.lang.Integer instead of int/scala.Int).

What is even more interesting that if I disable
"spark.sql.codegen.wholeStage" than program works as intended.

Any ideas on this?

Thanks,
Pasha

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message