spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Medan <ehrann.meh...@gmail.com>
Subject [spark-sql] What is the right way to represent an “Any” type in Spark SQL?
Date Fri, 27 Mar 2015 21:31:36 GMT
Hi everyone,

I had a lot of questions today, sorry if I'm spamming the list, but I
thought it's better than posting all questions in one thread. Let me know
if I should throttle my posts ;)

Here is my question:

When I try to have a case class that has Any in it (e.g. I have a property
map and values can be either String, Int or Boolean, and since we don't
have union types, Any is the closest thing)

When I try to register such an RDD as a table in 1.2.1 (or convert to
DataFrame in 1.3 and then register as a table)

I get this weird exception:

Exception in thread "main" scala.MatchError: Any (of class
scala.reflect.internal.Types$ClassNoArgsTypeRef) at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:112)

Which from my interpretaion simply means that Any is not a valid type that
Spark SQL can support in it's schema

I already sent a pull request <https://github.com/apache/spark/pull/5235> to
solve the cryptic exception but my question is - *is there a way to support
an "Any" type in Spark SQL?*

disclaimer - also posted at
http://stackoverflow.com/questions/29310405/what-is-the-right-way-to-represent-an-any-type-in-spark-sql

Mime
View raw message