Hello Spark community -
Does Spark 2.0 Datasets *not support* Scala Value classes (basically "extends AnyVal" with a bunch of limitations) ?
I am trying to do something like this:
case class FeatureId(value: Int) extends AnyVal
val seq = Seq(FeatureId(1),FeatureId(2),FeatureId(3))
val ds = spark.createDataset(seq)
This will compile, but then it will break at runtime with a cryptic error about "cannot find int at value". If I remove the "extends AnyVal" part, then everything works.
Value classes are a great performance boost / static type checking feature in Scala, but are they prohibited in Spark Datasets?