spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject Select per Dataset attribute (Scala) not possible? Why no Seq().as[type] for Datasets?
Date Thu, 31 Mar 2016 12:02:44 GMT

I can't seem to use Dataset using case classes (or tuples) to select per field:

scala> final case class Text(id: Int, text: String)
warning: there was one unchecked warning; re-run with -unchecked for details
defined class Text

scala> val ds = Seq(Text(0, "hello"), Text(1, "world"))[Text]
ds: org.apache.spark.sql.Dataset[Text] = [id: int, text: string]

// query per field as symbol works fine
| id|
|  0|
|  1|

// but not per field as Scala attribute
<console>:40: error: missing parameter type for expanded function
((x$1) => x$

Is this supposed to work in Spark 2.0 (today's build)?

BTW, Why is Seq(Text(0, "hello"), Text(1, "world")).as[Text] not possible?

Jacek Laskowski
Mastering Apache Spark
Follow me at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message