spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Select per Dataset attribute (Scala) not possible? Why no Seq().as[type] for Datasets?
Date Thu, 31 Mar 2016 12:02:44 GMT
Hi,

I can't seem to use Dataset using case classes (or tuples) to select per field:

scala> final case class Text(id: Int, text: String)
warning: there was one unchecked warning; re-run with -unchecked for details
defined class Text

scala> val ds = Seq(Text(0, "hello"), Text(1, "world")).toDF.as[Text]
ds: org.apache.spark.sql.Dataset[Text] = [id: int, text: string]

// query per field as symbol works fine
scala> ds.select('id).show
+---+
| id|
+---+
|  0|
|  1|
+---+

// but not per field as Scala attribute
scala> ds.select(_.id).show
<console>:40: error: missing parameter type for expanded function
((x$1) => x$1.id)
       ds.select(_.id).show
                 ^

Is this supposed to work in Spark 2.0 (today's build)?

BTW, Why is Seq(Text(0, "hello"), Text(1, "world")).as[Text] not possible?

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message