spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Select per Dataset attribute (Scala) not possible? Why no Seq().as[type] for Datasets?
Date Thu, 31 Mar 2016 21:56:31 GMT
Hi Ted,

Sure! It works with map, but not with select. Wonder if it's by design
or...will soon be fixed? Thanks again for your help.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Mar 31, 2016 at 10:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> I tried this:
>
> scala> final case class Text(id: Int, text: String)
> warning: there was one unchecked warning; re-run with -unchecked for details
> defined class Text
>
> scala> val ds = Seq(Text(0, "hello"), Text(1, "world")).toDF.as[Text]
> ds: org.apache.spark.sql.Dataset[Text] = [id: int, text: string]
>
> scala> ds.map(t => t.id).show
> +-----+
> |value|
> +-----+
> |    0|
> |    1|
> +-----+
>
> On Thu, Mar 31, 2016 at 5:02 AM, Jacek Laskowski <jacek@japila.pl> wrote:
>>
>> Hi,
>>
>> I can't seem to use Dataset using case classes (or tuples) to select per
>> field:
>>
>> scala> final case class Text(id: Int, text: String)
>> warning: there was one unchecked warning; re-run with -unchecked for
>> details
>> defined class Text
>>
>> scala> val ds = Seq(Text(0, "hello"), Text(1, "world")).toDF.as[Text]
>> ds: org.apache.spark.sql.Dataset[Text] = [id: int, text: string]
>>
>> // query per field as symbol works fine
>> scala> ds.select('id).show
>> +---+
>> | id|
>> +---+
>> |  0|
>> |  1|
>> +---+
>>
>> // but not per field as Scala attribute
>> scala> ds.select(_.id).show
>> <console>:40: error: missing parameter type for expanded function
>> ((x$1) => x$1.id)
>>        ds.select(_.id).show
>>                  ^
>>
>> Is this supposed to work in Spark 2.0 (today's build)?
>>
>> BTW, Why is Seq(Text(0, "hello"), Text(1, "world")).as[Text] not possible?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message