Perfect ! That's what I was looking for.

Thanks Sun !

On Tue, Aug 2, 2016 at 6:58 PM, Sun Rui <sunrise_win@163.com> wrote:
import org.apache.spark.sql.catalyst.encoders.RowEncoder
implicit val encoder = RowEncoder(df.schema)
df.mapPartitions(_.take(1))

On Aug 3, 2016, at 04:55, Dragisa Krsmanovic <dragisak@ticketfly.com> wrote:

I am trying to use mapPartitions on DataFrame.

Example:

import spark.implicits._
val df: DataFrame = Seq((1,"one"), (2, "two")).toDF("id", "name")
df.mapPartitions(_.take(1))


I am getting:

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

Since DataFrame is Dataset[Row], I was expecting encoder for Row to be there.

What's wrong with my code ?
    

--

Dragiša Krsmanović | Platform Engineer | Ticketfly

dragisak@ticketfly.com

@ticketfly | ticketfly.com/blog | facebook.com/ticketfly





--

Dragiša Krsmanović | Platform Engineer | Ticketfly

dragisak@ticketfly.com

@ticketfly | ticketfly.com/blog | facebook.com/ticketfly