spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emma Boya Peng <>
Subject ClassCastException when saving a DataFrame to parquet file (saveAsParquetFile, Spark 1.3.1) using Scala
Date Fri, 21 Aug 2015 07:15:47 GMT

I was trying to programmatically specify a schema and apply it to a RDD of
Rows and save the resulting DataFrame as a parquet file.
Here's what I did:

1. Created an RDD of Rows from RDD[Array[String]]:
val gameId= Long.valueOf(line(0))
      val accountType = Long.valueOf(line(1))
      val worldId = Long.valueOf(line(2))
      val dtEventTime = line(3)
      val iEventId = line(4)
      val vVersionId = line(5)
      val vUin = line(6)
      val vClientIp = line(7)
      val vZoneId = line(8)
      val dtCreateTime = line(9)
      val iFeeFlag = Long.valueOf(line(10))
      val vLoginWay = line(11)

      return Row(gameId, accountType, worldId, dtEventTime, iEventId,
vVersionId, vUin, vClientIp,
                 vZoneId, dtCreateTime, vZoneId, dtCreateTime, iFeeFlag,

View raw message