spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From asethia <sethia.a...@gmail.com>
Subject Spark GroupBy Save to different files
Date Fri, 01 Sep 2017 14:54:17 GMT
Hi,

I have list of person records in following format:

case class Person(fName:String, city:String)

val l=List(Person("A","City1"),Person("B","City2"),Person("C","City1"))

val rdd:RDD[Person]=sc.parallelize(l)

val groupBy:RDD[(String, Iterable[Person])]=rdd.groupBy(_.city)

I would like to save these group by records in different files (for example
by city). Please can some one help me here.

I tried this but not able to create those files

 groupBy.foreach(x=>{
    x._2.toList.toDF().rdd.saveAsObjectFile(s"file:///tmp/files/${x._1}")
  })

Thanks
Arun



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message