spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From qinwei <wei....@dewmobile.net>
Subject why flatmap has shuffle
Date Wed, 12 Nov 2014 14:18:24 GMT






Hi, everyone!
    I consider flatmap as a narrow dependency , but why it has shuffle?
as shown on the web UI:
my code is as below :
val transferRDD = sc.textFile("hdfs://host:port/path")



    val rdd = transferRDD.map(line => {

                          val trunks = line.split("\t")

                          if(trunks.length == 32){

                            (trunks(11), trunks(13), Try(java.lang.Long.parseLong(trunks(9))).getOrElse(0l),
trunks(14), trunks(19))

                          }

                        }).filter(arg =>arg != ()).map(arg => arg.asInstanceOf[(String,
String, Long, String, String)]).filter(arg => arg._3 != 0)val flatMappedRDD = rdd.flatMap(arg
=> List((arg._1, (arg._2, arg._3, 1)), (arg._2, (arg._1, arg._3, 0))))
Thank for your help!


qinwei

Mime
View raw message