spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Subramanian <sanjaysubraman...@yahoo.com.INVALID>
Subject FlatMapValues
Date Wed, 31 Dec 2014 08:12:25 GMT
hey guys 
My dataset is like this 
025126,Chills,8.10,Injection site oedema,8.10,Injection site reaction,8.10,Malaise,8.10,Myalgia,8.10

Intended output is ==================025126,Chills
025126,Injection site oedema
025126,Injection site reaction
025126,Malaise
025126,Myalgia

My code is as follows but the flatMapValues does not work even after I have created the pair
RDD.************************************************************************reacRdd.map(line
=> line.split(',')).map(fields => {
  if (fields.length >= 11 && !fields(0).contains("VAERS_ID")) {
    (fields(0),(fields(1)+"\t"+fields(3)+"\t"+fields(5)+"\t"+fields(7)+"\t"+fields(9)))
  }
  else {
    ""
  }
  }).filter(line => line.toString.length() > 0).flatMapValues(skus => skus.split('\t')).saveAsTextFile("/data/vaers/msfx/reac/"
+ outFile)************************************************************************
thanks
sanjay
Mime
View raw message