spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Behera <amit.bd...@gmail.com>
Subject Re: groupByKey is not working
Date Fri, 30 Jan 2015 20:44:20 GMT
I am sorry Sean.

I am developing code in intelliJ Idea. so with the above dependencies I am
not able to find *groupByKey* when I am searching by ctrl+<space>


On Sat, Jan 31, 2015 at 2:04 AM, Sean Owen <sowen@cloudera.com> wrote:

> When you post a question anywhere, and say "it's not working", you
> *really* need to say what that means.
>
> On Fri, Jan 30, 2015 at 8:20 PM, Amit Behera <amit.bdk10@gmail.com> wrote:
> > hi all,
> >
> > my sbt file is like this:
> >
> > name := "Spark"
> >
> > version := "1.0"
> >
> > scalaVersion := "2.10.4"
> >
> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
> >
> > libraryDependencies += "net.sf.opencsv" % "opencsv" % "2.3"
> >
> >
> > code:
> >
> > object SparkJob
> > {
> >
> >   def pLines(lines:Iterator[String])={
> >     val parser=new CSVParser()
> >     lines.map(l=>{val vs=parser.parseLine(l)
> >       (vs(0),vs(1).toInt)})
> >   }
> >
> >   def main(args: Array[String]) {
> >     val conf = new SparkConf().setAppName("Spark Job").setMaster("local")
> >     val sc = new SparkContext(conf)
> >     val data = sc.textFile("/home/amit/testData.csv").cache()
> >     val result = data.mapPartitions(pLines).groupByKey
> >     //val list = result.filter(x=> {(x._1).contains("24050881")})
> >
> >   }
> >
> > }
> >
> >
> > Here groupByKey is not working . But same thing is working from
> spark-shell.
> >
> > Please help me
> >
> >
> > Thanks
> >
> > Amit
>

Mime
View raw message