spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Feduke <charles.fed...@gmail.com>
Subject Re: groupByKey is not working
Date Fri, 30 Jan 2015 20:50:43 GMT
You'll still need to:

import org.apache.spark.SparkContext._

Importing org.apache.spark._ does _not_ recurse into sub-objects or
sub-packages, it only brings in whatever is at the level of the package or
object imported.

SparkContext._ has some implicits, one of them for adding groupByKey to an
RDD[_] IIRC.

On Fri Jan 30 2015 at 3:48:22 PM Stephen Boesch <javadba@gmail.com> wrote:

> Amit - IJ will not find it until you add the import as Sean mentioned.  It
> includes implicits that intellij will not know about otherwise.
>
> 2015-01-30 12:44 GMT-08:00 Amit Behera <amit.bdk10@gmail.com>:
>
> I am sorry Sean.
>>
>> I am developing code in intelliJ Idea. so with the above dependencies I
>> am not able to find *groupByKey* when I am searching by ctrl+<space>
>>
>>
>> On Sat, Jan 31, 2015 at 2:04 AM, Sean Owen <sowen@cloudera.com> wrote:
>>
>>> When you post a question anywhere, and say "it's not working", you
>>> *really* need to say what that means.
>>>
>>>
>>> On Fri, Jan 30, 2015 at 8:20 PM, Amit Behera <amit.bdk10@gmail.com>
>>> wrote:
>>> > hi all,
>>> >
>>> > my sbt file is like this:
>>> >
>>> > name := "Spark"
>>> >
>>> > version := "1.0"
>>> >
>>> > scalaVersion := "2.10.4"
>>> >
>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
>>> >
>>> > libraryDependencies += "net.sf.opencsv" % "opencsv" % "2.3"
>>> >
>>> >
>>> > code:
>>> >
>>> > object SparkJob
>>> > {
>>> >
>>> >   def pLines(lines:Iterator[String])={
>>> >     val parser=new CSVParser()
>>> >     lines.map(l=>{val vs=parser.parseLine(l)
>>> >       (vs(0),vs(1).toInt)})
>>> >   }
>>> >
>>> >   def main(args: Array[String]) {
>>> >     val conf = new SparkConf().setAppName("Spark
>>> Job").setMaster("local")
>>> >     val sc = new SparkContext(conf)
>>> >     val data = sc.textFile("/home/amit/testData.csv").cache()
>>> >     val result = data.mapPartitions(pLines).groupByKey
>>> >     //val list = result.filter(x=> {(x._1).contains("24050881")})
>>> >
>>> >   }
>>> >
>>> > }
>>> >
>>> >
>>> > Here groupByKey is not working . But same thing is working from
>>> spark-shell.
>>> >
>>> > Please help me
>>> >
>>> >
>>> > Thanks
>>> >
>>> > Amit
>>>
>>
>>

Mime
View raw message