spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirav Patel <npa...@xactlycorp.com>
Subject Re: spark 1.5.2 - value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey, myData)]
Date Sat, 02 Apr 2016 10:21:51 GMT
In second class I re-declared following and compile error went away. Your
soln worked too.

 implicit val rowKeyOrdering = rowKeyOrd

Thanks
Nirav



On Wed, Mar 30, 2016 at 7:36 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Have you tried the following construct ?
>
> new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey()
>
> See core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
>
> On Wed, Mar 30, 2016 at 5:20 AM, Nirav Patel <npatel@xactlycorp.com>
> wrote:
>
>> Hi, I am trying to use filterByRange feature of spark OrderedRDDFunctions
>> in a hope that it will speed up filtering by scanning only required
>> partitions.
>> I have created Paired RDD with a RangePartitioner in one scala class and
>> in another class I am trying to access this RDD and do following:
>>
>> In first scala class called RDDInitializer  I do:
>>
>>  implicit val rowKeyOrdering = rowKeyOrd
>>
>>     val repartitionRdd = rowdataRdd.partitionBy(new RangePartitioner(
>> minPartitions.toInt, dataRdd, true))
>>
>>     dataRdd  = repartitionRdd.sortByKey()
>>
>>
>> In second scala class I do:
>>
>> import org.apache.spark.SparkContext._
>>
>> RDDInitializer.dataRdd.filterByRange(myKeyFirst, myKeyLast)
>> But I am getting following compile error:
>>
>> "value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey,
>> myData)]"
>>
>>
>> Looks like I can use all methods of OrderedRDDFunctions inside first
>> scala class where implicit rowKeyOrdering is defined but not in second
>> class.
>>
>>
>> Please help me resolve this compile error.
>>
>>
>> Thanks
>>
>> Nirav
>>
>>
>>
>>
>>
>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>
>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
>> <https://twitter.com/Xactly>  [image: Facebook]
>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>> <http://www.youtube.com/xactlycorporation>
>
>
>

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

<https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>

Mime
View raw message