spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From swetha kasireddy <swethakasire...@gmail.com>
Subject Re: How to lookup by a key in an RDD
Date Fri, 06 Nov 2015 17:03:06 GMT
Hi,

I have the following questions on IndexedRDD.

1.  Does the IndexedRDD support the key types of String? As per the current
documentation, it looks like it supports only Long?

2. Is IndexedRDD efficient when joined with another RDD. So, basically my
usecase  is that I need to create an IndexedRDD for a certain set of data
and then get those keys that are present in the IndexedRDD but not present
in some other RDD.
How would an IndexedRDD support such an usecase in an efficient manner?


Thanks,
Swetha

On Mon, Nov 2, 2015 at 9:56 PM, Deenar Toraskar <deenar.toraskar@gmail.com>
wrote:

> Swetha
>
> Currently IndexedRDD is an external library and not part of Spark Core.
> You can use it by adding a dependency and pull it in. There are plans to
> move it to Spark core tracked in
> https://issues.apache.org/jira/browse/SPARK-2365. See
> https://spark-summit.org/2015/events/indexedrdd-efficient-fine-grained-updates-for-rdds/
> and https://github.com/amplab/spark-indexedrdd
>
> *Think Reactive Ltd*
> deenar.toraskar@thinkreactive.co.uk
> 07714140812
>
>
>
> On 2 November 2015 at 23:29, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Please take a look at SPARK-2365
>>
>> On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy <
>> swethakasireddy@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is Indexed RDDs released yet?
>>>
>>> Thanks,
>>> Swetha
>>>
>>> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi <gylfi@berkeley.edu> wrote:
>>>
>>>> Hi.
>>>>
>>>> You may want to look into Indexed RDDs
>>>> https://github.com/amplab/spark-indexedrdd
>>>>
>>>> Regards,
>>>>     Gylfi.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>

Mime
View raw message