spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shixiong Zhu <zsxw...@gmail.com>
Subject Re: About implicit rddToPairRDDFunctions
Date Fri, 14 Nov 2014 05:20:29 GMT
OK. I'll take it.

Best Regards,
Shixiong Zhu

2014-11-14 12:34 GMT+08:00 Reynold Xin <rxin@databricks.com>:

> That seems like a great idea. Can you submit a pull request?
>
>
> On Thu, Nov 13, 2014 at 7:13 PM, Shixiong Zhu <zsxwing@gmail.com> wrote:
>
>> If we put the `implicit` into "pacakge object rdd" or "object rdd", when
>> we write `rdd.groupbykey()`, because rdd is an object of RDD, Scala
>> compiler will search `object rdd`(companion object) and `package object rdd`(pacakge
>> object) by default. We don't need to import them explicitly. Here is a
>> post about the implicit search logic:
>> http://eed3si9n.com/revisiting-implicits-without-import-tax
>>
>> To maintain the compatibility, we can keep `rddToPairRDDFunctions` in
>> the SparkContext but remove `implicit`. The disadvantage is there are
>> two copies of same codes.
>>
>>
>>
>>
>> Best Regards,
>> Shixiong Zhu
>>
>> 2014-11-14 3:57 GMT+08:00 Reynold Xin <rxin@databricks.com>:
>>
>>> Do people usually important o.a.spark.rdd._ ?
>>>
>>> Also in order to maintain source and binary compatibility, we would need
>>> to keep both right?
>>>
>>>
>>> On Thu, Nov 6, 2014 at 3:12 AM, Shixiong Zhu <zsxwing@gmail.com> wrote:
>>>
>>>> I saw many people asked how to convert a RDD to a PairRDDFunctions. I
>>>> would
>>>> like to ask a question about it. Why not put the following implicit into
>>>> "pacakge object rdd" or "object rdd"?
>>>>
>>>>   implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
>>>>       (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] =
>>>> null)
>>>> = {
>>>>     new PairRDDFunctions(rdd)
>>>>   }
>>>>
>>>> If so, the converting will be automatic and not need to
>>>> import org.apache.spark.SparkContext._
>>>>
>>>> I tried to search some discussion but found nothing.
>>>>
>>>> Best Regards,
>>>> Shixiong Zhu
>>>>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message