spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaonary Rabarisoa <jaon...@gmail.com>
Subject Re: unable to make a custom class as a key in a pairrdd
Date Fri, 24 Oct 2014 09:20:06 GMT
In the documentation it's said that we need to override the hashCode and
equals methods. Without overriding it does't work too. I get this error on
REPL and stand alone application

On Fri, Oct 24, 2014 at 3:29 AM, Prashant Sharma <scrapcodes@gmail.com>
wrote:

> Are you doing this in REPL ? Then there is a bug filed for this, I just
> can't recall the bug ID at the moment.
>
> Prashant Sharma
>
>
>
> On Fri, Oct 24, 2014 at 4:07 AM, Niklas Wilcke <
> 1wilcke@informatik.uni-hamburg.de> wrote:
>
>>  Hi Jao,
>>
>> I don't really know why this doesn't work but I have two hints.
>> You don't need to override hashCode and equals. The modifier case is
>> doing that for you. Writing
>>
>> case class PersonID(id: String)
>>
>> would be enough to get the class you want I think.
>> If I change the type of the id param to Int it works for me but I don't
>> know why.
>>
>> case class PersonID(id: Int)
>>
>> Looks like a strange behavior to me. Have a try.
>>
>> Good luck,
>> Niklas
>>
>>
>> On 23.10.2014 21:52, Jaonary Rabarisoa wrote:
>>
>>  Hi all,
>>
>>  I have the following case class that I want to use as a key in a
>> key-value rdd. I defined the equals and hashCode methode but it's not
>> working. What I'm doing wrong ?
>>
>>  *case class PersonID(id: String) {*
>>
>> *     override def hashCode = id.hashCode*
>>
>> *     override def equals(other: Any) = other match {*
>>
>> *         case that: PersonID => this.id <http://this.id> == that.id
>> <http://that.id> && this.getClass == that.getClass*
>> *         case _ => false*
>> *     }   *
>> * }   *
>>
>>
>> * val p = sc.parallelize((1 until 10).map(x => (PersonID("1"),x )))*
>>
>>
>>  *p.groupByKey.collect foreach println*
>>
>>  *(PersonID(1),CompactBuffer(5))*
>> *(PersonID(1),CompactBuffer(6))*
>> *(PersonID(1),CompactBuffer(7))*
>> *(PersonID(1),CompactBuffer(8, 9))*
>> *(PersonID(1),CompactBuffer(1))*
>> *(PersonID(1),CompactBuffer(2))*
>> *(PersonID(1),CompactBuffer(3))*
>> *(PersonID(1),CompactBuffer(4))*
>>
>>
>>  Best,
>>
>>  Jao
>>
>>
>>
>

Mime
View raw message