spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Archit Thakur <archit279tha...@gmail.com>
Subject Re: Problem with flatmap.
Date Thu, 30 Jan 2014 09:05:29 GMT
Needless to say, it works fine with int/string(primitive) type.


On Wed, Jan 29, 2014 at 2:04 PM, Archit Thakur <archit279thakur@gmail.com>wrote:

> Hi,
>
> I am facing a general problem with flatmap operation on rdd.
>
> I am doing
>
> MyRdd.flatmap(func(_))
> MyRdd.saveAsTextFile(..)
>
> func(Tuple2[Key, Value]): List[Tuple2[MyCustomKey, MyCustomValue]] = {
>
> //
>
> println(list)
> list
> }
>
> now if I check the list from the logs at worker and check the textfile it
> has created, it differs.
>
> Only the no. of records are same, but the actual records in the file
> differs from one in the logs.
>
> Does Spark modifies keys/values in between? What other operations does it
> perform with Key or Value?
>
> Thanks and Regards,
> Archit Thakur.
>
>

Mime
View raw message