spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhor Banga <vibhorba...@gmail.com>
Subject Re: Serialization problem in Spark
Date Thu, 05 Jun 2014 13:02:21 GMT
Any inputs on this will be helpful.

Thanks,
-Vibhor


On Thu, Jun 5, 2014 at 3:41 PM, Vibhor Banga <vibhorbanga@gmail.com> wrote:

> Hi,
>
> I am trying to do something like following in Spark:
>
> JavaPairRDD<byte[], MyObject> eventRDD = hBaseRDD.map(new
> PairFunction<Tuple2<ImmutableBytesWritable, Result>, byte[], MyObject >()
{
>             @Override
>             public Tuple2<byte[], MyObject >
> call(Tuple2<ImmutableBytesWritable, Result>
> immutableBytesWritableResultTuple2) throws Exception {
>                 return new
> Tuple2<byte[], MyObject >(immutableBytesWritableResultTuple2._1.get(),
> MyClass.get(immutableBytesWritableResultTuple2._2));
>             }
>         });
>
>         eventRDD.foreach(new VoidFunction<Tuple2<byte[], Event>>() {
>             @Override
>             public void call(Tuple2<byte[], Event> eventTuple2) throws
> Exception {
>
>                 processForEvent(eventTuple2._2);
>             }
>         });
>
>
> processForEvent() function flow contains some processing and ultimately
> writing to HBase Table. But I am getting serialisation issues with Hadoop
> and HBase inbuilt classes. How do I solve this ? Does using Kyro
> Serialisation help in this case ?
>
> Thanks,
> -Vibhor
>



-- 
Vibhor Banga
Software Development Engineer
Flipkart Internet Pvt. Ltd., Bangalore

Mime
View raw message