spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
Date Tue, 31 Mar 2015 20:37:46 GMT
Jeetendra:
Please extract the information you need from Result and return the
extracted portion - instead of returning Result itself.

Cheers

On Tue, Mar 31, 2015 at 1:14 PM, Nan Zhu <zhunanmcgill@gmail.com> wrote:

> The example in
> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala
might
> help
>
> Best,
>
> --
> Nan Zhu
> http://codingcat.me
>
> On Tuesday, March 31, 2015 at 3:56 PM, Sean Owen wrote:
>
> Yep, it's not serializable:
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html
>
> You can't return this from a distributed operation since that would
> mean it has to travel over the network and you haven't supplied any
> way to convert the thing into bytes.
>
> On Tue, Mar 31, 2015 at 8:51 PM, Jeetendra Gangele <gangele397@gmail.com>
> wrote:
>
> When I am trying to get the result from Hbase and running mapToPair
> function
> of RRD its giving the error
> java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
>
> Here is the code
>
> // private static JavaPairRDD<Integer, Result>
> getCompanyDataRDD(JavaSparkContext sc) throws IOException {
> // return sc.newAPIHadoopRDD(companyDAO.getCompnayDataConfiguration(),
> TableInputFormat.class, ImmutableBytesWritable.class,
> // Result.class).mapToPair(new
> PairFunction<Tuple2<ImmutableBytesWritable, Result>, Integer, Result>() {
> //
> // public Tuple2<Integer, Result> call(Tuple2<ImmutableBytesWritable,
> Result> t) throws Exception {
> // System.out.println("In getCompanyDataRDD"+t._2);
> //
> // String cknid = Bytes.toString(t._1.get());
> // System.out.println("processing cknids is:"+cknid);
> // Integer cknidInt = Integer.parseInt(cknid);
> // Tuple2<Integer, Result> returnTuple = new Tuple2<Integer,
> Result>(cknidInt, t._2);
> // return returnTuple;
> // }
> // });
> // }
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>

Mime
View raw message