spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nan Zhu <zhunanmcg...@gmail.com>
Subject Re: java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
Date Tue, 31 Mar 2015 20:14:01 GMT
The example in https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala
might help

Best, 

-- 
Nan Zhu
http://codingcat.me


On Tuesday, March 31, 2015 at 3:56 PM, Sean Owen wrote:

> Yep, it's not serializable:
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html
> 
> You can't return this from a distributed operation since that would
> mean it has to travel over the network and you haven't supplied any
> way to convert the thing into bytes.
> 
> On Tue, Mar 31, 2015 at 8:51 PM, Jeetendra Gangele <gangele397@gmail.com (mailto:gangele397@gmail.com)>
wrote:
> > When I am trying to get the result from Hbase and running mapToPair function
> > of RRD its giving the error
> > java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
> > 
> > Here is the code
> > 
> > // private static JavaPairRDD<Integer, Result>
> > getCompanyDataRDD(JavaSparkContext sc) throws IOException {
> > // return sc.newAPIHadoopRDD(companyDAO.getCompnayDataConfiguration(),
> > TableInputFormat.class, ImmutableBytesWritable.class,
> > // Result.class).mapToPair(new
> > PairFunction<Tuple2<ImmutableBytesWritable, Result>, Integer, Result>()
{
> > //
> > // public Tuple2<Integer, Result> call(Tuple2<ImmutableBytesWritable,
> > Result> t) throws Exception {
> > // System.out.println("In getCompanyDataRDD"+t._2);
> > //
> > // String cknid = Bytes.toString(t._1.get());
> > // System.out.println("processing cknids is:"+cknid);
> > // Integer cknidInt = Integer.parseInt(cknid);
> > // Tuple2<Integer, Result> returnTuple = new Tuple2<Integer,
> > Result>(cknidInt, t._2);
> > // return returnTuple;
> > // }
> > // });
> > // }
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org (mailto:user-unsubscribe@spark.apache.org)
> For additional commands, e-mail: user-help@spark.apache.org (mailto:user-help@spark.apache.org)
> 
> 



Mime
View raw message